refactor: trim the kvsync lease internals and their callers

Drop the lease's separate stop channel, the election's held flag, the reconcile try-leader glue, the unused ErrLeaseHeld, Lease.Holder and the EKS lease-group fallback, and route constructor checks through one panic.
chore: sort the reconcile lock imports for gofmt
2026-11-05 10:37:23 +00:00 · 2026-11-05 17:11:08 +10:00 · 2026-11-05 17:10:29 +10:00 · 2026-11-05 17:10:29 +10:00 · 2026-11-05 17:10:29 +10:00 · 2026-11-05 17:10:29 +10:00
2629 changed files with 115439 additions and 31415 deletions
@@ -1,15 +0,0 @@
-# CODEOWNERS — GitHub PR review enforcement
-#
-# Format: <path pattern>  <owner> [<owner> …]
-# Order matters — last matching pattern wins.
-#
-# Set the matching branch protection rule on `dev` and `main` to
-# "Require review from Code Owners" or this file is advisory only.
-
-# Network layer is governed by its invariant tests.
-# Changes here must respect those clauses
-# see spinifex/domains/network/invariants/
-# mulga docs/development/proposals/region-local-architecture/0006-spinifex-network-layer-contract.md (proposal, not accepted)
-/spinifex/domains/network/          @joshsiv-mulga
-/spinifex/runtime/roles/vpcd/ @joshsiv-mulga
-/spinifex/domains/network/invariants/ @joshsiv-mulga
@@ -6,20 +6,6 @@

 <!-- The notable changes, as bullets. -->

-## Account-scoped lifecycle impact
-
-<!-- Complete this section only when the PR changes account-scoped resource lifecycle, account termination, resource cleanup, or a lifecycle boundary. Otherwise write: N/A. -->
-
- Lifecycle owner and resource family:
- Owner-local ordering, recovery, retained-state, or force impact:
- Cross-domain dependency or ownership impact:
- Evidence added or rechecked:
- ADR-0008 lifecycle register / ADR-0001 migration-record / bead updates:
-
-## Mulga design
-
-<!-- One line, so a reviewer knows whether the design records in the private mulga repository need to change. Write either "Updated: <mulga PR or bead id>" or "Not needed: <reason>". This is a reminder, not a gate. -->
-
 ## Community Contribution License

 - [ ] All community contributions in this pull request are licensed to Mulga under the terms of the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0). By creating this pull request I represent that I have the right to license these contributions to Mulga under the Apache License 2.0.
@@ -0,0 +1,148 @@
+#!/usr/bin/env bash
+# e2e-select.sh push <before> <sha> | suites "<e2e-x ...>"
+# Prints the e2e gate's single_legs, multi_suites and multi_timeout outputs.
+# Exits non-zero on anything it cannot read, which fails the e2e status.
+set -euo pipefail
+
+SCRIPTS_DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
+# shellcheck source=suite-sets.sh
+. "${SCRIPTS_DIR}/suite-sets.sh"
+
+# Provision, install and teardown around the suites; the floor is what the
+# jobs have always had, so a budget can only be raised from it.
+SETUP_ALLOWANCE_MIN=15
+FLOOR_MIN=60
+
+BASE_RUNNER=ci-single
+SERVICE_RUNNER=ci-pseudo
+
+log() { printf '[e2e-select] %s\n' "$*" >&2; }
+die() { log "error: $*"; exit 1; }
+
+# in_list <word> <list>
+in_list() {
+  local w
+  for w in $2; do [ "$w" = "$1" ] && return 0; done
+  return 1
+}
+
+# keep <list> <eligible>: the words of <list> that are in <eligible>, deduplicated.
+keep() {
+  local out="" s
+  for s in $1; do
+    in_list "$s" "$2" || continue
+    in_list "$s" "$out" || out="${out:+$out }$s"
+  done
+  printf '%s' "$out"
+}
+
+# budget <list>: max over the suites' own timeouts plus the setup allowance.
+# A sum would demand hours for a set that shares one job and never takes them.
+budget() {
+  local want="${FLOOR_MIN}" s t
+  for s in $1; do
+    t="$(e2e_suite_timeout "$s")"
+    t=$((${t%m} + SETUP_ALLOWANCE_MIN))
+    [ "$t" -gt "$want" ] && want="$t"
+  done
+  printf '%s' "$want"
+}
+
+prefixed() {
+  local out="" s
+  for s in $1; do out="${out:+$out }e2e-$s"; done
+  printf '%s' "$out"
+}
+
+# leg <name> <runner> <suites>: one matrix entry, or nothing for no suites.
+leg() {
+  [ -n "$3" ] || return 0
+  jq -nc --arg leg "$1" --arg runner "$2" --arg suites "$(prefixed "$3")" \
+    --argjson timeout "$(budget "$3")" \
+    '{leg: $leg, runner: $runner, suites: $suites, timeout: $timeout}'
+}
+
+# emit <base-single> <service-single> <multi>
+emit() {
+  local legs
+  legs=$( { leg base "${BASE_RUNNER}" "$1"; leg service "${SERVICE_RUNNER}" "$2"; } | jq -sc .)
+  log "base leg: ${1:-none}"
+  log "service leg: ${2:-none}"
+  log "multi: ${3:-none}"
+  printf 'single_legs=%s\n' "$legs"
+  printf 'multi_suites=%s\n' "$(prefixed "$3")"
+  printf 'multi_timeout=%s\n' "$(budget "$3")"
+}
+
+# select_for_paths <newline-separated paths> [all]: base set plus matches.
+# With "all", every mapped suite is selected, for a push with no baseline.
+select_for_paths() {
+  local matched="" p s
+  if [ "${2:-}" = all ]; then
+    matched=$(e2e_path_suites_all | tr '\n' ' ')
+  else
+    while IFS= read -r p; do
+      [ -n "$p" ] || continue
+      for s in $(e2e_suites_for_path "$p"); do
+        in_list "$s" "$matched" || { matched="${matched:+$matched }$s"; log "${p} -> ${s}"; }
+      done
+    done <<<"$1"
+  fi
+  # A matched suite runs where it is eligible, preferring one node, so route53
+  # goes to the service leg and lb to the multi job.
+  local extra_single="" extra_multi=""
+  for s in $matched; do
+    if in_list "$s" "${E2E_SUITES_SINGLE}"; then
+      extra_single="${extra_single:+$extra_single }$s"
+    elif in_list "$s" "${E2E_SUITES_MULTI}"; then
+      extra_multi="${extra_multi:+$extra_multi }$s"
+    else
+      die "E2E_PATH_SUITES names '${s}', which no topology can run"
+    fi
+  done
+  local base_single base_multi
+  base_single=$(keep "${E2E_SUITES_PUSH_BASE}" "${E2E_SUITES_SINGLE}")
+  base_multi=$(keep "${E2E_SUITES_PUSH_BASE}" "${E2E_SUITES_MULTI}")
+  # A matched suite already in the base leg runs there, not twice.
+  extra_single=$(for s in $extra_single; do in_list "$s" "$base_single" || printf '%s ' "$s"; done)
+  emit "$base_single" "$(keep "$extra_single" "${E2E_SUITES_SINGLE}")" \
+    "$(keep "$base_multi $extra_multi" "${E2E_SUITES_MULTI}")"
+}
+
+mode="${1:-}"
+case "$mode" in
+  push)
+    [ $# -eq 3 ] || die "usage: $0 push <before> <sha>"
+    before="$2" sha="$3"
+    git cat-file -e "${sha}^{commit}" 2>/dev/null || die "commit ${sha} is not in this checkout"
+    if [ -z "${before//0/}" ]; then
+      log "no previous commit for this push; selecting every mapped suite"
+      select_for_paths "" all
+      exit 0
+    fi
+    if ! git cat-file -e "${before}^{commit}" 2>/dev/null; then
+      git fetch -q --filter=blob:none origin "${before}" 2>/dev/null || true
+    fi
+    if ! git cat-file -e "${before}^{commit}" 2>/dev/null; then
+      log "previous commit ${before} is gone; selecting every mapped suite"
+      select_for_paths "" all
+      exit 0
+    fi
+    # A rename counts as both paths, so moving code out of a service selects it.
+    changed=$(git diff --no-renames --name-only "${before}" "${sha}") || die "git diff ${before} ${sha} failed"
+    log "$(printf '%s\n' "$changed" | grep -c . || true) paths changed ${before:0:9}..${sha:0:9}"
+    select_for_paths "$changed"
+    ;;
+  suites)
+    [ $# -eq 2 ] || die "usage: $0 suites \"<e2e-x ...>\""
+    if [ -z "${2// /}" ]; then
+      select_for_paths ""
+      exit 0
+    fi
+    requested=$(printf '%s' "$2" | sed 's/e2e-//g')
+    emit "$(keep "$requested" "${E2E_SUITES_SINGLE}")" "" "$(keep "$requested" "${E2E_SUITES_MULTI}")"
+    ;;
+  *)
+    die "usage: $0 push <before> <sha> | suites \"<list>\""
+    ;;
+esac
@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+# Trims GOCACHE to the entries one CI run used. Go refreshes an entry's mtime
+# on use only when it is over an hour old, so "mark" backdates every restored
+# entry first and "trim" drops whatever this run neither read nor wrote.
+set -euo pipefail
+
+cache="$(go env GOCACHE)"
+mark="${RUNNER_TEMP:?}/go-build-cache.mark"
+
+# stale prints each entry this run did not touch and no touched action entry
+# names. Go skips rewriting an output it already holds, so a fresh action
+# entry can point at a backdated output.
+stale() {
+  local keep="${RUNNER_TEMP}/go-build-cache.keep"
+  find "$cache" -mindepth 2 -maxdepth 2 -name '*-a' -newer "$mark" -exec cat {} + |
+    awk '{ print "/" $3 "-d" }' >"$keep"
+  find "$cache" -mindepth 2 -maxdepth 2 ! -newer "$mark" | { grep -vFf "$keep" || true; }
+}
+
+case "${1:-}" in
+mark)
+  mkdir -p "$cache"
+  find "$cache" -mindepth 2 -maxdepth 2 -exec touch -m -d '2 hours ago' {} +
+  touch "$mark"
+  ;;
+stale | trim)
+  if [[ ! -f "$mark" ]]; then
+    echo "go-build-cache: no mark, leaving the cache untrimmed" >&2
+    exit 0
+  fi
+  if [[ "$1" == stale ]]; then
+    stale
+    exit 0
+  fi
+  before=$(du -sm "$cache" | cut -f1)
+  # An entry is a file, or a directory holding one cached executable whose
+  # mtime Go refreshes in place of the binary's, so it goes as a unit.
+  stale | xargs -r rm -rf --
+  echo "trimmed ${before}MB -> $(du -sm "$cache" | cut -f1)MB"
+  ;;
+*)
+  echo "usage: $0 mark|stale|trim" >&2
+  exit 2
+  ;;
+esac
@@ -1,87 +0,0 @@
-#!/bin/sh
-# hypervisor-disk-preflight_test.sh — drives hypervisor-disk-preflight.sh against
-# a stub ssh that answers for each hypervisor from a canned file, so every
-# verdict is checked without a real host.
-set -eu
-
-SCRIPT_DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
-PREFLIGHT="$SCRIPT_DIR/hypervisor-disk-preflight.sh"
-TMP=$(mktemp -d)
-trap 'rm -rf "$TMP"' EXIT
-
-fails=0
-pass() { echo "  ok   - $1"; }
-fail() { echo "  FAIL - $1"; fails=$((fails + 1)); }
-
-# The stub answers "<root-kb> <pool-kb> <path>" from $FAKE/<host>, or exits 255
-# when that host has no file, the way ssh does for a host it cannot reach.
-mkdir -p "$TMP/bin" "$TMP/hosts"
-cat > "$TMP/bin/ssh" <<'STUB'
-#!/bin/sh
-for a in "$@"; do case $a in *@*) host=${a#*@} ;; esac; done
-cat >/dev/null
-if [ -f "$FAKE/$host" ]; then cat "$FAKE/$host"; exit 0; fi
-echo "ssh: Could not resolve hostname $host" >&2
-exit 255
-STUB
-chmod +x "$TMP/bin/ssh"
-PATH="$TMP/bin:$PATH"
-FAKE="$TMP/hosts"
-export PATH FAKE
-
-cat > "$TMP/env.tfvars" <<'TFVARS'
-nodes = [
-  { name = "wattle", wan_ip = "192.0.2.1", storage_pool = "drive1-pool", base_image = "/x.qcow2" },
-  # { name = "retired", wan_ip = "192.0.2.9", storage_pool = "drive1-pool", base_image = "/x.qcow2" },
-  { name = "banksia", wan_ip = "192.0.2.2", storage_pool = "nvme1-pool", base_image = "/x.qcow2" },
-]
-TFVARS
-
-GiB=1048576
-plenty="$((50 * GiB)) $((200 * GiB)) /mnt/drive1/pool"
-
-# run <label> <want-rc> <want-text>
-run() {
-    rc=0
-    out=$(bash "$PREFLIGHT" "$TMP/env.tfvars" 2>&1) || rc=$?
-    if [ "$rc" -ne "$2" ]; then
-        fail "$1: exit $rc, want $2"
-        echo "$out" | sed 's/^/        /'
-        return
-    fi
-    case $out in
-        *"$3"*) pass "$1" ;;
-        *) fail "$1: output lacks '$3'"; echo "$out" | sed 's/^/        /' ;;
-    esac
-}
-
-echo "-- hypervisor-disk-preflight.sh"
-
-echo "$plenty" > "$FAKE/wattle"
-echo "$plenty" > "$FAKE/banksia"
-run "every host has room" 0 "banksia: root fs 51200 MiB free"
-
-echo "0 $((200 * GiB)) /mnt/drive1/pool" > "$FAKE/wattle"
-run "a full root filesystem fails fast" 1 "::error::hypervisor wattle has 0 MiB free on the filesystem holding /etc/libvirt/qemu"
-
-echo "$plenty" > "$FAKE/wattle"
-echo "$((50 * GiB)) $((3 * GiB)) /mnt/nvme1/pool" > "$FAKE/banksia"
-run "a nearly full pool fails fast" 1 "::error::hypervisor banksia has 3 GiB free in storage pool nvme1-pool"
-
-echo "$plenty" > "$FAKE/banksia"
-rm -f "$FAKE/wattle"
-run "an unreachable host is reported, not failed" 0 "::warning::hypervisor disk preflight: cannot reach wattle"
-
-echo "$plenty" > "$FAKE/wattle"
-echo "$((50 * GiB)) - -" > "$FAKE/banksia"
-run "a pool libvirt does not know is reported, not failed" 0 "pool nvme1-pool (-) not found"
-
-echo "0 0 /x" > "$FAKE/retired"
-echo "$plenty" > "$FAKE/banksia"
-run "a commented-out node is never checked" 0 "wattle: root fs"
-
-if [ "$fails" -ne 0 ]; then
-    echo "$fails failure(s)"
-    exit 1
-fi
-echo "  hypervisor-disk-preflight ok"
@@ -1,36 +0,0 @@
-#!/usr/bin/env python3
-"""Warn when a PR body has no filled "Mulga design" line. Reads PR_BODY; never fails the job."""
-
-import os
-import re
-
-COMMENT_RE = re.compile(r"<!--.*?-->", re.S)
-SECTION_RE = re.compile(r"^#{1,6}\s*Mulga design\s*$", re.I)
-LINE_RE = re.compile(r"^\s*(Updated|Not needed)\s*:\s*(.*?)\s*$", re.I)
-PLACEHOLDER_RE = re.compile(r"<[^<>]*>")
-
-
-def filled(body):
-    inside = False
-    for line in COMMENT_RE.sub("", body or "").splitlines():
-        if SECTION_RE.match(line):
-            inside = True
-        elif inside and line.lstrip().startswith("#"):
-            return False
-        elif inside:
-            m = LINE_RE.match(line)
-            if m and PLACEHOLDER_RE.sub("", m.group(2)).strip():
-                return True
-    return False
-
-
-def main():
-    if filled(os.environ.get("PR_BODY", "")):
-        print("The Mulga design line is filled in.")
-    else:
-        print('::warning title=Mulga design::Add a line under "## Mulga design": "Updated: <mulga PR or bead id>" or "Not needed: <reason>".')
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())
@@ -66,6 +66,67 @@ E2E_SUITES_MULTI="multinode lb cert quota route53 kvquorum lbrecovery instancere
 E2E_SUITES_SINGLE_RE="$(printf '%s' "${E2E_SUITES_SINGLE}" | tr ' ' '|')"
 E2E_SUITES_MULTI_RE="$(printf '%s' "${E2E_SUITES_MULTI}" | tr ' ' '|')"

+# The push gate always runs this set. A push adds to it the suites whose code
+# it changed, by E2E_PATH_SUITES below; a dispatch names its own set instead.
+E2E_SUITES_PUSH_BASE="single multinode cert iam partialblock"
+
+# "suite prefix" pairs: a push changing a path under prefix adds suite. The
+# recovery and fault suites stay nightly, and shared code adds nothing because
+# the base set covers it. Image directories carry each guest's own scripts.
+E2E_PATH_SUITES="
+rds cmd/rds-agent/
+rds internal/rdsgw/
+rds spinifex/agents/rds/
+rds spinifex/domains/rds/
+rds tests/e2e/rds/
+rds scripts/images/rds-mariadb/
+rds scripts/images/rds-postgres/
+ecs cmd/ecs-agent/
+ecs contracts/ecs/
+ecs internal/ecsgw/
+ecs spinifex/agents/ecs/
+ecs spinifex/domains/ecs/
+ecs tests/e2e/ecs/
+ecs scripts/images/ecs-agent/
+eks cmd/eks-
+eks contracts/eks/
+eks internal/eksgw/
+eks spinifex/agents/eks/
+eks spinifex/domains/eks/
+eks tests/e2e/eks/
+eks scripts/images/eks-node/
+rds scripts/images/common/
+ecs scripts/images/common/
+eks scripts/images/common/
+rds scripts/build-system-image.sh
+ecs scripts/build-system-image.sh
+eks scripts/build-system-image.sh
+route53 spinifex/domains/route53/
+route53 tests/e2e/route53/
+lb cmd/lb-agent/
+lb spinifex/agents/elbv2/
+lb spinifex/domains/elbv2/
+lb tests/e2e/lb/
+"
+
+# Every suite E2E_PATH_SUITES can name, once each, in table order.
+#   e2e_path_suites_all
+e2e_path_suites_all() {
+  printf '%s\n' "${E2E_PATH_SUITES}" | awk 'NF == 2 && !seen[$1]++ { print $1 }'
+}
+
+# The suites a change to <path> triggers, one per line.
+#   e2e_suites_for_path <path>
+e2e_suites_for_path() {
+  local path="$1" suite prefix
+  while read -r suite prefix; do
+    [ -n "${prefix}" ] || continue
+    case "${path}" in
+      "${prefix}"*) printf '%s\n' "${suite}" ;;
+    esac
+  done <<<"${E2E_PATH_SUITES}"
+}
+
 # What every e2e-nightly permutation cell must prove, deliberately narrower
 # than the two sets above: those cells have a 35-minute budget and exist to show
 # that every install / network / host-OS combination boots and serves. The
@@ -1,23 +0,0 @@
-import unittest
-
-from mulga_design_check import filled
-
-
-class FilledTest(unittest.TestCase):
-    def test_accepts_either_form(self):
-        self.assertTrue(filled("## Mulga design\nUpdated: mulga#275"))
-        self.assertTrue(filled("## Mulga design\n\nnot needed: internal refactor\n## Next"))
-
-    def test_rejects_unanswered(self):
-        self.assertFalse(filled(""))
-        self.assertFalse(filled("## Mulga design\n<!-- Updated: x -->"))
-        self.assertFalse(filled("## Mulga design\nUpdated: <mulga PR or bead id>"))
-        self.assertFalse(filled("## Mulga design\nUpdated:"))
-
-    def test_scoped_to_the_section(self):
-        self.assertFalse(filled("Updated: mulga#1"))
-        self.assertFalse(filled("## Mulga design\n## Other\nUpdated: mulga#1"))
-
-
-if __name__ == "__main__":
-    unittest.main()
@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+# Usage: wait-for-memory.sh <MiB>
+#
+# Waits until the runner's MemAvailable reaches <MiB>, then returns. Runners on
+# one host share its RAM, and a neighbour's build holds several GiB for a few
+# minutes, so one sample taken during it is not a verdict on the host.
+# Gives up after MEM_WAIT_TIMEOUT_S and names the biggest processes.
+#
+# Env: MEM_WAIT_TIMEOUT_S (900), MEM_WAIT_POLL_S (30), MEMINFO (/proc/meminfo).
+set -uo pipefail
+
+want_mb="${1:?usage: $0 <MiB>}"
+timeout_s="${MEM_WAIT_TIMEOUT_S:-900}"
+poll_s="${MEM_WAIT_POLL_S:-30}"
+meminfo="${MEMINFO:-/proc/meminfo}"
+
+avail_mb() { awk '/^MemAvailable:/ {print int($2 / 1024)}' "$meminfo"; }
+
+start=$SECONDS
+while :; do
+  have=$(avail_mb)
+  if [ -n "$have" ] && [ "$have" -ge "$want_mb" ]; then
+    echo "memory: ${have} MiB available, need ${want_mb} MiB"
+    exit 0
+  fi
+  waited=$(( SECONDS - start ))
+  if [ "$waited" -ge "$timeout_s" ]; then
+    echo "::error::need ${want_mb} MiB available, have ${have:-unknown} MiB after waiting ${waited}s"
+    ps -eo rss,pid,args --sort=-rss | head -11 | cut -c1-200
+    exit 1
+  fi
+  echo "memory: ${have:-unknown} MiB available, need ${want_mb} MiB; waiting (${waited}s of ${timeout_s}s)"
+  sleep "$poll_s"
+done
@@ -50,8 +50,6 @@ workflows:
    '.github/workflows/image-scripts.yml':
        - 'actions/checkout@v7.0.1'
        - 'actions/setup-go@v7.0.0'
-    '.github/workflows/mulga-design.yml':
-        - 'actions/checkout@v7.0.1'
    '.github/workflows/perf-3node.yml':
        - 'actions/checkout@v7.0.1'
        - 'actions/create-github-app-token@v3.3.0'
@@ -0,0 +1,21 @@
+# Codex review gate: calls bluebottle's reusable workflows from main, so a PR cannot change its own review.
+name: codex-review
+
+on:
+  pull_request_target:
+    branches: [dev]
+    types: [opened, synchronize, reopened, ready_for_review, edited]
+  issue_comment:
+    types: [created]
+
+permissions: {}
+
+jobs:
+  review:
+    if: github.event_name == 'pull_request_target'
+    uses: mulgadc/bluebottle/.github/workflows/codex-pr-review.yml@main
+    secrets: inherit
+  waive:
+    if: github.event_name == 'issue_comment' && github.event.issue.pull_request
+    uses: mulgadc/bluebottle/.github/workflows/codex-waive.yml@main
+    secrets: inherit
@@ -30,6 +30,10 @@ concurrency:
 jobs:
  prerelease:
    runs-on: ubuntu-26.04
+    env:
+      # Dev prereleases, newest published first: GitHub lists releases by tag name
+      # as text, so list order is not build order. The install Worker uses this rule.
+      DEV_RELEASES_JQ: '[.[] | select(.draft == false and .prerelease and .published_at != null and (.tag_name | test("-dev(\\.|$)")))] | sort_by(.published_at) | reverse'
    steps:
      # Full history and tags, because the tag name is git describe and a shallow
      # clone has no tag to describe against.
@@ -104,7 +108,7 @@ jobs:

      - name: Build microVM artifacts (lb-agent + kernel + initramfs + qboot + SQS store and worker images)
        if: steps.version.outputs.exists != 'true'
-        run: cd spinifex && make build-microvm-image build-sqs-store-microvm-image build-sqs-worker-microvm-image
+        run: cd spinifex && make build-microvm-image build-sqs-worker-microvm-image

      - name: Build amd64 Tarball
        if: steps.version.outputs.exists != 'true'
@@ -164,12 +168,12 @@ jobs:
              exit 1
          fi
          echo "latest is still $latest"
-          dev="$(gh api "repos/$GITHUB_REPOSITORY/releases?per_page=20" --jq 'map(select(.prerelease))[0].tag_name')"
+          dev="$(gh api "repos/$GITHUB_REPOSITORY/releases?per_page=100" --jq "$DEV_RELEASES_JQ | .[0].tag_name")"
          [ "$dev" = "$TAG" ] || { echo "::error::the dev channel resolves to $dev, not $TAG"; exit 1; }

-      # Kept rather than unbounded: the Worker reads the first page of releases to
-      # resolve the channel, so enough prereleases would eventually push the real
-      # ones off it. Five leaves a few recent builds to roll back to.
+      # Kept rather than unbounded: the Worker reads one page of releases to resolve
+      # the channel, so enough prereleases would eventually push the newest off it.
+      # Five leaves a few recent builds to roll back to.
      - name: Prune old dev prereleases
        if: steps.version.outputs.exists != 'true'
        env:
@@ -177,7 +181,7 @@ jobs:
        run: |
          set -euo pipefail
          gh api "repos/$GITHUB_REPOSITORY/releases?per_page=100" \
-              --jq 'map(select(.prerelease and (.tag_name | endswith("-dev")))) | .[5:] | .[].tag_name' \
+              --jq "$DEV_RELEASES_JQ | .[5:] | .[].tag_name" \
              > stale.txt
          while read -r tag; do
              [ -n "$tag" ] || continue
@@ -71,9 +71,16 @@ jobs:
      ARTIFACT_DIR: /tmp/spinifex-e2e-nightly-cell-${{ inputs.cell || '20' }}-${{ github.run_id }}
      SPINIFEX_OTEL_ATTRS: ${{ inputs.otel-attrs || format('ci.run_id={0},ci.workflow={1},ci.cell={2},ci.matrix=baremetal/pxe/real-hw', github.run_id, github.workflow, inputs.cell || '20') }}
    steps:
+      # A cancelled job's root steps outlive it, since the runner cannot reach
+      # them once it kills sudo, and would keep writing here and to the cache.
+      # A rootfs killed before bin/bash exists is one the build refuses to reuse.
      - name: Clean Workspace
        run: |
+          sudo systemctl stop spinifex-bm-pxe.scope spinifex-bm-e2e.scope 2>/dev/null || true
          sudo umount -R -l "${BM_ISO_CACHE}/rootfs" 2>/dev/null || true
+          if [ -d "${BM_ISO_CACHE}/rootfs" ] && [ ! -x "${BM_ISO_CACHE}/rootfs/bin/bash" ]; then
+            sudo rm -rf --one-file-system "${BM_ISO_CACHE}/rootfs"
+          fi
          if [ ! -c /dev/null ]; then sudo rm -f /dev/null; sudo mknod -m 666 /dev/null c 1 3; fi
          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*

@@ -113,10 +120,15 @@ jobs:
            "$GITHUB_WORKSPACE" \
            "${RUNNER_TEMP}/spinifex-tests.tar" \
            "${RUNNER_TEMP}/bm-bins"
+          # Built here, as the runner user, so the PXE build finds it current.
+          make -C "$GITHUB_WORKSPACE" build-microvm-image

      - name: Build PXE Artifacts
        run: |
-          sudo make -C mulga/scripts/iso-builder \
+          # -o build-spinifex: the binaries come from Build Artifacts, and a
+          # rebuild here would write root-owned files into the workspace.
+          sudo systemd-run --scope --collect --quiet --unit=spinifex-bm-pxe -- \
+            make -C mulga/scripts/iso-builder -o build-spinifex \
            build-rootfs build-squashfs build-initramfs \
            CACHE="${BM_ISO_CACHE}" \
            SPINIFEX_BIN="${RUNNER_TEMP}/bm-bins/spx" \
@@ -124,6 +136,10 @@ jobs:
            SPINIFEX_SRC="${GITHUB_WORKSPACE}" \
            VIPERBLOCK_SRC="${RUNNER_TEMP}/bm-bins/viperblock" \
            VERSION="$(cd "$GITHUB_WORKSPACE" && git describe --tags --always --exclude '*-dev' 2>/dev/null || echo dev)"
+          if root_owned="$(find "$GITHUB_WORKSPACE" -user root -print -quit)" && [ -n "$root_owned" ]; then
+            echo "::error::the PXE build wrote root-owned files into the workspace, e.g. $root_owned"
+            exit 1
+          fi

      - name: Write ePDU credentials
        run: |
@@ -195,13 +211,18 @@ jobs:
          done
          export BM_TEST_TIMEOUT
          echo "suites=${BM_SUITES} test-timeout=${BM_TEST_TIMEOUT}"
-          sudo -E mulga/scripts/baremetal/run-baremetal-e2e.sh \
+          sudo -E systemd-run --scope --collect --quiet --unit=spinifex-bm-e2e -- \
+            mulga/scripts/baremetal/run-baremetal-e2e.sh \
            --vmlinuz   "${BM_ISO_CACHE}/vmlinuz" \
            --initrd    "${BM_ISO_CACHE}/initrd.img" \
            --squashfs  "${BM_ISO_CACHE}/spinifex.squashfs" \
            --tests-tar "${RUNNER_TEMP}/spinifex-tests.tar" \
            2>&1 | tee "${ARTIFACT_DIR}/test-baremetal.log"

+      - name: Stop this job's root steps
+        if: always()
+        run: sudo systemctl stop spinifex-bm-pxe.scope spinifex-bm-e2e.scope 2>/dev/null || true
+
      - name: Generate JUnit report
        if: always()
        run: |
@@ -6,11 +6,32 @@ name: E2E Nightly
 # `main` and `dev` as separate nightlies, and it would always execute `main`'s
 # copy of this file — leaving a fix to the workflow itself untestable until it
 # had already shipped. A dispatch takes both the file and the source from its ref.
+#
+# Cells (matrix cells are defined in .github/e2e-cells.json):
+#
+#   Single node                       Three nodes
+#    1  Debian 13, DHCP                 8  Debian 13, DHCP
+#    2  Ubuntu 26.04, static            9  Ubuntu 26.04, static
+#    3  Ubuntu 26.04, DHCP             10  Ubuntu 26.04, DHCP
+#   17  tofu examples                  29  storage fault injection
+#   18  reboot                         30  routed NAT
+#   19  routed NAT                     31  instance recovery
+#   25  eks                            32  load balancer recovery
+#   26  ecs                            33  KV bucket quorum under node loss
+#   27  rds                            37  Oracle Linux 10, DHCP
+#   34  Oracle Linux 10, DHCP          38  Oracle Linux 10, routed NAT
+#   35  Oracle Linux 10, reboot
+#   36  Oracle Linux 10, routed NAT
+#
+#   Standalone jobs
+#   20  bare metal (PXE)               23  predastore throughput
+#   21  ISO, single node               24  predastore fault injection
+#   22  ISO, three nodes               28  guest disk performance (bare metal)
 on:
  workflow_dispatch:
    inputs:
      cells:
-        description: "Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 34-36 Oracle Linux single node (basic, reboot, routed NAT), 37-38 Oracle Linux on three nodes (multi-node subset with IPsec, routed NAT), 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection, 30 routed NAT on three nodes, 31 instance recovery on three nodes, 32 load balancer recovery on three nodes, 33 KV bucket quorum under node loss on three nodes"
+        description: "Comma-separated cell numbers, no spaces, e.g. 1,25,27 (blank = all). The cell list is at the top of .github/workflows/e2e-nightly.yml."
        required: false

 permissions:
@@ -1017,7 +1038,7 @@ jobs:
        id: siblings
        env:
          BRANCH: ${{ env.GIT_BRANCH }}
-        run: bash .github/resolve-sibling-refs.sh
+        run: bash .github/scripts/resolve-sibling-refs.sh
        working-directory: mulga/spinifex

      - uses: actions/checkout@v7.0.1
@@ -1057,9 +1078,6 @@ jobs:
            { echo "::error::br-wan not permitted in /etc/qemu/bridge.conf"; exit 1; }
          command -v sshpass >/dev/null ||
            { echo "::error::sshpass not installed"; exit 1; }
-          free_mb=$(free -m | awk '/^Mem:/ {print $7}')
-          [ "$free_mb" -ge 8192 ] ||
-            { echo "::error::need 8 GiB available, have ${free_mb} MiB"; exit 1; }
          # A 40G image on tmpfs competes with the 8 GiB the VM itself needs.
          fstype=$(df --output=fstype "$TMPDIR" | tail -1 | tr -d ' ')
          [ "$fstype" != tmpfs ] ||
@@ -1090,6 +1108,10 @@ jobs:
              ssh-keygen -t ed25519 -N "" -f "$HOME/.ssh/$k" -C "$k-ci"
          done

+      # The node's 8 GiB, claimed when the VM boots rather than at preflight.
+      - name: Wait for memory
+        run: mulga/spinifex/.github/scripts/wait-for-memory.sh 8192
+
      # Boots the ISO, installs unattended, reads the lease off the serial
      # console, runs the in-guest smoke test, and installs the key for the
      # next step.
@@ -1203,6 +1225,16 @@ jobs:
          sleep 2
          rm -f "$TMPDIR"/tmp.*.qcow2 "$TMPDIR"/tmp.*.log "$TMPDIR"/tmp.*-node.log

+      # The ISO build runs as root inside the workspace, and the next job on
+      # this runner, from any workflow, cannot remove what it leaves there.
+      - name: Release the workspace
+        if: always()
+        run: |
+          sudo umount -R -l "$GITHUB_WORKSPACE"/mulga/scripts/iso-builder/.cache/rootfs 2>/dev/null || true
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/mulga/scripts/iso-builder/.cache \
+              "$GITHUB_WORKSPACE"/mulga/scripts/iso-builder/dist
+          sudo chown -R "$(id -u):$(id -g)" "$GITHUB_WORKSPACE"
+
  iso-multi:
    name: cell-22 (iso/dhcp/three-node)
    # After cell-21 rather than beside it: same concurrency group, and running
@@ -1264,7 +1296,7 @@ jobs:
        id: siblings
        env:
          BRANCH: ${{ env.GIT_BRANCH }}
-        run: bash .github/resolve-sibling-refs.sh
+        run: bash .github/scripts/resolve-sibling-refs.sh
        working-directory: mulga/spinifex

      - uses: actions/checkout@v7.0.1
@@ -1291,15 +1323,10 @@ jobs:
          ref: ${{ steps.siblings.outputs.bluebottle }}
          path: mulga/bluebottle

-      # 3 x 8 GiB plus 9 sparse 40G disks, before the instances the cluster
-      # launches on top.
      - name: Preflight
        run: |
          set -euo pipefail
          mkdir -p "$TMPDIR"
-          free_mb=$(free -m | awk '/^Mem:/ {print $7}')
-          [ "$free_mb" -ge 26624 ] ||
-            { echo "::error::need 26 GiB available, have ${free_mb} MiB"; exit 1; }
          fstype=$(df --output=fstype "$TMPDIR" | tail -1 | tr -d ' ')
          [ "$fstype" != tmpfs ] ||
            { echo "::error::$TMPDIR is tmpfs — nine node disks would be in RAM"; exit 1; }
@@ -1322,6 +1349,11 @@ jobs:
      # formation port. Both land with mulga-yxtda and the bridged WAN mode;
      # until then this cell proves the cluster forms and survives, not that a
      # customer can use it.
+      # 3 x 8 GiB plus headroom for the instances the cluster launches on top,
+      # claimed when the nodes boot rather than at preflight.
+      - name: Wait for memory
+        run: mulga/spinifex/.github/scripts/wait-for-memory.sh 26624
+
      - name: Install, form and verify the cluster
        working-directory: mulga/scripts/iso-builder
        run: |
@@ -1354,6 +1386,16 @@ jobs:
          rm -rf /tmp/cell22-artifacts-$GITHUB_RUN_ID /tmp/cell22-$GITHUB_RUN_ID.log \
                 "$WORK_DIR"

+      # The ISO build runs as root inside the workspace, and the next job on
+      # this runner, from any workflow, cannot remove what it leaves there.
+      - name: Release the workspace
+        if: always()
+        run: |
+          sudo umount -R -l "$GITHUB_WORKSPACE"/mulga/scripts/iso-builder/.cache/rootfs 2>/dev/null || true
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/mulga/scripts/iso-builder/.cache \
+              "$GITHUB_WORKSPACE"/mulga/scripts/iso-builder/dist
+          sudo chown -R "$(id -u):$(id -g)" "$GITHUB_WORKSPACE"
+
  # Predastore throughput ("cell 23"). Predastore has no runners of its own and
  # spinifex depends on it, so the benchmark lives here and checks predastore
  # out as a sibling.
@@ -1402,7 +1444,7 @@ jobs:
        id: siblings
        env:
          BRANCH: ${{ github.ref_name }}
-        run: bash .github/resolve-sibling-refs.sh
+        run: bash .github/scripts/resolve-sibling-refs.sh
        working-directory: spinifex

      - name: Checkout Predastore
@@ -1526,7 +1568,7 @@ jobs:
        id: siblings
        env:
          BRANCH: ${{ github.ref_name }}
-        run: bash .github/resolve-sibling-refs.sh
+        run: bash .github/scripts/resolve-sibling-refs.sh
        working-directory: spinifex

      - name: Checkout Predastore
@@ -12,13 +12,15 @@ on:
      - go.sum
      - Makefile
      - tests/e2e/**
+      - scripts/images/**
+      - scripts/build-system-image.sh
      - .github/workflows/e2e.yml
  workflow_dispatch:
    inputs:
      suites:
        description: "Space-separated Make targets to run (e.g. 'e2e-single e2e-multinode e2e-cert e2e-lb e2e-eks e2e-iam e2e-rds')"
        required: false
-        default: "e2e-single e2e-multinode e2e-cert e2e-lb e2e-iam"
+        default: "e2e-single e2e-multinode e2e-cert e2e-iam"
      test_run:
        description: "Optional regex for -test.run (e.g. '^TestMultinodeIPSec$'). Empty = run all tests in suite."
        required: false
@@ -31,86 +33,75 @@ permissions:

 env:
  GIT_BRANCH: ${{ github.ref_name }}
-  SUITES: ${{ inputs.suites || 'e2e-single e2e-multinode e2e-cert e2e-lb e2e-iam e2e-partialblock' }}
  TEST_RUN: ${{ inputs.test_run || '' }}
  SPINIFEX_OTLP_ENDPOINT: ${{ vars.SPINIFEX_OTLP_ENDPOINT || 'http://192.168.1.13:8200' }}
  SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }}

 jobs:
-  # The inner timeout must fire before the outer one. `go test` reports where it
-  # stopped and the suite's own failure; a job cancelled on timeout reports only
-  # "The operation was canceled" and takes the artifacts with it.
-  #
-  # A job budget cannot be a literal here because the suite set is a dispatch
-  # input: storagefault asks for 90m and diskperf for 180m, both eligible on one
-  # node, so any fixed number is either wrong for them or wasteful for everyone
-  # else. Resolving it from suite-sets.sh keeps the one source of truth and
-  # leaves the default push set at exactly the 60m it has always had.
+  # A push runs the base set plus the suites its changed paths map to in
+  # suite-sets.sh; a dispatch runs its suites input, or the base set if empty.
+  # Budgets come from the suites' own timeouts there.
  plan:
-    name: Resolve Job Budgets
+    name: Select Suites
    runs-on: ubuntu-26.04
    timeout-minutes: 5
    outputs:
-      single_timeout: ${{ steps.budgets.outputs.single_timeout }}
-      multi_timeout: ${{ steps.budgets.outputs.multi_timeout }}
+      single_legs: ${{ steps.select.outputs.single_legs }}
+      multi_suites: ${{ steps.select.outputs.multi_suites }}
+      multi_timeout: ${{ steps.select.outputs.multi_timeout }}
    steps:
+      # Full history without file contents: the diff needs both commits' trees.
      - uses: actions/checkout@v7.0.1
        with:
          path: spinifex
+          fetch-depth: 0
+          filter: blob:none
          sparse-checkout: .github/scripts

-      - name: Resolve Budgets
-        id: budgets
+      - name: Select suites
+        id: select
+        working-directory: spinifex
        env:
-          # Provision, install and teardown around the suites, plus the headroom
-          # a multi-suite job needs for whatever ran before the one that hangs.
-          SETUP_ALLOWANCE_MIN: 15
-          # What the job has always had. Kept as a floor so resolving the budget
-          # can only ever raise it, never quietly shorten an existing run.
-          FLOOR_MIN: 60
+          EVENT: ${{ github.event_name }}
+          BEFORE: ${{ github.event.before }}
+          IN_SUITES: ${{ inputs.suites }}
        run: |
-          . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
-          # Only max() is a sound bound: the suites share one job, so a sum would
-          # demand hours for a set that has never taken them. Max is what makes
-          # the hanging suite panic on its own timeout first, which is the point.
-          budget() {
-            local want="${FLOOR_MIN}" s t
-            for s in $(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -xE "$1"); do
-              t="$(e2e_suite_timeout "${s}")"
-              t=$(( ${t%m} + SETUP_ALLOWANCE_MIN ))
-              [ "${t}" -gt "${want}" ] && want="${t}"
-            done
-            echo "${want}"
-          }
-          single=$(budget "${E2E_SUITES_SINGLE_RE}")
-          multi=$(budget "${E2E_SUITES_MULTI_RE}")
-          echo "single_timeout=${single}" >> "$GITHUB_OUTPUT"
-          echo "multi_timeout=${multi}" >> "$GITHUB_OUTPUT"
-          echo "suites='${SUITES}' -> single ${single}m, multi ${multi}m"
+          set -euo pipefail
+          SELECT=.github/scripts/e2e-select.sh
+          if [ "${EVENT}" = push ]; then
+            bash "${SELECT}" push "${BEFORE}" "${GITHUB_SHA}" | tee -a "$GITHUB_OUTPUT"
+          else
+            bash "${SELECT}" suites "${IN_SUITES}" | tee -a "$GITHUB_OUTPUT"
+          fi

  e2e_single:
-    name: Single-Node E2E
-    # The dispatch check gates on exactly the suites this job's Run step
-    # allow-lists, i.e. E2E_SUITES_SINGLE in .github/scripts/suite-sets.sh —
-    # an `if:` expression cannot source that file, so this one list is
-    # hand-mirrored and must be updated with it. Gating on a suite the job
-    # won't run — lb, which only executes in the multinode job — would
-    # provision a tofu VM and pay the Spinifex install only to run nothing and
-    # tear down. Push triggers keep running via the null-check on
-    # inputs.suites.
-    if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-single') || contains(github.event.inputs.suites, 'e2e-iam') || contains(github.event.inputs.suites, 'e2e-cert') || contains(github.event.inputs.suites, 'e2e-eks') || contains(github.event.inputs.suites, 'e2e-ecs') || contains(github.event.inputs.suites, 'e2e-storagegrowth') || contains(github.event.inputs.suites, 'e2e-partialblock') || contains(github.event.inputs.suites, 'e2e-rds')
+    # The base leg keeps the job and artifact names it always had: mulga's
+    # registry intake finds a run's results by them.
+    name: ${{ matrix.leg == 'base' && 'Single-Node E2E' || format('Single-Node E2E ({0})', matrix.leg) }}
+    # `base` runs on ci-single; `service`, on ci-pseudo, carries the matched
+    # single-node suites so a long one stays off the base leg's critical path.
+    if: needs.plan.outputs.single_legs != '[]'
    needs: plan
-    runs-on: [self-hosted, ci-single]
-    timeout-minutes: ${{ fromJSON(needs.plan.outputs.single_timeout) }}
+    runs-on: [self-hosted, "${{ matrix.runner }}"]
+    timeout-minutes: ${{ matrix.timeout }}
+    strategy:
+      fail-fast: false
+      matrix:
+        include: ${{ fromJSON(needs.plan.outputs.single_legs) }}
    env:
-      TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
-      ARTIFACT_DIR: /tmp/spinifex-e2e-artifacts-${{ github.run_id }}
+      SUITES: ${{ matrix.suites }}
+      # Both legs can run on banksia at once, so every host path carries the leg.
+      TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}-${{ matrix.leg }}.log
+      ARTIFACT_DIR: /tmp/spinifex-e2e-artifacts-${{ github.run_id }}-${{ matrix.leg }}
+      # The base leg keeps the cell name it always had, so its history in
+      # Elastic and telemetry continues; any other leg appends its name.
+      E2E_CELL: ${{ matrix.leg == 'base' && 'single' || format('single-{0}', matrix.leg) }}
      # Job-level env wins over the workflow-level default. This job and
      # e2e_multi share one run id and both name their hosts node1/node2/node3,
      # so without ci.cell their telemetry collapses into one bucket. This job
      # is where "Bootstrap via Binary Install" bakes SPINIFEX_OTEL_ATTRS into
      # the VM's telemetry.env, so the label has to be set here.
-      SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=single
+      SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=${{ matrix.leg == 'base' && 'single' || format('single-{0}', matrix.leg) }}
    steps:
      - name: Reap Orphan Procs
        run: |
@@ -180,7 +171,7 @@ jobs:
      - name: Resolve CI job id
        uses: $/.github/actions/otel-job-id
        with:
-          job-name: Single-Node E2E
+          job-name: ${{ matrix.leg == 'base' && 'Single-Node E2E' || format('Single-Node E2E ({0})', matrix.leg) }}

      - name: Build Install Artifacts
        working-directory: mulga/scripts/tofu-cluster
@@ -224,6 +215,12 @@ jobs:
          echo "wan_ip=${WAN_IP}" >> "$GITHUB_OUTPUT"
          echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"

+      # build-system-image.sh reuses a raw image under ten minutes old from
+      # its build directory, and every runner on banksia shares the default
+      # one in /tmp, so a job could test another commit's guest agent.
+      - name: Use a per-job image build directory
+        run: echo "SYSTEM_IMAGE_BUILD_DIR=${RUNNER_TEMP}/system-image-build" >> "$GITHUB_ENV"
+
      # The eks suite's CreateCluster boots a control-plane VM from the
      # eks-node AMI, so that AMI must exist in the VM's image store before the
      # suite runs. build-system-image.sh customizes the image with libguestfs
@@ -247,7 +244,7 @@ jobs:
          group_open "make build-eks-node-image"
          make build-eks-node-image
          group_close
-          ok "raw: $(du -h /tmp/eks-node-image-build/eks-node-alpine.raw | cut -f1)"
+          ok "raw: $(du -h "${SYSTEM_IMAGE_BUILD_DIR}/eks-node-image-build/eks-node-alpine.raw" | cut -f1)"

      - name: Import eks-node AMI on VM
        if: contains(env.SUITES, 'e2e-eks')
@@ -259,13 +256,16 @@ jobs:
          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
          banner "import eks-node AMI to VM image store"
+          # The VM's /tmp is a tmpfs and scp writes each raw at its full size,
+          # so every import removes its file before the next image arrives.
          scp "${SSH_OPTS[@]}" \
-            /tmp/eks-node-image-build/eks-node-alpine.raw \
+            "${SYSTEM_IMAGE_BUILD_DIR}/eks-node-image-build/eks-node-alpine.raw" \
            "$VM":/tmp/eks-node-alpine.raw
          ssh "${SSH_OPTS[@]}" "$VM" \
            "spx admin images import --file /tmp/eks-node-alpine.raw \
               --distro alpine --version ${ALPINE_VERSION} --arch x86_64 \
-               --boot-mode bios --ami-name ${AMI_NAME} --tag ${SYSTEM_TAG}" >/dev/null 2>&1
+               --boot-mode bios --ami-name ${AMI_NAME} --tag ${SYSTEM_TAG}; \
+               rc=\$?; rm -f /tmp/eks-node-alpine.raw; exit \$rc" >/dev/null 2>&1
          ok "eks-node AMI imported"

      # The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full
@@ -299,6 +299,40 @@ jobs:
          EOF
          ok "kubectl installed on VM"

+      # The ecs suite boots its container instance from the ecs-node AMI, found
+      # by its managed-by tag, so the tag matters. The build directory is named
+      # for the manifest's IMAGE_NAME (ecs-agent), not for the make target.
+      - name: Build ecs-node AMI (libguestfs, runner)
+        if: contains(env.SUITES, 'e2e-ecs')
+        working-directory: spinifex
+        run: |
+          set -euo pipefail
+          # direct backend = no libvirt. Set here rather than in the script:
+          # it has to be in the environment the build itself runs in.
+          export LIBGUESTFS_BACKEND=direct
+          "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ensure-libguestfs.sh"
+          make build-ecs-node-image
+
+      - name: Import ecs-node AMI on VM
+        if: contains(env.SUITES, 'e2e-ecs')
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          eval "$(grep -E '^(ALPINE_VERSION|AMI_NAME|SYSTEM_TAG)=' \
+            spinifex/scripts/images/ecs-agent/manifest.conf)"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          banner "import ecs-node AMI to VM image store"
+          scp "${SSH_OPTS[@]}" \
+            "${SYSTEM_IMAGE_BUILD_DIR}/ecs-agent-image-build/ecs-agent-alpine.raw" \
+            "$VM":/tmp/ecs-agent-alpine.raw
+          ssh "${SSH_OPTS[@]}" "$VM" \
+            "spx admin images import --file /tmp/ecs-agent-alpine.raw \
+               --distro alpine --version ${ALPINE_VERSION} --arch x86_64 \
+               --boot-mode bios --ami-name ${AMI_NAME} --tag ${SYSTEM_TAG}; \
+               rc=\$?; rm -f /tmp/ecs-agent-alpine.raw; exit \$rc" >/dev/null 2>&1
+          ok "ecs-node AMI imported"
+
      # CreateDBInstance boots the DB VM from an RDS AMI, so both engines' images
      # must be in the VM's image store before the suite runs — same split as the
      # eks-node pair above: build on the runner (needs source + network), import
@@ -321,7 +355,7 @@ jobs:
            group_open "make build-${IMAGE}-image"
            make "build-${IMAGE}-image"
            group_close
-            ok "raw: $(du -h "/tmp/${IMAGE}-image-build/${IMAGE}-alpine.raw" | cut -f1)"
+            ok "raw: $(du -h "${SYSTEM_IMAGE_BUILD_DIR}/${IMAGE}-image-build/${IMAGE}-alpine.raw" | cut -f1)"
          done

      - name: Import RDS AMIs on VM
@@ -343,12 +377,13 @@ jobs:
            for tag in ${SYSTEM_TAG}; do TAG_ARGS="${TAG_ARGS} --tag ${tag}"; done
            banner "import ${IMAGE} AMI to VM image store"
            scp "${SSH_OPTS[@]}" \
-              "/tmp/${IMAGE}-image-build/${IMAGE}-alpine.raw" \
+              "${SYSTEM_IMAGE_BUILD_DIR}/${IMAGE}-image-build/${IMAGE}-alpine.raw" \
              "$VM":"/tmp/${IMAGE}-alpine.raw"
            ssh "${SSH_OPTS[@]}" "$VM" \
              "spx admin images import --file /tmp/${IMAGE}-alpine.raw \
                 --distro alpine --version ${ALPINE_VERSION} --arch x86_64 \
-                 --boot-mode bios --ami-name ${AMI_NAME}${TAG_ARGS}" >/dev/null 2>&1
+                 --boot-mode bios --ami-name ${AMI_NAME}${TAG_ARGS}; \
+                 rc=\$?; rm -f /tmp/${IMAGE}-alpine.raw; exit \$rc" >/dev/null 2>&1
            ok "${IMAGE} AMI imported"
          done

@@ -526,7 +561,7 @@ jobs:
        uses: $/.github/actions/junit-to-es
        with:
          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
-          cell: single
+          cell: ${{ env.E2E_CELL }}

      - name: Analyze e2e failures
        if: always()
@@ -546,7 +581,7 @@ jobs:
      - uses: actions/upload-artifact@v7.0.2
        if: always()
        with:
-          name: e2e-artifacts-${{ github.run_id }}
+          name: ${{ matrix.leg == 'base' && format('e2e-artifacts-{0}', github.run_id) || format('e2e-artifacts-{0}-{1}', github.run_id, matrix.leg) }}
          path: ${{ env.ARTIFACT_DIR }}
          if-no-files-found: warn
          retention-days: 14
@@ -589,7 +624,7 @@ jobs:
        if: always()
        uses: actions/upload-artifact@v7.0.2
        with:
-          name: tofu-log-single-${{ github.run_id }}
+          name: ${{ matrix.leg == 'base' && format('tofu-log-single-{0}', github.run_id) || format('tofu-log-single-{0}-{1}', github.run_id, matrix.leg) }}
          path: ${{ env.TOFU_LOG }}
          if-no-files-found: ignore
          retention-days: 14
@@ -605,15 +640,13 @@ jobs:

  e2e_multi:
    name: Multi-Node E2E
-    # The dispatch check gates on exactly the suites this job's Run step
-    # allow-lists (multinode|lb|cert) so lb — which only runs here — still
-    # provisions the 3-node env when requested on its own, and the env stays
-    # unprovisioned when none of them are asked for.
-    if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-multinode') || contains(github.event.inputs.suites, 'e2e-lb') || contains(github.event.inputs.suites, 'e2e-cert')
+    # Runs when the plan job selected any multi-node suite, and only then.
+    if: needs.plan.outputs.multi_suites != ''
    needs: plan
    runs-on: [self-hosted, ci-multi]
    timeout-minutes: ${{ fromJSON(needs.plan.outputs.multi_timeout) }}
    env:
+      SUITES: ${{ needs.plan.outputs.multi_suites }}
      TOFU_LOG: /tmp/e2e-tofu-multi-${{ github.run_id }}.log
      ARTIFACT_DIR: /tmp/spinifex-e2e-multi-artifacts-${{ github.run_id }}
      # See the matching note on e2e_single: ci.cell is what separates this
@@ -1070,7 +1103,10 @@ jobs:
  e2e_status:
    name: Report E2E Status
    if: always()
-    needs: [e2e_single, e2e_multi]
+    # plan is a dependency because a failed plan skips both e2e jobs, and a
+    # skipped job counts as passed below: without it a broken selection would
+    # report green having run nothing.
+    needs: [plan, e2e_single, e2e_multi]
    runs-on: ubuntu-26.04
    timeout-minutes: 5
    steps:
@@ -1079,15 +1115,16 @@ jobs:
          GH_TOKEN: ${{ github.token }}
          REPORT_SHA: ${{ github.sha }}
        run: |
-          single_ok=true; multi_ok=true
+          plan_ok=true; single_ok=true; multi_ok=true
+          [ "${{ needs.plan.result }}" = success ] || plan_ok=false
          case "${{ needs.e2e_single.result }}" in success|skipped) ;; *) single_ok=false ;; esac
          case "${{ needs.e2e_multi.result }}" in success|skipped) ;; *) multi_ok=false ;; esac
-          if $single_ok && $multi_ok; then
+          if $plan_ok && $single_ok && $multi_ok; then
            STATE="success"
            DESC="E2E suites passed"
          else
            STATE="failure"
-            DESC="E2E failed: single=${{ needs.e2e_single.result }} multi=${{ needs.e2e_multi.result }}"
+            DESC="E2E failed: plan=${{ needs.plan.result }} single=${{ needs.e2e_single.result }} multi=${{ needs.e2e_multi.result }}"
          fi
          gh api "repos/${{ github.repository }}/statuses/${REPORT_SHA}" \
            -f state="$STATE" \
@@ -1104,7 +1141,7 @@ jobs:
  # status-report check.
  e2e_es_summary:
    name: ES CI Run Summary
-    needs: [e2e_single, e2e_multi]
+    needs: [plan, e2e_single, e2e_multi]
    if: always()
    runs-on: [self-hosted, ci-single]
    timeout-minutes: 5
@@ -1116,6 +1153,11 @@ jobs:
      # this host, so no credentials are needed for the POST.
      MULGA_ES_SINK: ${{ vars.MULGA_ES_SINK || '192.168.1.13:9200' }}
    steps:
+      # The workspace outlives the run on a self-hosted runner, and the status
+      # files are collected recursively, so an earlier run's would count here.
+      - name: Clear previous summary downloads
+        run: rm -rf "${GITHUB_WORKSPACE}/es-summary-single" "${GITHUB_WORKSPACE}/es-summary-multi"
+
      # Either leg may not have run (e.g. a workflow_dispatch that only
      # requested single-node suites), so a missing artifact is tolerated —
      # the aggregation script below treats an absent suite-status.txt as
@@ -1124,7 +1166,7 @@ jobs:
        uses: actions/download-artifact@v8.0.2
        continue-on-error: true
        with:
-          name: e2e-artifacts-${{ github.run_id }}
+          pattern: e2e-artifacts-${{ github.run_id }}*
          path: ${{ github.workspace }}/es-summary-single

      - name: Download multi-node artifacts
@@ -1144,6 +1186,7 @@ jobs:
          BRANCH: ${{ github.head_ref || github.ref_name }}
          SHA: ${{ github.sha }}
          SERVER_URL: ${{ github.server_url }}
+          PLAN_RESULT: ${{ needs.plan.result }}
          SINGLE_RESULT: ${{ needs.e2e_single.result }}
          MULTI_RESULT: ${{ needs.e2e_multi.result }}
          SINGLE_DIR: ${{ github.workspace }}/es-summary-single
@@ -1151,7 +1194,8 @@ jobs:
        run: |
          set -euo pipefail

-          SINGLE_STATUS="${SINGLE_DIR}/suite-status.txt"
+          # One subdirectory per single-node leg's artifact.
+          SINGLE_STATUS=$(find "${SINGLE_DIR}" -name suite-status.txt 2>/dev/null | sort || true)
          MULTI_STATUS="${MULTI_DIR}/suite-status.txt"

          # Collect "<suite>: FAILED" lines from both legs' suite-status.txt
@@ -1160,7 +1204,9 @@ jobs:
          # rather than a bash array so this stays correct under `set -u`
          # even when nothing failed.
          FAILED_SUITES=""
-          for f in "$SINGLE_STATUS" "$MULTI_STATUS"; do
+          # SINGLE_STATUS is a newline-separated list, so it is split here.
+          # shellcheck disable=SC2086
+          for f in ${SINGLE_STATUS} "$MULTI_STATUS"; do
            [ -f "$f" ] || continue
            while IFS= read -r line; do
              case "$line" in
@@ -1180,7 +1226,7 @@ jobs:
          case "$MULTI_RESULT" in success | skipped) ;; *) multi_ok=false ;; esac

          STATE=success
-          if [ "$single_ok" != true ] || [ "$multi_ok" != true ] || [ "$FAILED_COUNT" -gt 0 ]; then
+          if [ "$PLAN_RESULT" != success ] || [ "$single_ok" != true ] || [ "$multi_ok" != true ] || [ "$FAILED_COUNT" -gt 0 ]; then
            STATE=failure
          fi

@@ -1,33 +0,0 @@
-name: Mulga design
-
-# Reminds the author to say whether the Mulga design records need to change.
-# It reads only the pull request body, uses no secrets, and warns without ever
-# failing: the check is a prompt, not a gate.
-
-on:
-  pull_request:
-    branches:
-      - dev
-    types: [opened, edited, reopened]
-
-permissions:
-  contents: read
-
-jobs:
-  mulga_design:
-    name: Mulga design line
-    if: ${{ !endsWith(github.actor, '[bot]') }}
-    runs-on: ubuntu-26.04
-    steps:
-      - name: Checkout Code
-        uses: actions/checkout@v7.0.1
-        with:
-          sparse-checkout: .github/scripts
-
-      - name: Check the Mulga design line
-        env:
-          PR_BODY: ${{ github.event.pull_request.body }}
-        working-directory: .github/scripts
-        run: |
-          python3 -m unittest -q test_mulga_design_check
-          python3 mulga_design_check.py
@@ -113,7 +113,7 @@ jobs:
        id: siblings
        env:
          BRANCH: ${{ env.GIT_BRANCH }}
-        run: bash .github/resolve-sibling-refs.sh
+        run: bash .github/scripts/resolve-sibling-refs.sh
        working-directory: spinifex

      - uses: actions/checkout@v7.0.1
@@ -84,7 +84,7 @@ jobs:

      - name: Resolve sibling release tags from go.mod
        id: siblings
-        run: bash spinifex/.github/resolve-release-sibling-refs.sh spinifex/go.mod
+        run: bash spinifex/.github/scripts/resolve-release-sibling-refs.sh spinifex/go.mod

      - name: Checkout Viperblock
        uses: actions/checkout@v7.0.1
@@ -18,7 +18,7 @@ jobs:

      - name: Resolve sibling release tags from go.mod
        id: siblings
-        run: bash spinifex/.github/resolve-release-sibling-refs.sh spinifex/go.mod
+        run: bash spinifex/.github/scripts/resolve-release-sibling-refs.sh spinifex/go.mod

      - name: Checkout Predastore
        uses: actions/checkout@v7.0.1
@@ -58,7 +58,7 @@ jobs:
          cache: false

      - name: Build microVM artifacts (lb-agent + kernel + initramfs + qboot + SQS store and worker images)
-        run: cd spinifex && make build-microvm-image build-sqs-store-microvm-image build-sqs-worker-microvm-image
+        run: cd spinifex && make build-microvm-image build-sqs-worker-microvm-image

      - name: Build amd64 Tarball
        env:
@@ -85,7 +85,7 @@ jobs:

      - name: Resolve sibling release tags from go.mod
        id: siblings
-        run: bash spinifex/.github/resolve-release-sibling-refs.sh spinifex/go.mod
+        run: bash spinifex/.github/scripts/resolve-release-sibling-refs.sh spinifex/go.mod

      - name: Checkout Predastore
        uses: actions/checkout@v7.0.1
@@ -335,7 +335,7 @@ jobs:

      - name: Resolve sibling release tags from go.mod
        id: siblings
-        run: bash spinifex/.github/resolve-release-sibling-refs.sh spinifex/go.mod
+        run: bash spinifex/.github/scripts/resolve-release-sibling-refs.sh spinifex/go.mod

      - name: Checkout Predastore
        uses: actions/checkout@v7.0.1
@@ -34,7 +34,7 @@ jobs:
        id: siblings
        env:
          BRANCH: ${{ github.head_ref || github.ref_name }}
-        run: bash .github/resolve-sibling-refs.sh
+        run: bash .github/scripts/resolve-sibling-refs.sh
        working-directory: spinifex

      - name: Checkout Viperblock
@@ -20,6 +20,12 @@ on:
 permissions:
  contents: read

+# A newer push to the same pull request supersedes its run. Pushes and dispatches
+# get a group of their own, so every commit on dev and main keeps its result.
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && format('pr-{0}', github.event.pull_request.number) || github.run_id }}
+  cancel-in-progress: true
+
 jobs:
  build_and_test:
    name: Build and Test
@@ -36,7 +42,7 @@ jobs:
        id: siblings
        env:
          BRANCH: ${{ github.head_ref || github.ref_name }}
-        run: bash .github/resolve-sibling-refs.sh
+        run: bash .github/scripts/resolve-sibling-refs.sh
        working-directory: spinifex

      - name: Checkout Viperblock
@@ -99,6 +105,13 @@ jobs:
          key: gobuild-${{ github.job }}-${{ github.run_id }}
          restore-keys: gobuild-${{ github.job }}-

+      # Go refreshes an entry's mtime on use only when it is over an hour old, so
+      # restored entries are backdated past that before any Go command runs and
+      # the trim below can tell this run's working set from the last run's.
+      - name: Mark Go build cache
+        if: ${{ github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/main' }}
+        run: spinifex/.github/scripts/go-build-cache.sh mark
+
      - name: Setup Go Workspace
        run: go work init ./spinifex ./viperblock ./predastore ./northstar ./bluebottle

@@ -131,19 +144,12 @@ jobs:
      - name: Check Diff Coverage
        run: cd spinifex && scripts/diff-coverage.sh coverage.out --base HEAD~1

-      # Compiled output stays valid whatever the tests said, and skipping the
-      # save on a red run would leave the next one rebuilding it, so this is
-      # gated on the branch rather than on success.
-      # Go bumps an entry's mtime when a run uses it, so this run's working set
-      # survives and objects superseded by later commits do not. Runs after the
-      # work, so it costs this job nothing and only shrinks what the next reads.
+      # Keeps what this run read or wrote and drops the rest, so the saved entry
+      # is one run's working set however often dev is pushed. A red run may have
+      # stopped early and used only part of the set, so it saves untrimmed.
      - name: Trim Go build cache
-        if: ${{ !cancelled() && (github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/main') }}
-        run: |
-          C="$(go env GOCACHE)"
-          before=$(du -sm "$C" | cut -f1)
-          find "$C" -type f -mmin +240 -delete
-          echo "trimmed ${before}MB -> $(du -sm "$C" | cut -f1)MB"
+        if: ${{ success() && (github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/main') }}
+        run: spinifex/.github/scripts/go-build-cache.sh trim

      - name: Save Go build cache
        if: ${{ !cancelled() && (github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/main') }}
@@ -154,6 +160,7 @@ jobs:

  race_detection:
    name: Race Detection
+    if: ${{ github.event_name != 'pull_request' }}
    runs-on: ubuntu-26.04

    steps:
@@ -166,7 +173,7 @@ jobs:
        id: siblings
        env:
          BRANCH: ${{ github.head_ref || github.ref_name }}
-        run: bash .github/resolve-sibling-refs.sh
+        run: bash .github/scripts/resolve-sibling-refs.sh
        working-directory: spinifex

      - name: Checkout Viperblock
@@ -229,22 +236,25 @@ jobs:
          key: gobuild-${{ github.job }}-${{ github.run_id }}
          restore-keys: gobuild-${{ github.job }}-

+      # Go refreshes an entry's mtime on use only when it is over an hour old, so
+      # restored entries are backdated past that before any Go command runs and
+      # the trim below can tell this run's working set from the last run's.
+      - name: Mark Go build cache
+        if: ${{ github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/main' }}
+        run: spinifex/.github/scripts/go-build-cache.sh mark
+
      - name: Setup Go Workspace
        run: go work init ./spinifex ./viperblock ./predastore ./northstar ./bluebottle

      - name: Run Tests with Race Detector
        run: make -C spinifex test-race GOTESTFLAGS=-count=1

-      # Go bumps an entry's mtime when a run uses it, so this run's working set
-      # survives and objects superseded by later commits do not. Runs after the
-      # work, so it costs this job nothing and only shrinks what the next reads.
+      # Keeps what this run read or wrote and drops the rest, so the saved entry
+      # is one run's working set however often dev is pushed. A red run may have
+      # stopped early and used only part of the set, so it saves untrimmed.
      - name: Trim Go build cache
-        if: ${{ !cancelled() && (github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/main') }}
-        run: |
-          C="$(go env GOCACHE)"
-          before=$(du -sm "$C" | cut -f1)
-          find "$C" -type f -mmin +240 -delete
-          echo "trimmed ${before}MB -> $(du -sm "$C" | cut -f1)MB"
+        if: ${{ success() && (github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/main') }}
+        run: spinifex/.github/scripts/go-build-cache.sh trim

      - name: Save Go build cache
        if: ${{ !cancelled() && (github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/main') }}
@@ -268,7 +278,7 @@ jobs:
        id: siblings
        env:
          BRANCH: ${{ github.head_ref || github.ref_name }}
-        run: bash .github/resolve-sibling-refs.sh
+        run: bash .github/scripts/resolve-sibling-refs.sh
        working-directory: spinifex

      - name: Checkout Viperblock
@@ -331,6 +341,13 @@ jobs:
          key: gobuild-${{ github.job }}-${{ github.run_id }}
          restore-keys: gobuild-${{ github.job }}-

+      # Go refreshes an entry's mtime on use only when it is over an hour old, so
+      # restored entries are backdated past that before any Go command runs and
+      # the trim below can tell this run's working set from the last run's.
+      - name: Mark Go build cache
+        if: ${{ github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/main' }}
+        run: spinifex/.github/scripts/go-build-cache.sh mark
+
      - name: Setup Go Workspace
        run: go work init ./spinifex ./viperblock ./predastore ./northstar ./bluebottle

@@ -347,16 +364,16 @@ jobs:
        if: ${{ !cancelled() }}
        run: make -C spinifex test-integration

-      # Go bumps an entry's mtime when a run uses it, so this run's working set
-      # survives and objects superseded by later commits do not. Runs after the
-      # work, so it costs this job nothing and only shrinks what the next reads.
+      - name: Run Gated Suites
+        if: ${{ !cancelled() }}
+        run: make -C spinifex test-gated GOTESTFLAGS=-count=1
+
+      # Keeps what this run read or wrote and drops the rest, so the saved entry
+      # is one run's working set however often dev is pushed. A red run may have
+      # stopped early and used only part of the set, so it saves untrimmed.
      - name: Trim Go build cache
-        if: ${{ !cancelled() && (github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/main') }}
-        run: |
-          C="$(go env GOCACHE)"
-          before=$(du -sm "$C" | cut -f1)
-          find "$C" -type f -mmin +240 -delete
-          echo "trimmed ${before}MB -> $(du -sm "$C" | cut -f1)MB"
+        if: ${{ success() && (github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/main') }}
+        run: spinifex/.github/scripts/go-build-cache.sh trim

      - name: Save Go build cache
        if: ${{ !cancelled() && (github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/main') }}
@@ -30,6 +30,12 @@ on:
 permissions:
  contents: read

+# A newer push to the same pull request supersedes its run. Pushes and dispatches
+# get a group of their own, so every commit on dev and main keeps its result.
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && format('pr-{0}', github.event.pull_request.number) || github.run_id }}
+  cancel-in-progress: true
+
 jobs:
  ui_build_and_test:
    name: UI Build and Test
@@ -21,12 +21,13 @@ build/microvm/vmlinuz
 build/microvm/initramfs.cpio.gz
 build/microvm/qboot.rom
 build/microvm/qboot.LICENSE
-build/microvm/sqs-store/vmlinuz
-build/microvm/sqs-store/initramfs.cpio.gz
-build/microvm/sqs-store/.stage.*/
 build/microvm/sqs-worker/vmlinuz
 build/microvm/sqs-worker/initramfs.cpio.gz
 build/microvm/sqs-worker/.stage.*/
+build/microvm/lambda-runtime/vmlinuz
+build/microvm/lambda-runtime/initramfs.cpio.gz
+build/microvm/lambda-runtime/.stage.*/
+build/microvm/lambda-testlink-runtime/
 build/qboot/
 !spinifex/runtime/roles/spinifexui/frontend/dist/
 tmp/
@@ -42,11 +42,15 @@ ifdef QUIET
  _COVQ  = 2>&1 | { grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' | grep -v 'coverage: 0\.0%' || true; }; exit $${PIPESTATUS[0]}
  _RACEQ = 2>&1 | { grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' || true; }; exit $${PIPESTATUS[0]}
  _SECQ  = >
+  _HDR   = :
+  _RUNSH = sh "$$t" >"$$log" 2>&1 || { echo "-- $$t"; cat "$$log"; exit 1; }
 else
  _Q     =
  _COVQ  =
  _RACEQ =
  _SECQ  = 2>&1 | tee
+  _HDR   = echo -e
+  _RUNSH = echo "-- $$t"; sh "$$t" || exit 1
 endif

 build: go_build build-installer build-lb-agent generate-aws-model-coverage
@@ -78,6 +82,35 @@ go_build:
 	@echo -e "\n....Building $(GO_PROJECT_NAME)"
 	go build $(GO_BUILD_MOD) -ldflags "$(LDFLAGS)" -o ./bin/$(GO_PROJECT_NAME) cmd/spinifex/main.go

+# Controlled-test only: builds the tagged daemon and its hard-kill initializer
+# together, pinning the initializer's digest into that daemon build. The daemon
+# keeps its symbols so the pin can be read back from the binary.
+# LAMBDA_IMAGE_DIGEST is the test-link runtime image digest that
+# build-lambda-testlink-image prints; it is read raw and validated before use.
+LAMBDA_TESTLINK_DIR := ./bin/lambda-testlink
+LAMBDA_TESTLINK_PKG := github.com/mulgadc/spinifex/spinifex/runtime/daemon
+LAMBDA_TESTLINK_PIN := $(LAMBDA_TESTLINK_PKG).lambdaHardKillSHA256
+LAMBDA_DIGEST_RE := ^[0-9a-f]{64}$$
+build-lambda-testlink: override export LAMBDA_IMAGE_DIGEST := $(value LAMBDA_IMAGE_DIGEST)
+build-lambda-testlink:
+	@[[ "$$LAMBDA_IMAGE_DIGEST" =~ $(LAMBDA_DIGEST_RE) ]] || { echo "LAMBDA_IMAGE_DIGEST must be 64 lowercase hex digits"; exit 1; }
+	@host=$$(uname -m); goos=$$(go env GOOS); goarch=$$(go env GOARCH); \
+	test "$$host" = x86_64 && test "$$goos/$$goarch" = linux/amd64 || { echo "the test link supports only an x86_64 host building linux/amd64, not $$host building $$goos/$$goarch"; exit 1; }
+	@rm -f $(LAMBDA_TESTLINK_DIR)/hardkill.so $(LAMBDA_TESTLINK_DIR)/$(GO_PROJECT_NAME)
+	@mkdir -p $(LAMBDA_TESTLINK_DIR).tmp
+	@rm -f $(LAMBDA_TESTLINK_DIR).tmp/hardkill.so $(LAMBDA_TESTLINK_DIR).tmp/$(GO_PROJECT_NAME)
+	cc -std=c17 -O2 -Wall -Wextra -Werror -shared -fPIC -o $(LAMBDA_TESTLINK_DIR).tmp/hardkill.so spinifex/runtime/compute/lambda/testlink/hardkill/hardkill.c -lrt
+	scripts/check-glibc-ceiling.sh 2.34 $(LAMBDA_TESTLINK_DIR).tmp/hardkill.so
+	@digest=$$(sha256sum $(LAMBDA_TESTLINK_DIR).tmp/hardkill.so | cut -d' ' -f1); \
+	test $${#digest} -eq 64 || { echo "initializer digest missing"; exit 1; }; \
+	echo "initializer sha256 $$digest"; \
+	pin="-X $(LAMBDA_TESTLINK_PIN)=$$digest -X $(LAMBDA_TESTLINK_PKG).lambdaImageDigest=$$LAMBDA_IMAGE_DIGEST"; \
+	LAMBDA_TESTLINK_INITIALIZER=$(abspath $(LAMBDA_TESTLINK_DIR).tmp/hardkill.so) go test -count=1 -tags lambdatestlink -ldflags "$$pin" -run '^TestLambda(HardKillPin|ImageDigestPin)$$' ./spinifex/runtime/daemon/ && \
+	go build $(GO_BUILD_MOD) -tags lambdatestlink -ldflags "$(subst -s -w ,,$(LDFLAGS)) $$pin" -o $(LAMBDA_TESTLINK_DIR).tmp/$(GO_PROJECT_NAME) cmd/spinifex/main.go
+	@mkdir -p $(LAMBDA_TESTLINK_DIR)
+	mv $(LAMBDA_TESTLINK_DIR).tmp/hardkill.so $(LAMBDA_TESTLINK_DIR).tmp/$(GO_PROJECT_NAME) $(LAMBDA_TESTLINK_DIR)/
+	@rmdir $(LAMBDA_TESTLINK_DIR).tmp
+
 build-installer:
 	@echo -e "\n....Building spinifex-installer"
 	go build -ldflags "-s -w" -o ./bin/spinifex-installer cmd/installer/main.go
@@ -153,37 +186,64 @@ $(MICROVM_ARTIFACTS) &: $(MICROVM_INPUTS)
 bin/lb-agent:
 	$(MAKE) build-lb-agent

-# The SQS store microVM image: a separate artifact from the LB image above, not
-# installed or released by any target yet.
-SQS_STORE_MICROVM_OUT_DIR := $(MICROVM_OUT_DIR)/sqs-store
-SQS_STORE_MICROVM_ARTIFACTS := $(SQS_STORE_MICROVM_OUT_DIR)/vmlinuz $(SQS_STORE_MICROVM_OUT_DIR)/initramfs.cpio.gz
-SQS_STORE_MICROVM_INPUTS := scripts/build-sqs-store-microvm-image.sh scripts/lib/sqs-store-microvm-publish.sh $(SQS_STORE_MICROVM_OUT_DIR)/init.sh $(SQS_STORE_MICROVM_OUT_DIR)/inittab $(addprefix $(SQS_STORE_MICROVM_OUT_DIR)/,sqs-store-lib.sh sqs-store-prepare sqs-store-run sqs-store-stop) bin/nats-server
-
-$(SQS_STORE_MICROVM_ARTIFACTS) &: $(SQS_STORE_MICROVM_INPUTS)
-	./scripts/build-sqs-store-microvm-image.sh
-
-# The dedicated SQS store server, built from the go.mod-pinned module so the
-# image ships exactly the reviewed version. Static for the musl guest.
+# The dedicated SQS server, built from the go.mod-pinned module so the image
+# ships exactly the reviewed version. Static for the musl guest.
 bin/nats-server: go.mod go.sum
 	@mkdir -p bin
 	GOWORK=off CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags '-s -w' -o $@ github.com/nats-io/nats-server/v2

-build-sqs-store-microvm-image: $(SQS_STORE_MICROVM_ARTIFACTS) ## Build the SQS store microVM kernel and initramfs
+# go_local_srcs lists the main module package dirs and files guest package $(1)
+# links, scanned with the guest build settings; a dir's mtime catches a deleted
+# or renamed file. A failed scan names a missing prerequisite, so make stops.
+go_local_srcs = $(shell GOWORK=off CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go list -deps -f '{{if and .Module .Module.Main}}{{.Dir}} {{range .GoFiles}}{{$$.Dir}}/{{.}} {{end}}{{range .EmbedFiles}}{{$$.Dir}}/{{.}} {{end}}{{end}}' $(1) || echo go-list-failed-for-$(1))
+.SECONDEXPANSION:

-# The shared SQS worker microVM image: the store image's /init and kernel with
-# the worker binary instead of nats-server. Not installed or released yet.
+# The SQS microVM image: one guest running its own SQS nats-server and the SQS
+# worker as separate processes. A separate artifact from the LB image above.
 SQS_WORKER_MICROVM_OUT_DIR := $(MICROVM_OUT_DIR)/sqs-worker
 SQS_WORKER_MICROVM_ARTIFACTS := $(SQS_WORKER_MICROVM_OUT_DIR)/vmlinuz $(SQS_WORKER_MICROVM_OUT_DIR)/initramfs.cpio.gz
-SQS_WORKER_MICROVM_INPUTS := scripts/build-sqs-store-microvm-image.sh scripts/lib/sqs-store-microvm-publish.sh $(SQS_STORE_MICROVM_OUT_DIR)/init.sh $(SQS_WORKER_MICROVM_OUT_DIR)/inittab $(addprefix $(SQS_WORKER_MICROVM_OUT_DIR)/,sqs-worker-lib.sh sqs-worker-prepare sqs-worker-ready sqs-worker-stop) bin/sqs-worker
+SQS_WORKER_MICROVM_INPUTS := scripts/build-sqs-store-microvm-image.sh scripts/lib/microvm-image.sh scripts/lib/sqs-store-microvm-publish.sh $(SQS_WORKER_MICROVM_OUT_DIR)/init.sh $(SQS_WORKER_MICROVM_OUT_DIR)/inittab $(addprefix $(SQS_WORKER_MICROVM_OUT_DIR)/,sqs-worker-lib.sh sqs-worker-prepare sqs-worker-store sqs-worker-ready sqs-worker-stop) bin/nats-server bin/sqs-worker

 $(SQS_WORKER_MICROVM_ARTIFACTS) &: $(SQS_WORKER_MICROVM_INPUTS)
-	SQS_MICROVM_PROFILE=worker ./scripts/build-sqs-store-microvm-image.sh
+	./scripts/build-sqs-store-microvm-image.sh

-bin/sqs-worker: go.mod go.sum $(shell find cmd/sqs-worker spinifex/agents/sqs -name '*.go' ! -name '*_test.go')
+bin/sqs-worker: go.mod go.sum $$(call go_local_srcs,./cmd/sqs-worker)
 	@mkdir -p bin
 	GOWORK=off CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags '-s -w' -o $@ ./cmd/sqs-worker

-build-sqs-worker-microvm-image: $(SQS_WORKER_MICROVM_ARTIFACTS) ## Build the shared SQS worker microVM kernel and initramfs
+build-sqs-worker-microvm-image: $(SQS_WORKER_MICROVM_ARTIFACTS) ## Build the SQS microVM kernel and initramfs
+
+# The Lambda runtime microVM image. Built on demand; not shipped or installed.
+LAMBDA_RUNTIME_MICROVM_OUT_DIR := $(MICROVM_OUT_DIR)/lambda-runtime
+LAMBDA_RUNTIME_MICROVM_ARTIFACTS := $(LAMBDA_RUNTIME_MICROVM_OUT_DIR)/vmlinuz $(LAMBDA_RUNTIME_MICROVM_OUT_DIR)/initramfs.cpio.gz
+LAMBDA_RUNTIME_MICROVM_INPUTS := scripts/build-lambda-runtime-microvm-image.sh scripts/lib/microvm-image.sh $(addprefix $(LAMBDA_RUNTIME_MICROVM_OUT_DIR)/,init.sh inittab run-agent) bin/lambda-agent
+
+$(LAMBDA_RUNTIME_MICROVM_ARTIFACTS) &: $(LAMBDA_RUNTIME_MICROVM_INPUTS)
+	./scripts/build-lambda-runtime-microvm-image.sh
+
+bin/lambda-agent: go.mod go.sum $$(call go_local_srcs,./cmd/lambda-agent)
+	@mkdir -p bin
+	GOWORK=off CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags '-s -w' -o $@ ./cmd/lambda-agent
+
+build-lambda-runtime-microvm-image: $(LAMBDA_RUNTIME_MICROVM_ARTIFACTS) ## Build the Lambda runtime microVM kernel and initramfs (not shipped)
+	GOWORK=off go run ./cmd/lambda-image-agent-tag $(LAMBDA_RUNTIME_MICROVM_OUT_DIR)/initramfs.cpio.gz absent
+
+# Controlled-test only: a separately built guest whose agent has the
+# lambdatestlink tag, in its own outputs, never touching bin/lambda-agent or the
+# default image. It prints the digest to pass as LAMBDA_IMAGE_DIGEST.
+LAMBDA_TESTLINK_IMAGE_DIR := $(MICROVM_OUT_DIR)/lambda-testlink-runtime
+build-lambda-testlink-image: ## Build the test-link Lambda guest image and print its digest (not shipped)
+	@host=$$(uname -m); goos=$$(go env GOOS); goarch=$$(go env GOARCH); \
+	test "$$host" = x86_64 && test "$$goos/$$goarch" = linux/amd64 || { echo "the test link supports only an x86_64 host building linux/amd64, not $$host building $$goos/$$goarch"; exit 1; }
+	$(MAKE) --no-print-directory $(MICROVM_OUT_DIR)/qboot.rom
+	@mkdir -p bin $(LAMBDA_TESTLINK_IMAGE_DIR)
+	GOWORK=off GOFLAGS= CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -tags lambdatestlink -ldflags '-s -w' -o bin/lambda-agent-testlink ./cmd/lambda-agent
+	./scripts/build-lambda-runtime-microvm-image.sh bin/lambda-agent-testlink $(LAMBDA_TESTLINK_IMAGE_DIR)
+	GOWORK=off go run ./cmd/lambda-image-agent-tag $(LAMBDA_TESTLINK_IMAGE_DIR)/initramfs.cpio.gz present
+	install -m 0644 $(MICROVM_OUT_DIR)/qboot.rom $(LAMBDA_TESTLINK_IMAGE_DIR)/qboot.rom
+	@digest=$$(GOWORK=off go run -tags lambdatestlink ./cmd/lambda-image-digest $(LAMBDA_TESTLINK_IMAGE_DIR)) && \
+	[[ "$$digest" =~ $(LAMBDA_DIGEST_RE) ]] || { echo "image digest not computed"; exit 1; }; \
+	echo "LAMBDA_IMAGE_DIGEST=$$digest"

 # qboot is the microvm firmware. The source tarball ships with each release
 # because qboot is GPL-2.0.
@@ -196,13 +256,7 @@ $(QBOOT_ARTIFACTS) &: scripts/build-qboot.sh

 build-microvm-image: $(MICROVM_ARTIFACTS) $(QBOOT_ARTIFACTS) ## Build microVM kernel, initramfs and qboot firmware (incremental — skips when up to date)

-install-sqs-store-microvm: $(SQS_STORE_MICROVM_ARTIFACTS) ## Install the SQS store microVM image to /usr/share/spinifex/microvm/sqs-store/
-	sudo install -d /usr/share/spinifex/microvm/sqs-store
-	sudo rm -f /usr/share/spinifex/microvm/sqs-store/initramfs.cpio.gz
-	sudo install -m 0644 $(SQS_STORE_MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/sqs-store/vmlinuz
-	sudo install -m 0644 $(SQS_STORE_MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/sqs-store/initramfs.cpio.gz
-
-install-sqs-worker-microvm: $(SQS_WORKER_MICROVM_ARTIFACTS) ## Install the SQS worker microVM image to /usr/share/spinifex/microvm/sqs-worker/
+install-sqs-worker-microvm: $(SQS_WORKER_MICROVM_ARTIFACTS) ## Install the SQS microVM image to /usr/share/spinifex/microvm/sqs-worker/
 	sudo install -d /usr/share/spinifex/microvm/sqs-worker
 	sudo rm -f /usr/share/spinifex/microvm/sqs-worker/initramfs.cpio.gz
 	sudo install -m 0644 $(SQS_WORKER_MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/sqs-worker/vmlinuz
@@ -237,26 +291,30 @@ preflight:
 NODE_SCRIPTS := scripts/setup.sh scripts/setup-ovn.sh scripts/install-node.sh scripts/smoke-test.sh scripts/node-reset.sh scripts/uninstall-spx.sh scripts/lib/distro.sh scripts/lib/stage-sqs-store-microvm.sh

 test-build-scripts:
-	@echo -e "\n....Running build/scripts/**/*_test.sh...."
-	@for t in $$(find build/scripts -name '*_test.sh' | sort); do \
-		echo "-- $$t"; \
-		sh "$$t" || exit 1; \
+	@$(_HDR) "\n....Running build/scripts/**/*_test.sh...."
+	@log=$$(mktemp); trap 'rm -f "$$log"' EXIT; for t in $$(find build/scripts -name '*_test.sh' | sort); do \
+		$(_RUNSH); \
 	done
-	@echo -e "\n....Running scripts/*_test.sh, scripts/lib/*_test.sh and scripts/appliance/*_test.sh...."
-	@for t in $$( (find scripts -maxdepth 1 -name '*_test.sh'; find scripts/lib scripts/appliance -name '*_test.sh') | sort); do \
-		echo "-- $$t"; \
-		sh "$$t" || exit 1; \
+	@$(_HDR) "\n....Running scripts/*_test.sh, scripts/lib/*_test.sh and scripts/appliance/*_test.sh...."
+	@log=$$(mktemp); trap 'rm -f "$$log"' EXIT; for t in $$( (find scripts -maxdepth 1 -name '*_test.sh'; find scripts/lib scripts/appliance -name '*_test.sh') | sort); do \
+		$(_RUNSH); \
 	done
-	@echo -e "\n....Running .github/scripts/*_test.sh...."
-	@for t in $$(find .github/scripts -maxdepth 1 -name '*_test.sh' | sort); do \
-		echo "-- $$t"; \
-		sh "$$t" || exit 1; \
+	@$(_HDR) "\n....Running scripts/terraform/latitude-spx/*_test.sh...."
+	@log=$$(mktemp); trap 'rm -f "$$log"' EXIT; for t in $$(find scripts/terraform/latitude-spx -maxdepth 1 -name '*_test.sh' | sort); do \
+		$(_RUNSH); \
 	done
-	shellcheck -S warning .github/scripts/hypervisor-disk-preflight.sh .github/scripts/hypervisor-disk-preflight_test.sh
-	@echo -e "\n....Running shellcheck over build/scripts/**/*.sh + the node scripts...."
-	shellcheck -S warning $$(find build/scripts -name '*.sh' | sort) $(NODE_SCRIPTS)
-	shellcheck -S warning -x scripts/build-oracle-appliance.sh $$(find scripts/appliance -name '*.sh' | sort)
-	@echo "  test-build-scripts ok"
+	@$(_HDR) "\n....Running .github/scripts/*_test.sh...."
+	@log=$$(mktemp); trap 'rm -f "$$log"' EXIT; for t in $$(find .github/scripts -maxdepth 1 -name '*_test.sh' | sort); do \
+		$(_RUNSH); \
+	done
+	$(_Q)shellcheck -S warning .github/scripts/hypervisor-disk-preflight.sh \
+		.github/scripts/wait-for-memory.sh \
+		.github/scripts/go-build-cache.sh \
+		.github/scripts/e2e-select.sh
+	@$(_HDR) "\n....Running shellcheck over build/scripts/**/*.sh + the node scripts...."
+	$(_Q)shellcheck -S warning $$(find build/scripts -name '*.sh' | sort) $(NODE_SCRIPTS)
+	$(_Q)shellcheck -S warning -x scripts/build-oracle-appliance.sh $$(find scripts/appliance -name '*.sh' | sort)
+	@$(_HDR) "  test-build-scripts ok"

 # E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
 # default `go test ./spinifex/...`. Runs with mocked AWS clients — no
@@ -287,6 +345,16 @@ test-integration:
 	@$(MAKE) --no-print-directory generate-aws-model-coverage
 	@echo "AWS operation coverage: $(AWS_MODEL_OPERATION_COVERAGE_DIR)"

+# Suites the unit tiers skip: the three-node SQS R3 scenarios behind
+# SPINIFEX_SQS_R3 and the lambdatestlink-tagged tests. CI runs these on every PR.
+test-gated:
+	@echo -e "\n....Running env- and tag-gated suites...."
+	$(_Q)LOG_IGNORE=1 SPINIFEX_SQS_R3=1 go test $(GOTESTFLAGS) -timeout 180s -run '^TestR3$$' ./spinifex/domains/sqs/message/jetstream/ $(_RACEQ)
+	$(_Q)LOG_IGNORE=1 go test $(GOTESTFLAGS) -tags=lambdatestlink -timeout 120s ./cmd/lambda-agent/ ./spinifex/agents/lambda/hostlink/ ./spinifex/runtime/compute/lambda/guestconn/ ./spinifex/runtime/compute/lambda/testlink/ ./spinifex/domains/network/host/ $(_RACEQ)
+	$(_Q)LOG_IGNORE=1 go test $(GOTESTFLAGS) -tags=lambdatestlink -timeout 120s -run '^TestLambdaTestLink(Executor|Reply)_' ./spinifex/runtime/roles/awsgw/ $(_RACEQ)
+	$(_Q)LOG_IGNORE=1 go test $(GOTESTFLAGS) -tags=lambdatestlink,lambdatestlinkseam -timeout 120s -run '^TestLambdaTestLinkSeam_' ./spinifex/runtime/roles/awsgw/ $(_RACEQ)
+	$(_Q)LOG_IGNORE=1 go test $(GOTESTFLAGS) -tags=lambdatestlink -timeout 120s -run '^TestLambdaTestLink' ./spinifex/runtime/daemon/ $(_RACEQ)
+
 generate-aws-model-coverage:
 	@go run ./cmd/aws-model-coverage -out $(AWS_MODEL_OPERATION_COVERAGE_DIR) -json $(AWS_MODEL_OPERATION_COVERAGE_DIR)/coverage.json

@@ -514,7 +582,6 @@ distro-amd64:
 		echo "[distro-amd64] WARNING: microvm artifacts missing — tarball will not include them"; \
 		echo "[distro-amd64]          run 'make build-microvm-image' before 'make distro-amd64'"; \
 	fi
-	./scripts/lib/stage-sqs-store-microvm.sh $(SQS_STORE_MICROVM_OUT_DIR) dist/amd64
 	./scripts/lib/stage-sqs-store-microvm.sh $(SQS_WORKER_MICROVM_OUT_DIR) dist/amd64 sqs-worker
 	tar -czf dist/spinifex-$(VERSION)-linux-amd64.tar.gz -C dist/amd64 .
 	sha256sum dist/spinifex-$(VERSION)-linux-amd64.tar.gz > dist/spinifex-$(VERSION)-linux-amd64.tar.gz.sha256
@@ -534,7 +601,7 @@ distro-arm64:
 distro-clean:
 	rm -rf dist/

-.PHONY: test-package-check build build-ui test-ui-e2e ui-e2e-container ui-baselines build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-rds-postgres-image import-rds-postgres-image build-rds-mariadb-image import-rds-mariadb-image build-microvm-image build-sqs-store-microvm-image build-sqs-worker-microvm-image install-microvm install-sqs-store-microvm install-sqs-worker-microvm go_build preflight test test-cover test-race diff-coverage bench fuzz test-actions test-images test-build-scripts test-harness test-integration generate-aws-model-coverage aws-model-coverage test-segscan-oracle \
+.PHONY: test-package-check build build-lambda-testlink build-lambda-testlink-image build-ui test-ui-e2e ui-e2e-container ui-baselines build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-rds-postgres-image import-rds-postgres-image build-rds-mariadb-image import-rds-mariadb-image build-microvm-image build-sqs-worker-microvm-image build-lambda-runtime-microvm-image install-microvm install-sqs-worker-microvm go_build preflight test test-cover test-race diff-coverage bench fuzz test-actions test-images test-build-scripts test-harness test-integration test-gated generate-aws-model-coverage aws-model-coverage test-segscan-oracle \
 	deploy reinstall clean \
 	install-system install-go install-aws quickinstall \
 	lint fix govulncheck nilaway \
@@ -70,8 +70,8 @@ The platform serves **405 AWS API operations** across the services below. See ex
 | **RDS** | Databases | PostgreSQL and MariaDB, with automated backups | Available |
 | **Route 53** | DNS | [Northstar](https://github.com/mulgadc/northstar): authoritative DNS with hosted zones | Available |
 | **Bedrock** | AI deployment | vLLM for text generation and Hugging Face TEI for embeddings and reranking | Q4 2026 |
-| **Lambda** | Serverless functions | | Q1 2027 |
-| **SQS** | Message queues | | Q1 2027 |
+| **Lambda** | Serverless functions | QEMU `microvm` execution environments | Q1 2027 |
+| **SQS** | Message queues | A dedicated NATS JetStream store in a per-Region QEMU `microvm` guest | Q1 2027 |

 Roadmap services ship under the same AWS API surface, so code written for AWS today works the moment they land. [Track what's shipped in the release notes](https://github.com/mulgadc/spinifex/releases).

@@ -113,7 +113,7 @@ aws ec2 describe-instance-types
 ## Web Console

 <p align="center">
-  <img src=".github/assets/spinifex-ui.jpg" alt="Spinifex web console dashboard" width="900">
+  <img src=".github/assets/dashboard.webp" alt="Spinifex web console dashboard" width="900">
 </p>

 Spinifex ships with a built-in web console for your instances, volumes, buckets, networks, and IAM. See [Launching the Web UI](https://docs.mulgadc.com/docs/setting-up-your-cluster#7-launching-the-web-ui).
@@ -15,9 +15,6 @@ var allowlist = []allowance{
 	{From: "operator/accountlifecycle", To: "domains/ec2/awsapi/igw", Debt: "accountteardown (dependency)", Reason: "EC2 reapers call the awsapi action adapters; needs domain-owned cleanup capabilities"},
 	{From: "operator/accountlifecycle", To: "domains/ec2/awsapi/image", Debt: "accountteardown (dependency)", Reason: "EC2 reapers call the awsapi action adapters; needs domain-owned cleanup capabilities"},
 	{From: "operator/accountlifecycle", To: "domains/ec2/awsapi/instance", Debt: "accountteardown (dependency)", Reason: "EC2 reapers call the awsapi action adapters; needs domain-owned cleanup capabilities"},
-	{From: "operator/accountlifecycle", To: "domains/ec2/awsapi/key", Debt: "accountteardown (dependency)", Reason: "EC2 reapers call the awsapi action adapters; needs domain-owned cleanup capabilities"},
-	{From: "operator/accountlifecycle", To: "domains/ec2/awsapi/launchtemplate", Debt: "accountteardown (dependency)", Reason: "EC2 reapers call the awsapi action adapters; needs domain-owned cleanup capabilities"},
-	{From: "operator/accountlifecycle", To: "domains/ec2/awsapi/placementgroup", Debt: "accountteardown (dependency)", Reason: "EC2 reapers call the awsapi action adapters; needs domain-owned cleanup capabilities"},
 	{From: "operator/accountlifecycle", To: "domains/ec2/awsapi/routetable", Debt: "accountteardown (dependency)", Reason: "EC2 reapers call the awsapi action adapters; needs domain-owned cleanup capabilities"},
 	{From: "operator/accountlifecycle", To: "domains/ec2/awsapi/snapshot", Debt: "accountteardown (dependency)", Reason: "EC2 reapers call the awsapi action adapters; needs domain-owned cleanup capabilities"},
 	{From: "operator/accountlifecycle", To: "domains/ec2/awsapi/volume", Debt: "accountteardown (dependency)", Reason: "EC2 reapers call the awsapi action adapters; needs domain-owned cleanup capabilities"},
@@ -38,6 +35,7 @@ var allowlist = []allowance{
 	{From: "cmd/aws-model-coverage", To: "domains/ecs/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
 	{From: "cmd/aws-model-coverage", To: "domains/eks/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
 	{From: "cmd/aws-model-coverage", To: "domains/iam/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
+	{From: "cmd/aws-model-coverage", To: "domains/sqs/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
 	{From: "cmd/aws-model-coverage", To: "domains/sts/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},

 	{From: "runtime/daemon", To: "domains/ec2/awsapi/instance", Debt: "Instance adapter consumers outside EC2", Reason: "runtime/daemon/eks_cp_control.go and rds_deps.go call the instance adapter; needs an EC2 capability"},
@@ -1,6 +1,6 @@
 ARG VERSION=dev

-FROM golang:1.27.3-bookworm@sha256:972b4110ba0f47539a975c31e52c64765e1ff19f2d74469ae4a595c6676e80a5 AS builder
+FROM golang:1.27.3-bookworm@sha256:d9b6a1beb9474adc2a1ce842573639b44b2a0dd49b6669f51d3821baf9ec37c9 AS builder

 ARG VERSION

@@ -0,0 +1,68 @@
+#!/bin/sh
+mount -t proc  proc  /proc
+mount -t sysfs sys   /sys
+mount -t devtmpfs dev /dev
+
+# modprobe needs a usable modules.dep; insmod by explicit path does not, so
+# fall back to it. Handles .ko and .ko.gz.
+load_mod() {
+    modprobe "$1" 2>/dev/null && return 0
+    for ko in $(find /lib/modules -name "$1.ko" -o -name "$1.ko.gz" 2>/dev/null); do
+        insmod "$ko" 2>/dev/null && return 0
+        case "$ko" in
+            *.gz) gunzip -c "$ko" >/tmp/mod.ko 2>/dev/null \
+                  && insmod /tmp/mod.ko 2>/dev/null && return 0 ;;
+        esac
+    done
+    return 1
+}
+# lambda-agent reads its identity from fw_cfg and unloads qemu_fw_cfg before
+# anything else runs. virtio_net needs net_failover and failover, and the
+# insmod fallback resolves no dependencies, so load the chain bottom-up.
+load_mod qemu_fw_cfg || echo "[init] qemu_fw_cfg did not load" >/dev/console
+load_mod failover || true
+load_mod net_failover || true
+load_mod virtio_net || echo "[init] virtio_net did not load" >/dev/console
+
+ip link set lo up
+
+# The guest's addresses come from the same netcfg entry as other system VMs.
+NETCFG=/sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw
+i=0
+while [ "$i" -lt 25 ] && [ ! -f "$NETCFG" ]; do
+    sleep 0.04
+    i=$((i + 1))
+done
+if [ -f "$NETCFG" ]; then
+    # shellcheck source=/dev/null
+    . "$NETCFG"
+    for n in 0 1 2 3 4 5; do
+        eval mac=\$NIC${n}_MAC
+        [ -z "$mac" ] && continue
+        iface=""
+        for _ in 1 2 3 4 5; do
+            iface=$(ip -o link | awk -v m="$mac" 'tolower($0) ~ tolower(m) {print $2}' | tr -d :)
+            [ -n "$iface" ] && break
+            sleep 0.02
+        done
+        if [ -z "$iface" ]; then
+            echo "[init] no interface for MAC $mac (NIC$n)" >/dev/console
+            continue
+        fi
+        cidr="" gw="" default="" rdst="" rvia=""
+        eval cidr=\$NIC${n}_CIDR
+        eval gw=\$NIC${n}_GW
+        eval default=\$NIC${n}_DEFAULT
+        eval rdst=\$NIC${n}_ROUTE_DST
+        eval rvia=\$NIC${n}_ROUTE_VIA
+        ip link set "$iface" up
+        [ -n "$cidr" ] && ip addr add "$cidr" dev "$iface"
+        [ "$default" = "1" ] && [ -n "$gw" ] && ip route add default via "$gw" dev "$iface"
+        [ -n "$rdst" ] && [ -n "$rvia" ] && ip route add "$rdst" via "$rvia" dev "$iface"
+    done
+else
+    # lambda-agent then fails to reach the host and the guest powers off.
+    echo "[init] fw_cfg netcfg not available" >/dev/console
+fi
+
+exec /sbin/init
@@ -2,9 +2,5 @@
 # (poweroff) handler installed before sysinit runs, so a press can never reach
 # the pre-exec shell PID 1. A press before this line is lost; the host re-sends.
 ::sysinit:/sbin/modprobe tiny_power_button power_signal=12
-::sysinit:/usr/local/lib/sqs-store/sqs-store-prepare
-::respawn:/usr/local/lib/sqs-store/sqs-store-run
+::once:/usr/local/lib/lambda-runtime/run-agent
 ::ctrlaltdel:/sbin/reboot
-# One action: the store stops and its disk unmounts first, and the generic
-# umount -a -r runs only if the store is gone.
-::shutdown:/usr/local/lib/sqs-store/sqs-store-stop
@@ -0,0 +1,6 @@
+#!/bin/sh
+# Runs the Lambda guest agent, then powers the guest off whatever its status.
+/usr/local/bin/lambda-agent >/dev/console 2>&1
+status=$?
+echo "[run-agent] lambda-agent exited with status $status" >/dev/console
+poweroff -f
@@ -1,224 +0,0 @@
-# sqs-store-lib.sh — shared by the SQS store guest's prepare, run and stop
-# steps. Sourced, never executed. Every path and command is overridable so the
-# host test can drive the same code against a fake fw_cfg, sysfs and tools.
-#
-# Guest contract, read from fw_cfg under opt/spinifex/sqs-store/:
-#   data-serial  virtio-blk serial of the already-formatted ext4 data disk
-#   listen-addr  IPv4 address on a configured guest NIC; the only listener
-#   client-nkey  NATS user NKey public key allowed to connect; never a seed
-# Anything absent, malformed or ambiguous refuses the store. Nothing formats.
-
-SQS_FWCFG=${SQS_FWCFG:-/sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/sqs-store}
-SQS_SYSBLOCK=${SQS_SYSBLOCK:-/sys/block}
-SQS_DEVDIR=${SQS_DEVDIR:-/dev}
-# The host test cannot create block nodes, so it relaxes this to -e.
-SQS_NODE_TEST=${SQS_NODE_TEST:--b}
-SQS_RUN=${SQS_RUN:-/run/sqs-store}
-SQS_DATA=${SQS_DATA:-/data}
-SQS_CONSOLE=${SQS_CONSOLE:-/dev/console}
-SQS_MOUNTS=${SQS_MOUNTS:-/proc/mounts}
-SQS_PORT=${SQS_PORT:-4222}
-SQS_STOP_BUDGET_TENTHS=${SQS_STOP_BUDGET_TENTHS:-120}
-NATS_BIN=${NATS_BIN:-/usr/local/bin/nats-server}
-E2FSCK=${E2FSCK:-e2fsck}
-MOUNT=${MOUNT:-mount}
-UMOUNT=${UMOUNT:-umount}
-IP=${IP:-ip}
-
-sqs_log() { echo "[sqs-store] $*" >> "$SQS_CONSOLE"; }
-sqs_fatal() { sqs_log "FATAL: $*"; return 1; }
-
-# sqs_fwcfg <name>: print one fw_cfg value, refusing a missing, empty, oversized
-# or multi-token file. The value is not echoed on failure.
-sqs_fwcfg() {
-    f="$SQS_FWCFG/$1/raw"
-    [ -f "$f" ] || { sqs_fatal "fw_cfg $1 not supplied"; return 1; }
-    [ "$(wc -c < "$f")" -le 128 ] || { sqs_fatal "fw_cfg $1 is oversized"; return 1; }
-    v=$(cat "$f")
-    case "$v" in
-        "" | *[[:space:]]*) sqs_fatal "fw_cfg $1 must be exactly one value"; return 1 ;;
-    esac
-    printf '%s' "$v"
-}
-
-# A volume serial is the volume ID with dashes stripped: vol + 17 hex.
-sqs_valid_serial() {
-    [ "${#1}" -eq 20 ] || return 1
-    case "$1" in vol*) ;; *) return 1 ;; esac
-    case "${1#vol}" in *[!0-9a-f]*) return 1 ;; esac
-}
-
-# A NATS user public NKey: "U" then 55 base32 characters. The checksum is
-# left to the pinned server's own config test.
-sqs_valid_user_nkey() {
-    [ "${#1}" -eq 56 ] || return 1
-    case "$1" in U*) ;; *) return 1 ;; esac
-    case "$1" in *[!A-Z2-7]*) return 1 ;; esac
-}
-
-sqs_valid_ipv4() {
-    case "$1" in *[!0-9.]* | .* | *. | *..*) return 1 ;; esac
-    old_ifs=$IFS; IFS=.
-    # shellcheck disable=SC2086 # split on dots on purpose
-    set -- $1
-    IFS=$old_ifs
-    [ "$#" -eq 4 ] || return 1
-    for o in "$@"; do
-        [ "${#o}" -le 3 ] && [ "$o" -le 255 ] || return 1
-        case "$o" in 0?*) return 1 ;; esac
-    done
-}
-
-# sqs_find_disk <serial>: print the /dev node of the one block device whose
-# sysfs serial matches exactly. Zero or several matches refuse.
-sqs_find_disk() {
-    found="" n=0
-    for b in "$SQS_SYSBLOCK"/*; do
-        [ -r "$b/serial" ] || continue
-        [ "$(cat "$b/serial")" = "$1" ] || continue
-        found=${b##*/}
-        n=$((n + 1))
-    done
-    [ "$n" -eq 1 ] || { sqs_fatal "expected one disk with serial $1, found $n"; return 1; }
-    test "$SQS_NODE_TEST" "$SQS_DEVDIR/$found" || { sqs_fatal "disk $found has no block device node"; return 1; }
-    printf '%s' "$SQS_DEVDIR/$found"
-}
-
-# sqs_has_addr <ipv4>: the address is assigned to a non-loopback interface.
-sqs_has_addr() {
-    "$IP" -o -4 addr show 2>/dev/null \
-        | awk -v a="$1" '$2 != "lo" { split($4, p, "/"); if (p[1] == a) f = 1 } END { exit !f }'
-}
-
-# sqs_alive <pid>: running or stopped, not exited (a zombie counts as exited).
-sqs_alive() { [ -r "/proc/$1/status" ] && ! grep -q '^State:[[:space:]]*Z' "/proc/$1/status" 2>/dev/null; }
-
-# sqs_starttime <pid>: kernel start time (stat field 22). It survives exec, and
-# pairing it with the pid reduces PID-reuse risk; without a pidfd the check and
-# the signal stay separate steps, so it is not an absolute guarantee.
-sqs_starttime() { sed 's/.*) //' "/proc/$1/stat" 2>/dev/null | cut -d' ' -f20; }
-
-# sqs_same <pid> <starttime>: the recorded process is still alive.
-sqs_same() { [ -n "$2" ] && sqs_alive "$1" && [ "$(sqs_starttime "$1")" = "$2" ]; }
-
-# sqs_write_config <addr> <nkey> <file>: the dedicated server's config. Its
-# max_payload admits a 1 MiB SQS message with its headers (ADR-0011:S4); the
-# 1MB NATS default does not.
-sqs_write_config() {
-    cat > "$3" <<EOF
-listen: "$1:$SQS_PORT"
-server_name: "sqs-store"
-max_payload: 2MiB
-jetstream {
-  store_dir: "$SQS_DATA"
-  sync_interval: always
-}
-authorization {
-  users = [ { nkey: "$2" } ]
-}
-EOF
-}
-
-# sqs_prepare: validate the contract, check and mount the data disk, write the
-# server config and mark the store ready. Returns non-zero, unready, on any refusal.
-sqs_prepare() {
-    rm -f "$SQS_RUN/ready"
-    mkdir -p "$SQS_RUN"
-    serial=$(sqs_fwcfg data-serial) || return 1
-    sqs_valid_serial "$serial" || { sqs_fatal "data-serial is not a volume serial"; return 1; }
-    addr=$(sqs_fwcfg listen-addr) || return 1
-    sqs_valid_ipv4 "$addr" || { sqs_fatal "listen-addr is not an IPv4 address"; return 1; }
-    nkey=$(sqs_fwcfg client-nkey) || return 1
-    sqs_valid_user_nkey "$nkey" || { sqs_fatal "client-nkey is not a user public NKey"; return 1; }
-    sqs_has_addr "$addr" || { sqs_fatal "listen-addr $addr is not on any guest NIC"; return 1; }
-
-    # The server checks the config, including the NKey checksum, before the
-    # disk is touched.
-    umask 077
-    sqs_write_config "$addr" "$nkey" "$SQS_RUN/nats.conf" || return 1
-    if ! "$NATS_BIN" -t -c "$SQS_RUN/nats.conf" > "$SQS_RUN/config-test.out" 2>&1; then
-        sqs_fatal "server config test failed: $(sed 's/U[A-Z2-7]\{55\}/<nkey>/g' "$SQS_RUN/config-test.out" | tr '\n' ' ')"
-        return 1
-    fi
-
-    dev=$(sqs_find_disk "$serial") || return 1
-    rc=0
-    "$E2FSCK" -p "$dev" > "$SQS_RUN/fsck.out" 2>&1 || rc=$?
-    sqs_log "e2fsck -p $dev rc=$rc"
-    # 0 clean, 1 corrected: proceed. Anything higher, including a disk that is
-    # not ext4 at all, refuses; the guest never formats.
-    [ "$rc" -le 1 ] || { sqs_fatal "e2fsck refused $dev: $(tr '\n' ' ' < "$SQS_RUN/fsck.out")"; return 1; }
-    mkdir -p "$SQS_DATA"
-    "$MOUNT" -t ext4 -o nodev,nosuid,noexec "$dev" "$SQS_DATA" || { sqs_fatal "mount $dev failed"; return 1; }
-    : > "$SQS_RUN/ready"
-    sqs_log "prepared: $dev mounted at $SQS_DATA, config written for $addr:$SQS_PORT; server not started yet"
-}
-
-# sqs_run: the respawned supervisor body. Its identity is recorded before the
-# stopping check: a stop that misses it set stopping first, so this parks; a
-# stop that reads it signals this process whether or not it has exec'd yet.
-sqs_run() {
-    mkdir -p "$SQS_RUN"
-    echo "$$ $(sqs_starttime $$)" > "$SQS_RUN/pid"
-    if [ -e "$SQS_RUN/stopping" ]; then
-        sqs_log "run: shutdown in progress, not starting"
-        return 2
-    fi
-    if [ ! -e "$SQS_RUN/ready" ]; then
-        sqs_log "run: store not prepared, not starting"
-        return 2
-    fi
-    sqs_log "run: starting pid=$$"
-    # ADR-0011:S11 bounds the store's Go heap; the guest must be larger.
-    GOMEMLIMIT=512MiB exec "$NATS_BIN" -c "$SQS_RUN/nats.conf"
-}
-
-# sqs_stop: first shutdown action. Block any respawn, SIGTERM the recorded run
-# process (a server, or a start not yet exec'd), SIGKILL only after the budget,
-# then sync and unmount. Each signal rechecks the identity; a survivor keeps the disk mounted.
-sqs_stop() {
-    mkdir -p "$SQS_RUN"
-    : > "$SQS_RUN/stopping"
-    sqs_log "stop: begin"
-    rec=$(cat "$SQS_RUN/pid" 2>/dev/null || true)
-    pid=${rec%% *} st=${rec#* }
-    [ "$st" != "$rec" ] || st=""
-    if [ -n "$pid" ] && sqs_same "$pid" "$st"; then
-        kill -TERM "$pid" 2>/dev/null || true
-        sqs_log "stop: SIGTERM sent to run pid=$pid"
-        i=0
-        while sqs_same "$pid" "$st" && [ "$i" -lt "$SQS_STOP_BUDGET_TENTHS" ]; do sleep 0.1; i=$((i + 1)); done
-        if sqs_same "$pid" "$st"; then
-            sqs_log "stop: run pid=$pid still running after $((SQS_STOP_BUDGET_TENTHS / 10))s, SIGKILL"
-            kill -KILL "$pid" 2>/dev/null || true
-            j=0
-            while sqs_same "$pid" "$st" && [ "$j" -lt 30 ]; do sleep 0.1; j=$((j + 1)); done
-        fi
-        if sqs_same "$pid" "$st"; then
-            sqs_log "stop: run pid=$pid STILL ALIVE, leaving $SQS_DATA mounted"
-            return 1
-        fi
-        sqs_log "stop: store exited after $((i / 10)).$((i % 10))s"
-    else
-        sqs_log "stop: no running store"
-    fi
-    sync
-    if grep -q " $SQS_DATA " "$SQS_MOUNTS" 2>/dev/null; then
-        rc=0
-        "$UMOUNT" "$SQS_DATA" || rc=$?
-        sqs_log "stop: umount $SQS_DATA rc=$rc"
-    fi
-}
-
-# sqs_shutdown: the only shutdown action. The generic read-only remount runs
-# only after the store stopped; a surviving writer keeps every mount as it is.
-sqs_shutdown() {
-    if ! sqs_stop; then
-        sync
-        sqs_log "shutdown: store survived, skipping umount -a -r"
-        return 1
-    fi
-    rc=0
-    "$UMOUNT" -a -r || rc=$?
-    echo "[init] shutdown: umount -a -r rc=$rc" >> "$SQS_CONSOLE"
-}
@@ -1,9 +0,0 @@
-#!/bin/sh
-# sysinit: load the disk drivers, then validate the guest contract, check and
-# mount the data disk and write the server config. A refusal leaves the store
-# unprepared, so the run step never starts it.
-. /usr/local/lib/sqs-store/sqs-store-lib.sh
-modprobe virtio_blk 2>/dev/null || true
-modprobe ext4 2>/dev/null || true
-sqs_prepare || sqs_log "store refused; it will not start"
-exit 0
@@ -1,6 +0,0 @@
-#!/bin/sh
-# Respawned by init. A store that must not run parks here instead of exiting,
-# so init does not respawn it in a tight loop.
-. /usr/local/lib/sqs-store/sqs-store-lib.sh
-sqs_run
-exec sleep 2147483647
@@ -1,5 +0,0 @@
-#!/bin/sh
-# The only shutdown action: stop the store and unmount its disk, then run the
-# generic read-only remount only if the store is gone.
-. /usr/local/lib/sqs-store/sqs-store-lib.sh
-sqs_shutdown
@@ -33,6 +33,10 @@ load_mod failover || true
 load_mod net_failover || true
 load_mod virtio_net || true

+# The guest's SQS store listens on 127.0.0.1 and the worker dials it there;
+# the kernel leaves lo down, so bring it up before init starts either.
+ip link set lo up
+
 NETCFG=/sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw
 i=0
 while [ "$i" -lt 25 ]; do
@@ -87,8 +91,8 @@ for n in 0 1 2 3 4 5; do
    [ -n "$rdst" ] && [ -n "$rvia" ] && ip route add "$rdst" via "$rvia" dev "$iface"
 done

-# The store image takes no lb-agent environment. A CA bundle is optional here:
-# nothing in this image dials out yet.
+# The image takes no lb-agent environment. The worker reads the cluster CA
+# from fw_cfg itself; this copy only serves other tools in the guest.
 CA_CERT=/sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/ca-cert/raw
 mkdir -p /etc/ssl/certs
 [ -f "$CA_CERT" ] && cp "$CA_CERT" /etc/ssl/certs/ca-certificates.crt
@@ -3,6 +3,7 @@
 # the pre-exec shell PID 1. A press before this line is lost; the host re-sends.
 ::sysinit:/sbin/modprobe tiny_power_button power_signal=12
 ::sysinit:/usr/local/lib/sqs-worker/sqs-worker-prepare
+::respawn:/usr/local/lib/sqs-worker/sqs-worker-store
 ::respawn:/usr/local/lib/sqs-worker/sqs-worker-ready
 ::ctrlaltdel:/sbin/reboot
 ::shutdown:/usr/local/lib/sqs-worker/sqs-worker-stop
@@ -1,19 +1,23 @@
-# sqs-worker-lib.sh — shared by the SQS worker guest's readiness and stop
+# sqs-worker-lib.sh — shared by the SQS guest's store, readiness and stop
 # steps. Sourced, never executed. Every path and command is overridable so the
 # host test can drive the same code against test-owned processes and stub tools.
 #
-# store-ready exists only while the running watcher's last check passed. A
-# check proves the store accepted the worker key, not the store server's
-# identity, and a store failure is noticed up to one interval plus call time late.
+# The guest runs its own SQS nats-server on loopback and the worker beside it.
+# The server starts only after boot settled the receipt key; store-ready exists
+# only while the running watcher's last check of that server passed.

 SQSW_RUN=${SQSW_RUN:-/run/sqs-worker}
 SQSW_DATA=${SQSW_DATA:-/data}
 SQSW_CONSOLE=${SQSW_CONSOLE:-/dev/console}
 SQSW_MOUNTS=${SQSW_MOUNTS:-/proc/mounts}
 SQSW_BIN=${SQSW_BIN:-/usr/local/bin/sqs-worker}
+SQSW_LIB=${SQSW_LIB:-/usr/local/lib/sqs-worker/sqs-worker-lib.sh}
+SQSW_NATS_BIN=${SQSW_NATS_BIN:-/usr/local/bin/nats-server}
+SQSW_STORE_PORT=${SQSW_STORE_PORT:-4222}
 SQSW_STOP_BUDGET_TENTHS=${SQSW_STOP_BUDGET_TENTHS:-100}
 SQSW_RESPAWN_GAP=${SQSW_RESPAWN_GAP:-5}
-SQSW_PARK_INTERVAL=${SQSW_PARK_INTERVAL:-30}
+SQSW_PARK=${SQSW_PARK:-sleep 2147483647}
+SQSW_BOOT_RETRY=${SQSW_BOOT_RETRY:-30}
 UMOUNT=${UMOUNT:-umount}

 sqsw_log() { echo "[sqs-worker] $*" >> "$SQSW_CONSOLE"; }
@@ -28,28 +32,102 @@ sqsw_starttime() { sed 's/.*) //' "/proc/$1/stat" 2>/dev/null | cut -d' ' -f20;
 # sqsw_same <pid> <starttime>: the recorded process is still alive.
 sqsw_same() { [ -n "$2" ] && sqsw_alive "$1" && [ "$(sqsw_starttime "$1")" = "$2" ]; }

+# sqsw_write_config <file>: the guest's own server. It listens only on
+# loopback, with no monitor, route or cluster listener. Its max_payload admits
+# a 1 MiB SQS message with its headers (ADR-0011:S4); the 1MB default does not.
+sqsw_write_config() {
+    cat > "$1" <<CONF
+listen: "127.0.0.1:$SQSW_STORE_PORT"
+server_name: "sqs-store"
+max_payload: 2MiB
+jetstream {
+  store_dir: "$SQSW_DATA/jetstream"
+  sync_interval: always
+}
+CONF
+}
+
+# sqsw_prepare_store: write and test the server config once boot has settled
+# the receipt key. Returns non-zero, leaving the store unstarted, otherwise.
+sqsw_prepare_store() {
+    rm -f "$SQSW_RUN/store.conf"
+    [ -s "$SQSW_RUN/receipt-ready" ] || { sqsw_log "store: receipt key not settled, store not prepared"; return 1; }
+    umask 077
+    sqsw_write_config "$SQSW_RUN/store.conf.new" || return 1
+    if ! "$SQSW_NATS_BIN" -t -c "$SQSW_RUN/store.conf.new" > "$SQSW_RUN/config-test.out" 2>&1; then
+        sqsw_log "store: FATAL: server config test failed: $(tr '\n' ' ' < "$SQSW_RUN/config-test.out")"
+        return 1
+    fi
+    mv "$SQSW_RUN/store.conf.new" "$SQSW_RUN/store.conf"
+    sqsw_log "store: config written for 127.0.0.1:$SQSW_STORE_PORT; server not started yet"
+}
+
+# sqsw_park <why>: init respawns a step that exits, so a step that must not run
+# logs once and waits for shutdown instead of looping.
+sqsw_park() {
+    sqsw_log "$1"
+    exec $SQSW_PARK
+}
+
+# sqsw_record <name>: record this process's identity before the stopping
+# check, so a stop that misses the record has already set stopping.
+sqsw_record() { echo "$$ $(sqsw_starttime $$)" > "$SQSW_RUN/$1.pid"; }
+
+# sqsw_boot: settle the disk and receipt key once per guest boot. A transient
+# failure (exit 75) waits and exits so init runs it again; boot never remounts
+# a disk already mounted. Any other failure is final for this guest boot.
+sqsw_boot() {
+    [ ! -e "$SQSW_RUN/boot-refused" ] || sqsw_park "store: boot refused, not starting"
+    [ -s "$SQSW_RUN/receipt-ready" ] && return 0
+    # The boot child records itself before its stopping check, as the other
+    # steps do, so shutdown either sees its record or it never starts.
+    SQSW_RUN="$SQSW_RUN" SQSW_BIN="$SQSW_BIN" sh -c '. "$1"; sqsw_record boot; [ ! -e "$SQSW_RUN/stopping" ] || exit 75; exec "$SQSW_BIN" boot' \
+        sh "$SQSW_LIB" >> "$SQSW_CONSOLE" 2>&1 &
+    child=$!
+    rc=0
+    wait "$child" || rc=$?
+    rm -f "$SQSW_RUN/boot.pid"
+    [ ! -e "$SQSW_RUN/stopping" ] || sqsw_park "store: shutdown during boot, not starting"
+    case "$rc" in
+        0) return 0 ;;
+        75) sqsw_log "store: boot not settled, retrying in ${SQSW_BOOT_RETRY}s"
+            sleep "$SQSW_BOOT_RETRY"
+            exit 1 ;;
+        *) : > "$SQSW_RUN/boot-refused"
+           sqsw_park "store: boot refused (rc=$rc), not starting" ;;
+    esac
+}
+
+# sqsw_store: the respawned server body. It starts only on a prepared config.
+sqsw_store() {
+    mkdir -p "$SQSW_RUN"
+    sqsw_record store
+    [ ! -e "$SQSW_RUN/stopping" ] || sqsw_park "store: shutdown in progress, not starting"
+    sqsw_boot
+    if [ ! -s "$SQSW_RUN/store.conf" ] && ! sqsw_prepare_store; then
+        : > "$SQSW_RUN/boot-refused"
+        sqsw_park "store: not prepared, not starting"
+    fi
+    sqsw_log "store: starting pid=$$"
+    # ADR-0011:S11 bounds the store's Go heap; the guest must be larger.
+    GOMEMLIMIT=512MiB exec "$SQSW_NATS_BIN" -c "$SQSW_RUN/store.conf"
+}
+
 # sqsw_run: the respawned readiness body. A predecessor that was killed or
-# crashed could not remove its marker, so every start removes it first. Its
-# identity is recorded before the stopping check, as the store's run step does.
+# crashed could not remove its marker, so every start removes it first.
 sqsw_run() {
    mkdir -p "$SQSW_RUN"
-    echo "$$ $(sqsw_starttime $$)" > "$SQSW_RUN/ready.pid"
+    sqsw_record ready
    if [ -e "$SQSW_RUN/store-ready" ]; then
        rm -f "$SQSW_RUN/store-ready"
        sqsw_log "ready: removed a store-ready marker left by an earlier watcher"
    fi
-    # Returning here would make init respawn and log at once, forever. Park
-    # instead, logging once: shutdown stops this recorded process, and a key
-    # that appears later lets the watcher start.
-    if [ -e "$SQSW_RUN/stopping" ] || [ ! -s "$SQSW_RUN/public-key" ]; then
-        if [ -e "$SQSW_RUN/stopping" ]; then
-            sqsw_log "ready: shutdown in progress, not starting"
-        else
-            sqsw_log "ready: no worker key; readiness not started, waiting"
-        fi
-        while [ -e "$SQSW_RUN/stopping" ] || [ ! -s "$SQSW_RUN/public-key" ]; do
-            sleep "$SQSW_PARK_INTERVAL"
-        done
+    [ ! -e "$SQSW_RUN/stopping" ] || sqsw_park "ready: shutdown in progress, not starting"
+    [ ! -e "$SQSW_RUN/boot-refused" ] || sqsw_park "ready: boot refused, readiness not started"
+    # Boot may still be retrying; check again after a gap rather than parking.
+    if [ ! -s "$SQSW_RUN/store.conf" ]; then
+        sleep "$SQSW_RESPAWN_GAP"
+        exit 0
    fi
    # Init respawns at once; a watcher that keeps exiting restarts at most
    # once per gap instead of in a tight loop.
@@ -62,37 +140,47 @@ sqsw_run() {
    exec "$SQSW_BIN" ready >> "$SQSW_CONSOLE" 2>&1
 }

-# sqsw_stop: block any respawn, drop the marker, SIGTERM the recorded watcher,
-# SIGKILL only after the budget, and unmount the worker disk only once the
-# watcher is confirmed gone. A survivor keeps the disk mounted.
+# sqsw_end <name>: SIGTERM the recorded process, SIGKILL only after the
+# budget. Returns non-zero while it is still alive.
+sqsw_end() {
+    rec=$(cat "$SQSW_RUN/$1.pid" 2>/dev/null || true)
+    pid=${rec%% *} st=${rec#* }
+    [ "$st" != "$rec" ] || st=""
+    if [ -z "$pid" ] || ! sqsw_same "$pid" "$st"; then
+        sqsw_log "stop: no running $1"
+        return 0
+    fi
+    kill -TERM "$pid" 2>/dev/null || true
+    sqsw_log "stop: SIGTERM sent to $1 pid=$pid"
+    i=0
+    while sqsw_same "$pid" "$st" && [ "$i" -lt "$SQSW_STOP_BUDGET_TENTHS" ]; do sleep 0.1; i=$((i + 1)); done
+    if sqsw_same "$pid" "$st"; then
+        sqsw_log "stop: $1 pid=$pid still running after $((SQSW_STOP_BUDGET_TENTHS / 10))s, SIGKILL"
+        kill -KILL "$pid" 2>/dev/null || true
+        j=0
+        while sqsw_same "$pid" "$st" && [ "$j" -lt 30 ]; do sleep 0.1; j=$((j + 1)); done
+    fi
+    if sqsw_same "$pid" "$st"; then
+        sqsw_log "stop: $1 pid=$pid STILL ALIVE"
+        return 1
+    fi
+    sqsw_log "stop: $1 pid=$pid exited"
+    rm -f "$SQSW_RUN/$1.pid"
+}
+
+# sqsw_stop: block any respawn, drop the marker, stop the watcher, the server
+# and any boot in progress, and unmount the disk only once all are confirmed
+# gone. A survivor keeps the disk mounted.
 sqsw_stop() {
    mkdir -p "$SQSW_RUN"
    : > "$SQSW_RUN/stopping"
    rm -f "$SQSW_RUN/store-ready"
-    rec=$(cat "$SQSW_RUN/ready.pid" 2>/dev/null || true)
-    pid=${rec%% *} st=${rec#* }
-    [ "$st" != "$rec" ] || st=""
-    if [ -n "$pid" ] && sqsw_same "$pid" "$st"; then
-        kill -TERM "$pid" 2>/dev/null || true
-        sqsw_log "stop: SIGTERM sent to readiness watcher pid=$pid"
-        i=0
-        while sqsw_same "$pid" "$st" && [ "$i" -lt "$SQSW_STOP_BUDGET_TENTHS" ]; do sleep 0.1; i=$((i + 1)); done
-        if sqsw_same "$pid" "$st"; then
-            sqsw_log "stop: readiness watcher pid=$pid still running after $((SQSW_STOP_BUDGET_TENTHS / 10))s, SIGKILL"
-            kill -KILL "$pid" 2>/dev/null || true
-            j=0
-            while sqsw_same "$pid" "$st" && [ "$j" -lt 30 ]; do sleep 0.1; j=$((j + 1)); done
-        fi
-        if sqsw_same "$pid" "$st"; then
-            rm -f "$SQSW_RUN/store-ready"
-            sqsw_log "stop: readiness watcher pid=$pid STILL ALIVE, leaving $SQSW_DATA mounted"
-            return 1
-        fi
-        sqsw_log "stop: readiness watcher pid=$pid exited"
-    else
-        sqsw_log "stop: no running readiness watcher"
+    if ! sqsw_end ready || ! sqsw_end store || ! sqsw_end boot; then
+        rm -f "$SQSW_RUN/store-ready"
+        sqsw_log "stop: leaving $SQSW_DATA mounted"
+        return 1
    fi
-    rm -f "$SQSW_RUN/store-ready" "$SQSW_RUN/ready.pid"
+    rm -f "$SQSW_RUN/store-ready"
    sync
    if grep -q " $SQSW_DATA " "$SQSW_MOUNTS" 2>/dev/null; then
        rc=0
@@ -102,11 +190,11 @@ sqsw_stop() {
 }

 # sqsw_shutdown: the only shutdown action. The generic read-only remount runs
-# only after the watcher stopped; a survivor keeps every mount as it is.
+# only after both processes stopped; a survivor keeps every mount as it is.
 sqsw_shutdown() {
    if ! sqsw_stop; then
        sync
-        sqsw_log "shutdown: readiness watcher survived, skipping umount -a -r"
+        sqsw_log "shutdown: a guest process survived, skipping umount -a -r"
        return 1
    fi
    rc=0
@@ -1,8 +1,9 @@
 #!/bin/sh
-# sysinit: load the disk drivers, then select, check and mount the worker disk
-# and load or create the worker's store key on it. Any refusal is logged; the
-# key is never read or created on a disk that is not proven mounted.
+# sysinit: load the disk drivers and clear this boot's markers. The respawned
+# store step mounts the disk, settles the receipt key and starts the store.
+. /usr/local/lib/sqs-worker/sqs-worker-lib.sh
 modprobe virtio_blk 2>/dev/null || true
 modprobe ext4 2>/dev/null || true
-/usr/local/bin/sqs-worker boot >> /dev/console 2>&1 || echo "[sqs-worker] boot refused" >> /dev/console
+mkdir -p "$SQSW_RUN"
+rm -f "$SQSW_RUN/receipt-ready" "$SQSW_RUN/store.conf" "$SQSW_RUN/boot-refused"
 exit 0
@@ -1,6 +1,5 @@
 #!/bin/sh
-# Respawned by init. A watcher that must not run parks here instead of exiting,
-# so init does not respawn it in a tight loop.
+# Respawned by init: the worker, reporting ready only while the local store
+# answers. A watcher that must not run parks instead of exiting.
 . /usr/local/lib/sqs-worker/sqs-worker-lib.sh
 sqsw_run
-exec sleep 2147483647
@@ -1,5 +1,5 @@
 #!/bin/sh
-# The only shutdown action: stop the readiness watcher and unmount the worker
-# disk, then run the generic read-only remount only if the watcher is gone.
+# The only shutdown action: stop the worker, then the store, and unmount the
+# data disk, then run the generic read-only remount only if both are gone.
 . /usr/local/lib/sqs-worker/sqs-worker-lib.sh
 sqsw_shutdown
@@ -0,0 +1,5 @@
+#!/bin/sh
+# Respawned by init: the guest's own SQS nats-server, started only on a
+# prepared config. A store that must not run parks instead of exiting.
+. /usr/local/lib/sqs-worker/sqs-worker-lib.sh
+sqsw_store
@@ -1,4 +1,4 @@
-# spinifex-unit-version: 5
+# spinifex-unit-version: 6
 [Unit]
 Description=Spinifex Daemon (VM Orchestration)
 PartOf=spinifex.target
@@ -15,6 +15,7 @@ Slice=spinifex-system.slice
 User=spinifex-daemon
 Group=spinifex
 ExecStartPre=/usr/local/bin/spx admin hostnet check
+ExecStartPre=-+/usr/local/bin/spx admin node-facts collect
 ExecStart=/usr/local/bin/spx service spinifex start
 Restart=on-failure
 RestartSec=5
@@ -17,6 +17,7 @@ import (
 	iamawsapi "github.com/mulgadc/spinifex/spinifex/domains/iam/awsapi"
 	rdsawsapi "github.com/mulgadc/spinifex/spinifex/domains/rds/awsapi"
 	route53awsapi "github.com/mulgadc/spinifex/spinifex/domains/route53/awsapi"
+	sqsawsapi "github.com/mulgadc/spinifex/spinifex/domains/sqs/awsapi"
 	stsawsapi "github.com/mulgadc/spinifex/spinifex/domains/sts/awsapi"
 	"github.com/mulgadc/spinifex/spinifex/ingress/aws/dispatch"

@@ -66,6 +67,7 @@ func compareAll() ([]awsmodel.OperationCoverage, error) {
 		ecsawsapi.ServiceName:     ecsawsapi.OperationInventory(),
 		eksawsapi.ServiceName:     eksawsapi.OperationInventory(),
 		iamawsapi.ServiceName:     iamawsapi.OperationInventory(),
+		sqsawsapi.ServiceName:     sqsawsapi.OperationInventory(),
 		stsawsapi.ServiceName:     stsawsapi.OperationInventory(),
 	})
 	// The loop below walks the embedded models, so a served service with no
@@ -6,6 +6,7 @@
 //
 //	eks-gateway-fetch -resource addons   # GET /clusters/{cluster}/internal-addons/{accountId}
 //	eks-gateway-fetch -resource recovery -instance-id i-… # GET /clusters/{cluster}/internal-recovery/{accountId}/{instanceId}
+//	eks-gateway-fetch -resource workers  # GET /clusters/{cluster}/internal-workers/{accountId}
 //
 // Flags default to environment variables seeded by cloud-init:
 // EKS_GATEWAY_URL, EKS_GATEWAY_CA, EKS_ACCESS_KEY, EKS_SECRET_KEY, EKS_REGION,
@@ -33,7 +34,7 @@ func main() {
 	flag.StringVar(&cfg.AccountID, "account-id", os.Getenv("EKS_ACCOUNT_ID"), "Cluster account ID")
 	flag.StringVar(&cfg.ClusterName, "cluster", os.Getenv("EKS_CLUSTER_NAME"), "Cluster name")
 	flag.StringVar(&cfg.InstanceID, "instance-id", os.Getenv("EKS_INSTANCE_ID"), "This member's EC2 instance ID (recovery resource)")
-	flag.StringVar(&cfg.Resource, "resource", "addons", "Resource to fetch: addons | recovery")
+	flag.StringVar(&cfg.Resource, "resource", "addons", "Resource to fetch: addons | recovery | workers")
 	flag.Parse()

 	if err := gatewayfetch.Run(context.Background(), cfg); err != nil {
@@ -0,0 +1,32 @@
+// Command eks-pod-identity-webhook is the IRSA mutating admission webhook run
+// on every EKS control-plane server; the implementation is in
+// agents/eks/podidentity. It listens on loopback only and is called by the
+// local kube-apiserver.
+package main
+
+import (
+	"context"
+	"log/slog"
+	"os"
+	"os/signal"
+	"syscall"
+
+	_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
+
+	"github.com/mulgadc/spinifex/spinifex/agents/eks/podidentity"
+)
+
+func main() {
+	log := slog.New(slog.NewTextHandler(os.Stderr, nil))
+	cfg, err := podidentity.ParseServiceConfig(os.Args[1:], os.Getenv, os.Stderr)
+	if err != nil {
+		log.Error("invalid configuration", "error", err)
+		os.Exit(2)
+	}
+	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
+	defer stop()
+	if err := podidentity.Run(ctx, cfg, log); err != nil {
+		log.Error("pod identity webhook stopped", "error", err)
+		os.Exit(1)
+	}
+}
@@ -0,0 +1,49 @@
+package main
+
+import (
+	"context"
+	"crypto/tls"
+	"errors"
+	"fmt"
+
+	lambdav1 "github.com/mulgadc/spinifex/contracts/lambda/v1"
+	"github.com/mulgadc/spinifex/spinifex/agents/lambda/hostlink"
+)
+
+// Exit statuses: the guest refused to start, the control connection failed,
+// or the host closed it.
+const (
+	exitRefused = 2
+	exitLink    = 1
+	exitClosed  = 0
+)
+
+type startup struct {
+	fwCfgDir  string
+	fwCfgRoot string
+	unload    func() error
+}
+
+func defaultStartup() startup {
+	return startup{fwCfgDir: hostlink.FwCfgDir, fwCfgRoot: hostlink.FwCfgRoot, unload: hostlink.UnloadFwCfg}
+}
+
+// connect loads the identity, seals fw_cfg whether or not the load succeeded,
+// then dials the host. Under TLS 1.3 a host refusal is seen only on first read.
+func (s startup) connect(ctx context.Context) (*tls.Conn, lambdav1.Scope, int, error) {
+	id, loadErr := hostlink.LoadIdentity(s.fwCfgDir)
+	if err := hostlink.SealFwCfg(s.fwCfgRoot, s.unload); err != nil {
+		return nil, "", exitRefused, errors.Join(loadErr, err)
+	}
+	if loadErr != nil {
+		return nil, "", exitRefused, loadErr
+	}
+	conn, err := hostlink.Dial(ctx, id)
+	if err != nil {
+		if ctx.Err() != nil {
+			err = fmt.Errorf("%w: %w", err, context.Cause(ctx))
+		}
+		return nil, "", exitLink, err
+	}
+	return conn, id.Scope, 0, nil
+}
@@ -0,0 +1,66 @@
+//go:build lambdatestlink
+
+package main
+
+import (
+	"errors"
+	"fmt"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"time"
+
+	lambdav1 "github.com/mulgadc/spinifex/contracts/lambda/v1"
+	"github.com/mulgadc/spinifex/spinifex/agents/lambda/hostlink"
+)
+
+const entryGrant = "test-grant"
+
+// testGrant is the grant the host launched this VM under.
+type testGrant struct {
+	grantID, attemptID, environmentID string
+	generation                        uint64
+}
+
+// readGrant reads the test-grant fw_cfg entry: grant ID, attempt ID,
+// environment ID and generation, one per line.
+func readGrant(fwCfgDir string) (testGrant, error) {
+	b, err := os.ReadFile(filepath.Join(fwCfgDir, entryGrant, "raw"))
+	if err != nil {
+		return testGrant{}, fmt.Errorf("read %s: %w", entryGrant, err)
+	}
+	lines := strings.Split(strings.TrimSuffix(string(b), "\n"), "\n")
+	if len(lines) != 4 {
+		return testGrant{}, errors.New(entryGrant + " entry malformed")
+	}
+	g := testGrant{grantID: lines[0], attemptID: lines[1], environmentID: lines[2]}
+	g.generation, err = strconv.ParseUint(lines[3], 10, 64)
+	if err != nil || g.generation == 0 || g.grantID == "" || g.attemptID == "" || g.environmentID == "" {
+		return testGrant{}, errors.New(entryGrant + " entry malformed")
+	}
+	return g, nil
+}
+
+// checkBinding refuses a run unless the grant names the authenticated environment
+// generation, the request carries that grant and its deadline is still ahead. The
+// clock check is a local guard only: it cannot detect skew or a superseded grant.
+func checkBinding(scope lambdav1.Scope, g testGrant, req hostlink.Request, now time.Time) error {
+	parts := strings.Split(string(scope), "/")
+	if len(parts) != 5 {
+		return errors.New("authenticated scope malformed")
+	}
+	if parts[3] != g.environmentID || parts[4] != strconv.FormatUint(g.generation, 10) {
+		return errors.New("grant is not for the authenticated environment generation")
+	}
+	if req.GrantID != g.grantID || req.AttemptID != g.attemptID {
+		return errors.New("request does not carry this VM's grant")
+	}
+	if req.DeadlineMs <= 0 {
+		return errors.New("request has no deadline")
+	}
+	if !now.Before(time.UnixMilli(req.DeadlineMs)) {
+		return errors.New("request deadline already passed")
+	}
+	return nil
+}
@@ -0,0 +1,98 @@
+package main
+
+import (
+	"context"
+	"crypto/tls"
+	"net"
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+
+	lambdav1 "github.com/mulgadc/spinifex/contracts/lambda/v1"
+	"github.com/mulgadc/spinifex/spinifex/runtime/compute/lambda/guestconn"
+	"github.com/stretchr/testify/require"
+)
+
+const scope lambdav1.Scope = "ap-southeast-2/000000000001/fn-1/env-1/1"
+
+// guest is a fake fw_cfg tree; unload removes it as unloading the module would.
+type guest struct {
+	root     string
+	unloads  int
+	sealedAt chan struct{}
+}
+
+func newGuest(t *testing.T, entries map[string][]byte) *guest {
+	t.Helper()
+	g := &guest{root: filepath.Join(t.TempDir(), "qemu_fw_cfg"), sealedAt: make(chan struct{}, 1)}
+	for name, b := range entries {
+		dir := filepath.Join(g.root, "by_name", lambdav1.FwCfgPrefix, name)
+		require.NoError(t, os.MkdirAll(dir, 0o700))
+		require.NoError(t, os.WriteFile(filepath.Join(dir, "raw"), b, 0o600))
+	}
+	return g
+}
+
+func (g *guest) startup() startup {
+	return startup{
+		fwCfgDir:  filepath.Join(g.root, "by_name", lambdav1.FwCfgPrefix),
+		fwCfgRoot: g.root,
+		unload: func() error {
+			g.unloads++
+			g.sealedAt <- struct{}{}
+			return os.RemoveAll(g.root)
+		},
+	}
+}
+
+type host struct {
+	ln   *net.TCPListener
+	gen  *guestconn.Generation
+	conn chan accepted
+}
+
+func newHost(t *testing.T) *host {
+	t.Helper()
+	ln, err := net.ListenTCP("tcp", &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1)})
+	require.NoError(t, err)
+	t.Cleanup(func() { ln.Close() })
+	gen, err := guestconn.NewGeneration(scope, time.Hour)
+	require.NoError(t, err)
+	return &host{ln: ln, gen: gen, conn: make(chan accepted, 1)}
+}
+
+func (h *host) entries() map[string][]byte { return h.gen.GuestEntries(h.ln.Addr().String()) }
+
+type accepted struct {
+	conn *tls.Conn
+	err  error
+}
+
+// accept runs the host end until ctx ends and reports what it accepted.
+func (h *host) accept(ctx context.Context) {
+	go func() {
+		c, err := h.gen.AcceptGuest(ctx, h.ln, 2*time.Second)
+		h.conn <- accepted{c, err}
+	}()
+}
+
+// guest waits for the host to authenticate the guest.
+func (h *host) guest(t *testing.T) *tls.Conn {
+	t.Helper()
+	a := wait(t, h.conn, "host to authenticate the guest")
+	require.NoError(t, a.err)
+	return a.conn
+}
+
+// wait bounds every test wait so a stalled host or startup fails promptly.
+func wait[T any](t *testing.T, ch <-chan T, what string) T {
+	t.Helper()
+	select {
+	case v := <-ch:
+		return v
+	case <-time.After(10 * time.Second):
+		t.Fatalf("timed out waiting for %s", what)
+		panic("unreachable")
+	}
+}
@@ -0,0 +1,24 @@
+// lambda-agent runs inside a Lambda guest.
+//
+// A normal build reads its per-generation identity from fw_cfg, unloads
+// fw_cfg so nothing started later can read the guest key, and opens the
+// authenticated control connection to the host. It accepts no invocations and
+// never starts the function's bootstrap: the message protocol and grant
+// fencing do not exist yet. The agent never listens for the host.
+//
+// A lambdatestlink build opens the same connection, then relays one invocation
+// over a test carrier for local proofs; the carrier is not the guest protocol.
+package main
+
+import (
+	"context"
+	"flag"
+	"os"
+)
+
+func main() { os.Exit(run()) }
+
+func run() int {
+	flag.Parse()
+	return start(context.Background())
+}
@@ -0,0 +1,286 @@
+//go:build lambdatestlink
+
+package main
+
+import (
+	"archive/zip"
+	"bytes"
+	"crypto/sha256"
+	"encoding/hex"
+	"errors"
+	"fmt"
+	"io"
+	"io/fs"
+	"os"
+	"os/exec"
+	"path"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"time"
+	"unicode"
+
+	"golang.org/x/sys/unix"
+)
+
+// Dev-only code delivery for the test path: the host attaches the ZIP as a
+// read-only disk and passes its exact size and SHA-256 through fw_cfg. This
+// is not the production delivery, and these bounds are not AWS limits.
+const (
+	entryPackageSize   = "test-package-size"
+	entryPackageSHA256 = "test-package-sha256"
+	entryHandler       = "test-handler"
+
+	devMaxZipBytes    = 16 << 20
+	devMaxUnzipBytes  = 64 << 20
+	devMaxEntries     = 1024
+	packageDevice     = "/dev/vda"
+	taskTmpfsSizeOpts = "mode=0755,size=72m"
+)
+
+type packageSpec struct {
+	size    int64
+	sha256  [sha256.Size]byte
+	handler string
+}
+
+type unpackLimits struct {
+	entries int
+	bytes   int64
+}
+
+var devLimits = unpackLimits{entries: devMaxEntries, bytes: devMaxUnzipBytes}
+
+// readPackageSpec reads the expected size and SHA-256 from fw_cfg; it must
+// run before fw_cfg is sealed.
+func readPackageSpec(fwCfgDir string) (packageSpec, error) {
+	var spec packageSpec
+	read := func(name string) (string, error) {
+		b, err := os.ReadFile(filepath.Join(fwCfgDir, name, "raw"))
+		if err != nil {
+			return "", fmt.Errorf("read %s: %w", name, err)
+		}
+		return strings.TrimSpace(string(b)), nil
+	}
+	s, err := read(entryPackageSize)
+	if err != nil {
+		return spec, err
+	}
+	spec.size, err = strconv.ParseInt(s, 10, 64)
+	if err != nil || spec.size <= 0 || spec.size > devMaxZipBytes {
+		return spec, fmt.Errorf("package size %q outside 1..%d bytes", s, devMaxZipBytes)
+	}
+	h, err := read(entryPackageSHA256)
+	if err != nil {
+		return spec, err
+	}
+	b, err := hex.DecodeString(h)
+	if err != nil || len(b) != sha256.Size {
+		return spec, errors.New("package sha256 is not 64 hex digits")
+	}
+	copy(spec.sha256[:], b)
+	if spec.handler, err = read(entryHandler); err != nil {
+		return spec, err
+	}
+	if spec.handler == "" || len(spec.handler) > 128 || strings.IndexFunc(spec.handler, unicode.IsSpace) >= 0 ||
+		strings.ContainsRune(spec.handler, 0) {
+		return spec, errors.New("handler must be 1..128 characters with no whitespace")
+	}
+	return spec, nil
+}
+
+// taskPackage puts the package under dir: a fresh nosuid,nodev tmpfs, made
+// read-only once the files are in place.
+type taskPackage struct {
+	device     string
+	dir        string
+	loadDriver func() error
+	mountDir   func(dir string) error
+	sealDir    func(dir string) error
+}
+
+func devTaskPackage() taskPackage {
+	return taskPackage{device: packageDevice, dir: taskRoot, loadDriver: loadBlockDriver, mountDir: mountTaskDir, sealDir: sealTaskDir}
+}
+
+func (p taskPackage) install(spec packageSpec) error {
+	if err := p.loadDriver(); err != nil {
+		return fmt.Errorf("load block driver: %w", err)
+	}
+	dev, err := os.Open(p.device)
+	if err != nil {
+		return fmt.Errorf("open package disk: %w", err)
+	}
+	defer dev.Close()
+	if err := p.mountDir(p.dir); err != nil {
+		return fmt.Errorf("mount %s: %w", p.dir, err)
+	}
+	if err := unpack(dev, spec, p.dir, devLimits); err != nil {
+		return err
+	}
+	if err := p.sealDir(p.dir); err != nil {
+		return fmt.Errorf("make %s read-only: %w", p.dir, err)
+	}
+	return nil
+}
+
+// unpack reads exactly spec.size bytes once, checks their SHA-256, then
+// validates every entry before writing any of them under dir.
+func unpack(r io.ReaderAt, spec packageSpec, dir string, lim unpackLimits) error {
+	buf := make([]byte, spec.size)
+	if _, err := io.ReadFull(io.NewSectionReader(r, 0, spec.size), buf); err != nil {
+		return fmt.Errorf("read package: %w", err)
+	}
+	if sha256.Sum256(buf) != spec.sha256 {
+		return errors.New("package sha256 mismatch")
+	}
+	zr, err := zip.NewReader(bytes.NewReader(buf), spec.size)
+	if err != nil {
+		return fmt.Errorf("open package: %w", err)
+	}
+	if len(zr.File) > lim.entries {
+		return fmt.Errorf("package has %d entries, limit %d", len(zr.File), lim.entries)
+	}
+	names := make([]string, len(zr.File))
+	var declared int64
+	for i, f := range zr.File {
+		name, err := entryName(f.Name)
+		if err != nil {
+			return err
+		}
+		names[i] = name
+		switch mode := f.Mode(); {
+		case mode.IsDir():
+		case mode.IsRegular():
+			// Compared against what remains, so a huge size cannot wrap the sum.
+			if f.UncompressedSize64 > uint64(lim.bytes-declared) {
+				return fmt.Errorf("%q: unpacked size over the %d-byte limit", f.Name, lim.bytes)
+			}
+			declared += int64(f.UncompressedSize64)
+		default:
+			return fmt.Errorf("%q: %s entries are refused", f.Name, mode.Type())
+		}
+	}
+
+	root, err := os.OpenRoot(dir)
+	if err != nil {
+		return err
+	}
+	defer root.Close()
+	var written int64
+	for i, f := range zr.File {
+		n, err := extract(root, names[i], f, lim.bytes-written)
+		written += n
+		if err != nil {
+			return fmt.Errorf("%q: %w", f.Name, err)
+		}
+	}
+	return nil
+}
+
+// extract writes one entry, refusing it if its actual bytes exceed budget
+// whatever its header claims.
+func extract(root *os.Root, name string, f *zip.File, budget int64) (int64, error) {
+	if f.Mode().IsDir() {
+		return 0, mkdirs(root, name)
+	}
+	if d := path.Dir(name); d != "." {
+		if err := mkdirs(root, d); err != nil {
+			return 0, err
+		}
+	}
+	perm := os.FileMode(0o644)
+	if f.Mode()&0o111 != 0 {
+		perm = 0o755
+	}
+	in, err := f.Open()
+	if err != nil {
+		return 0, err
+	}
+	defer in.Close()
+	out, err := root.OpenFile(name, os.O_CREATE|os.O_EXCL|os.O_WRONLY, perm)
+	if err != nil {
+		return 0, err
+	}
+	// The umask applies at create; fchmod does not, so the mode is exact.
+	if err := out.Chmod(perm); err != nil {
+		out.Close()
+		return 0, err
+	}
+	n, err := io.Copy(out, io.LimitReader(in, budget+1))
+	if cerr := out.Close(); err == nil {
+		err = cerr
+	}
+	if err == nil && n > budget {
+		err = errors.New("unpacked bytes over the limit")
+	}
+	return n, err
+}
+
+// mkdirs creates each component of dir as mode 0755 whatever the umask. An
+// existing component must be a real directory, never a symlink.
+func mkdirs(root *os.Root, dir string) error {
+	cur := ""
+	for _, part := range strings.Split(dir, "/") {
+		cur = path.Join(cur, part)
+		if err := root.Mkdir(cur, 0o755); err != nil {
+			if !errors.Is(err, fs.ErrExist) {
+				return err
+			}
+			fi, err := root.Lstat(cur)
+			if err != nil {
+				return err
+			}
+			if !fi.IsDir() {
+				return fmt.Errorf("%q exists and is not a directory", cur)
+			}
+		}
+		if err := root.Chmod(cur, 0o755); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// entryName returns a clean relative path, refusing anything that could name
+// a location outside the task root.
+func entryName(name string) (string, error) {
+	if name == "" || strings.ContainsAny(name, "\\\x00") || path.IsAbs(name) {
+		return "", fmt.Errorf("%q: entry name refused", name)
+	}
+	c := path.Clean(name)
+	if c == "." || c == ".." || strings.HasPrefix(c, "../") {
+		return "", fmt.Errorf("%q: entry name refused", name)
+	}
+	return c, nil
+}
+
+// loadBlockDriver loads virtio_blk, which the image ships but init does not
+// load, then waits briefly for the package disk to appear.
+func loadBlockDriver() error {
+	if err := exec.Command("/sbin/modprobe", "virtio_blk").Run(); err != nil {
+		var uts unix.Utsname
+		if uerr := unix.Uname(&uts); uerr != nil {
+			return uerr
+		}
+		ko := "/lib/modules/" + unix.ByteSliceToString(uts.Release[:]) + "/kernel/drivers/block/virtio_blk.ko"
+		if err := exec.Command("/sbin/insmod", ko).Run(); err != nil {
+			return fmt.Errorf("modprobe and insmod virtio_blk failed: %w", err)
+		}
+	}
+	for range 50 {
+		if _, err := os.Stat(packageDevice); err == nil {
+			return nil
+		}
+		time.Sleep(20 * time.Millisecond)
+	}
+	return fmt.Errorf("%s did not appear", packageDevice)
+}
+
+func mountTaskDir(dir string) error {
+	return unix.Mount("tmpfs", dir, "tmpfs", unix.MS_NOSUID|unix.MS_NODEV, taskTmpfsSizeOpts)
+}
+
+func sealTaskDir(dir string) error {
+	return unix.Mount("", dir, "", unix.MS_REMOUNT|unix.MS_RDONLY|unix.MS_NOSUID|unix.MS_NODEV, "")
+}
@@ -0,0 +1,215 @@
+//go:build lambdatestlink
+
+package main
+
+import (
+	"archive/zip"
+	"bytes"
+	"crypto/sha256"
+	"hash/crc32"
+	"io/fs"
+	"math"
+	"os"
+	"path/filepath"
+	"sort"
+	"strings"
+	"syscall"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// testZip builds a ZIP; an entry named bootstrap is marked executable.
+func testZip(t *testing.T, files map[string]string) []byte {
+	t.Helper()
+	var buf bytes.Buffer
+	zw := zip.NewWriter(&buf)
+	names := make([]string, 0, len(files))
+	for n := range files {
+		names = append(names, n)
+	}
+	sort.Strings(names)
+	for _, n := range names {
+		h := &zip.FileHeader{Name: n, Method: zip.Deflate}
+		h.SetMode(0o644)
+		if n == "bootstrap" {
+			h.SetMode(0o755)
+		}
+		w, err := zw.CreateHeader(h)
+		require.NoError(t, err)
+		_, err = w.Write([]byte(files[n]))
+		require.NoError(t, err)
+	}
+	require.NoError(t, zw.Close())
+	return buf.Bytes()
+}
+
+// rawZip writes entries with headers set by the caller, so a test can claim
+// sizes or modes the data does not have.
+func rawZip(t *testing.T, entries ...*zip.FileHeader) []byte {
+	t.Helper()
+	var buf bytes.Buffer
+	zw := zip.NewWriter(&buf)
+	for _, h := range entries {
+		w, err := zw.CreateRaw(h)
+		require.NoError(t, err)
+		_, err = w.Write([]byte("x"))
+		require.NoError(t, err)
+	}
+	require.NoError(t, zw.Close())
+	return buf.Bytes()
+}
+
+func specFor(b []byte) packageSpec {
+	return packageSpec{size: int64(len(b)), sha256: sha256.Sum256(b)}
+}
+
+// Not parallel: the umask is process-wide, so the test sets a restrictive one
+// and restores it.
+func TestUnpackInstallsFilesWithFixedModes(t *testing.T) {
+	old := syscall.Umask(0o077)
+	t.Cleanup(func() { syscall.Umask(old) })
+	b := testZip(t, map[string]string{"bootstrap": "#!/bin/sh\n", "lib/data.txt": "data"})
+	dir := t.TempDir()
+	require.NoError(t, unpack(bytes.NewReader(b), specFor(b), dir, devLimits))
+
+	pfi, err := os.Stat(filepath.Join(dir, "lib"))
+	require.NoError(t, err)
+	assert.Equal(t, fs.FileMode(0o755), pfi.Mode().Perm(), "implicit parent dir")
+
+	fi, err := os.Stat(filepath.Join(dir, "bootstrap"))
+	require.NoError(t, err)
+	assert.Equal(t, fs.FileMode(0o755), fi.Mode().Perm(), "executable")
+	fi, err = os.Stat(filepath.Join(dir, "lib", "data.txt"))
+	require.NoError(t, err)
+	assert.Equal(t, fs.FileMode(0o644), fi.Mode().Perm())
+	got, err := os.ReadFile(filepath.Join(dir, "lib", "data.txt"))
+	require.NoError(t, err)
+	assert.Equal(t, "data", string(got))
+}
+
+func TestUnpackRefusesBeforeWritingAnything(t *testing.T) {
+	symlink := &zip.FileHeader{Name: "link", Method: zip.Store, CRC32: crc32.ChecksumIEEE([]byte("x")), CompressedSize64: 1, UncompressedSize64: 1}
+	symlink.SetMode(fs.ModeSymlink | 0o777)
+	fifo := &zip.FileHeader{Name: "pipe", Method: zip.Store, CRC32: crc32.ChecksumIEEE([]byte("x")), CompressedSize64: 1, UncompressedSize64: 1}
+	fifo.SetMode(fs.ModeNamedPipe | 0o644)
+	// The second size would wrap a plain running sum back under the limit.
+	big := func(name string, n uint64) *zip.FileHeader {
+		return &zip.FileHeader{Name: name, Method: zip.Store, CRC32: crc32.ChecksumIEEE([]byte("x")), CompressedSize64: 1, UncompressedSize64: n}
+	}
+	cases := map[string]struct {
+		zip  []byte
+		want string
+	}{
+		"parent traversal": {testZip(t, map[string]string{"a": "ok", "../escape": "x"}), "entry name refused"},
+		"nested traversal": {testZip(t, map[string]string{"a": "ok", "lib/../../escape": "x"}), "entry name refused"},
+		"absolute path":    {testZip(t, map[string]string{"a": "ok", "/etc/escape": "x"}), "entry name refused"},
+		"backslash":        {testZip(t, map[string]string{"a": "ok", `..\escape`: "x"}), "entry name refused"},
+		"symlink":          {rawZip(t, big("a", 1), symlink), "entries are refused"},
+		"named pipe":       {rawZip(t, big("a", 1), fifo), "entries are refused"},
+		"declared size":    {rawZip(t, big("a", devMaxUnzipBytes+1)), "over the"},
+		"wrapping sizes":   {rawZip(t, big("a", devMaxUnzipBytes), big("b", math.MaxUint64)), "over the"},
+	}
+	for name, tc := range cases {
+		t.Run(name, func(t *testing.T) {
+			dir := t.TempDir()
+			err := unpack(bytes.NewReader(tc.zip), specFor(tc.zip), dir, devLimits)
+			assert.ErrorContains(t, err, tc.want)
+			left, rerr := os.ReadDir(dir)
+			require.NoError(t, rerr)
+			assert.Empty(t, left, "files written before the refusal")
+		})
+	}
+}
+
+func TestUnpackRefusesTooManyEntries(t *testing.T) {
+	b := testZip(t, map[string]string{"a": "1", "b": "2", "c": "3"})
+	err := unpack(bytes.NewReader(b), specFor(b), t.TempDir(), unpackLimits{entries: 2, bytes: devMaxUnzipBytes})
+	assert.ErrorContains(t, err, "limit 2")
+}
+
+// The header claims a size within the limit, but the data is longer: the
+// actual bytes are refused, whatever the header says.
+func TestUnpackRefusesBytesBeyondTheLimit(t *testing.T) {
+	data := bytes.Repeat([]byte("a"), 64)
+	var buf bytes.Buffer
+	zw := zip.NewWriter(&buf)
+	h := &zip.FileHeader{Name: "a", Method: zip.Store, CRC32: crc32.ChecksumIEEE(data), CompressedSize64: uint64(len(data)), UncompressedSize64: 8}
+	w, err := zw.CreateRaw(h)
+	require.NoError(t, err)
+	_, err = w.Write(data)
+	require.NoError(t, err)
+	require.NoError(t, zw.Close())
+	b := buf.Bytes()
+
+	dir := t.TempDir()
+	err = unpack(bytes.NewReader(b), specFor(b), dir, unpackLimits{entries: 4, bytes: 16})
+	require.Error(t, err)
+	fi, serr := os.Stat(filepath.Join(dir, "a"))
+	if serr == nil {
+		assert.LessOrEqual(t, fi.Size(), int64(17))
+	}
+}
+
+func TestUnpackChecksSizeAndHash(t *testing.T) {
+	b := testZip(t, map[string]string{"bootstrap": "x"})
+	wrongHash := specFor(b)
+	wrongHash.sha256[0] ^= 1
+	assert.ErrorContains(t, unpack(bytes.NewReader(b), wrongHash, t.TempDir(), devLimits), "sha256 mismatch")
+
+	longer := specFor(b)
+	longer.size++
+	assert.ErrorContains(t, unpack(bytes.NewReader(b), longer, t.TempDir(), devLimits), "read package")
+}
+
+func TestReadPackageSpecBounds(t *testing.T) {
+	sum := "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
+	cases := map[string]struct {
+		size, hash, handler string
+		want                string
+	}{
+		"valid":            {"10", sum, "bootstrap.h", ""},
+		"zero":             {"0", sum, "bootstrap.h", "outside"},
+		"too large":        {"16777217", sum, "bootstrap.h", "outside"},
+		"not hex":          {"10", "zz", "bootstrap.h", "64 hex digits"},
+		"short":            {"10", sum[:62], "bootstrap.h", "64 hex digits"},
+		"empty handler":    {"10", sum, "", "handler"},
+		"handler space":    {"10", sum, "a b", "handler"},
+		"128-char handler": {"10", sum, strings.Repeat("h", 128), ""},
+		"129-char handler": {"10", sum, strings.Repeat("h", 129), "handler"},
+	}
+	for name, tc := range cases {
+		t.Run(name, func(t *testing.T) {
+			dir := t.TempDir()
+			for n, v := range map[string]string{entryPackageSize: tc.size, entryPackageSHA256: tc.hash, entryHandler: tc.handler} {
+				require.NoError(t, os.MkdirAll(filepath.Join(dir, n), 0o700))
+				require.NoError(t, os.WriteFile(filepath.Join(dir, n, "raw"), []byte(v), 0o600))
+			}
+			spec, err := readPackageSpec(dir)
+			if tc.want == "" {
+				require.NoError(t, err)
+				assert.Equal(t, int64(10), spec.size)
+				assert.Equal(t, tc.handler, spec.handler)
+				return
+			}
+			assert.ErrorContains(t, err, tc.want)
+		})
+	}
+}
+
+// A pre-existing symlink where a directory belongs is refused, not followed
+// or chmodded.
+func TestMkdirsRefusesSymlinkComponent(t *testing.T) {
+	dir := t.TempDir()
+	outside := t.TempDir()
+	require.NoError(t, os.Chmod(outside, 0o700))
+	require.NoError(t, os.Symlink(outside, filepath.Join(dir, "lib")))
+	root, err := os.OpenRoot(dir)
+	require.NoError(t, err)
+	defer root.Close()
+	assert.ErrorContains(t, mkdirs(root, "lib/sub"), "not a directory")
+	fi, err := os.Stat(outside)
+	require.NoError(t, err)
+	assert.Equal(t, fs.FileMode(0o700), fi.Mode().Perm(), "symlink target was chmodded")
+}
@@ -0,0 +1,56 @@
+//go:build !lambdatestlink
+
+package main
+
+import (
+	"context"
+	"crypto/tls"
+	"errors"
+	"fmt"
+	"io"
+	"log/slog"
+)
+
+// errNoProtocol is returned when the host sends anything: this build defines no
+// message protocol, so it accepts no work.
+var errNoProtocol = errors.New("host sent data, but this build accepts no invocations")
+
+func start(ctx context.Context) int {
+	code, err := defaultStartup().run(ctx)
+	if err != nil {
+		slog.Error("lambda-agent stopping", "exit", code, "err", err)
+	}
+	return code
+}
+
+// run authenticates to the host and holds the connection until it closes.
+func (s startup) run(ctx context.Context) (int, error) {
+	conn, _, code, err := s.connect(ctx)
+	if err != nil {
+		return code, err
+	}
+	defer conn.Close()
+	slog.Info("control connection open; invocations are not enabled in this build", "host", conn.RemoteAddr().String())
+	if err := awaitHostClose(ctx, conn); err != nil {
+		return exitLink, err
+	}
+	return exitClosed, nil
+}
+
+// awaitHostClose reads until the host closes the connection. Under TLS 1.3 a
+// host that rejects the guest certificate is only seen here, on first read.
+func awaitHostClose(ctx context.Context, conn *tls.Conn) error {
+	stop := context.AfterFunc(ctx, func() { _ = conn.Close() })
+	defer stop()
+	n, err := conn.Read(make([]byte, 1))
+	switch {
+	case n > 0:
+		return errNoProtocol
+	case errors.Is(err, io.EOF):
+		return nil
+	case ctx.Err() != nil:
+		return fmt.Errorf("control connection: %w", context.Cause(ctx))
+	default:
+		return fmt.Errorf("control connection: %w", err)
+	}
+}
@@ -0,0 +1,171 @@
+//go:build !lambdatestlink
+
+package main
+
+import (
+	"context"
+	"errors"
+	"net"
+	"os"
+	"testing"
+	"time"
+
+	lambdav1 "github.com/mulgadc/spinifex/contracts/lambda/v1"
+	"github.com/mulgadc/spinifex/spinifex/runtime/compute/lambda/guestconn"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+type result struct {
+	code int
+	err  error
+}
+
+func runAsync(ctx context.Context, s startup) <-chan result {
+	out := make(chan result, 1)
+	go func() {
+		code, err := s.run(ctx)
+		out <- result{code, err}
+	}()
+	return out
+}
+
+func TestStartupAuthenticatesAfterSealingFwCfg(t *testing.T) {
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	h := newHost(t)
+	h.accept(ctx)
+	g := newGuest(t, h.entries())
+	done := runAsync(ctx, g.startup())
+
+	conn := h.guest(t)
+	select {
+	case <-g.sealedAt:
+	default:
+		t.Fatal("guest dialled before fw_cfg was sealed")
+	}
+	_, err := os.Stat(g.root)
+	assert.ErrorIs(t, err, os.ErrNotExist, "fw_cfg still readable while connected")
+
+	require.NoError(t, conn.Close())
+	res := wait(t, done, "startup to return")
+	assert.Equal(t, exitClosed, res.code)
+	assert.NoError(t, res.err)
+	assert.Equal(t, 1, g.unloads)
+}
+
+func TestStartupRefusesHostData(t *testing.T) {
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	h := newHost(t)
+	h.accept(ctx)
+	done := runAsync(ctx, newGuest(t, h.entries()).startup())
+
+	conn := h.guest(t)
+	defer conn.Close()
+	_, err := conn.Write([]byte{0})
+	require.NoError(t, err)
+	res := wait(t, done, "startup to return")
+	assert.Equal(t, exitLink, res.code)
+	assert.ErrorIs(t, res.err, errNoProtocol)
+}
+
+// The host pins another generation's guest; the guest learns of the refusal
+// on its first read and fails with a link error, never a clean close.
+func TestStartupFailsWhenHostRefusesGuest(t *testing.T) {
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	h := newHost(t)
+	h.accept(ctx)
+	other, err := guestconn.NewGeneration(scope, time.Hour)
+	require.NoError(t, err)
+	entries := h.entries()
+	wrong := other.GuestEntries(h.ln.Addr().String())
+	entries[lambdav1.EntryGuestKey], entries[lambdav1.EntryGuestCert] = wrong[lambdav1.EntryGuestKey], wrong[lambdav1.EntryGuestCert]
+	g := newGuest(t, entries)
+
+	res := wait(t, runAsync(ctx, g.startup()), "startup to return")
+	assert.Equal(t, exitLink, res.code)
+	require.Error(t, res.err)
+	assert.Contains(t, res.err.Error(), "bad certificate")
+	assert.Equal(t, 1, g.unloads)
+	select {
+	case a := <-h.conn:
+		require.Error(t, a.err, "host accepted a guest it does not pin")
+	default:
+	}
+}
+
+func TestStartupRefusesWithoutDialling(t *testing.T) {
+	h := newHost(t)
+	cases := map[string]struct {
+		entries map[string][]byte
+		unload  func(g *guest) func() error
+		want    string
+	}{
+		"missing entry": {
+			entries: func() map[string][]byte { e := h.entries(); delete(e, lambdav1.EntryHostCert); return e }(),
+			want:    "read host-cert",
+		},
+		"fw_cfg stays readable": {
+			entries: h.entries(),
+			unload:  func(*guest) func() error { return func() error { return errors.New("module is built in") } },
+			want:    "fw_cfg still readable",
+		},
+		"unload claims success but tree remains": {
+			entries: h.entries(),
+			unload:  func(*guest) func() error { return func() error { return nil } },
+			want:    "unload reported success",
+		},
+	}
+	for name, tc := range cases {
+		t.Run(name, func(t *testing.T) {
+			g := newGuest(t, tc.entries)
+			s := g.startup()
+			if tc.unload != nil {
+				s.unload = tc.unload(g)
+			}
+			code, err := s.run(context.Background())
+			assert.Equal(t, exitRefused, code)
+			assert.ErrorContains(t, err, tc.want)
+		})
+	}
+
+	// Nothing dialled the host in any case above.
+	require.NoError(t, h.ln.SetDeadline(time.Now().Add(50*time.Millisecond)))
+	_, err := h.ln.Accept()
+	var ne net.Error
+	require.ErrorAs(t, err, &ne)
+	assert.True(t, ne.Timeout())
+}
+
+// A missing entry still seals fw_cfg: the key may be present even when
+// another entry is not.
+func TestStartupSealsEvenWhenIdentityIsIncomplete(t *testing.T) {
+	h := newHost(t)
+	e := h.entries()
+	delete(e, lambdav1.EntryHostAddr)
+	g := newGuest(t, e)
+	code, err := g.startup().run(context.Background())
+	assert.Equal(t, exitRefused, code)
+	assert.ErrorContains(t, err, "read host-addr")
+	assert.Equal(t, 1, g.unloads)
+	_, statErr := os.Stat(g.root)
+	assert.ErrorIs(t, statErr, os.ErrNotExist)
+}
+
+func TestStartupStopsWithContext(t *testing.T) {
+	ctx, cancel := context.WithCancelCause(context.Background())
+	defer cancel(nil)
+	h := newHost(t)
+	h.accept(ctx)
+	done := runAsync(ctx, newGuest(t, h.entries()).startup())
+	conn := h.guest(t)
+	defer conn.Close()
+
+	reason := errors.New("guest shutting down")
+	cancel(reason)
+	res := wait(t, done, "startup to return")
+	assert.Equal(t, exitLink, res.code)
+	assert.ErrorIs(t, res.err, reason)
+}
@@ -0,0 +1,193 @@
+//go:build lambdatestlink
+
+package main
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"log/slog"
+	"net"
+	"net/http"
+	"os"
+	"os/exec"
+	"time"
+
+	"github.com/mulgadc/spinifex/spinifex/agents/lambda/agent"
+	"github.com/mulgadc/spinifex/spinifex/agents/lambda/hostlink"
+	"github.com/mulgadc/spinifex/spinifex/agents/lambda/runtimeapi"
+)
+
+const (
+	runtimeAddr = "127.0.0.1:9001"
+	taskRoot    = "/var/task"
+)
+
+// testlink relays one invocation over the test carrier on the authenticated
+// connection; the carrier is not the guest protocol.
+type testlink struct {
+	startup
+	pkg            taskPackage
+	runtimeAddr    string
+	startBootstrap func(ctx context.Context, env []string) (*exec.Cmd, error)
+}
+
+func start(ctx context.Context) int {
+	t := testlink{
+		startup:     defaultStartup(),
+		pkg:         devTaskPackage(),
+		runtimeAddr: runtimeAddr,
+		startBootstrap: func(ctx context.Context, env []string) (*exec.Cmd, error) {
+			return agent.StartBootstrap(ctx, taskRoot+"/bootstrap", env, os.Stdout, os.Stderr)
+		},
+	}
+	code, err := t.run(ctx)
+	if err != nil {
+		slog.Error("lambda-agent stopping", "exit", code, "err", err)
+	}
+	return code
+}
+
+func (t testlink) run(parent context.Context) (int, error) {
+	// The package values are fw_cfg entries, so read them before connect seals it.
+	spec, specErr := readPackageSpec(t.fwCfgDir)
+	grant, grantErr := readGrant(t.fwCfgDir)
+	conn, scope, code, err := t.connect(parent)
+	if err != nil {
+		return code, errors.Join(err, specErr, grantErr)
+	}
+	defer conn.Close()
+	if err := errors.Join(specErr, grantErr); err != nil {
+		return exitRefused, err
+	}
+	link := hostlink.NewTestStream(conn)
+
+	// The handshake can return before the host refuses the guest, so nothing
+	// tenant-facing starts until the host's first valid frame arrives.
+	first, err := link.Next(parent)
+	if errors.Is(err, hostlink.ErrShutdown) {
+		return exitClosed, nil
+	}
+	if err != nil {
+		return exitLink, fmt.Errorf("first frame from host: %w", err)
+	}
+	if parent.Err() != nil {
+		return exitLink, context.Cause(parent)
+	}
+	// Nothing of the tenant's is installed or run for an unbound request.
+	if err := checkBinding(scope, grant, first, time.Now()); err != nil {
+		return exitRefused, err
+	}
+	// A host that sent no credential is refused before anything is installed.
+	if first.Credential == nil || !first.Credential.Complete() {
+		return exitRefused, errors.New("first frame carries no execution credential")
+	}
+	cred := *first.Credential
+	first.Credential = nil
+	if err := t.pkg.install(spec); err != nil {
+		return exitRefused, fmt.Errorf("install package: %w", err)
+	}
+
+	srv := runtimeapi.New()
+	ln, err := net.Listen("tcp", t.runtimeAddr)
+	if err != nil {
+		return exitLink, fmt.Errorf("listen runtime api: %w", err)
+	}
+	hs := &http.Server{Handler: srv.Handler(), ReadHeaderTimeout: 10 * time.Second}
+	go func() { _ = hs.Serve(ln) }()
+	defer hs.Close()
+
+	ctx, cancel := context.WithCancelCause(parent)
+	defer cancel(nil)
+	cmd, err := t.startBootstrap(ctx, bootstrapEnv(ln.Addr().String(), spec.handler, cred))
+	if err != nil {
+		return exitLink, fmt.Errorf("start bootstrap: %w", err)
+	}
+	exited := make(chan struct{})
+	go func() {
+		defer close(exited)
+		err := cmd.Wait()
+		// Once a response is being received the result stands, so only an
+		// exit before that cancels the invocation.
+		if srv.BootstrapExited() {
+			cancel(fmt.Errorf("bootstrap exited: %w", err))
+		}
+	}()
+
+	err = agent.Run(ctx, &oneInvocation{Link: link, first: &first}, srv)
+	cancel(errors.New("agent stopping"))
+	<-exited
+	if err != nil {
+		return exitLink, err
+	}
+	return exitClosed, nil
+}
+
+// bootstrapEnv is the bootstrap's whole environment: the runtime variables and
+// the execution role's three credential variables.
+func bootstrapEnv(runtimeAPI, handler string, cred hostlink.Credential) []string {
+	return []string{
+		"AWS_LAMBDA_RUNTIME_API=" + runtimeAPI,
+		"_HANDLER=" + handler,
+		"LAMBDA_TASK_ROOT=" + taskRoot,
+		"PATH=/bin:/sbin:/usr/bin",
+		"AWS_ACCESS_KEY_ID=" + cred.AccessKeyID,
+		"AWS_SECRET_ACCESS_KEY=" + cred.SecretAccessKey,
+		"AWS_SESSION_TOKEN=" + cred.SessionToken,
+	}
+}
+
+// shutdownGrace is how long past the invocation deadline the agent waits for
+// the host's shutdown frame once its result is written.
+const shutdownGrace = 2 * time.Second
+
+// oneInvocation serves the host's first request and then stops the agent:
+// each test VM runs a single invocation. After the result is written it waits
+// for the host's shutdown frame or EOF, so the result is not cut off by the
+// poweroff that follows the agent's exit.
+type oneInvocation struct {
+	hostlink.Link
+	first      *hostlink.Request
+	deadlineMs int64
+}
+
+var _ hostlink.Link = (*oneInvocation)(nil)
+
+func (o *oneInvocation) Next(ctx context.Context) (hostlink.Request, error) {
+	if err := ctx.Err(); err != nil {
+		return hostlink.Request{}, err
+	}
+	if o.first != nil {
+		req := *o.first
+		o.first = nil
+		o.deadlineMs = req.DeadlineMs
+		return req, nil
+	}
+	return hostlink.Request{}, o.awaitShutdown(ctx)
+}
+
+// awaitShutdown reads the host's next frame and always ends the agent: with
+// ErrShutdown for a shutdown frame, EOF, a stray frame or an expired wait.
+func (o *oneInvocation) awaitShutdown(ctx context.Context) error {
+	end := time.UnixMilli(o.deadlineMs)
+	if now := time.Now(); end.Before(now) {
+		end = now
+	}
+	wctx, cancel := context.WithDeadline(ctx, end.Add(shutdownGrace))
+	defer cancel()
+	_, err := o.Link.Next(wctx)
+	switch {
+	case errors.Is(err, hostlink.ErrShutdown), errors.Is(err, io.EOF):
+		return hostlink.ErrShutdown
+	case err == nil:
+		slog.Warn("lambda-agent: frame after the invocation was not dispatched")
+		return hostlink.ErrShutdown
+	case ctx.Err() != nil:
+		return err
+	case wctx.Err() != nil:
+		slog.Warn("lambda-agent: no shutdown from the host; exiting")
+		return hostlink.ErrShutdown
+	}
+	return err
+}
@@ -0,0 +1,506 @@
+//go:build lambdatestlink
+
+package main
+
+import (
+	"bytes"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"io"
+	"net/http"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"testing"
+	"time"
+
+	lambdav1 "github.com/mulgadc/spinifex/contracts/lambda/v1"
+	"github.com/mulgadc/spinifex/spinifex/agents/lambda/hostlink"
+	"github.com/mulgadc/spinifex/spinifex/runtime/compute/lambda/guestconn"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+const helperEnv = "LAMBDA_AGENT_TEST_BOOTSTRAP"
+
+// helperHandlerOut names the file the helper bootstrap writes its _HANDLER to.
+const helperHandlerOut = "LAMBDA_AGENT_TEST_HANDLER_OUT"
+
+// TestHelperBootstrap is the fake function bootstrap, run as a subprocess: it
+// echoes one invocation through the Runtime API, then waits for the next.
+func TestHelperBootstrap(t *testing.T) {
+	if os.Getenv(helperEnv) != "1" {
+		return
+	}
+	if out := os.Getenv(helperHandlerOut); out != "" {
+		require.NoError(t, os.WriteFile(out, []byte(os.Getenv("_HANDLER")), 0o600))
+	}
+	base := "http://" + os.Getenv("AWS_LAMBDA_RUNTIME_API") + "/2018-06-01/runtime/invocation/"
+	resp, err := http.Get(base + "next")
+	require.NoError(t, err)
+	body, err := io.ReadAll(resp.Body)
+	require.NoError(t, err)
+	resp.Body.Close()
+	id := resp.Header.Get("Lambda-Runtime-Aws-Request-Id")
+	post, err := http.Post(base+id+"/response", "application/json", bytes.NewReader(body))
+	require.NoError(t, err)
+	post.Body.Close()
+	if os.Getenv(helperExitAfterPost) == "1" {
+		return
+	}
+	_, _ = http.Get(base + "next")
+}
+
+// helperExitAfterPost makes the helper bootstrap exit once its response is posted.
+const helperExitAfterPost = "LAMBDA_AGENT_TEST_EXIT_AFTER_POST"
+
+// testGrantEntry grants g-1 for attempt a-1 on the fixture scope's env-1/1.
+const testGrantEntry = "g-1\na-1\nenv-1\n1\n"
+
+// testHandler is the handler the fixture's host configures.
+const testHandler = "bootstrap.fixture"
+
+type bootstrapStarts struct {
+	n       int
+	started chan struct{}
+	task    string
+	sealed  bool
+	// handlerOut receives the _HANDLER the bootstrap process itself saw.
+	handlerOut string
+	// env is the environment the agent gave the bootstrap.
+	env []string
+}
+
+// testlink builds a guest whose fw_cfg also carries a controlled package,
+// installed into a temp dir without mounting anything.
+func (b *bootstrapStarts) testlink(t *testing.T, entries map[string][]byte) testlink {
+	t.Helper()
+	zipBytes := testZip(t, map[string]string{"bootstrap": "#!/bin/sh\n", "lib/data.txt": "data"})
+	disk := filepath.Join(t.TempDir(), "disk")
+	require.NoError(t, os.WriteFile(disk, zipBytes, 0o600))
+	sum := sha256.Sum256(zipBytes)
+	entries[entryPackageSize] = []byte(strconv.Itoa(len(zipBytes)))
+	entries[entryPackageSHA256] = []byte(hex.EncodeToString(sum[:]))
+	if _, ok := entries[entryHandler]; !ok {
+		entries[entryHandler] = []byte(testHandler)
+	}
+	if _, ok := entries[entryGrant]; !ok {
+		entries[entryGrant] = []byte(testGrantEntry)
+	}
+	g := newGuest(t, entries)
+	b.started = make(chan struct{}, 1)
+	b.task = t.TempDir()
+	b.handlerOut = filepath.Join(t.TempDir(), "handler")
+	return testlink{
+		startup: g.startup(),
+		pkg: taskPackage{
+			device: disk, dir: b.task,
+			loadDriver: func() error { return nil },
+			mountDir:   func(string) error { return nil },
+			sealDir:    func(string) error { b.sealed = true; return nil },
+		},
+		runtimeAddr: "127.0.0.1:0",
+		startBootstrap: func(ctx context.Context, env []string) (*exec.Cmd, error) {
+			b.n++
+			b.env = env
+			b.started <- struct{}{}
+			cmd := exec.CommandContext(ctx, os.Args[0], "-test.run=^TestHelperBootstrap$")
+			cmd.Env = append(env, helperEnv+"=1", helperHandlerOut+"="+b.handlerOut)
+			return cmd, cmd.Start()
+		},
+	}
+}
+
+func runTestlinkAsync(ctx context.Context, tl testlink) <-chan result {
+	out := make(chan result, 1)
+	go func() {
+		code, err := tl.run(ctx)
+		out <- result{code, err}
+	}()
+	return out
+}
+
+type result struct {
+	code int
+	err  error
+}
+
+func TestTestlinkServesOneInvocationThenStops(t *testing.T) {
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	h := newHost(t)
+	h.accept(ctx)
+	var b bootstrapStarts
+	done := runTestlinkAsync(ctx, b.testlink(t, h.entries()))
+
+	conn := h.guest(t)
+	defer conn.Close()
+	select {
+	case <-b.started:
+		t.Fatal("bootstrap started before the host sent a frame")
+	case <-time.After(100 * time.Millisecond):
+	}
+	hs := hostlink.NewTestStream(conn)
+	req := hostlink.Request{Credential: &testCredential, GrantID: "g-1", AttemptID: "a-1", RequestID: "r-1", DeadlineMs: time.Now().Add(5 * time.Second).UnixMilli(), Payload: []byte(`{"a":40,"b":2}`)}
+	require.NoError(t, hs.HostSend(req))
+	got, err := hs.HostResult(ctx)
+	require.NoError(t, err)
+	assert.Equal(t, hostlink.Response{GrantID: "g-1", AttemptID: "a-1", RequestID: "r-1", Payload: req.Payload}, got)
+
+	// A second invocation on the same VM is never served. The agent may
+	// already have closed the connection, so a failed send is a refusal too.
+	second := req
+	second.RequestID = "r-2"
+	sendErr := hs.HostSend(second)
+	res := wait(t, done, "agent to stop after one invocation")
+	assert.Equal(t, exitClosed, res.code)
+	assert.NoError(t, res.err)
+	if sendErr == nil {
+		_, err = hs.HostResult(ctx)
+		assert.Error(t, err, "agent answered a second invocation")
+	}
+	assert.Equal(t, 1, b.n)
+	seen, err := os.ReadFile(b.handlerOut)
+	require.NoError(t, err, "bootstrap process did not report its handler")
+	assert.Equal(t, testHandler, string(seen), "bootstrap process not given the configured handler")
+	assert.True(t, b.sealed, "task dir not made read-only")
+	assert.FileExists(t, filepath.Join(b.task, "lib", "data.txt"))
+	assert.ElementsMatch(t, []string{
+		"AWS_LAMBDA_RUNTIME_API=" + envValue(b.env, "AWS_LAMBDA_RUNTIME_API"),
+		"_HANDLER=" + testHandler,
+		"LAMBDA_TASK_ROOT=" + taskRoot,
+		"PATH=/bin:/sbin:/usr/bin",
+		"AWS_ACCESS_KEY_ID=" + testCredential.AccessKeyID,
+		"AWS_SECRET_ACCESS_KEY=" + testCredential.SecretAccessKey,
+		"AWS_SESSION_TOKEN=" + testCredential.SessionToken,
+	}, b.env, "the bootstrap gets exactly the runtime and credential variables")
+	assert.NotEmpty(t, envValue(b.env, "AWS_LAMBDA_RUNTIME_API"))
+}
+
+var testCredential = hostlink.Credential{AccessKeyID: "ASIAGUESTKEY", SecretAccessKey: "guest-secret", SessionToken: "guest-token"}
+
+func envValue(env []string, name string) string {
+	for _, kv := range env {
+		if v, ok := strings.CutPrefix(kv, name+"="); ok {
+			return v
+		}
+	}
+	return ""
+}
+
+// A first frame without a complete credential installs and runs nothing.
+func TestTestlinkFrameWithoutCredentialNeverStartsBootstrap(t *testing.T) {
+	for name, cred := range map[string]*hostlink.Credential{
+		"none":       nil,
+		"incomplete": {AccessKeyID: "ASIAGUESTKEY"},
+	} {
+		t.Run(name, func(t *testing.T) {
+			ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+			defer cancel()
+			h := newHost(t)
+			h.accept(ctx)
+			var b bootstrapStarts
+			done := runTestlinkAsync(ctx, b.testlink(t, h.entries()))
+			conn := h.guest(t)
+			defer conn.Close()
+			require.NoError(t, hostlink.NewTestStream(conn).HostSend(hostlink.Request{Credential: cred, GrantID: "g-1", AttemptID: "a-1",
+				RequestID: "r-1", DeadlineMs: time.Now().Add(5 * time.Second).UnixMilli(), Payload: []byte(`{}`)}))
+
+			res := wait(t, done, "agent to stop")
+			assert.Equal(t, exitRefused, res.code)
+			assert.ErrorContains(t, res.err, "no execution credential")
+			assert.Zero(t, b.n)
+			assert.False(t, b.sealed)
+			assert.NoFileExists(t, filepath.Join(b.task, "bootstrap"), "package installed")
+		})
+	}
+}
+
+// A bootstrap that posts its response and exits at once does not cost the
+// host the result; the agent then waits for the host's shutdown.
+func TestTestlinkBootstrapExitingAfterResponseStillDelivers(t *testing.T) {
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	h := newHost(t)
+	h.accept(ctx)
+	var b bootstrapStarts
+	tl := b.testlink(t, h.entries())
+	start := tl.startBootstrap
+	tl.startBootstrap = func(ctx context.Context, env []string) (*exec.Cmd, error) {
+		return start(ctx, append(env, helperExitAfterPost+"=1"))
+	}
+	done := runTestlinkAsync(ctx, tl)
+
+	conn := h.guest(t)
+	defer conn.Close()
+	hs := hostlink.NewTestStream(conn)
+	payload := bytes.Repeat([]byte("0123456789"), 300_000)
+	req := hostlink.Request{Credential: &testCredential, GrantID: "g-1", AttemptID: "a-1", RequestID: "r-1", DeadlineMs: time.Now().Add(5 * time.Second).UnixMilli(), Payload: payload}
+	require.NoError(t, hs.HostSend(req))
+	got, err := hs.HostResult(ctx)
+	require.NoError(t, err)
+	assert.Equal(t, hostlink.Response{GrantID: "g-1", AttemptID: "a-1", RequestID: "r-1", Payload: payload}, got)
+
+	require.NoError(t, hs.HostShutdown())
+	res := wait(t, done, "agent to stop after the host's shutdown")
+	assert.Equal(t, exitClosed, res.code)
+	assert.NoError(t, res.err)
+}
+
+// The agent stays up for the host's shutdown rather than exiting as soon as
+// its result is written, and never dispatches a stray frame.
+func TestOneInvocationAfterResult(t *testing.T) {
+	first := hostlink.Request{RequestID: "r-1", DeadlineMs: time.Now().Add(time.Minute).UnixMilli()}
+	cases := map[string]struct {
+		next func(context.Context) (hostlink.Request, error)
+		want error
+	}{
+		"shutdown frame": {func(context.Context) (hostlink.Request, error) { return hostlink.Request{}, hostlink.ErrShutdown }, hostlink.ErrShutdown},
+		"eof":            {func(context.Context) (hostlink.Request, error) { return hostlink.Request{}, io.EOF }, hostlink.ErrShutdown},
+		"a second request is not dispatched": {func(context.Context) (hostlink.Request, error) {
+			return hostlink.Request{RequestID: "r-2"}, nil
+		}, hostlink.ErrShutdown},
+	}
+	for name, tc := range cases {
+		t.Run(name, func(t *testing.T) {
+			o := &oneInvocation{Link: stubLink{tc.next}, first: &first}
+			_, err := o.Next(context.Background())
+			require.NoError(t, err)
+			req, err := o.Next(context.Background())
+			require.ErrorIs(t, err, tc.want)
+			assert.Empty(t, req.RequestID)
+		})
+	}
+	t.Run("host never sends shutdown", func(t *testing.T) {
+		past := first
+		past.DeadlineMs = time.Now().Add(-time.Second).UnixMilli()
+		o := &oneInvocation{Link: stubLink{func(ctx context.Context) (hostlink.Request, error) {
+			<-ctx.Done()
+			return hostlink.Request{}, ctx.Err()
+		}}, first: &past}
+		_, err := o.Next(context.Background())
+		require.NoError(t, err)
+		start := time.Now()
+		_, err = o.Next(context.Background())
+		require.ErrorIs(t, err, hostlink.ErrShutdown)
+		assert.GreaterOrEqual(t, time.Since(start), shutdownGrace-100*time.Millisecond, "waited the grace")
+	})
+	t.Run("parent cancel is not a shutdown", func(t *testing.T) {
+		o := &oneInvocation{Link: stubLink{func(ctx context.Context) (hostlink.Request, error) {
+			<-ctx.Done()
+			return hostlink.Request{}, ctx.Err()
+		}}, first: &first}
+		_, _ = o.Next(context.Background())
+		ctx, cancel := context.WithCancel(context.Background())
+		cancel()
+		_, err := o.Next(ctx)
+		require.ErrorIs(t, err, context.Canceled)
+	})
+}
+
+type stubLink struct {
+	next func(context.Context) (hostlink.Request, error)
+}
+
+var _ hostlink.Link = stubLink{}
+
+func (l stubLink) Next(ctx context.Context) (hostlink.Request, error) { return l.next(ctx) }
+
+func (stubLink) Complete(context.Context, hostlink.Response) error { return nil }
+
+func TestTestlinkShutdownBeforeInvocationNeverStartsBootstrap(t *testing.T) {
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	h := newHost(t)
+	h.accept(ctx)
+	var b bootstrapStarts
+	done := runTestlinkAsync(ctx, b.testlink(t, h.entries()))
+
+	conn := h.guest(t)
+	defer conn.Close()
+	require.NoError(t, hostlink.NewTestStream(conn).HostShutdown())
+	res := wait(t, done, "agent to stop")
+	assert.Equal(t, exitClosed, res.code)
+	assert.NoError(t, res.err)
+	assert.Zero(t, b.n)
+}
+
+// The guest's handshake completes, but the host pins another generation's
+// guest; the refusal arrives on first read and the bootstrap never starts.
+func TestTestlinkHostRefusalNeverStartsBootstrap(t *testing.T) {
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	h := newHost(t)
+	h.accept(ctx)
+	other, err := guestconn.NewGeneration(scope, time.Hour)
+	require.NoError(t, err)
+	entries := h.entries()
+	wrong := other.GuestEntries(h.ln.Addr().String())
+	entries[lambdav1.EntryGuestKey], entries[lambdav1.EntryGuestCert] = wrong[lambdav1.EntryGuestKey], wrong[lambdav1.EntryGuestCert]
+	var b bootstrapStarts
+
+	res := wait(t, runTestlinkAsync(ctx, b.testlink(t, entries)), "agent to stop")
+	assert.Equal(t, exitLink, res.code)
+	require.Error(t, res.err)
+	assert.Contains(t, res.err.Error(), "bad certificate")
+	assert.Zero(t, b.n)
+}
+
+func TestTestlinkHostCloseBeforeFrameNeverStartsBootstrap(t *testing.T) {
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	h := newHost(t)
+	h.accept(ctx)
+	var b bootstrapStarts
+	done := runTestlinkAsync(ctx, b.testlink(t, h.entries()))
+
+	require.NoError(t, h.guest(t).Close())
+	res := wait(t, done, "agent to stop")
+	assert.Equal(t, exitLink, res.code)
+	assert.ErrorContains(t, res.err, "first frame from host")
+	assert.Zero(t, b.n)
+}
+
+// A cancelled context is reported before the buffered request is delivered,
+// and the request stays undelivered.
+func TestOneInvocationHonoursCancelledContext(t *testing.T) {
+	ctx, cancel := context.WithCancel(context.Background())
+	cancel()
+	o := &oneInvocation{first: &hostlink.Request{RequestID: "r-1"}}
+	_, err := o.Next(ctx)
+	require.ErrorIs(t, err, context.Canceled)
+
+	req, err := o.Next(context.Background())
+	require.NoError(t, err)
+	assert.Equal(t, "r-1", req.RequestID)
+}
+
+// A package that fails its check is refused after authentication, and the
+// bootstrap never starts.
+func TestTestlinkBadPackageNeverStartsBootstrap(t *testing.T) {
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	h := newHost(t)
+	h.accept(ctx)
+	var b bootstrapStarts
+	tl := b.testlink(t, h.entries())
+	tl.pkg.device = filepath.Join(t.TempDir(), "other")
+	require.NoError(t, os.WriteFile(tl.pkg.device, testZip(t, map[string]string{"bootstrap": "x"}), 0o600))
+	done := runTestlinkAsync(ctx, tl)
+
+	conn := h.guest(t)
+	defer conn.Close()
+	require.NoError(t, hostlink.NewTestStream(conn).HostSend(hostlink.Request{Credential: &testCredential, GrantID: "g-1", AttemptID: "a-1", RequestID: "r-1",
+		DeadlineMs: time.Now().Add(5 * time.Second).UnixMilli()}))
+	res := wait(t, done, "agent to stop")
+	assert.Equal(t, exitRefused, res.code)
+	assert.ErrorContains(t, res.err, "install package")
+	assert.Zero(t, b.n)
+	assert.False(t, b.sealed)
+}
+
+// Missing package values still seal fw_cfg, and the agent refuses before
+// waiting for any frame.
+func TestTestlinkMissingPackageSpecRefuses(t *testing.T) {
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	h := newHost(t)
+	h.accept(ctx)
+	var b bootstrapStarts
+	tl := b.testlink(t, h.entries())
+	require.NoError(t, os.RemoveAll(filepath.Join(tl.fwCfgDir, entryPackageSHA256)))
+
+	res := wait(t, runTestlinkAsync(ctx, tl), "agent to stop")
+	assert.Equal(t, exitRefused, res.code)
+	assert.ErrorContains(t, res.err, "read "+entryPackageSHA256)
+	assert.Zero(t, b.n)
+	_, err := os.Stat(tl.fwCfgRoot)
+	assert.ErrorIs(t, err, os.ErrNotExist, "fw_cfg not sealed")
+}
+
+// A request that does not carry this VM's grant, or a grant for another
+// environment generation than the one authenticated, installs and runs
+// nothing of the tenant's.
+func TestTestlinkUnboundRequestNeverStartsBootstrap(t *testing.T) {
+	cases := map[string]struct {
+		grant            string
+		grantID, attempt string
+		deadlineMs       int64
+		want             string
+	}{
+		"frame with no deadline":      {testGrantEntry, "g-1", "a-1", -1, "no deadline"},
+		"frame with expired deadline": {testGrantEntry, "g-1", "a-1", time.Now().Add(-time.Second).UnixMilli(), "deadline already passed"},
+		"frame for another grant":     {testGrantEntry, "g-2", "a-1", 0, "does not carry this VM's grant"},
+		"frame for another attempt":   {testGrantEntry, "g-1", "a-2", 0, "does not carry this VM's grant"},
+		"frame with no grant":         {testGrantEntry, "", "", 0, "does not carry this VM's grant"},
+		"grant for another env":       {"g-1\na-1\nenv-2\n1\n", "g-1", "a-1", 0, "not for the authenticated environment"},
+		"grant for another gen":       {"g-1\na-1\nenv-1\n2\n", "g-1", "a-1", 0, "not for the authenticated environment"},
+	}
+	for name, tc := range cases {
+		t.Run(name, func(t *testing.T) {
+			ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+			defer cancel()
+			h := newHost(t)
+			h.accept(ctx)
+			var b bootstrapStarts
+			entries := h.entries()
+			entries[entryGrant] = []byte(tc.grant)
+			done := runTestlinkAsync(ctx, b.testlink(t, entries))
+			conn := h.guest(t)
+			defer conn.Close()
+			deadline := time.Now().Add(5 * time.Second).UnixMilli()
+			switch {
+			case tc.deadlineMs < 0:
+				deadline = 0
+			case tc.deadlineMs > 0:
+				deadline = tc.deadlineMs
+			}
+			req := hostlink.Request{Credential: &testCredential, GrantID: tc.grantID, AttemptID: tc.attempt, RequestID: "r-1",
+				DeadlineMs: deadline, Payload: []byte(`{}`)}
+			require.NoError(t, hostlink.NewTestStream(conn).HostSend(req))
+
+			res := wait(t, done, "agent to stop")
+			assert.Equal(t, exitRefused, res.code)
+			assert.ErrorContains(t, res.err, tc.want)
+			assert.Zero(t, b.n)
+			assert.False(t, b.sealed)
+			assert.NoFileExists(t, filepath.Join(b.task, "bootstrap"), "package installed")
+		})
+	}
+}
+
+// A missing or malformed grant entry refuses before any frame is awaited.
+func TestTestlinkBadGrantEntryRefuses(t *testing.T) {
+	cases := map[string]*string{
+		"missing":       nil,
+		"too few lines": ptr("g-1\na-1\nenv-1\n"),
+		"generation 0":  ptr("g-1\na-1\nenv-1\n0\n"),
+		"empty grant":   ptr("\na-1\nenv-1\n1\n"),
+	}
+	for name, grant := range cases {
+		t.Run(name, func(t *testing.T) {
+			ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+			defer cancel()
+			h := newHost(t)
+			h.accept(ctx)
+			var b bootstrapStarts
+			entries := h.entries()
+			if grant != nil {
+				entries[entryGrant] = []byte(*grant)
+			}
+			tl := b.testlink(t, entries)
+			if grant == nil {
+				require.NoError(t, os.RemoveAll(filepath.Join(tl.fwCfgDir, entryGrant)))
+			}
+			res := wait(t, runTestlinkAsync(ctx, tl), "agent to stop")
+			assert.Equal(t, exitRefused, res.code)
+			assert.ErrorContains(t, res.err, entryGrant)
+			assert.Zero(t, b.n)
+		})
+	}
+}
+
+func ptr(s string) *string { return &s }
@@ -0,0 +1,224 @@
+// Command lambda-image-agent-tag checks whether the lambda-agent archived in a
+// built Lambda guest initramfs carries the lambdatestlink build tag. Build
+// information is recorded by the build; it is not proof of how the agent behaves.
+package main
+
+import (
+	"bufio"
+	"bytes"
+	"compress/gzip"
+	"debug/buildinfo"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+	"slices"
+	"strconv"
+	"strings"
+)
+
+const (
+	agentPath    = "usr/local/bin/lambda-agent"
+	maxArchive   = 1 << 30
+	maxAgent     = 256 << 20
+	maxName      = 4096
+	newcHeader   = 110
+	trailerName  = "TRAILER!!!"
+	modeTypeMask = 0o170000
+	modeDir      = 0o040000
+	modeRegular  = 0o100000
+)
+
+var agentParents = []string{"usr", "usr/local", "usr/local/bin"}
+
+func main() {
+	if len(os.Args) != 3 || (os.Args[2] != "present" && os.Args[2] != "absent") {
+		fmt.Fprintln(os.Stderr, "usage: lambda-image-agent-tag <initramfs.cpio.gz> present|absent")
+		os.Exit(2)
+	}
+	if err := run(os.Args[1], os.Args[2] == "present"); err != nil {
+		fmt.Fprintf(os.Stderr, "%s: %v\n", os.Args[1], err)
+		os.Exit(1)
+	}
+}
+
+func run(path string, want bool) error {
+	fi, err := os.Lstat(path)
+	if err != nil {
+		return err
+	}
+	if !fi.Mode().IsRegular() {
+		return errors.New("not a regular file")
+	}
+	f, err := os.Open(path)
+	if err != nil {
+		return err
+	}
+	defer f.Close()
+	agent, err := archivedAgent(f)
+	if err != nil {
+		return err
+	}
+	return checkTag(agent, want)
+}
+
+// archivedAgent returns the single agent entry after checking listed names,
+// entry types, the agent's link count and zero-only padding after the trailer.
+// It is not a full kernel-equivalent parser; 070702 CRCs, for one, are unchecked.
+func archivedAgent(r io.Reader) ([]byte, error) {
+	zr, err := gzip.NewReader(bufio.NewReader(r))
+	if err != nil {
+		return nil, fmt.Errorf("not a readable gzip archive: %w", err)
+	}
+	raw, err := io.ReadAll(io.LimitReader(zr, maxArchive+1))
+	if err != nil {
+		return nil, fmt.Errorf("not a readable gzip archive: %w", err)
+	}
+	if len(raw) > maxArchive {
+		return nil, fmt.Errorf("archive larger than %d bytes", maxArchive)
+	}
+
+	seen := map[string]bool{}
+	var agent []byte
+	off := 0
+	for {
+		e, next, err := parseEntry(raw, off)
+		if err != nil {
+			return nil, fmt.Errorf("entry at offset %d: %w", off, err)
+		}
+		off = next
+		if e.name == trailerName {
+			break
+		}
+		if !canonical(e.name) {
+			return nil, fmt.Errorf("entry name %q is not canonical", e.name)
+		}
+		if seen[e.name] {
+			return nil, fmt.Errorf("entry %q appears more than once", e.name)
+		}
+		seen[e.name] = true
+		for _, p := range agentParents {
+			if e.name == p && e.mode&modeTypeMask != modeDir {
+				return nil, fmt.Errorf("%s is not a directory", p)
+			}
+		}
+		if e.name == agentPath {
+			for _, p := range agentParents {
+				if !seen[p] {
+					return nil, fmt.Errorf("%s has no %s directory entry before it", agentPath, p)
+				}
+			}
+			if e.mode&modeTypeMask != modeRegular {
+				return nil, fmt.Errorf("%s is not a regular file", agentPath)
+			}
+			if e.nlink != 1 {
+				return nil, fmt.Errorf("%s has %d links, want 1", agentPath, e.nlink)
+			}
+			if len(e.data) > maxAgent {
+				return nil, fmt.Errorf("%s is larger than %d bytes", agentPath, maxAgent)
+			}
+			agent = e.data
+		}
+	}
+	if i := bytes.IndexFunc(raw[off:], func(r rune) bool { return r != 0 }); i >= 0 {
+		return nil, fmt.Errorf("data after the archive trailer at offset %d", off+i)
+	}
+	if agent == nil {
+		return nil, fmt.Errorf("no %s entry", agentPath)
+	}
+	return agent, nil
+}
+
+type entry struct {
+	name  string
+	mode  uint64
+	nlink uint64
+	data  []byte
+}
+
+func parseEntry(raw []byte, off int) (entry, int, error) {
+	if len(raw)-off < newcHeader {
+		return entry{}, 0, errors.New("truncated header")
+	}
+	h := raw[off : off+newcHeader]
+	if m := string(h[:6]); m != "070701" && m != "070702" {
+		return entry{}, 0, fmt.Errorf("magic %q is not newc", m)
+	}
+	field := func(i int) (uint64, error) {
+		return strconv.ParseUint(string(h[6+8*i:14+8*i]), 16, 32)
+	}
+	mode, err1 := field(1)
+	nlink, err2 := field(4)
+	size, err3 := field(6)
+	nameSize, err4 := field(11)
+	if err := errors.Join(err1, err2, err3, err4); err != nil {
+		return entry{}, 0, fmt.Errorf("bad header field: %w", err)
+	}
+	if nameSize < 2 || nameSize > maxName {
+		return entry{}, 0, fmt.Errorf("name size %d out of range", nameSize)
+	}
+	nameEnd := off + newcHeader + int(nameSize)
+	dataOff := align4(nameEnd)
+	if dataOff > len(raw) || size > uint64(len(raw)-dataOff) {
+		return entry{}, 0, errors.New("truncated entry")
+	}
+	dataEnd := dataOff + int(size)
+	if dataEnd > len(raw) || align4(dataEnd) > len(raw) {
+		return entry{}, 0, errors.New("truncated entry")
+	}
+	name := raw[off+newcHeader : nameEnd]
+	if name[len(name)-1] != 0 || bytes.IndexByte(name[:len(name)-1], 0) >= 0 {
+		return entry{}, 0, errors.New("name is not one NUL-terminated string")
+	}
+	return entry{name: string(name[:len(name)-1]), mode: mode, nlink: nlink, data: raw[dataOff:dataEnd]}, align4(dataEnd), nil
+}
+
+func align4(n int) int { return (n + 3) &^ 3 }
+
+// canonical accepts "." and relative names with no empty, "." or ".." component.
+func canonical(name string) bool {
+	if name == "." {
+		return true
+	}
+	if name == "" || strings.HasPrefix(name, "/") {
+		return false
+	}
+	for c := range strings.SplitSeq(name, "/") {
+		if c == "" || c == "." || c == ".." {
+			return false
+		}
+	}
+	return true
+}
+
+func checkTag(agent []byte, want bool) error {
+	info, err := buildinfo.Read(bytes.NewReader(agent))
+	if err != nil {
+		return fmt.Errorf("no Go build information in %s: %w", agentPath, err)
+	}
+	return decide(info, want)
+}
+
+// decide refuses build information with no recorded settings, since their
+// absence does not show the tag was absent.
+func decide(info *buildinfo.BuildInfo, want bool) error {
+	if len(info.Settings) == 0 {
+		return fmt.Errorf("%s records no build settings", agentPath)
+	}
+	if hasTag(info, "lambdatestlink") != want {
+		return fmt.Errorf("%s: lambdatestlink build tag present=%v, want %v", agentPath, !want, want)
+	}
+	return nil
+}
+
+func hasTag(info *buildinfo.BuildInfo, tag string) bool {
+	for _, s := range info.Settings {
+		if s.Key != "-tags" {
+			continue
+		}
+		if slices.Contains(strings.Split(s.Value, ","), tag) {
+			return true
+		}
+	}
+	return false
+}
@@ -0,0 +1,154 @@
+package main
+
+import (
+	"bytes"
+	"compress/gzip"
+	"debug/buildinfo"
+	"fmt"
+	"os"
+	"path/filepath"
+	"runtime/debug"
+	"testing"
+
+	"github.com/stretchr/testify/require"
+)
+
+type fileEntry struct {
+	name  string
+	mode  uint32
+	nlink uint32
+	data  string
+}
+
+func dir(name string) fileEntry { return fileEntry{name: name, mode: modeDir | 0o755, nlink: 2} }
+func file(name, data string) fileEntry {
+	return fileEntry{name: name, mode: modeRegular | 0o755, nlink: 1, data: data}
+}
+func symlink(name, target string) fileEntry {
+	return fileEntry{name: name, mode: 0o120777, nlink: 1, data: target}
+}
+
+// newc writes entries and a trailer in the format the kernel unpacks.
+func newc(entries ...fileEntry) []byte {
+	var b bytes.Buffer
+	for i, e := range append(entries, fileEntry{name: trailerName, nlink: 1}) {
+		fmt.Fprintf(&b, "070701%08X%08X%08X%08X%08X%08X%08X%08X%08X%08X%08X%08X%08X",
+			i+1, e.mode, 0, 0, e.nlink, 0, len(e.data), 0, 0, 0, 0, len(e.name)+1, 0)
+		b.WriteString(e.name)
+		b.WriteByte(0)
+		pad(&b)
+		b.WriteString(e.data)
+		pad(&b)
+	}
+	return b.Bytes()
+}
+
+func pad(b *bytes.Buffer) {
+	for b.Len()%4 != 0 {
+		b.WriteByte(0)
+	}
+}
+
+func gz(parts ...[]byte) []byte {
+	var b bytes.Buffer
+	for _, p := range parts {
+		w := gzip.NewWriter(&b)
+		_, _ = w.Write(p)
+		_ = w.Close()
+	}
+	return b.Bytes()
+}
+
+func withAgent(agent fileEntry) []byte {
+	return newc(dir("."), dir("usr"), dir("usr/local"), dir("usr/local/bin"), agent, file("init", "#!/bin/sh"))
+}
+
+func TestArchivedAgent_ReturnsTheOneArchivedAgent(t *testing.T) {
+	got, err := archivedAgent(bytes.NewReader(gz(withAgent(file(agentPath, "agent-bytes")))))
+	require.NoError(t, err)
+	require.Equal(t, "agent-bytes", string(got))
+
+	padded := append(withAgent(file(agentPath, "agent-bytes")), make([]byte, 1024)...)
+	got, err = archivedAgent(bytes.NewReader(gz(padded)))
+	require.NoError(t, err, "zero padding after the trailer")
+	require.Equal(t, "agent-bytes", string(got))
+}
+
+func TestArchivedAgent_Refuses(t *testing.T) {
+	parents := []fileEntry{dir("."), dir("usr"), dir("usr/local"), dir("usr/local/bin")}
+	agent := file(agentPath, "untagged")
+	small := newc(append(parents, agent)...)
+	second := newc(append(parents[1:], file(agentPath, "tagged"))...)
+	// The second archive starts inside what would be the first one's final block.
+	sameBlock := append(append([]byte{}, small...), second...)
+	for len(sameBlock)%512 != 0 {
+		sameBlock = append(sameBlock, 0)
+	}
+
+	for name, archive := range map[string][]byte{
+		"aliased duplicate":            gz(newc(append(parents, agent, file("././"+agentPath, "tagged"))...)),
+		"leading slash alias":          gz(newc(append(parents, agent, file("/"+agentPath, "tagged"))...)),
+		"dot-dot component":            gz(newc(append(parents, agent, file("usr/local/../local/bin/lambda-agent", "x"))...)),
+		"empty component":              gz(newc(append(parents, agent, file("usr//local/bin/lambda-agent", "x"))...)),
+		"exact duplicate":              gz(newc(append(parents, agent, file(agentPath, "tagged"))...)),
+		"duplicate directory":          gz(newc(dir("usr"), dir("usr"), dir("usr/local"), dir("usr/local/bin"), agent)),
+		"missing directory entry":      gz(newc(dir("usr"), dir("usr/local/bin"), agent)),
+		"directory after the agent":    gz(newc(dir("usr"), dir("usr/local"), agent, dir("usr/local/bin"))),
+		"parent is a symlink":          gz(newc(dir("usr"), dir("usr/local"), symlink("usr/local/bin", "elsewhere"), agent)),
+		"agent is a symlink":           gz(newc(append(parents, symlink(agentPath, "/bin/sh"))...)),
+		"hard-linked agent":            gz(newc(append(parents, fileEntry{name: agentPath, mode: modeRegular | 0o755, nlink: 2, data: "x"})...)),
+		"no agent":                     gz(newc(parents...)),
+		"second archive appended":      gz(small, second),
+		"second archive in same block": gz(sameBlock),
+		"non-zero byte after trailer":  gz(append(append([]byte{}, small...), 0, 0, 1)),
+		"truncated archive":            gz(small[:len(small)-20]),
+		"not newc":                     gz([]byte("not cpio at all, but long enough to fill one newc header: " + string(make([]byte, 120)))),
+		"not gzip":                     small,
+	} {
+		t.Run(name, func(t *testing.T) {
+			_, err := archivedAgent(bytes.NewReader(archive))
+			require.Error(t, err)
+		})
+	}
+}
+
+func TestRun_RefusesASymlinkedImagePath(t *testing.T) {
+	d := t.TempDir()
+	img := filepath.Join(d, "initramfs.cpio.gz")
+	require.NoError(t, os.WriteFile(img, gz(withAgent(file(agentPath, "x"))), 0o644))
+	require.NoError(t, os.Symlink(img, filepath.Join(d, "link")))
+	require.ErrorContains(t, run(filepath.Join(d, "link"), false), "not a regular file")
+}
+
+func TestHasTag_MatchesWholeTagsOnly(t *testing.T) {
+	info := func(tags string) *buildinfo.BuildInfo {
+		return &buildinfo.BuildInfo{Settings: []debug.BuildSetting{{Key: "-buildmode", Value: "exe"}, {Key: "-tags", Value: tags}}}
+	}
+	require.True(t, hasTag(info("lambdatestlink,fips140v1.0"), "lambdatestlink"))
+	require.False(t, hasTag(info("fips140v1.0"), "lambdatestlink"))
+	require.False(t, hasTag(info("lambdatestlinkx,xlambdatestlink"), "lambdatestlink"))
+	require.False(t, hasTag(&buildinfo.BuildInfo{}, "lambdatestlink"))
+}
+
+// The test binary carries real build information, so checkTag reads it and
+// refuses the opposite answer; bytes with no build information are refused.
+func TestCheckTag_ReadsRealBuildInformation(t *testing.T) {
+	exe, err := os.Executable()
+	require.NoError(t, err)
+	raw, err := os.ReadFile(exe)
+	require.NoError(t, err)
+	info, err := buildinfo.Read(bytes.NewReader(raw))
+	require.NoError(t, err)
+	tagged := hasTag(info, "lambdatestlink")
+	require.NoError(t, checkTag(raw, tagged))
+	require.Error(t, checkTag(raw, !tagged))
+	require.Error(t, checkTag([]byte("not a Go binary"), false))
+}
+
+func TestDecide_RefusesMissingBuildSettings(t *testing.T) {
+	require.ErrorContains(t, decide(&buildinfo.BuildInfo{}, false), "no build settings")
+	require.ErrorContains(t, decide(&buildinfo.BuildInfo{}, true), "no build settings")
+	untagged := &buildinfo.BuildInfo{Settings: []debug.BuildSetting{{Key: "-buildmode", Value: "exe"}}}
+	require.NoError(t, decide(untagged, false))
+	require.Error(t, decide(untagged, true))
+}
@@ -0,0 +1,26 @@
+//go:build lambdatestlink
+
+// Command lambda-image-digest prints the test-link runtime image digest of one
+// directory, selecting the boot files the same way the daemon does. The digest
+// identifies the bytes; it does not authenticate where they came from.
+package main
+
+import (
+	"fmt"
+	"os"
+
+	"github.com/mulgadc/spinifex/spinifex/runtime/compute/lambda/testlink"
+)
+
+func main() {
+	if len(os.Args) != 2 {
+		fmt.Fprintln(os.Stderr, "usage: lambda-image-digest <image-dir>")
+		os.Exit(2)
+	}
+	digest, err := testlink.DirDigest(os.Args[1])
+	if err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
+	fmt.Println(digest)
+}
@@ -1,17 +1,22 @@
-// sqs-worker runs inside the shared SQS worker microVM.
+// sqs-worker runs inside the SQS microVM, next to its own SQS NATS server.
 //
 // Usage:
 //
 //	sqs-worker boot
 //	sqs-worker ready
 //
-// boot selects and mounts the worker's data disk named by fw_cfg, asks the
-// gateway for its current first-store decision with its instance-role
-// credentials, then loads or creates the store key on that disk and
-// publishes only the public key. ready then keeps checking the bound store's
-// JetStream with that key until it is stopped, keeping a readiness marker only
-// while the check passes; it proves the store accepted the worker key, not
-// the store server's identity.
+// boot selects and mounts the guest's data disk named by fw_cfg, asks the
+// gateway for its receipt key decision with its instance-role credentials,
+// then loads the recorded key, or creates one on a new volume and records its
+// fingerprint, before the guest's SQS store first starts on that disk. ready
+// then keeps checking the guest's own loopback JetStream, keeping a readiness
+// marker only while it answers, and serves SQS owner calls against it, pulling
+// them from the gateway with the same credentials.
+//
+// Serving is refused unless the gateway reports a single-node host cluster.
+// A disk whose receipt key is missing or does not match the recorded one
+// refuses to serve; recovering it needs a procedure not yet designed, and
+// nothing deletes or replaces the volume.
 package main

 import (
@@ -23,10 +28,14 @@ import (
 	"syscall"
 	"time"

+	"github.com/nats-io/nats.go/jetstream"
+
 	"github.com/mulgadc/spinifex/internal/gwsign"
 	"github.com/mulgadc/spinifex/internal/sqsworkergw"
 	"github.com/mulgadc/spinifex/spinifex/agents/sqs/workerboot"
 	"github.com/mulgadc/spinifex/spinifex/agents/sqs/workerkey"
+	"github.com/mulgadc/spinifex/spinifex/domains/sqs/workerserve"
+	"github.com/mulgadc/spinifex/spinifex/foundation/state/clustersize"
 )

 // The cluster CA the daemon supplies; the only root the client trusts.
@@ -36,7 +45,11 @@ const caPath = "/sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/ca-cert/raw"
 // no static-key fallback.
 func newClaimer() workerboot.NewClaimer {
 	return func(region, gatewayURL string) (workerkey.Claimer, error) {
-		return newClient(region, gatewayURL)
+		c, err := newClient(region, gatewayURL)
+		if err != nil {
+			return nil, err
+		}
+		return workerboot.GatewayClaimer{Client: c}, nil
 	}
 }

@@ -48,13 +61,6 @@ func newClient(region, gatewayURL string) (*sqsworkergw.Client, error) {
 	return sqsworkergw.New(gatewayURL, region, ca, gwsign.NewInstanceRole())
 }

-type describer struct{ c *sqsworkergw.Client }
-
-func (d describer) DescribeStore(ctx context.Context) (workerboot.Store, error) {
-	st, err := d.c.DescribeStore(ctx)
-	return workerboot.Store(st), err
-}
-
 func main() {
 	if len(os.Args) != 2 || (os.Args[1] != "boot" && os.Args[1] != "ready") {
 		fmt.Fprintln(os.Stderr, "usage: sqs-worker boot|ready")
@@ -66,11 +72,18 @@ func main() {
 		run = ready
 	}
 	if err := run(logf); err != nil {
+		if os.Args[1] == "boot" && workerboot.Transient(err) {
+			logf("boot not settled, will retry: " + err.Error())
+			os.Exit(exitTransient)
+		}
 		logf("FATAL: " + err.Error())
 		os.Exit(1)
 	}
 }

+// exitTransient tells the guest's supervisor that boot may be run again.
+const exitTransient = 75
+
 func config(logf func(string)) workerboot.Config {
 	return workerboot.Config{
 		FwCfgDir:      "/sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/sqs-worker",
@@ -99,19 +112,26 @@ func boot(logf func(string)) error {
 func ready(logf func(string)) error {
 	ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
 	defer cancel()
+	// The store is one guest, so its streams and buckets are R1.
+	if err := clustersize.Declare(1); err != nil {
+		return err
+	}
 	err := workerboot.Watch(ctx, workerboot.ReadyConfig{
-		Config: config(logf),
-		NewDescriber: func(region, gatewayURL string) (workerboot.StoreDescriber, error) {
-			c, err := newClient(region, gatewayURL)
-			if err != nil {
-				return nil, err
-			}
-			return describer{c}, nil
-		},
+		Config:      config(logf),
 		Checker:     workerboot.NATSChecker{},
 		CallTimeout: 30 * time.Second,
 		Refresh:     30 * time.Second,
 		Refused:     5 * time.Minute,
+		Serve: &workerboot.ServeConfig{
+			Connector: workerboot.NATSConnector{},
+			Run: func(ctx context.Context, region, gatewayURL string, js jetstream.JetStream, receiptKey []byte) error {
+				c, err := newClient(region, gatewayURL)
+				if err != nil {
+					return err
+				}
+				return workerserve.Run(ctx, c, js, receiptKey, logf)
+			},
+		},
 	})
 	if errors.Is(err, context.Canceled) {
 		return nil
@@ -10,7 +10,7 @@ quota state.
 a request/reply route on which a gateway, in the `awsgw-quota-seal` queue
 group, irreversibly seals one account's quota override record.
 `domains/admission/quota` owns it: `awsgw` serves it.
-`accountteardown` is the consumer: it calls this subject to fence every
+`operator/accountlifecycle` is the consumer: it calls this subject to fence every
 override writer at the quota owner before the account record is removed.

 The request names the account in its body (`account_id`) and may carry an
@@ -31,10 +31,20 @@ Any current gateway may answer; the leader-accepted revision is the evidence
 whichever one does, and the override bucket is opened at seal time even on a
 gateway whose quotas were disabled at startup.

+## Seal-quota-override capability
+
+`AccountQuotaOverrideSealCapabilitySubject(node)` (`admission.v1.AccountQuotaOverrideSealCapability.<node>`) is a per-node, read-only request/reply route on which `operator/accountlifecycle` asks one configured gateway whether it supports the quota override seal, immediately before every seal attempt.
+The quota owner (`domains/admission/quota`) owns it: `awsgw` serves it with no queue group, so a request addressed to one node reaches only subscribers configured as that node, and two gateways sharing a node ID both answer.
+The consumer's quota seal reaper probes every configured node that runs `awsgw` and sends no seal request unless every one answers compatible.
+
+`AccountQuotaOverrideSealCapabilityVersion` is a claim about the whole binary's writers, not about the responder: version 1 means every quota override writer in the binary is revision-guarded and refuses a sealed record, and the binary serves the seal subject.
+A gateway binary without this responder has no subscriber, so the probe for that node reads as incomplete (`no-reply`), the same as a stopped or partitioned node.
+The reply proves nothing beyond the moment it was sent: a gateway downgraded or replaced after a successful probe is outside the guarantee.
+
 ### Compatibility boundary

-The subject name carries the contract major; `account_quota_seal_test.go`
-preserves the version-1 JSON shape.
+The subject name carries the contract major; `account_quota_seal_test.go` and `account_quota_seal_capability_test.go`
+preserve the version-1 JSON shapes and the subject builder.
 Producer and consumer are in the coordinated Spinifex release, so the window
 is N and N-1.
 A gateway that predates the subject has no subscription, so the request gets
@@ -0,0 +1,39 @@
+package admissionv1
+
+import "strings"
+
+// AccountQuotaOverrideSealCapabilityVersion is the version this binary reports.
+// Version 1: every quota override writer in the binary is revision-guarded and
+// refuses a sealed record, and the seal subject is served.
+const AccountQuotaOverrideSealCapabilityVersion = 1
+
+// AccountQuotaOverrideSealCapabilitySubject is one gateway's read-only
+// seal-capability route, addressed per node so a probe reaches exactly the
+// node it asks about.
+func AccountQuotaOverrideSealCapabilitySubject(node string) string {
+	return "admission.v1.AccountQuotaOverrideSealCapability." + node
+}
+
+// ValidSealCapabilityNode reports whether node is exactly one NATS subject
+// token. Responder and probe both check it before building a subject, so
+// neither can address a wildcard or another node's route.
+func ValidSealCapabilityNode(node string) bool {
+	return node != "" && !strings.ContainsAny(node, ".*>\t\n\r ")
+}
+
+// AccountQuotaOverrideSealCapabilityRequest asks node whether it supports the
+// quota override seal. CorrelationRef is an opaque caller reference echoed for
+// logs; it grants nothing and is never a fence or authority to write.
+type AccountQuotaOverrideSealCapabilityRequest struct {
+	Node           string `json:"node"`
+	CorrelationRef string `json:"correlation_ref,omitempty"`
+}
+
+// AccountQuotaOverrideSealCapabilityReply reports the responding node's
+// capability version. It carries no account data and proves nothing beyond
+// the moment it was sent.
+type AccountQuotaOverrideSealCapabilityReply struct {
+	Node              string `json:"node"`
+	CapabilityVersion int    `json:"capability_version"`
+	CorrelationRef    string `json:"correlation_ref,omitempty"`
+}
@@ -0,0 +1,43 @@
+package admissionv1
+
+import (
+	"encoding/json"
+	"testing"
+
+	"github.com/stretchr/testify/require"
+)
+
+func TestAccountQuotaOverrideSealCapabilitySubject(t *testing.T) {
+	require.Equal(t, "admission.v1.AccountQuotaOverrideSealCapability.node-1", AccountQuotaOverrideSealCapabilitySubject("node-1"))
+	require.Equal(t, "admission.v1.AccountQuotaOverrideSealCapability.node-2", AccountQuotaOverrideSealCapabilitySubject("node-2"))
+}
+
+func TestAccountQuotaOverrideSealCapabilityRequestJSONContract(t *testing.T) {
+	data, err := json.Marshal(AccountQuotaOverrideSealCapabilityRequest{Node: "node-1", CorrelationRef: "req-1"})
+	require.NoError(t, err)
+	require.JSONEq(t, `{"node":"node-1","correlation_ref":"req-1"}`, string(data))
+
+	data, err = json.Marshal(AccountQuotaOverrideSealCapabilityRequest{Node: "node-1"})
+	require.NoError(t, err)
+	require.JSONEq(t, `{"node":"node-1"}`, string(data))
+}
+
+func TestAccountQuotaOverrideSealCapabilityReplyJSONContract(t *testing.T) {
+	reply := AccountQuotaOverrideSealCapabilityReply{Node: "node-1", CapabilityVersion: 1, CorrelationRef: "req-1"}
+	data, err := json.Marshal(reply)
+	require.NoError(t, err)
+	require.JSONEq(t, `{"node":"node-1","capability_version":1,"correlation_ref":"req-1"}`, string(data))
+
+	data, err = json.Marshal(AccountQuotaOverrideSealCapabilityReply{Node: "node-1", CapabilityVersion: 1})
+	require.NoError(t, err)
+	require.JSONEq(t, `{"node":"node-1","capability_version":1}`, string(data))
+}
+
+func TestValidSealCapabilityNode(t *testing.T) {
+	for _, node := range []string{"node-1", "node_2", "n3"} {
+		require.True(t, ValidSealCapabilityNode(node), node)
+	}
+	for _, node := range []string{"", "a.b", "*", ">", "node-*", "node >", "node\t1", "node\n1"} {
+		require.False(t, ValidSealCapabilityNode(node), "%q", node)
+	}
+}
@@ -0,0 +1,255 @@
+package clusterv1
+
+import "regexp"
+
+const (
+	// NodeFactsSubject fans out a request for each daemon's hardware and
+	// cabling description. Its reply payload is NodeFacts.
+	NodeFactsSubject = "spinifex.node.facts"
+
+	nodeNetTestPrefix = "spinifex.node.nettest."
+)
+
+var nodeIDPattern = regexp.MustCompile(`^[A-Za-z0-9_-]{1,64}$`)
+
+// ValidNodeID reports whether a node name is safe to place in a subject: no
+// NATS token separator or wildcard can get through.
+func ValidNodeID(node string) bool {
+	return nodeIDPattern.MatchString(node)
+}
+
+// NodeFactsOfSubject asks one node alone for its NodeFacts.
+func NodeFactsOfSubject(node string) string {
+	return NodeFactsSubject + "." + node
+}
+
+// NetTestServeSubject asks one node to start a one-shot throughput server.
+func NetTestServeSubject(node string) string {
+	return nodeNetTestPrefix + node + ".serve"
+}
+
+// NetTestRunSubject asks one node to run the client side of a throughput test.
+func NetTestRunSubject(node string) string {
+	return nodeNetTestPrefix + node + ".run"
+}
+
+// Planes a node's NICs can carry, named as the installer names them.
+const (
+	PlaneWAN = "wan"
+	PlaneLAN = "lan"
+	PlaneVPC = "vpc"
+)
+
+// NetTestPort is the TCP port reserved for node-to-node throughput tests.
+const NetTestPort = 5001
+
+// NodeFacts describes one node's hardware, OS and cabling. Fields a host
+// cannot report are left empty, and Sources says why.
+type NodeFacts struct {
+	Node        string            `json:"node"`
+	CollectedAt string            `json:"collected_at,omitempty"`
+	FormFactor  string            `json:"form_factor"`
+	Virt        string            `json:"virt,omitempty"`
+	System      SystemFacts       `json:"system"`
+	OS          OSFacts           `json:"os"`
+	CPU         CPUFacts          `json:"cpu"`
+	Memory      MemoryFacts       `json:"memory"`
+	NICs        []NICFacts        `json:"nics"`
+	Planes      []PlaneFacts      `json:"planes"`
+	Disks       []DiskFacts       `json:"disks,omitempty"`
+	Filesystems []FilesystemFacts `json:"filesystems,omitempty"`
+	Sources     map[string]string `json:"sources"`
+}
+
+// SystemFacts is the machine's identity from DMI.
+type SystemFacts struct {
+	Vendor       string `json:"vendor,omitempty"`
+	Product      string `json:"product,omitempty"`
+	Family       string `json:"family,omitempty"`
+	Version      string `json:"version,omitempty"`
+	Serial       string `json:"serial,omitempty"`
+	UUID         string `json:"uuid,omitempty"`
+	ChassisType  string `json:"chassis_type,omitempty"`
+	ChassisAsset string `json:"chassis_asset,omitempty"`
+	BoardVendor  string `json:"board_vendor,omitempty"`
+	BoardProduct string `json:"board_product,omitempty"`
+	BoardSerial  string `json:"board_serial,omitempty"`
+	BIOSVendor   string `json:"bios_vendor,omitempty"`
+	BIOSVersion  string `json:"bios_version,omitempty"`
+	BIOSDate     string `json:"bios_date,omitempty"`
+	DeviceModel  string `json:"device_model,omitempty"`
+}
+
+// OSFacts names the running OS and Spinifex build.
+type OSFacts struct {
+	Name          string `json:"name,omitempty"`
+	Kernel        string `json:"kernel,omitempty"`
+	Arch          string `json:"arch,omitempty"`
+	Version       string `json:"spx_version,omitempty"`
+	Commit        string `json:"spx_commit,omitempty"`
+	UptimeSeconds int64  `json:"uptime_seconds,omitempty"`
+	Hostname      string `json:"hostname,omitempty"`
+}
+
+// CPUFacts summarises the sockets. Threads and Cores are totals. Features are
+// the flags worth naming; Flags is every flag the CPU reports.
+type CPUFacts struct {
+	Model          string   `json:"model,omitempty"`
+	Vendor         string   `json:"vendor,omitempty"`
+	Sockets        int      `json:"sockets"`
+	CoresPerSocket int      `json:"cores_per_socket,omitempty"`
+	Cores          int      `json:"cores"`
+	Threads        int      `json:"threads"`
+	MaxMHz         int      `json:"max_mhz,omitempty"`
+	CurrentMHz     int      `json:"current_mhz,omitempty"`
+	Virtualization string   `json:"virtualization,omitempty"`
+	Features       []string `json:"features,omitempty"`
+	Flags          []string `json:"flags,omitempty"`
+}
+
+// MemoryFacts groups identical DIMMs so a 32-slot board reads as one line.
+type MemoryFacts struct {
+	TotalBytes    uint64        `json:"total_bytes"`
+	SlotsUsed     int           `json:"slots_used"`
+	SlotsTotal    int           `json:"slots_total"`
+	MaxCapacity   string        `json:"max_capacity,omitempty"`
+	ECC           string        `json:"ecc,omitempty"`
+	Groups        []MemoryGroup `json:"groups"`
+	OSTotalBytes  uint64        `json:"os_total_bytes,omitempty"`
+	OSAvailBytes  uint64        `json:"os_available_bytes,omitempty"`
+	HugePageBytes uint64        `json:"huge_page_bytes,omitempty"`
+}
+
+// MemoryGroup is a set of DIMMs alike in every field shown. ConfiguredMTs
+// below SpeedMTs means the board runs them slower than rated.
+type MemoryGroup struct {
+	Count         int      `json:"count"`
+	SizeBytes     uint64   `json:"size_bytes"`
+	Type          string   `json:"type,omitempty"`
+	TypeDetail    string   `json:"type_detail,omitempty"`
+	FormFactor    string   `json:"form_factor,omitempty"`
+	SpeedMTs      int      `json:"speed_mts,omitempty"`
+	ConfiguredMTs int      `json:"configured_mts,omitempty"`
+	Manufacturer  string   `json:"manufacturer,omitempty"`
+	PartNumber    string   `json:"part_number,omitempty"`
+	Rank          string   `json:"rank,omitempty"`
+	Serials       []string `json:"serials,omitempty"`
+}
+
+// DiskFacts is one physical drive as sysfs reports it. Transport is nvme, sata,
+// usb or virtio, and empty when the device path names none, as a RAID
+// controller's virtual disk does.
+type DiskFacts struct {
+	Name       string `json:"name"`
+	Path       string `json:"path"`
+	SizeBytes  uint64 `json:"size_bytes"`
+	Model      string `json:"model,omitempty"`
+	Transport  string `json:"transport,omitempty"`
+	Rotational bool   `json:"rotational,omitempty"`
+	Removable  bool   `json:"removable,omitempty"`
+}
+
+// FilesystemFacts is one mounted block-device or ZFS filesystem. BindOf names
+// the directory a bind mount shows; Disk names the drives under it. Error says
+// why Usage is missing or stale.
+type FilesystemFacts struct {
+	Mount  string           `json:"mount"`
+	Source string           `json:"source"`
+	Disk   string           `json:"disk,omitempty"`
+	Type   string           `json:"type"`
+	BindOf string           `json:"bind_of,omitempty"`
+	Usage  *FilesystemUsage `json:"usage,omitempty"`
+	Error  string           `json:"error,omitempty"`
+	// ID identifies the mount on its node, so a filesystem replaced at the
+	// same path is read afresh. Empty for a mount the daemon cannot see.
+	ID string `json:"-"`
+}
+
+// FilesystemUsage is a filesystem's last statfs reading, as df counts it.
+type FilesystemUsage struct {
+	SizeBytes  uint64 `json:"size_bytes"`
+	UsedBytes  uint64 `json:"used_bytes"`
+	AvailBytes uint64 `json:"avail_bytes"`
+}
+
+// NICFacts is one physical network interface and, where LLDP reports it, the
+// switch port it is cabled to.
+type NICFacts struct {
+	Name      string        `json:"name"`
+	MAC       string        `json:"mac,omitempty"`
+	Driver    string        `json:"driver,omitempty"`
+	Firmware  string        `json:"firmware,omitempty"`
+	BusInfo   string        `json:"bus_info,omitempty"`
+	Model     string        `json:"model,omitempty"`
+	SpeedMbps int           `json:"speed_mbps,omitempty"`
+	MaxMbps   int           `json:"max_mbps,omitempty"`
+	Duplex    string        `json:"duplex,omitempty"`
+	Port      string        `json:"port,omitempty"`
+	MTU       int           `json:"mtu,omitempty"`
+	State     string        `json:"state"`
+	Master    string        `json:"master,omitempty"`
+	Planes    []string      `json:"planes,omitempty"`
+	Neighbor  *LLDPNeighbor `json:"neighbor,omitempty"`
+}
+
+// LLDPNeighbor is what the switch on the other end of a NIC advertises.
+type LLDPNeighbor struct {
+	ChassisID         string   `json:"chassis_id"`
+	SystemName        string   `json:"system_name,omitempty"`
+	SystemDescription string   `json:"system_description,omitempty"`
+	PortID            string   `json:"port_id,omitempty"`
+	PortDescription   string   `json:"port_description,omitempty"`
+	Capabilities      []string `json:"capabilities,omitempty"`
+}
+
+// PlaneFacts is the address a plane uses on this node and the NICs under it.
+type PlaneFacts struct {
+	Plane   string   `json:"plane"`
+	Bridge  string   `json:"bridge,omitempty"`
+	Address string   `json:"address,omitempty"`
+	Prefix  int      `json:"prefix,omitempty"`
+	NICs    []string `json:"nics,omitempty"`
+	MTU     int      `json:"mtu,omitempty"`
+}
+
+// NetTestServeRequest starts a server bound to the node's address on Plane.
+type NetTestServeRequest struct {
+	Plane           string `json:"plane"`
+	DurationSeconds int    `json:"duration_seconds"`
+}
+
+// NetTestServeResponse names the address the server listens on.
+type NetTestServeResponse struct {
+	Address string `json:"address,omitempty"`
+	Port    int    `json:"port,omitempty"`
+	Error   string `json:"error,omitempty"`
+}
+
+// NetTestRunRequest runs the client towards Address from the node's own
+// address on Plane. Address must be what Target reports for that plane.
+type NetTestRunRequest struct {
+	Plane           string `json:"plane"`
+	Target          string `json:"target"`
+	Address         string `json:"address"`
+	Port            int    `json:"port"`
+	DurationSeconds int    `json:"duration_seconds"`
+	Streams         int    `json:"streams"`
+}
+
+// NetTestRunResponse is one direction of a pair's measurement.
+type NetTestRunResponse struct {
+	SourceAddress string    `json:"source_address,omitempty"`
+	BitsPerSecond float64   `json:"bits_per_second"`
+	Bytes         int64     `json:"bytes"`
+	Seconds       float64   `json:"seconds"`
+	Streams       int       `json:"streams"`
+	StreamBPS     []float64 `json:"stream_bps,omitempty"`
+	RTTMinMillis  float64   `json:"rtt_min_ms,omitempty"`
+	RTTAvgMillis  float64   `json:"rtt_avg_ms,omitempty"`
+	RTTMaxMillis  float64   `json:"rtt_max_ms,omitempty"`
+	PacketLoss    float64   `json:"packet_loss_percent,omitempty"`
+	LinkMbps      int       `json:"link_mbps,omitempty"`
+	MTU           int       `json:"mtu,omitempty"`
+	Tool          string    `json:"tool,omitempty"`
+	Error         string    `json:"error,omitempty"`
+}
@@ -23,7 +23,7 @@ subject shape are wire compatibility. Callers and receivers must use this
 package rather than rebuild those details. `commands_test.go` preserves the
 version-1 JSON shape and subject form.

-This package owns shared command payloads and the drain-volume response. It
+This package owns shared command payloads. It
 does not define every action's reply body, authorize the request, store EC2
 state or implement NATS transport. Those remain responsibilities of the EC2
 domain and its runtime composition.
@@ -37,6 +37,13 @@ contract or an approved compatibility plan; it is not a package refactor.
 Only the subject is owned here today.
 The request (`account_id`) and reply (`vpc_id`, `error`) bodies are still declared as anonymous structs on each side; moving them into this package is a contract change that needs its own characterization first.

+## Volume held elsewhere
+
+A start refused because another node holds one of the instance's volumes is an `IncorrectState` error payload on the `ec2.start`, `ec2.start.<node>` and `ec2.cmd.<instance-id>` replies, and the AWS body cannot say why.
+The owning node adds the `X-Volume-Held-Elsewhere: true` header to that reply, the `ec2.start` forwarder relays it, and `StartInstances` wraps `ErrVolumeHeldElsewhere` around the error it rebuilds, so a caller matches it with `errors.Is` rather than reading the message.
+The owning node sets the header only from the EBS provider's `volume_held_elsewhere` code, which every provider returns for this refusal; no reader matches the message text.
+`VolumeHeldElsewhereText` is the wording viperblockd puts in that refusal's message, not a signal.
+
 ## Platform inventory

 `PlatformInventorySubject` (`ec2.v1.PlatformInventory`) is a request/reply route on which a daemon in the `spinifex-workers` queue group reports one account's key pairs, placement groups, launch templates and their `ec2-tags` entries from the EC2 owner stores.
@@ -29,7 +29,6 @@ type EC2InstanceCommand struct {
 	Attributes                EC2CommandAttributes       `json:"attributes"`
 	AttachVolumeData          *AttachVolumeData          `json:"attach_volume_data,omitempty"`
 	DetachVolumeData          *DetachVolumeData          `json:"detach_volume_data,omitempty"`
-	DrainVolumeData           *DrainVolumeData           `json:"drain_volume_data,omitempty"`
 	AttachENIData             *AttachENIData             `json:"attach_eni_data,omitempty"`
 	DetachENIData             *DetachENIData             `json:"detach_eni_data,omitempty"`
 	IamProfileAssociationData *IamProfileAssociationData `json:"iam_profile_association_data,omitempty"`
@@ -45,7 +44,6 @@ type EC2CommandAttributes struct {
 	StartInstance               bool `json:"start_instance"`
 	AttachVolume                bool `json:"attach_volume"`
 	DetachVolume                bool `json:"detach_volume"`
-	DrainVolume                 bool `json:"drain_volume,omitempty"`
 	RebootInstance              bool `json:"reboot_instance"`
 	AttachENI                   bool `json:"attach_eni"`
 	DetachENI                   bool `json:"detach_eni"`
@@ -69,33 +67,6 @@ type DetachVolumeData struct {
 	Force    bool   `json:"force,omitempty"`
 }

-// DrainVolumeData carries parameters for a drain-volume command, which flushes
-// the volume's in-flight writes to S3 before a snapshot reads them. The command
-// is addressed to the instance the volume is attached to because the drain
-// socket only exists on the node hosting it.
-type DrainVolumeData struct {
-	VolumeID string `json:"volume_id"`
-}
-
-const (
-	// DrainVolumeStatusDrained is the ack a node returns once the volume's
-	// in-flight writes have reached S3.
-	DrainVolumeStatusDrained = "drained"
-
-	// DrainVolumeStatusNotRunning is the ack a node returns when it still holds
-	// the instance but the VM is not running, so no process is writing to the
-	// volume and its sealed checkpoint is already current. An attachment record
-	// outlives the writer — stop deliberately leaves boot volumes attached — so
-	// the caller cannot tell this from the record alone.
-	DrainVolumeStatusNotRunning = "not-running"
-)
-
-// DrainVolumeResponse is the reply to a drain-volume command.
-type DrainVolumeResponse struct {
-	VolumeID string `json:"volume_id"`
-	Status   string `json:"status"`
-}
-
 // AttachENIData carries parameters for an attach-network-interface command.
 type AttachENIData struct {
 	NetworkInterfaceID string `json:"network_interface_id"`
@@ -16,7 +16,6 @@ func TestEC2InstanceCommandJSONContract(t *testing.T) {
 			StartInstance:               true,
 			AttachVolume:                true,
 			DetachVolume:                true,
-			DrainVolume:                 true,
 			RebootInstance:              true,
 			AttachENI:                   true,
 			DetachENI:                   true,
@@ -28,7 +27,6 @@ func TestEC2InstanceCommandJSONContract(t *testing.T) {
 		},
 		AttachVolumeData:          &AttachVolumeData{VolumeID: "vol-1", Device: "/dev/sdf"},
 		DetachVolumeData:          &DetachVolumeData{VolumeID: "vol-2", Device: "/dev/sdg", Force: true},
-		DrainVolumeData:           &DrainVolumeData{VolumeID: "vol-3"},
 		AttachENIData:             &AttachENIData{NetworkInterfaceID: "eni-1", DeviceIndex: 2},
 		DetachENIData:             &DetachENIData{AttachmentID: "eni-attach-1", Force: true},
 		IamProfileAssociationData: &IamProfileAssociationData{InstanceProfileArn: "arn:aws:iam::123456789012:instance-profile/example"},
@@ -47,7 +45,6 @@ func TestEC2InstanceCommandJSONContract(t *testing.T) {
 			"start_instance": true,
 			"attach_volume": true,
 			"detach_volume": true,
-			"drain_volume": true,
 			"reboot_instance": true,
 			"attach_eni": true,
 			"detach_eni": true,
@@ -59,7 +56,6 @@ func TestEC2InstanceCommandJSONContract(t *testing.T) {
 		},
 		"attach_volume_data": {"volume_id": "vol-1", "device": "/dev/sdf"},
 		"detach_volume_data": {"volume_id": "vol-2", "device": "/dev/sdg", "force": true},
-		"drain_volume_data": {"volume_id": "vol-3"},
 		"attach_eni_data": {"network_interface_id": "eni-1", "device_index": 2},
 		"detach_eni_data": {"attachment_id": "eni-attach-1", "force": true},
 		"iam_profile_association_data": {"instance_profile_arn": "arn:aws:iam::123456789012:instance-profile/example"},
@@ -69,16 +65,6 @@ func TestEC2InstanceCommandJSONContract(t *testing.T) {
 	}`, string(data))
 }

-func TestDrainVolumeResponseJSONContract(t *testing.T) {
-	data, err := json.Marshal(DrainVolumeResponse{
-		VolumeID: "vol-1",
-		Status:   DrainVolumeStatusDrained,
-	})
-	require.NoError(t, err)
-	require.JSONEq(t, `{"volume_id":"vol-1","status":"drained"}`, string(data))
-	require.Equal(t, "not-running", DrainVolumeStatusNotRunning)
-}
-
 func TestInstanceCommandSubjects(t *testing.T) {
 	require.Equal(t, "ec2.cmd.i-0123456789abcdef0", InstanceCommandSubject("i-0123456789abcdef0"))
 	require.Equal(t, "ec2.cmd.*", InstanceCommandSubjectWildcard)
@@ -1,7 +1,13 @@
 package ec2v1

-// VolumeHeldElsewhereText is the text of a start refused because another node
-// holds one of the instance's volumes. It crosses NATS as message text, so
-// readers match on it; the provider that refuses and every reader share this
-// string so they cannot drift.
+import "errors"
+
+// VolumeHeldElsewhereText is the message viperblockd gives a lease refusal. It
+// is wording for the operator; callers classify the refusal by its code.
 const VolumeHeldElsewhereText = "volume is leased by another owner"
+
+// ErrVolumeHeldElsewhere marks a start refused because another node still
+// holds one of the instance's volumes. The owning node sets it from the EBS
+// provider's held-elsewhere code, and a start reply carries it across NATS as
+// the X-Volume-Held-Elsewhere header rather than as message text.
+var ErrVolumeHeldElsewhere = errors.New("a volume is leased by another node")
@@ -126,8 +126,8 @@ type Assign struct {
 	ENIPrivateIP  string `json:"eniPrivateIp,omitempty"`
 	ENISubnetID   string `json:"eniSubnetId,omitempty"`
 	// TaskRoleARN is the task's IAM role (from the taskdef); when set, the agent
-	// serves its credentials at 169.254.170.2 under CredID, defaulting CredID to
-	// the taskID when the scheduler leaves it empty.
+	// serves its credentials at 169.254.170.2 under CredID, a scheduler-minted
+	// secret. An assign without a CredID gets no credentials.
 	CredID      string `json:"credId,omitempty"`
 	TaskRoleARN string `json:"taskRoleArn,omitempty"`
 	// ExecutionRoleARN is the task's execution role (from the taskdef); when set,
@@ -0,0 +1,66 @@
+# Lambda v1 contracts
+
+This package will own the Lambda process contracts declared by ADR-0009:S13.
+It holds the L-GUEST declaration and, in `guest.go`, the connection vocabulary both ends share: the scope, the URI SAN roles, the ALPN identifier and the fw_cfg entry names. The transport and authentication are decided (ADR-0009:S17); the invocation message types and framing are not chosen, so no message types are defined here.
+It is not an AWS tenant-facing API.
+
+## L-GUEST: node executor to guest agent
+
+**Owner:** the Lambda domain.
+
+**Producers and consumers:** the node-local executor (`runtime/compute/lambda`) and the in-guest Lambda agent (`agents/lambda`), in both directions.
+
+**Operations:** bootstrap with scoped material, readiness, invoke, completion, bounded logs, shutdown.
+
+**Transport:** TCP over the microVM's network (ADR-0009:S17).
+The agent dials one connection to a listener the executor owns for the environment, and the guest exposes no listener for host control.
+At most one authenticated control connection is active per environment, and the agent may reconnect; how a prior connection and its in-flight work are handled is open (ADR-0009 M1).
+The network is shared and untrusted: no source address, port or bridge membership is identity or authority.
+Tenant code may be able to reach the listener, but it cannot open the agent's connection or key material, so any connection it makes fails authentication.
+
+**Authentication:** mutual TLS 1.3, bound to the environment ID and realization generation, using material the user runtime cannot read (ADR-0009:S17).
+For each generation the executor creates ephemeral self-signed host and guest certificates whose single URI SAN is `spinifex-lambda://<Region>/<account>/<function ID>/<environment ID>/<generation>/<role>`, with role `host` or `guest`.
+Each end validates the peer certificate against the one pinned peer certificate and separately requires that URI SAN with the opposite role; hostname verification is not relied on.
+ALPN carries the L-GUEST major version, `spinifex-lambda-guest/1`, and an unsupported version fails the handshake. Builds with the `lambdatestlink` tag offer only `spinifex-lambda-testlink/2` on both host and guest, whose first frame carries the execution-role credential, so a test-link end and a release end fail the handshake; release builds are unchanged.
+Under TLS 1.3 the guest's handshake can complete before the host refuses its certificate, so the guest treats the connection as accepted only after the host's first message.
+
+**Bootstrap material:** the executor passes these fw_cfg entries under `opt/spinifex/lambda/`, and the agent reads them as root and unloads `qemu_fw_cfg` before user code starts.
+
+| Entry | Content |
+|---|---|
+| `guest-key` | Guest private key, PEM |
+| `guest-cert` | Guest certificate, PEM |
+| `host-cert` | The one pinned host certificate, PEM |
+| `expect` | `<Region>/<account>/<function ID>/<environment ID>/<generation>` |
+| `host-addr` | Host listener as `host:port` |
+
+The guest's own interface settings come from `opt/spinifex/netcfg`, the entry other system VMs use.
+The agent unloads `qemu_fw_cfg` even when an entry is missing, and refuses to start if the fw_cfg tree is still readable afterwards.
+Until the message format exists, the agent authenticates and holds the connection but accepts no host message and starts no user code.
+
+**Authority:** one environment's attempt only.
+The user runtime never holds these keys, and sharing the guest with the agent gives user code no platform authority.
+
+**Idempotency and fencing:** an invoke carries the attempt-grant identity and the attempt ID.
+The agent accepts a run command only under a current grant, and a completion under a superseded grant is discarded and reported.
+How a grant is enforced and proven ended is open (ADR-0009 M1).
+
+**Errors and versions:** the agent and executor fail closed on an unknown protocol version or an identity mismatch.
+
+**Readiness:** environment readiness needs authenticated, current-generation agent evidence; a running hypervisor process does not count.
+
+**Payloads:** the contract must carry a synchronous payload and response of up to 6 MB and an asynchronous payload of up to 1 MB, plus framing, over a path whose size bound is declared.
+How payloads move is open (ADR-0009 M3).
+
+### Compatibility boundary
+
+The agent ships inside the runtime image pinned by each published version, so it is outside the coordinated release train.
+The executor supports every L-GUEST major carried by a still-supported runtime image.
+Retiring an image follows a published runtime-deprecation lifecycle that blocks create and update before it blocks invoke.
+
+Evidence required before the first agent image ships: a function on an old image invoked by a new executor, and the reverse during rollback.
+
+## L-RUNTIME
+
+The guest agent serves the AWS Lambda Runtime API, version `2018-06-01`, to the user runtime at `AWS_LAMBDA_RUNTIME_API`.
+Spinifex does not version this AWS contract, and conformance is measured against AWS.
@@ -0,0 +1,54 @@
+// Package lambdav1 holds the Lambda process-contract vocabulary shared by the
+// node executor and the guest agent. See README.md for the declarations.
+package lambdav1
+
+import (
+	"fmt"
+	"strings"
+)
+
+// ALPNv1 names L-GUEST major version 1; a peer offering any other fails the handshake.
+const ALPNv1 = "spinifex-lambda-guest/1"
+
+// FwCfgPrefix prefixes every fw_cfg entry the executor passes the guest.
+const FwCfgPrefix = "opt/spinifex/lambda/"
+
+// fw_cfg entry names under FwCfgPrefix.
+const (
+	EntryGuestKey  = "guest-key"
+	EntryGuestCert = "guest-cert"
+	EntryHostCert  = "host-cert"
+	EntryScope     = "expect"
+	EntryHostAddr  = "host-addr"
+)
+
+// Roles named as the last URI SAN path segment.
+const (
+	RoleHost  = "host"
+	RoleGuest = "guest"
+)
+
+const uriScheme = "spinifex-lambda"
+
+// Scope names one environment generation:
+// Region/account/function ID/environment ID/generation.
+type Scope string
+
+// ParseScope accepts exactly five non-empty, slash-separated segments.
+func ParseScope(s string) (Scope, error) {
+	parts := strings.Split(s, "/")
+	if len(parts) != 5 {
+		return "", fmt.Errorf("lambdav1: scope %q: want 5 segments, got %d", s, len(parts))
+	}
+	for _, p := range parts {
+		if p == "" || strings.ContainsAny(p, " \t\r\n?#%") {
+			return "", fmt.Errorf("lambdav1: scope %q: empty or invalid segment", s)
+		}
+	}
+	return Scope(s), nil
+}
+
+// URI is the single URI SAN a certificate for role in this scope carries.
+func (s Scope) URI(role string) string {
+	return uriScheme + "://" + string(s) + "/" + role
+}
@@ -0,0 +1,20 @@
+package lambdav1
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestParseScope(t *testing.T) {
+	s, err := ParseScope("ap-southeast-2/000000000001/fn-1/env-1/gen-1")
+	require.NoError(t, err)
+	assert.Equal(t, "spinifex-lambda://ap-southeast-2/000000000001/fn-1/env-1/gen-1/host", s.URI(RoleHost))
+	for _, bad := range []string{
+		"", "a/b/c/d", "a/b/c/d/e/f", "a//c/d/e", "a/b/c/d/e/", "a/b/c/d/e?x", "a/b/c/d/e#x", "a b/b/c/d/e",
+	} {
+		_, err := ParseScope(bad)
+		assert.Error(t, err, bad)
+	}
+}
@@ -25,6 +25,10 @@ const (
 	DeletionStateRunning   = "RUNNING"
 	DeletionStateCompleted = "COMPLETED"
 	DeletionStateFailed    = "FAILED"
+
+	// DeletionStateExercised ends a non-finalizing run: the account is left
+	// TERMINATING and a later DeleteAccount takes the same operation over.
+	DeletionStateExercised = "EXERCISED"
 )

 // CreateAccountRequest is the /admin/CreateAccount request body.
@@ -52,6 +56,10 @@ type DeleteAccountRequest struct {
 	ClientToken string `json:"clientToken"`
 	Force       bool   `json:"force,omitempty"`
 	DryRun      bool   `json:"dryRun,omitempty"`
+
+	// NonFinalizing quiesces the account, cleans the quiesced EC2 platform
+	// kinds and stops. No legacy stage runs and the account record stays.
+	NonFinalizing bool `json:"nonFinalizing,omitempty"`
 }

 // DescribeAccountDeletionRequest is the /admin/DescribeAccountDeletion body.
@@ -74,3 +82,129 @@ type ListAccountsResponse struct {
 	Accounts []AccountSummary `json:"accounts"`
 	Count    int              `json:"count"`
 }
+
+// IAMQuiescence is the coordinator's recorded view of IAM quiescence for one
+// fence. Recording only: it admits nothing. QuiescedScope is filled only by a run
+// that guarantees a final inventory re-read, and holds as of EvaluatedAt.
+type IAMQuiescence struct {
+	OperationID   string            `json:"operationId"`
+	Fence         uint64            `json:"fence"`
+	Generation    uint64            `json:"generation"`
+	QuiescedScope []string          `json:"quiescedScope"`
+	Unresolved    []UnresolvedScope `json:"unresolved"`
+	Consumers     []ConsumerBasis   `json:"consumers"`
+	Writers       []AccountedWriter `json:"writers"`
+	Inventory     []InventoryStatus `json:"inventory"`
+	EvaluatedAt   string            `json:"evaluatedAt"`
+}
+
+// UnresolvedScope names a scope or writer that blocks quiescence, with a fixed
+// reason. It is never a timeout.
+type UnresolvedScope struct {
+	Scope  string `json:"scope"`
+	Class  string `json:"class"`
+	Reason string `json:"reason"`
+}
+
+// ConsumerBasis cites an enforcement consumer and the accepted contract that
+// lets it count as having applied the generation.
+type ConsumerBasis struct {
+	Consumer string `json:"consumer"`
+	Basis    string `json:"basis"`
+}
+
+// AccountedWriter is a writer that cannot be fenced and is covered by the
+// owner's inventory instead. Ref names the tracked work that closes the gap.
+type AccountedWriter struct {
+	Writer string `json:"writer"`
+	Basis  string `json:"basis"`
+	Ref    string `json:"ref"`
+}
+
+// InventoryStatus is the owner inventory's status for one kind at evaluation.
+type InventoryStatus struct {
+	Kind   string `json:"kind"`
+	Status string `json:"status"`
+}
+
+// Classes of an UnresolvedScope.
+const (
+	UnresolvedClassFence    = "fence"
+	UnresolvedClassConsumer = "consumer"
+	UnresolvedClassWriter   = "writer"
+)
+
+// Fixed reasons of an UnresolvedScope.
+const (
+	ReasonFenceMismatch                = "fence-mismatch"
+	ReasonIAMReadbackFailed            = "iam-readback-failed"
+	ReasonNoAcceptedEnforcement        = "no-accepted-enforcement-contract"
+	ReasonNoCompleteOwnerInventory     = "no-complete-owner-inventory"
+	ReasonParticipantNotDeclared       = "participant-not-declared"
+	ReasonOwnerInventoryIncomplete     = "owner-inventory-incomplete"
+	ReasonFinalInventoryRereadNotWired = "final-inventory-reread-not-wired"
+)
+
+// ExerciseResult is what a non-finalizing run did after IAM quiescence.
+type ExerciseResult struct {
+	// Skipped is the fixed reason no cleanup ran; Cleanup is then absent.
+	Skipped    string           `json:"skipped,omitempty"`
+	Cleanup    *PlatformCleanup `json:"cleanup,omitempty"`
+	Accounting []KindAccounting `json:"accounting,omitempty"`
+}
+
+// PlatformCleanup is the EC2 platform cleanup result. It carries resource IDs,
+// fixed reasons and counts only, never names, tags, key material or data.
+type PlatformCleanup struct {
+	Kinds  []CleanupKind  `json:"kinds"`
+	Items  []CleanupItem  `json:"items,omitempty"`
+	Counts []CleanupCount `json:"counts,omitempty"`
+}
+
+// CleanupKind is one kind's scan status after the cleanup's own re-scan.
+type CleanupKind struct {
+	Kind              string   `json:"kind"`
+	Status            string   `json:"status"`
+	IncompleteReasons []string `json:"incompleteReasons,omitempty"`
+}
+
+// CleanupItem is one resource by its ID with what the cleanup did to it.
+type CleanupItem struct {
+	ResourceID  string `json:"resourceId"`
+	Kind        string `json:"kind"`
+	Disposition string `json:"disposition"`
+	Reason      string `json:"reason,omitempty"`
+	MemberCount int    `json:"memberCount,omitempty"`
+}
+
+// CleanupCount is the number of records of one class that have no public ID.
+type CleanupCount struct {
+	Kind        string `json:"kind"`
+	Class       string `json:"class"`
+	Disposition string `json:"disposition"`
+	Reason      string `json:"reason,omitempty"`
+	Count       int    `json:"count"`
+}
+
+// KindAccounting is the final inventory re-read held against the cleanup result
+// for one kind. Quiescence is not rewritten from it.
+type KindAccounting struct {
+	Kind      string `json:"kind"`
+	Status    string `json:"status"`
+	Reason    string `json:"reason,omitempty"`
+	Remaining int    `json:"remaining"`
+}
+
+// Accounting statuses and fixed reasons of a KindAccounting.
+const (
+	AccountingAccounted   = "accounted"
+	AccountingUnaccounted = "unaccounted"
+
+	AccountingReasonRereadIncomplete = "reread-incomplete"
+	AccountingReasonUnnamedRemainder = "unnamed-remainder"
+)
+
+// Fixed reasons an exercise ran no cleanup.
+const (
+	ExerciseSkippedNoQuiescedKind = "no-quiesced-kind"
+)
@@ -43,6 +43,27 @@ func TestAdminWire_DeletionStates(t *testing.T) {
 	require.Equal(t, "RUNNING", DeletionStateRunning)
 	require.Equal(t, "COMPLETED", DeletionStateCompleted)
 	require.Equal(t, "FAILED", DeletionStateFailed)
+	require.Equal(t, "EXERCISED", DeletionStateExercised)
+}
+
+func TestAdminWire_NonFinalizingDeleteRequest(t *testing.T) {
+	goldenJSON(t, DeleteAccountRequest{AccountID: "000000000042", AccountName: "acme", ClientToken: "tok-3", NonFinalizing: true},
+		`{"accountId":"000000000042","accountName":"acme","clientToken":"tok-3","nonFinalizing":true}`)
+}
+
+func TestAdminWire_ExerciseResult(t *testing.T) {
+	goldenJSON(t, ExerciseResult{
+		Cleanup: &PlatformCleanup{
+			Kinds:  []CleanupKind{{Kind: "key-pair", Status: "observed-clean-at-scan-time"}},
+			Items:  []CleanupItem{{ResourceID: "pg-1", Kind: "placement-group", Disposition: "blocked", Reason: "instance-members", MemberCount: 1}},
+			Counts: []CleanupCount{{Kind: "key-pair", Class: "public-key-object-without-metadata", Disposition: "deleted", Count: 2}},
+		},
+		Accounting: []KindAccounting{{Kind: "placement-group", Status: AccountingAccounted, Remaining: 1}},
+	}, `{"cleanup":{"kinds":[{"kind":"key-pair","status":"observed-clean-at-scan-time"}],
+		"items":[{"resourceId":"pg-1","kind":"placement-group","disposition":"blocked","reason":"instance-members","memberCount":1}],
+		"counts":[{"kind":"key-pair","class":"public-key-object-without-metadata","disposition":"deleted","count":2}]},
+		"accounting":[{"kind":"placement-group","status":"accounted","remaining":1}]}`)
+	goldenJSON(t, ExerciseResult{Skipped: ExerciseSkippedNoQuiescedKind}, `{"skipped":"no-quiesced-kind"}`)
 }

 func TestAdminWire_ListAccountsResponse(t *testing.T) {
@@ -0,0 +1,53 @@
+package telemetryv1
+
+import "regexp"
+
+// Node live subjects mirror the instance ones, keyed by node name. Only the
+// super-admin account's gateway handler subscribes to them.
+const (
+	nodeLiveWatchSubjectPrefix  = "telemetry.watch.node."
+	nodeLiveSampleSubjectPrefix = "telemetry.live.node."
+)
+
+var liveNodePattern = regexp.MustCompile(`^[A-Za-z0-9_-]{1,64}$`)
+
+// ValidLiveNode reports whether a node name is safe to place in a subject.
+func ValidLiveNode(node string) bool {
+	return liveNodePattern.MatchString(node)
+}
+
+// NodeLiveWatchSubject is where a viewer asks a node to keep sampling itself.
+// A request on it is answered with a NodeLiveRing.
+func NodeLiveWatchSubject(node string) string {
+	return nodeLiveWatchSubjectPrefix + node
+}
+
+// NodeLiveSampleSubject carries one NodeLiveSample per period while watched.
+func NodeLiveSampleSubject(node string) string {
+	return nodeLiveSampleSubjectPrefix + node
+}
+
+// NodeLiveRing answers a watch with the samples newer than its SinceMillis.
+type NodeLiveRing struct {
+	Samples []NodeLiveSample `json:"samples"`
+}
+
+// NodeLiveSample is one host measurement. Rates are per second over the
+// interval ending at TimeMillis.
+type NodeLiveSample struct {
+	TimeMillis int64   `json:"t"`
+	CPUPercent float64 `json:"cpu"`
+	// MemoryUsedPercent is used memory less page cache, as free(1) reports it.
+	MemoryUsedPercent float64 `json:"mem"`
+	Load1             float64 `json:"load1"`
+	// Planes holds the traffic of each plane's physical NICs, keyed wan, lan, vpc.
+	Planes map[string]NodeLivePlane `json:"planes"`
+}
+
+// NodeLivePlane is one plane's traffic from the host's side.
+type NodeLivePlane struct {
+	NetworkIn  float64 `json:"netIn"`
+	NetworkOut float64 `json:"netOut"`
+	PacketsIn  float64 `json:"pktIn"`
+	PacketsOut float64 `json:"pktOut"`
+}
@@ -51,19 +51,6 @@ request/reply replies.
 removed its local state and export. These routes and JSON shapes are deployed
 legacy compatibility behaviour, not a generic EBS-provider interface.

-## Fence route
-
-`VolumeFencedSubject(node)` is `ebs.<node>.fenced`, or `ebs.fenced` with no
-node for a single-node daemon. The Viperblock adapter publishes
-`VolumeFencedEvent` on it after tearing down an export whose volume lease
-moved to another node; the local daemon subscribes to stop the guest that
-was using it. Node-addressed, like mount and unmount: the guest is local, and
-no other node's daemon has anything to stop.
-
-The event is sent as a request and repeated until the daemon replies with `VolumeFencedAck`, because the fence often runs while the node's NATS server is restarting and a single publish can reach no subscriber.
-The daemon replies only when the event carries a reply subject, so a plain publish is still accepted.
-Repeats are harmless, since fencing a guest that is already fenced changes nothing.
-
 ## Abandon route

 `VolumeAbandonSubject(node)` is `ebs.<node>.abandon`. The daemon uses
@@ -9,9 +9,8 @@ import (
 	"github.com/stretchr/testify/require"
 )

-// TestVolumeAbandonSubject_BothForms pins the abandon route the same way the
-// fenced route is pinned: node-addressed, with an empty-node fallback for a
-// single-node daemon.
+// TestVolumeAbandonSubject_BothForms pins the abandon route: node-addressed,
+// with an empty-node fallback for a single-node daemon.
 func TestVolumeAbandonSubject_BothForms(t *testing.T) {
 	assert.Equal(t, "ebs.node-a.abandon", viperblocklegacyv1.VolumeAbandonSubject("node-a"))
 	assert.Equal(t, "ebs.abandon", viperblocklegacyv1.VolumeAbandonSubject(""))
@@ -1,29 +0,0 @@
-package viperblocklegacyv1
-
-// VolumeFencedSubject is where a fenced volume is announced so the daemon on
-// this node can stop the guest that was using it. Node-addressed: the guest is
-// local, and every other node's daemon has nothing to do about it.
-func VolumeFencedSubject(node string) string {
-	if node == "" {
-		return "ebs.fenced"
-	}
-	return "ebs." + node + ".fenced"
-}
-
-// VolumeFencedEvent announces that a node tore down a volume's export because
-// the lease moved. Winner is who holds it now, for the operator reading the
-// guest's stop reason.
-type VolumeFencedEvent struct {
-	Volume string `json:"volume"`
-	Node   string `json:"node"`
-	Winner string `json:"winner"`
-	Reason string `json:"reason"`
-}
-
-// VolumeFencedAck is the daemon's answer to a VolumeFencedEvent. Any answer
-// ends the sender's retries; Instance names the guest stopped, if one was, and
-// Error says why the daemon could not act on the event.
-type VolumeFencedAck struct {
-	Instance string `json:"instance,omitempty"`
-	Error    string `json:"error,omitempty"`
-}
@@ -1,54 +0,0 @@
-package viperblocklegacyv1_test
-
-import (
-	"encoding/json"
-	"testing"
-
-	viperblocklegacyv1 "github.com/mulgadc/spinifex/contracts/viperblockd/legacy/v1"
-	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
-)
-
-// TestVolumeFencedSubject_IsNodeAddressed pins the routing. A fenced guest is on
-// the node that lost the volume, so a subject without the node in it would reach
-// daemons with nothing to stop and, on a queue group, miss the one that has.
-func TestVolumeFencedSubject_IsNodeAddressed(t *testing.T) {
-	assert.Equal(t, "ebs.node-a.fenced", viperblocklegacyv1.VolumeFencedSubject("node-a"))
-	assert.Equal(t, "ebs.fenced", viperblocklegacyv1.VolumeFencedSubject(""),
-		"a single-node daemon has no node name, and still has to hear its own fences")
-}
-
-// TestVolumeFencedEvent_JSON pins the exact field spelling of the fenced
-// event, published by viperblockd and consumed by the daemon.
-func TestVolumeFencedEvent_JSON(t *testing.T) {
-	golden := `{"volume":"vol-1","node":"node-a","winner":"node-b","reason":"volume lease moved to node-b while this node had the volume open"}`
-	event := viperblocklegacyv1.VolumeFencedEvent{
-		Volume: "vol-1",
-		Node:   "node-a",
-		Winner: "node-b",
-		Reason: "volume lease moved to node-b while this node had the volume open",
-	}
-
-	marshaled, err := json.Marshal(event)
-	require.NoError(t, err)
-	assert.JSONEq(t, golden, string(marshaled))
-
-	var decoded viperblocklegacyv1.VolumeFencedEvent
-	require.NoError(t, json.Unmarshal([]byte(golden), &decoded))
-	assert.Equal(t, event, decoded)
-}
-
-// TestVolumeFencedAck_JSON pins the daemon's answer, which is what ends
-// viperblockd's retries of a fence notice.
-func TestVolumeFencedAck_JSON(t *testing.T) {
-	golden := `{"instance":"i-1","error":"bad event"}`
-	ack := viperblocklegacyv1.VolumeFencedAck{Instance: "i-1", Error: "bad event"}
-
-	marshaled, err := json.Marshal(ack)
-	require.NoError(t, err)
-	assert.JSONEq(t, golden, string(marshaled))
-
-	empty, err := json.Marshal(viperblocklegacyv1.VolumeFencedAck{})
-	require.NoError(t, err)
-	assert.JSONEq(t, `{}`, string(empty), "an ack for a volume with no local guest carries nothing")
-}
@@ -19,7 +19,7 @@ Every service is `PartOf=spinifex.target` and ordered by its own `After=`/`Wants
 | `spinifex-predastore.service` | `spx service predastore start` | S3-compatible object storage, serving that host's nodes and the S3 gate among them |
 | `spinifex-northstar.service` | `spx service northstar start` | Authoritative DNS for `*.spx3.net`, recursive via upstream forwarders, syncing zones from its S3 bucket |
 | `spinifex-viperblock.service` | `spx service viperblock start` | Block storage with the NBD plugin. `KillMode=process`: nbdkit serving a guest survives a restart of this unit |
-| `spinifex-daemon.service` | `spx service spinifex start` | VM orchestration, NATS subscriptions, health endpoint. `KillMode=process`: guest QEMU survives a restart and the new daemon reattaches. `SIGHUP` reloads config |
+| `spinifex-daemon.service` | `spx service spinifex start` | VM orchestration, NATS subscriptions, health endpoint. Two `ExecStartPre` steps run first: `spx admin hostnet check`, and `spx admin node-facts collect`, which runs as root and whose failure does not stop the daemon. `KillMode=process`: guest QEMU survives a restart and the new daemon reattaches. `SIGHUP` reloads config |
 | `spinifex-awsgw.service` | `spx service awsgw start` | AWS-compatible gateway: SigV4 auth, IAM policy enforcement, TLS. Also serves the Private Admin API |
 | `spinifex-vpcd.service` | `spx service vpcd start` | VPC daemon: translates `vpc.*` NATS events to OVN logical switches, ports and routers |
 | `spinifex-ui.service` | `spx service spinifex-ui start` | Web UI serving the React frontend |
@@ -95,7 +95,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t

 | Command | Flags | Description |
 |---------|-------|-------------|
-| `spx admin init` | `--nodes`, `--node`, `--bind`, `--advertise` (external IP off-host clients dial; auto-detected from the default route), `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--predastore-compaction-interval` (seconds; test clusters set a short one), `--services`, `--formation-timeout`, `--token-ttl`, `--force`, `--discard-jetstream` (with `--nodes` >= 2, remove this node's existing JetStream streams once every node has joined; without it a multi-node init over a store that holds streams is refused, and it never proceeds while NATS is running), `--skip-host-dns`, `--email` (operator address for update and security notices), `--no-telemetry`, `--gpu-passthrough`, `--ipsec` (default true; encrypt intra-AZ Geneve via OVN native IPsec cluster-wide — disable only for a trusted single-rack lab; on a host whose IPsec stack is libreswan it defaults to false and `--ipsec=true` is refused), `--host` (leader to join; falls back to multicast discovery), `--external-mode` (`pool` \| `nat` routed NAT for non-bridgeable uplinks, single-node; pair with `setup-ovn.sh --nat-uplink`; note: system instances (ECS/EKS/load-balancer agents) are unsupported in nat v1 — they require `pool` mode), `--external-iface` (WAN NIC for br-external, auto-detected), `--external-source` (`dhcp` \| `static`), `--external-pool` (`start-end`), `--external-prefix-len` (default 24), `--external-gateway` (auto-detected), `--external-dns` (comma-separated IPv4 addresses, at most 3; the DNS servers written to the external pool's `dns_servers` and handed to guests, verbatim; when omitted they are detected from the host resolver at init time, so set it to give every node of a cluster the same value), `--external-bind-bridge` (default `br-wan` when `--external-source=dhcp`), `--gateway-ip` (OVN gateway router's external IP for SNAT; defaults to the pool's range start, required for `nat` mode without DHCP) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` → points the host resolver at this node's local northstar so the Spinifex zones (ELB/EKS names) resolve from the node itself (opt out with `--skip-host-dns`). `--force` on an already-initialized node is idempotent for crypto: it preserves the existing keys (master.key, predastore/viperblock encryption keys), system + admin credentials, and CA, and refreshes only the config files and the CA-signed server certificate (so a changed bind IP / SANs is picked up without breaking already-joined nodes or CA-baked AMIs). A genuine clean slate requires removing the data dirs first. |
+| `spx admin init` | `--nodes`, `--node`, `--bind`, `--advertise` (external IP off-host clients dial; auto-detected from the default route), `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--predastore-compaction-interval` (seconds; test clusters set a short one), `--services`, `--formation-timeout`, `--token-ttl`, `--force`, `--discard-jetstream` (with `--nodes` >= 2, remove this node's existing JetStream streams once every node has joined; without it a multi-node init over a store that holds streams is refused, and it never proceeds while NATS is running), `--skip-host-dns`, `--email` (operator address for update and security notices), `--no-telemetry`, `--gpu-passthrough`, `--ipsec` (default true; encrypt intra-AZ Geneve via OVN native IPsec cluster-wide — disable only for a trusted single-rack lab; on a host whose IPsec stack is libreswan it defaults to false and `--ipsec=true` is refused), `--external-mode` (`pool` \| `nat` routed NAT for non-bridgeable uplinks, single-node; pair with `setup-ovn.sh --nat-uplink`; note: system instances (ECS/EKS/load-balancer agents) are unsupported in nat v1 — they require `pool` mode), `--external-iface` (WAN NIC for br-external, auto-detected), `--external-source` (`dhcp` \| `static`), `--external-pool` (`start-end`), `--external-prefix-len` (default 24), `--external-gateway` (auto-detected), `--external-dns` (comma-separated IPv4 addresses, at most 3; the DNS servers written to the external pool's `dns_servers` and handed to guests, verbatim; when omitted they are detected from the host resolver at init time, so set it to give every node of a cluster the same value), `--external-bind-bridge` (default `br-wan` when `--external-source=dhcp`), `--gateway-ip` (OVN gateway router's external IP for SNAT; defaults to the pool's range start, required for `nat` mode without DHCP) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` → points the host resolver at this node's local northstar so the Spinifex zones (ELB/EKS names) resolve from the node itself (opt out with `--skip-host-dns`). `--force` on an already-initialized node is idempotent for crypto: it preserves the existing keys (master.key, predastore/viperblock encryption keys), system + admin credentials, and CA, and refreshes only the config files and the CA-signed server certificate (so a changed bind IP / SANs is picked up without breaking already-joined nodes or CA-baked AMIs). A genuine clean slate requires removing the data dirs first. |
 | `spx admin join` | `--host` (required), `--node` (required), `--token` (required), `--bind`, `--advertise` (defaults to `--bind`, or the auto-detected WAN IP when `--bind` is 0.0.0.0), `--port`, `--region`, `--az`, `--cluster-bind`, `--cluster-routes`, `--data-dir`, `--services`, `--skip-host-dns`, `--email`, `--no-telemetry`, `--predastore-compaction-interval`, `--join-timeout` (how long to keep retrying while the formation server is unreachable, default 20m), `--force` (join even though this node is already initialized or holds JetStream streams, **discarding its own CA, master key and JetStream store**; the store is removed only after formation succeeds, and the join is refused while NATS is running even with `--force`) | Connects to leader node with join token (Authorization: Bearer header) → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations → points the host resolver at this node's local northstar so the Spinifex zones resolve from the node itself (opt out with `--skip-host-dns`) |

 ### Version
@@ -129,13 +129,15 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 |---------|-------|-------------|
 | `spx admin preflight` | `--quiet` (print only non-OK rows) | Reports whether this host has the privileged helper scripts and sudoers grants the installed build depends on. `spx version` knows the build but not the host: `setup.sh` and `setup-ovn.sh` install those assets, and a binary-swap deploy (`make deploy`, or a manual copy) skips them. It also reports any running service still executing a replaced binary — overwriting `/usr/local/bin/spx` while `spinifex.target` is up does not move services onto it, so each keeps executing the replaced inode until it restarts and a node can serve a mixture of two builds indefinitely. Checks compare content, not existence, so a present-but-stale asset is caught. On x86_64 it also checks that the QEMU binary the daemon will run lists the `microvm` machine type and that the microvm firmware `/usr/share/spinifex/microvm/qboot.rom` is installed, since load balancers boot on both. It runs the host network check `spinifex-daemon` and `spinifex-vpcd` run before they start, and fails while firewalld or ufw is active, since the Spinifex nftables policy is the only supported node firewall. It reports a Warning, without failing, when systemd-resolved is active but `/etc/resolv.conf` is not its stub, since lookups then bypass the route that sends Spinifex names to northstar. Exits non-zero if any asset is Missing, Stale, Ungranted, Unsupported or Failed. |
 | `spx admin node js-probe` | `--timeout` (default: 10s) | Round-trips a canary key through the cluster-state KV bucket and exits non-zero if the write or the read-back fails. JetStream can latch a write error per message block and keep serving reads — nats-server's filestore records the failure on the block and returns it on every later write, and nothing clears it short of a restart. The HTTP health endpoint reports the server-wide JetStream state, which that latch never touches, so only an actual write observes it. |
+| `spx admin node-facts collect` | — | Runs `dmidecode`, `ethtool`, `udevadm`, `networkctl lldp` and `systemd-detect-virt`, and writes their raw output to `/etc/spinifex/node-facts/`, readable by the spinifex group. `spinifex-daemon.service` runs it as a root `ExecStartPre` because the daemon itself is unprivileged; the daemon parses the files when the operator console asks. A probe that fails writes no new file, so a file from an earlier successful run can remain and the console reads it as available; a source with no file is shown as unavailable. |
 | `spx admin banner` | `--boot-check` | Writes the node information banner to `/etc/issue` (shown before login on the physical/serial console) and `/etc/motd` (shown after SSH login). With `--boot-check` it also detects a management IP that has changed since last boot and updates `/etc/spinifex/node.conf` accordingly. |

 ### Storage Inspection

 | Command | Flags | Description |
 |---------|-------|-------------|
-| `spx admin volumes orphans` | — | Lists volumes the storage provider holds that the control plane has no record of. Their blocks consume space but they have no API handle, so they can be neither described nor deleted through the EC2 API. **Reports only, never deletes**: the evidence that an orphan is still wanted is the record that went missing, so removal is an operator decision taken per volume. |
+| `spx admin volumes orphans` | — | Lists volumes the storage provider holds that the control plane has no record of. Their blocks consume space but they have no API handle, so they can be neither described nor deleted through the EC2 API. **Reports only, never deletes**: the evidence that an orphan is still wanted is the record that went missing, so removal is an operator decision taken per volume. It also lists UEFI variable stores nothing owns: a `uefi/i-*/` prefix with no instance record and no store record, a `uefi/ami-*/` prefix with no AMI document, and a store record that is not `deleting` and has no instance record. These are reported and never deleted, because an absent instance record is not proof of a terminate. |
+| `spx admin volumes obligations` | — | Lists the open unpublish obligations, oldest first: a volume export that a force detach, or a terminate whose unmount did not complete, left on the node that held the attachment. Each row names the volume, the node, the instance and the age, plus attempts and the last error once the node has tried. The node settles each one once it answers; an obligation stays open while its node is down and closes without the node only once the node has been removed from the cluster. **Reports only.** |
 | `spx admin volumes unsealed` | — | Lists volumes a node has open, or had open, without a confirmed seal to the backend — the node named holds writes predastore may not have. Starting such a volume elsewhere is refused while that node is alive (its daemon heartbeat is under 60 seconds old, or cannot be read) and is otherwise allowed, in which case it opens from the last checkpoint that did reach the backend. This is the list of volumes where that would cost something. |
 | `spx admin storage status` | `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA) | Sends the signed `GetStorageStatus` action to the AWS gateway with credentials from the standard AWS chain (e.g. `AWS_PROFILE=spinifex`) and prints what the console's Service Metrics page shows: predastore encoding, meta and blob node counts, cluster health, a meta node table (health, raft state, leader, term, commit and applied index) and the blob nodes. Cluster health is `Degraded` when any meta node is unreachable or the nodes reporting raft state do not agree on one leader. A meta node read only through its host's `/readyz` shows healthy with blank raft fields. |

@@ -152,12 +154,13 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 | `spx admin account create` | `--name` (required) | Connects to NATS → refuses if an account already holds the name (case- and whitespace-insensitive) → reserves the name in `spinifex-account-names` KV → CAS loop on `spinifex-account-counter:next_id` for sequential 12-digit ID → creates Account record in `spinifex-accounts` KV → creates default VPC → creates `admin` user, AdministratorAccess policy (Action:*, Resource:*) and one access key → prints credentials and writes a `spinifex-<name>` AWS CLI profile |
 | `spx admin account create --remote` | `--name` (required), `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA), `--client-token` (default: generated), `--source` (default: `spx-cli`) | Same account, created through `POST /admin/CreateAccount` over SigV4 instead of NATS — the path the public signup form uses, so a shell can exercise it. Credentials come from the standard AWS chain (env vars or `AWS_PROFILE`), never from the cluster master key; the caller must be an IAM user in the super-admin account holding `spinifex:CreateAccount`. Reuse the printed `--client-token` to retry: a fresh token would create a second account. |
 | `spx admin account list` | — | Connects to NATS → IAMService.ListAccounts() → prints table with Account ID, Name, Status, Created |
-| `spx admin account list --remote` | `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA) | Same listing read through `POST /admin/ListAccounts` over SigV4 instead of NATS, so an off-cluster operator or the load-test harness sees its tenants without SSH access to a node. Credentials come from the standard AWS chain; the caller must be an IAM user in the super-admin account holding `spinifex:ListAccounts`. |
+| `spx admin account list --remote` | `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA) | Same listing read through `POST /admin/ListAccounts` over SigV4 instead of NATS, so an off-cluster operator or the load-test harness sees its tenants without SSH access to a node. Credentials come from the standard AWS credential chain: on a cluster node that means `AWS_PROFILE=spinifex` (there is no default profile, so without it the command fails to find credentials), or equivalent credentials; the caller must be an IAM user in the super-admin account holding `spinifex:ListAccounts`. |
 | `spx admin account quota get <account-id>` | `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA) | Reads the limits in force for one account through `POST /admin/GetAccountQuota` over SigV4, and reports for each dimension whether it came from the `[quota]` block in `awsgw.toml` (`config`) or from an override stored against the account (`override`). Credentials come from the standard AWS chain; the caller must be an IAM user in the super-admin account holding `spinifex:GetAccountQuota`. |
 | `spx admin account quota set <account-id>` | `--vcpus`, `--vpcs`, `--subnets`, `--eips`, `--volumes`, `--volumes-gib`, `--rds-instances`, `--load-balancers`, `--clear`, plus `--endpoint`, `--region`, `--ca-bundle` | Overrides individual limits for one account through `POST /admin/PutAccountQuota`, leaving every dimension not named inheriting the configured baseline. `-1` uncaps a dimension; `0` is a real limit that denies every request on it. `--clear` removes the whole override record and cannot be combined with a limit flag. The account must already exist. Prints the resulting limits, so a caller never has to guess what it applied. The caller must hold `spinifex:PutAccountQuota`. |
-| `spx admin account delete <account-id>` | `--dry-run`, `--force`, `--yes` (requires `--name`), `--name` (account name, must match the stored record) | Prints the inventory it is about to remove → confirms the account name (prompted unless `--name` is given) → marks the account `TERMINATING`, which blocks its credentials cluster-wide → deletes its resources in dependency order (compute, attachments, storage, network, platform, identity, account), waiting for each stage to actually drain before starting the next → releases the name reservation and quota counter → deletes the account record. Anything that will not delete is reported as **stuck** with the reason, and the account stays `TERMINATING` so the residue keeps an owner. `000000000000` and `000000000001` are refused. `--dry-run` lists and changes nothing. The compute stage reaps ECS clusters, EKS nodegroups and clusters, and RDS instances before it reaps plain instances, because their backing VMs belong to the system account and a tenant-scoped instance listing cannot see them. The storage stage reaps the account's S3 buckets: uploads in flight are aborted, then the objects are removed a page at a time, then the bucket. Only buckets the account owns are ever listed, so the shared platform buckets are out of reach rather than filtered out. Platform-managed serving endpoints are reaped ahead of instances, because only their own delete releases the ENI and attached data volume alongside the VM, and spot instance requests are cancelled first so a live request cannot launch a replacement for something already terminated. The network stage reaps load balancers, then target groups, then NAT gateways and egress-only internet gateways before the subnets, security groups and addresses they hold. The platform stage reaps ACM certificates. |
+| `spx admin account delete <account-id>` | `--dry-run`, `--force`, `--non-finalizing` (hidden), `--yes` (requires `--name`), `--name` (account name, must match the stored record), `--endpoint`, `--region`, `--ca-bundle`, `--client-token` | Always runs through `POST /admin/DeleteAccount` over SigV4 and follows `DescribeAccountDeletion` to completion; the CLI never tears an account down in its own process, so a CLI delete and an API delete cannot race. Credentials come from the standard AWS chain; the caller must be an IAM user in the super-admin account holding `spinifex:DeleteAccount` and `spinifex:DescribeAccountDeletion`. Prints the inventory it is about to remove → confirms the account name (prompted unless `--name` is given) → marks the account `TERMINATING`, which blocks its credentials cluster-wide → deletes its resources in dependency order (compute, attachments, storage, network, platform, identity, account), waiting for each stage to actually drain before starting the next → seals the account's quota override record so no later write can recreate it, which needs every gateway node the cluster configures to answer a read-only probe first (otherwise nothing is sealed and the account stays `TERMINATING`) → releases the name reservation and quota counter → deletes the account record. Anything that will not delete is reported as **stuck** with the reason, and the account stays `TERMINATING` so the residue keeps an owner. `000000000000` and `000000000001` are refused. `--dry-run` lists and changes nothing. The compute stage reaps ECS clusters, EKS nodegroups and clusters, and RDS instances before it reaps plain instances, because their backing VMs belong to the system account and a tenant-scoped instance listing cannot see them. The storage stage reaps the account's S3 buckets: uploads in flight are aborted, then the objects are removed a page at a time, then the bucket. Only buckets the account owns are ever listed, so the shared platform buckets are out of reach rather than filtered out. Platform-managed serving endpoints are reaped ahead of instances, because only their own delete releases the ENI and attached data volume alongside the VM, and spot instance requests are cancelled first so a live request cannot launch a replacement for something already terminated. The network stage reaps load balancers, then target groups, then NAT gateways and egress-only internet gateways before the subnets, security groups and addresses they hold. The platform stage reaps ACM certificates. |
 | `spx admin account delete --force` | as above | Escalates only for resources the ordinary API refuses: it clears a volume attachment in the control plane without the guest's cooperation (`ec2.ForceDetachVolume`), hard-destroys after a graceful attempt times out, and treats an already-missing resource as deleted. It never reorders the stages and never widens scope beyond the account. Use it once the guest is being destroyed — this is the escape from a volume stuck attached to an instance that will not terminate, where neither can otherwise be deleted. |
-| `spx admin account delete --remote` | as above, plus `--endpoint`, `--region`, `--ca-bundle`, `--client-token` | Same teardown driven through `POST /admin/DeleteAccount` over SigV4 instead of NATS, then follows `DescribeAccountDeletion` to completion, printing each stage as it lands. Credentials come from the standard AWS chain; the caller must be an IAM user in the super-admin account holding `spinifex:DeleteAccount` and `spinifex:DescribeAccountDeletion`. |
+| `spx admin account delete --non-finalizing` | as above | An exercise run, not a finalising delete, but the cleanup is real. Marks the account `TERMINATING`, deletes the resources of each EC2 platform kind the evaluation found quiesced (for example key pairs, empty placement groups and orphaned tags), re-reads the owner inventory against that cleanup, and ends `EXERCISED` with the account still `TERMINATING`. No other stage runs and the account record, name reservation and quota are left in place. It refuses `--force` and `--dry-run`. |
+| `spx admin account delete --remote` | as above | Accepted for existing scripts and ignored: every delete already goes through the admin API. |
 | `spx admin principal create <name>` | `--grant` (repeatable or comma-separated; default: every admin method — `CreateAccount`, `DeleteAccount`, `DescribeAccountDeletion`, `GetAccountQuota`, `ListAccounts`, `PutAccountQuota`) | Creates an IAM user in the super-admin account with an inline `spx-admin-methods` policy allowing the requested methods, revokes any access key left from a previous run, and mints one new key. Each method is named rather than granted as `spinifex:*`, so a later addition to the admin surface is not authorised by an existing key. The principal's grants are exactly that one policy: re-running with fewer `--grant` values narrows it, and any other inline policy on the user is removed, so a rename cannot leave a wider grant behind. The secret is printed once; store it in a secret manager or an AWS profile, never in a config file. `admin` is refused as a name — it is the account's own administrator and rotating its key would revoke the bootstrap credential. |
 | `spx admin principal list` | — | Lists every principal in the super-admin account with its access-key count and what it may call. An attached policy is reported by name (`AdministratorAccess (attached)`) rather than expanded, which is how an unscoped principal is spotted. |
 | `spx admin principal revoke <name>` | — | Deletes every access key the named principal holds, leaving the user and its policy in place. Immediate and cluster-wide; re-issue with `principal create`. |
@@ -194,18 +197,32 @@ Operational commands for inspecting cluster state. These fan out NATS requests t

 ### SQS System Instances

-These commands start the first SQS system instances on one node. They do not serve the SQS API, and a bound VM is not a ready service.
+These commands enable SQS on a single-node cluster and start its one system VM.
+A bound VM is not a ready service: `spx admin sqs status` reports readiness.

-Both send a signed `spinifex` admin action (`EnsureSQSWorker`, `EnsureSQSStore`) to the AWS gateway using the standard AWS credential chain, for example `AWS_PROFILE=spinifex`. The caller must be in the super-admin account (`000000000001`) and hold `spinifex:<Action>`. The VMs, volumes, private VPC and ENIs belong to the system account (`000000000000`). `--endpoint`, `--region` and `--ca-bundle` default to this node's gateway, region and `config/ca.pem`; off a node, `--endpoint` and `--region` are required. Each call waits up to 11 minutes.
+The SQS VM runs two processes on one encrypted data volume: the Region's dedicated SQS NATS server, which listens only on the guest's loopback address and is separate from platform NATS, and the worker, which serves SQS owner calls from that local store. The worker starts serving only after the local store answers and the receipt-signing key on the volume matches the fingerprint the platform recorded.
+
+SQS supports only a fresh single-node install. A worker record without a supported launch contract, or any record of the separate SQS store VM from an earlier dev build, cannot be upgraded: `enable` refuses it, and any existing volume is kept, not inspected, emptied or replaced. An existing separate store VM is refused on relaunch with its volume kept, and an existing worker VM is not restarted into a supported mode; upgrading such an install is unsupported.
+
+**Do not put tenant production data in SQS.** SQS-managed per-message encryption is not implemented: `GetQueueAttributes` reports `SqsManagedSseEnabled=false`, and a request to enable SSE-SQS or SSE-KMS is rejected.
+The SQS data volume must be encrypted, which is a separate layer under every queue and not SSE.
+There is no `DeleteQueue` or `PurgeQueue`, so a queue and its unexpired messages cannot be removed once created.
+Retention is checked when a message is received, against the queue's retention at that moment, and nothing deletes an expired message: the store drops a body only at its age backstop of 14 days and 1 hour.
+So a message past a short retention can be delivered again if `MessageRetentionPeriod` is later raised while its body is still stored.
+Account termination does not yet remove SQS queues or messages, and there is no disable: the VM's volume, receipt key and data are kept.
+
+Each sends a signed `spinifex` admin action (`EnableSQS`, `DescribeSQS`, `EnsureSQSWorker`) to the AWS gateway using the standard AWS credential chain, for example `AWS_PROFILE=spinifex`. The caller must be in the super-admin account (`000000000001`) and hold `spinifex:<Action>`. The VM, volume, private VPC and ENI belong to the system account (`000000000000`). `--endpoint`, `--region` and `--ca-bundle` default to this node's gateway, region and `config/ca.pem`; off a node, `--endpoint` and `--region` are required. `ensure-worker` and `enable` wait up to 11 minutes, and `status` up to 1 minute.

 | Command | Flags | Description |
 |---------|-------|-------------|
-| `spx admin sqs ensure-worker` | `--node` (required), `--instance-type` (required), `--endpoint`, `--region`, `--ca-bundle` | Launches the Region's one SQS worker VM on `--node`, or reports the VM already bound there: node, region, instance, instance type and generation. The worker is pinned to its first node; another node is refused with `InvalidParameterValue`, and there is no takeover. It gets a private system VPC with one /24 subnet and no internet gateway, NAT or Elastic IP, and an encrypted 1 GiB system-account data volume on which the guest generates its own store key. The platform records only the public key. A different `--instance-type` is accepted only while no volume, network interface or launch step has been attempted; once any has, it is `IncorrectState`, whether or not the step completed. |
-| `spx admin sqs ensure-store` | `--node` (required, the worker's node), `--instance-type` (required, more than 512 MiB memory), `--volume-size-gib` (required, 1–16384), `--endpoint`, `--region`, `--ca-bundle` | Launches the Region's SQS store VM on the worker's node with an encrypted system-account data volume, or reports the VM already bound there: node, region, instance, instance type and address. Needs the worker bound with its public key recorded, else `IncorrectState`. The store accepts only that key. A different size, type or key than the first call is `IncorrectState`. |
+| `spx admin sqs enable` | `--node` (required), `--instance-type` (required, more than 512 MiB memory), `--volume-size-gib` (required, 1–16384), `--endpoint`, `--region`, `--ca-bundle` | Records SQS as enabled for the Region, pinned to `--node`, then launches or reports the SQS VM there. The desired state is recorded first, so rerunning the command resumes an interrupted enable. More than one node in the cluster is refused before anything is recorded. A worker record with a missing or unsupported launch contract, or a record of the separate store VM, is also refused before anything is recorded or launched; any existing volume is kept, and upgrading it is not supported. |
+| `spx admin sqs status` | `--endpoint`, `--region`, `--ca-bundle` | Shows the requested state, node, generation, VM state, receipt key state, whether a worker is polling this gateway, whether this gateway serves the SQS API, and `Ready`, which is true only when all of them hold. `Unsupported` names a record whose missing or unsupported launch contract blocks enable. |
+| `spx admin sqs ensure-worker` | `--node` (required), `--instance-type` (required, more than 512 MiB memory), `--volume-size-gib` (required, 1–16384), `--endpoint`, `--region`, `--ca-bundle` | Launches the Region's one SQS VM on `--node`, or reports the VM already bound there: node, region, instance, instance type and generation. The VM is pinned to its first node; another node is refused with `InvalidParameterValue`, and there is no takeover. It gets a private system VPC with one /24 subnet and no internet gateway, NAT or Elastic IP, and an encrypted system-account data volume of `--volume-size-gib` that holds the queue data and the receipt-signing key the guest creates on first boot. The platform records only the key's fingerprint. A different `--instance-type` is accepted only while no volume, network interface or launch step has been attempted; once any has, it is `IncorrectState`, whether or not the step completed. A different `--volume-size-gib` than the recorded one is `IncorrectState`. |

 `ServiceUnavailable` never proves that no SQS resources exist, because an earlier attempt may already have created some. Its message says what it does establish. When no node serves the subject, this request was not delivered. When a volume create was not sent, that particular create did not happen. When the node cannot boot the guest yet or its launcher lacks a capability, this request stopped before allocating or recording anything. When the gateway times out waiting for the node's reply, nothing is known: the launch may still be running, so never assume it created nothing. In every case, call the command again with the same arguments. A retry first checks the durable record of the attempt and continues only when that is safe; it can proceed to the first launch after a create that was proven unsent, and otherwise reports the recorded outcome as bound or `IncorrectState`, or returns `ServiceUnavailable` again. `IncorrectState` from a volume, network interface or launch step means that step was recorded as sent but its outcome was never observed. It is never resent, the message names the volume when its ID is known, and no tool reconciles it yet: recovery is manual.

-`bound` says only that the VM was launched for the current record. The worker decides store readiness itself, by checking that the store is bound to its own public key and that a NATS connection with that key succeeds; nothing reports that readiness to the operator. The worker-to-store connection does not authenticate the store server. Real VPC routing, IMDS and gateway TLS from the guest, and worker-to-store traffic on a real interface have not been tested.
+`bound` says only that the VM was launched for the current record. The worker is ready only when the guest's local store answers, and it serves only while its receipt key matches the recorded fingerprint; a missing or different key once one is recorded refuses serving and is never replaced.
+Real VPC routing, IMDS and gateway TLS from the guest have not been tested on a live host.

 ## Private Admin API

@@ -65,9 +65,10 @@ The EC2 registration (`spinifex/domains/ec2/awsapi`, `NewRegistration`) parses t
 The gateway communicates with daemons via NATS request/response. Most calls go through `natsmsg.NATSRequest`, which marshals the input, attaches the account ID as a NATS header, and unmarshals the typed response:

 ```go
-func (s *NATSInstanceService) RunInstances(ctx context.Context, input *ec2.RunInstancesInput, accountID string) (*ec2.Reservation, error) {
-    topic := fmt.Sprintf("ec2.RunInstances.%s", aws.StringValue(input.InstanceType))
-    return natsmsg.NATSRequest[ec2.Reservation](ctx, s.natsConn, topic, input, 5*time.Minute, accountID)
+func runIntoReservation(ctx context.Context, input *ec2.RunInstancesInput, natsConn *nats.Conn, accountID, crID string) (*ec2.Reservation, error) {
+    subject := "ec2.RunInstances.cr." + crID
+    reservation, err := natsmsg.NATSRequest[ec2.Reservation](ctx, natsConn, subject, input, 5*time.Minute, accountID)
+    // ...
 }
 ```

@@ -111,14 +112,16 @@ When a daemon handles `RunInstances`:

 1. **Resource Check**: Validates CPU/memory availability via `ResourceManager`
 2. **Volume Generation**: Creates boot and EFI volumes via Viperblock
-3. **Volume Mount**: Sends a mount request to the colocated Viperblock worker on `ebs.{node}.mount`, receives NBD URI
+   A UEFI instance whose record carries a variable store pointer has no EFI volume: its `vars.fd` is fetched from the object under `uefi/` in the configured Predastore bucket that the pointer names, recorded in the `spinifex-uefi-stores` JetStream bucket (or reused when `seeded.json` on this node already names the same object) after a firmware architecture and VARS size check, and a record with both or neither is refused.
+   RunInstances still creates the EFI volume, nothing writes `vars.fd` back to the object store yet, and a terminated instance's store is removed by a cluster-wide reaper.
+3. **Volume Mount**: Publishes each volume through the EBS provider contract (`PublishVolume` on `ebs.provider.v1.{node}.mount`), receives its export
 4. **QEMU Launch**: Builds and executes QEMU command with NBD-backed disks
 5. **QMP Monitoring**: Establishes QEMU Machine Protocol connection for VM management
 6. **Command subscription**: Subscribes to `ec2.cmd.{instanceID}` for subsequent commands on this VM

 ```go
-// Volume mounting via NATS (runtime/daemon/vm_adapters.go)
-reply, err := ebsRequestWithTrace(ctx, a.nc, instance.AccountID, viperblocklegacyv1.MountSubject(a.node), ebsMountRequest, 30*time.Second)
+// Volume mounting through the EBS provider contract (runtime/daemon/vm_adapters.go)
+published, err := a.provider.PublishVolume(ebsprovider.WithAccountID(ctx, accountID), ebsprovider.PublishVolumeRequest{Versioned: ebsprovider.NewLatestVersioned(), VolumeID: req.Name, NodeID: a.node})

 // QEMU launch with NBD-backed drives in args (runtime/compute/vm/vm.go)
 cmd := exec.Command(qemuArchitecture, args...) // qemu-system-x86_64 or qemu-system-aarch64
@@ -224,6 +227,9 @@ It may keep writing for 40 seconds after the last write the store acknowledged,
 A slow acknowledgement therefore shortens the remaining validity rather than extending it, and an acquisition acknowledged after its validity has already run out is refused as retryable.

 For a holder that cannot renew, a timer is armed for the validity deadline, leaving a nominal 5 seconds before the entry can expire; when it fires, the export's nbdkit is fenced with SIGKILL (`volume_fence.go`).
+The kill comes first: the read that names the winner (bounded at 2 seconds) and the notice to the daemon follow it, so a hung lease store cannot hold the writer alive.
+The notice is a request that viperblockd repeats every 2 seconds until the local daemon replies with `VolumeFencedAck`, because the fence often runs while the node's NATS server is restarting.
+On the notice the daemon moves the guest using the volume to `error` with state reason `Server.VolumeFenced` and SIGKILLs its QEMU immediately rather than shutting it down gracefully, because there is no disk left to flush to; it only stops the guest process and removes its taps, and never terminates the instance or deletes a volume.
 The fence runs on its own goroutine, so Go scheduling, that goroutine and process termination all delay the kill, and enough delay can use up the whole margin; a kill held in uninterruptible sleep, which the fence waits up to 30 seconds for, is the extreme case.
 Once a lease is lost or has lapsed it stays lost: a renewal that succeeds afterwards is ignored, and another open on the same node cannot share it while the fence runs.

@@ -278,8 +284,8 @@ storage-provider contract.
 ### SQS message store (internal)

 `domains/sqs/message` defines the store-neutral interface the SQS domain reaches its message store through (ADR-0011:S6), with a JetStream adapter in `domains/sqs/message/jetstream` and a shared conformance suite in `domains/sqs/message/conformance`.
-It is an internal primitive: nothing outside the SQS domain uses it, no daemon or service is wired to it, and it does not itself deliver an SQS API.
-The dedicated JetStream server ADR-0011 describes is not deployed by this code, and Spinifex ships no host service, unit, user or firewall rule for it; the guest that could run it is described under *SQS store system instance (not wired)*.
+It is used only by the SQS worker inside the SQS system instance and does not itself deliver an SQS API.
+Spinifex ships no host service, unit, user or firewall rule for the dedicated JetStream server; the system instance that runs it is described under *SQS system instance (operator-launched)*.

 The interface covers durable and idempotent append, body and metadata reads, conditional state writes and deletes, per-namespace counts, namespace enumeration, an ordered change feed, a bounded paginated scan, namespace deletion and a sweep of tombstoned bodies.
 A namespace carries an immutable message-shard and state-shard allocation chosen when the queue is created, so a later change in shard counts never moves an existing queue.
@@ -298,124 +304,36 @@ If that consumer delivers nothing for about two seconds while the stream holds a

 Coverage is unit and conformance tests on in-process R1 and R3 servers; there is no end-to-end or multi-host validation of this code, and earlier multi-host evidence predates the final feed.

-### SQS store system instance (not wired)
+### SQS system instance (operator-launched)

-Spinifex can build, install, launch, restart and give storage to a dedicated SQS store microVM, but nothing calls any of it.
-No SQS API, domain service, admin command or daemon start-up path creates a store, so none of this is in production use.
+One system-owned microVM per Region runs both the SQS store and the SQS worker as separate processes, on one encrypted data volume pinned to one node (ADR-0011 Amendment 5, ADR-0012 Amendment 6).
+Only an operator activates it with the super-admin `EnsureSQSWorker` or `EnableSQS` actions; nothing starts it at daemon startup, and platform NATS is neither used nor changed for SQS data.
+This first profile is fresh single-node only: a host cluster of more than one node is refused before anything is recorded or allocated, and there is no disable.

-The guest image is built by `make build-sqs-store-microvm-image`, separate from the load balancer image, and bundles the `nats-server` version pinned in `go.mod` plus the ext4 tools.
-Release tarballs carry it as `microvm-sqs-store/`, and `setup.sh` installs it to `/usr/share/spinifex/microvm/sqs-store/`.
-A release without a complete image removes any previously installed one, so a launch is then refused rather than booting a stale image.
+`make build-sqs-worker-microvm-image` builds the one `sqs-worker` image: the static `sqs-worker` binary, the pinned `nats-server`, the ext4 tools and a network `/init`; releases ship it as `microvm-sqs-worker`, which `setup.sh` installs to `/usr/share/spinifex/microvm/sqs-worker`.
+At boot the guest reads three firmware config values under `opt/spinifex/sqs-worker/` (the data disk's serial, its Region and the gateway URL on this node's mgmt bridge) and refuses before touching any disk if one is missing or malformed.
+It selects exactly one disk with that serial, runs `e2fsck -p`, mounts it at `/data` and confirms from `/proc/self/mountinfo` that the whole disk is what is mounted; the guest never formats.

-At boot the guest reads three firmware config values: the data disk's serial, an IPv4 address on one of its interfaces, and one NATS user NKey public key.
-It mounts exactly one matching disk at `/data` after `e2fsck -p`, refuses a missing, duplicate, unformatted or unrepairable disk, and never formats.
-JetStream keeps its data on `/data` with `sync_interval: always` (ADR-0011:S3), `max_payload: 2MiB` admits a full 1 MiB SQS message with its headers (ADR-0011:S4), and the server listens only on that address and accepts only that key.
-On shutdown the store is stopped before `/data` is unmounted; a store that survives SIGKILL leaves every mount in place.
+The store is a dedicated `nats-server` listening only on `127.0.0.1`, with its JetStream directory at `/data/jetstream` and `GOMEMLIMIT=512MiB` (ADR-0011:S11), so nothing outside the VM can reach it and it needs no network identity of its own.
+The store starts only after the receipt key is settled, and the worker connects only after the store answers JetStream account info, so an unavailable store never appears ready.

-A launch uses the versioned subject `system.LaunchSQSStore.v1`, which a daemon without this code does not serve, so it fails with no responder instead of booting another guest.
-The legacy `system.LaunchInstance` path refuses the store kind on both send and receive.
-Every input check runs before capacity is reserved, and the data volume must be an available system-account volume.
-It is mounted through the normal EBS path and detached, never deleted, on terminate, which is the protection standard EBS gives today and nothing stronger.
-CPU mitigations stay enabled for the store, unlike the load balancer.
-A successful launch means the VM is running, not that NATS is ready.
+The receipt-signing key lives on the data disk under `/data/sqs-worker-key`, written to a temporary file, synced, renamed and its directory synced before its fingerprint is recorded.
+`domains/sqs/workerident` holds the platform's receipt-key state on the worker's identity record, beside its generation, claim, pinned node and volume: `pending` is set only when the record first creates its volume, and `recorded` holds the fingerprint.
+Under `pending` the guest may create a key, or reuse a valid key it already wrote before a crash, only while no `/data/jetstream` directory exists; any store data under `pending` refuses, because it proves the volume is not new.
+Under `recorded` a missing, unreadable or different key refuses the boot and is never regenerated; a record with no receipt-key state is refused as not proven new.
+`RecordReceiptKey` writes the fingerprint by compare-and-swap that re-checks the current instance, generation, claim and volume, and a lost reply is answered by reading the record again, never by a new key.
+A replacement worker reuses the pinned volume and keeps its predecessor's receipt-key state.

-The instance record keeps the store's non-secret guest settings so a restart on the node that launched it rewrites the firmware config files.
-A relaunch on any other node is refused, and cross-node recovery never claims a store.
-Both checks exist only in daemons with this code: an older daemon can still claim and relaunch a store elsewhere, and an explicit start served by another node claims the instance before its relaunch is refused.
+The worker reaches the gateway over TLS 1.3 trusting only the cluster CA, signing with instance-role credentials from the metadata service only.
+The internal `spinifex-sqs` actions are `ClaimReceiptKey`, `RecordReceiptKey`, `PollWork` and `CompleteWork`; the gateway admits only an assumed-role session of `sqsWorkerRole` in the system account and resolves the instance through `workerident.Authorize`, a point-in-time check against the current bound record.
+The worker pulls tenant work from its own node's gateway relay and posts results back, so no SQS payload crosses platform NATS; the relay holds a bounded, in-memory set of waiting requests and answers `ServiceUnavailable` when it is full or no worker is polling.

-`domains/sqs/storevolume` prepares a fresh data volume, because the guest never formats.
-It checks for `mkfs.ext4` and `qemu-img` before allocating, creates the volume in `creating`, writes an empty ext4 image to it over NBD on the local node, unpublishes it, and only then makes it `available`.
-While a volume is `creating`, attach, delete, modify, force-detach and snapshot are refused.
-A volume is deleted only after a confirmed unpublish; any publish error, an unconfirmed unpublish or a failed release keeps it and reports its ID, with its state reported as unknown after a failed release.
+`EnsureSQSWorker` takes a `NodeId`, an `InstanceType` and a `VolumeSizeGiB`; the instance type's memory must exceed the store's 512 MiB limit, and a later request for a different type or size is refused as `IncorrectState`.
+`EnableSQS` records the Region's desired state, pinned to one node, and then ensures the worker; `status` reports it ready only when the worker is bound, polling and the tenant API is registered.
+A store launch record or VM from the earlier separate-store design, or a worker record without a supported launch contract, is refused with its volume kept: enable stops before recording anything, and a relaunch of an old store VM is marked failed rather than started.

-Evidence is local only: unit tests, real QEMU boots against disposable scratch disks, and one run of the preparer against the `qemunbdd` provider whose volume kept acknowledged writes across a reboot.
-Viperblock, volume encryption, the EC2 volume record path, multiple nodes and host reachability of the guest address are not tested.
-Credential issuance and rotation, store placement and NATS readiness checks do not exist, and there is no failover: a lost node leaves its store unavailable.
-
-### SQS worker guest (operator-launched)
-
-The shared SQS worker guest holds the SQS store's client credential, so that no platform component does (ADR-0011:S2).
-Only an operator activates it, with the super-admin `EnsureSQSWorker` action described below; nothing starts it at daemon startup, and no worker service runs after boot.
-
-`make build-sqs-worker-microvm-image` builds it with the store image's script and network `/init` under `SQS_MICROVM_PROFILE=worker`; it carries the static `sqs-worker` binary and the ext4 tools, no `nats-server`, and an empty `/data` mountpoint.
-Releases build it beside the store image and ship it as `microvm-sqs-worker`, which `setup.sh` installs to `/usr/share/spinifex/microvm/sqs-worker` with the store image's rules: a release without it removes an earlier one, and an incomplete one is refused.
-At boot `sqs-worker boot` reads three firmware config values under `opt/spinifex/sqs-worker/`: the worker disk's serial, its Region and the gateway URL; none is a first-store decision, because a static value survives a guest reset.
-A missing, empty or malformed value refuses before any disk is touched.
-
-It selects exactly one disk with that serial, runs `e2fsck -p`, mounts it at `/data` and then confirms from `/proc/self/mountinfo` that this disk, as a whole ext4 filesystem, is what is mounted there; an empty mountpoint, a foreign mount or a failed check or mount refuses, and the guest never formats.
-Only then does it ask the gateway for the current decision with `ClaimFirstStore`, signed by `internal/sqsworkergw` with credentials from the instance metadata service only (`gwsign.NewInstanceRole`, no environment, profile or static-key fallback), over TLS 1.3 that trusts only the cluster CA from `opt/spinifex/ca-cert` and never follows a redirect.
-Under that decision `workerkey.Acquire` creates the key on `/data/sqs-worker-key`, owner-only and synced, records only its public key and then uses it, or loads the recorded key; a missing or different key once one is recorded refuses and is never replaced.
-The VM can start before its launch binds the instance, so its first calls are legitimately `AccessDenied`; that denial creates nothing and is asked again, as are a lost reply and a `ServiceUnavailable` or `Throttling` refusal, all within one bound (60 attempts 5 seconds apart), after which a worker that was never bound, or was replaced, refuses the boot.
-A signature or other refusal, or an answer that is neither "create" nor one valid public key, refuses the boot at once.
-
-`domains/sqs/firststore` is the SQS domain's first-store decision: one record per Region in the `spinifex-sqs-first-store` KV bucket moves from `claimed` to `key-recorded` to `launch-intent` under compare-and-swap, bound to the worker's instance ID, claim ID and generation.
-A claim authorizes key creation only while no public key is recorded; once one is, every claim returns only that key, and a store launch intent needs it recorded first.
-An older generation, a different identity at the same generation or an unreadable record authorizes nothing, and the record holds no private key material.
-Reads are stream message gets answered by the stream leader, never direct gets a lagging replica can answer, and every authorizing answer is also confirmed by a compare-and-swap at the revision it read, so a stale or unavailable read authorizes nothing.
-The authenticated caller that will carry this decision must derive the worker's instance ID and generation from authoritative instance state; a generation supplied in a request cannot itself authorize a takeover.
-
-`domains/sqs/workerident` is that authoritative instance state: one record per Region in the `spinifex-sqs-worker-identity` KV bucket holds a monotonic generation, claim ID, pinned node and encrypted data volume, and moves from `launching` to `bound` to `stopped`, or from `launching` to `failed`, under compare-and-swap.
-A launch intent is refused while the previous attempt is `launching` or `bound`, including one whose outcome is unknown, so only one caller ever holds an intent to start a VM on the pinned volume.
-A replacement takes the next generation only after `FailLaunch` or `RecordStopped` records that nothing holds the volume, and each request names the generation it supersedes, so a replayed request cannot take a generation it never observed; there is no automatic takeover.
-The launch path then raises the first-store fence to the successor's generation with `firststore.Fence` before `BeginLaunch`, which refuses a replacement until a leader read shows that fence; the fence only rises, and `firststore` refuses every call from a generation below it.
-Only the current claim ID is recorded, so claim IDs are minted fresh per attempt and a VM launch must be idempotent on generation and claim together.
-Binding writes the instance's index entry first and then binds the record to that instance.
-The index is only a lookup: `Authorize` accepts an instance only while the Region's record is `bound` to exactly that instance, generation and claim, confirmed by a compare-and-swap at the leader-read revision, so an old, orphaned, rewritten or unbound entry authorizes nothing.
-A same-instance reboot keeps its instance ID and identity; a replacement takes a new generation, and the first cut refuses another node or data volume and an unencrypted volume, although the encryption flag is the caller's claim and the launch wiring must verify it from the volume's metadata.
-Both buckets are opened through `OpenDurableKV`, which refuses an existing bucket whose effective stream configuration could expire, evict, lose, lag or refuse records: non-limits retention, an age, per-message or delete-marker TTL, memory storage, asynchronous flush, a size, count or message-size cap, a discard policy other than discard-new, unacknowledged writes, a sealed stream, a mirror or sources, or fewer replicas than the cluster needs.
-`workerkey.Acquire` drives that decision: it syncs a new key to disk, records its public key and only then returns it, so a crash before recording reuses the key already on disk.
-
-`domains/sqs/awsapi/worker` carries the decision to the worker as `ClaimFirstStore` and `RecordWorkerPublicKey`, served under the internal `spinifex-sqs` signing service, which no AWS SDK signs for; the public AmazonSQS action inventory is untouched.
-The gateway admits only an assumed-role session of `sqsWorkerRole` in the system account, then the role's policy, and only then resolves the session name, the instance ID STS set, through `workerident.Authorize`; the request carries no identity, and any parameter beyond the action's own is refused.
-The first-store identity is the current record's instance, claim and generation, and a worker bound to another Region is refused.
-A request that passed `Authorize` just before a confirmed stop and replacement still carries the old generation; the fence was raised before the successor began, so `firststore` refuses it whether or not the successor has claimed yet.
-The adapter never logs a caller-chosen parameter name, because one could carry a secret.
-`RecordWorkerPublicKey` accepts only a valid public user key, so a seed is refused without being recorded, logged or echoed.
-The gateway binds both buckets per request and never creates them, so it provisions nothing and answers `ServiceUnavailable` until a worker launch has created them; the worker launch below ensures the role.
-The private key stays on the worker disk; the guest logs and publishes only the public key to `/run/sqs-worker/public-key`.
-
-`domains/sqs/workerlaunch` realizes the worker on the node it runs on, its only node: a bound record is returned as is; otherwise it takes the intent, then on the first launch marks the intent's one volume create, prepares the volume and records its ID, reads the volume's own record and refuses it unless encrypted, ensures the role, finds or creates the network for the claim and reads the recorded store key, and marks the intent dispatched before launching.
-A dispatched launch is never sent again: a lost reply or error leaves the intent `launching` and dispatched, and every later call returns `ErrOutcomeUnknown` until the outcome is reconciled, since the launch path is not idempotent.
-A failure before dispatch leaves an undispatched intent that the next call resumes with the same claim; a confirmed stop or failure is replaced only after `firststore.Fence` excludes the old generation, reusing the pinned volume.
-The launch passes no store decision or key: the guest asks the gateway at boot.
-The intent exists before any volume, so concurrent first launches allocate nothing for the losers. A Prepare proven to have sent no create withdraws the mark and returns `ServiceUnavailable`, or `UnsupportedOperation` when the cluster storage key is not configured, and a retry may prepare the volume; a create that may have applied is never sent again, and every later call returns `IncorrectState` until it is reconciled. A volume the call knows of but cannot use is reported with its ID in a `VolumeError` and left in place, never deleted. An intent that has sent no create may be re-typed; one that has keeps its type until replaced. The worker role is created with EC2-only trust, and an existing role is checked for that trust before any policy is granted; the instance profile must carry exactly the worker role, and one holding any other role is refused unchanged.
-The daemon's `LaunchSQSWorkerInstance` boots the `sqs-worker` image through the same direct-boot path as the store, with the data volume attached at boot and kept on terminate, a required instance profile, and a required mgmt NIC; it re-reads the volume's own record and refuses one that is not available and encrypted, before anything is allocated.
-Its firmware config adds the gateway URL on this node's mgmt bridge and the cluster CA from the daemon's own trust store; the caller cannot supply either, and no value written is secret.
-A relaunch rebuilds that config from the VM record on the node that launched it and is refused anywhere else, and membership takeover already skips SQS-managed instances.
-`sqsWorkerRole` is ensured in the system account granting exactly `spinifex-sqs:ClaimFirstStore` and `spinifex-sqs:RecordWorkerPublicKey`, from the `domains/sqs/workerrole` contract the gateway gate also reads; an existing role whose trust admits more than the EC2 service is refused.
-
-Evidence is local only: unit tests with a fake firmware config, sysfs and mount table, tests running the boot through the real client against the real `spinifex-sqs` registration and records, and QEMU boots of the built image against disposable scratch disks and a host-side test gateway and metadata service, showing the first boot creates the key, reboots reuse it, a key lost after recording is never replaced, and the seed reaches no log or request.
-The worker's network is a private SQS system VPC per Region, built by `systemvpc.EnsurePrivate` from the `[sqs]` supernet (default `10.240.0.0/14`): one VPC and one /24 subnet, with no internet gateway, NAT gateway, EIP, route table or daemon-to-store node port. Each launch claim gets one ENI in that subnet, in the VPC default security group, found again by its claim on a retry; two ENIs for one claim are refused. The launch intent is marked before the one create is sent, so a create whose ENI is not yet visible is never sent again: the launch fails closed with `IncorrectState` until the ENI appears or the outcome is reconciled. Worker-to-store TCP and the guest's route to the metadata service are not yet proven by any traffic test.
-The super-admin `EnsureSQSWorker` action on the internal `spinifex` operator service takes a `NodeId`, which must be a node of this cluster and a single subject token, and an `InstanceType`, and sends a request to `sqs.worker.Ensure.v1.<node>` with no queue group, so only that node answers and the worker stays pinned.
-That node's daemon re-checks the node the request names and the instance type, and refuses as `ServiceUnavailable` before allocating a volume or network or recording an intent unless the worker image's kernel and initramfs are non-empty files, its mgmt bridge exists and the cluster CA is readable; a call after the image is installed then launches.
-It then runs its one composed launcher for its own Region, one launch at a time, in the daemon's own context with a 30-minute bound, so a caller's timeout does not cancel it.
-A reply lost to that timeout is reported as `ServiceUnavailable`; calling again reports the bound worker, or `IncorrectState` for a dispatched launch whose outcome is unknown, which is never resent; a Region pinned to another node is refused as `InvalidParameterValue`.
-The reply names the bound VM's node, Region, instance ID and generation; bound is not ready, and it says nothing about the guest's boot, its key or the store.
-Stop/detach evidence for `RecordStopped` is not wired, the guest's metadata route and gateway TLS on a real ENI and worker-to-store TCP are unproven, and Viperblock's at-rest key is held by the platform.
-
-The worker identity record holds the instance type of its generation; a request for another type is refused as `IncorrectState`, before dispatch and after binding, the reply names the recorded type, and only a replacement after a confirmed stop may change it.
-Operators call both actions with `spx admin sqs ensure-worker --node --instance-type` and `spx admin sqs ensure-store --node --instance-type --volume-size-gib`, signed for the `spinifex` service with the standard AWS credential chain like the other `spx admin` gateway commands.
-Each command refuses missing inputs before sending anything and reports only the bound VM; a bound VM is not a ready worker or store.
-The super-admin `EnsureSQSStore` action takes a `NodeId`, an `InstanceType` and a `VolumeSizeGiB` and sends a request to `sqs.store.Ensure.v1.<node>` with no queue group; only the node the Region's worker is bound to can launch the store.
-That node refuses an instance type whose memory does not exceed 512 MiB, the store's `GOMEMLIMIT` under ADR-0011:S11; this is a floor, not a validated size, and no type is defaulted.
-It refuses before recording or allocating anything unless the store image and the cluster storage key are present, the worker is bound on this node, and that worker's identity has recorded its public key in the first-store decision, which it then advances to a launch intent.
-The store's own launch record in `spinifex-sqs-store-launch` holds only that public key, the claim, the node and the operator's instance type and size, and marks each step before its uncertain create: the volume, the ENI in the worker's private subnet, and the launch.
-A later request for a different instance type or size is refused as `IncorrectState`, before dispatch and after binding, and the reply names the type the store was launched with.
-A volume preparation that failed before sending any create, such as a missing `mkfs.ext4` or a scratch or format failure, withdraws its mark and returns `ServiceUnavailable`, so a retry may try again once the host is fixed.
-Any other marked step whose result is not visible is reported as `IncorrectState` and never resent; a volume that is not encrypted by its own record is refused as `UnsupportedOperation` with its ID kept, and the daemon re-checks the volume's encryption and the guest's size at launch.
-The daemon never receives the worker's seed; the store's NATS server accepts only that public key, and the reply's bound VM and address are not readiness, which only the worker can check.
-
-After boot, the worker's long-running `sqs-worker ready` step mounts and verifies its disk again and calls the internal `spinifex-sqs` `DescribeStore` action, which admits only the current worker and returns the bound store's address and recorded public key, or pending.
-It loads its key only if it matches that public key, then connects to the store's NATS on port 4222 with it and asks JetStream for account info, each pass bounded to 30 seconds.
-The step runs until shutdown with no overall deadline, so a store the operator launches at any time is picked up: it asks again every 5 seconds while the store is pending, not answering or the reply was lost, and re-checks a ready store every 30 seconds.
-A pass writes only the address and public key to `/run/sqs-worker/store-ready`; any failed pass removes that file, and so does the step's own exit.
-A start that finds shutdown in progress or no worker key logs once and waits in place rather than exiting, so init does not respawn it in a loop; shutdown stops it like a running watcher.
-Init respawns the step, at most once every 5 seconds, and every start removes a marker left by a watcher that was killed or crashed before the new watcher runs.
-A ready store that fails is noticed up to one re-check interval plus one bounded call later, so the marker can be up to about a minute stale.
-Shutdown blocks further respawns, removes the marker, and stops the watcher by its recorded PID and start time, escalating to SIGKILL after a budget; it unmounts the worker disk only once the watcher is confirmed gone, and a survivor leaves every mount in place.
-A mismatched key, a store that refuses the key, a key missing from the disk or an unclassified gateway refusal is asked again only every 5 minutes, so it never spins and still recovers if the operator fixes it; a missing key is never replaced.
-The NKey authenticates the worker to the store, but nothing authenticates the store to the worker: there is no TLS, so a pass means only that something at that address accepted the key.
-This is store readiness for the worker alone; there is no SQS API, and no production sizing or support is claimed.
+Evidence is local: unit tests, the guest shell test and an AWS SDK test that creates a queue and sends, receives and deletes a message through the real worker loop against a real local JetStream store and a restart.
+A real guest boot, its metadata route on a real ENI and the signed path on a live node are unproven, Viperblock's at-rest key is held by the platform, and SQS-managed message encryption is not implemented, so the profile must not hold tenant production data.

 ## Configuration

@@ -60,7 +60,7 @@ intact; ADR-0001 explicitly defers that design.

 | Current path | Standing | Target ownership | Rule |
 |---|---|---|---|
-| `cmd/aws-model-coverage` | Qualification generator | Thin main; reusable code in `internal/tooling` | Produces the dispatch-versus-model inventory; never imported by product packages. |
+| `cmd/aws-model-coverage` | Qualification generator | Thin main; reusable code in `internal/tooling` | Produces the dispatch-versus-model inventory; never imported by product packages. Debt: it reads each domain's declared `OperationInventory` from `domains/<service>/awsapi` (ACM, EC2, ECR, ECS, EKS, ELBv2, IAM, RDS, Route 53, SQS, STS) and never constructs a registration. |
 | `cmd/dhcptest` | Development diagnostic | Thin main; reusable code in `internal/tooling` if extraction is useful | It is a DHCP probe, not a supported `spx admin` command. |
 | `cmd/spx-loadgen` | Qualification/load tool | Thin main; reusable code in `internal/testkit` | It drives test load and is not shipped to nodes. |
 | `tests/e2e/*/cmd/*`, `tests/bench/*/cmd/*` | Test and benchmark-private tools | Remain test-private | They are invoked by a test or benchmark harness, not a product process. |
@@ -371,6 +371,7 @@ while the branch is in flight; they do not describe the umbrella repository's
 | `spinifex#1378` | `spinifex/gateway` `oidc_discovery.go` and `registry.go` (the `/oidc/eks` routes and the `/v2` mount), the `DiscoverActiveNodes` method and its cache, and the `GatewayConfig` fields `Config`, `RootCAs`, `InternalSuffix`, `Version`, `Commit`, `IAMService`, `STSService`, `NATSConn` and `ExpectedNodes` | `spinifex/runtime/roles/awsgw` (`PublicRoutes`, `NodeDiscovery.ActiveNodes`, `composeGateway`); the gateway keeps one `PublicRoutes` mount hook and the `Admin` registration name supplied by the role |
 | `spinifex#1384` | `spinifex/gateway` (`GatewayConfig`, the SigV4 and policy middleware, `Request`, the admin dispatch, `AWSOperationInventory`) | `spinifex/ingress/aws/pipeline` (`Config`) and `spinifex/ingress/aws/dispatch` (`AWSOperationInventory`); the gateway layering rows are retired and R2 binds the pipeline |
 | `spinifex#1334` | `spinifex/gateway/oidc_discovery.go` (`eksAccountKV`, the discovery-document and JWKS read and construction) | `spinifex/domains/eks/cluster` (`OIDCPublisher`, `OIDCDiscoveryDocument`), composed by `spinifex/runtime/roles/awsgw` | The EKS domain now reads its own account bucket and builds the per-cluster discovery document and JWKS responses (ADR-0005:S4), and the gateway only maps the unauthenticated `/oidc/eks/...` routes onto an `OIDCPublisher` interface it is handed. Responses are byte-identical, pinned by `oidc_discovery_current_behaviour_test.go`, whose assertions are unchanged. The gateway no longer imports `domains/eks/cluster` in production code. `STS` keeps its own `ClusterJWKSResolver` (`OIDCJWKSReader`), untouched. Recorded debt, not repaired here: a stored JWKS is served verbatim with 200 even when it is not valid JSON. |
+| `spinifex#1245`, `spinifex#1390` | — | `contracts/admission/v1` | The quota override seal route (`admission.v1.SealAccountQuotaOverride`, served by `awsgw`) and the per-gateway read-only capability probe (`admission.v1.AccountQuotaOverrideSealCapability.<node>`) are an explicit versioned cross-process contract between the quota owner and `operator/accountlifecycle`. The consumer wiring is new: `NewClusterEngine` takes a `ClusterNodes` value with separate daemon and gateway node sets, `ClusterConfig.NodesRunning` is the single definition of which nodes run a service, and the seal reaper runs in `StageAccount` after the EC2 settings seal. A termination whose gateway probe is incomplete seals nothing and leaves the account `TERMINATING`. Contract, responder, probe and reaper tests pin the subjects, JSON shapes and fail-closed cases; `TestAccountTermination_QuotaOverrideEndsSealed` replaces the former characterization that overrides survive teardown. |
 The EC2 contract row is intentionally different from the directory moves: it creates
 a compatibility boundary. `ec2.cmd.*` retains its deployed one-token NATS
 subscription and permission shape, while callers use
@@ -488,8 +489,8 @@ Enforcement: the owner decided the layering test is blocking now, with a ratchet
 - `domains/rds/instance` → `domains/rds/parametergroup` (legacy edges): `parameterGroupReferences` binds `Dependants` by scanning instance records; this clears when the RDS instance gains its own owner and answers `Dependants` from it. `tags.go` writes the meta record through its generic tag registry; this clears when tagging invokes a parameter-group-owned tag mutation capability, and moving the tagging adapter to `domains/rds/awsapi` alone does not clear it. The Modify handler still calls `propagateParameterGroup` after the owner's write; propagation is instance-side realization and stays legacy until it is designed as asynchronous realization (group generation, instance staleness, retry). `parameterGroupValues` returns the persisted `Record` and overrides to instance code; when the instance owner is extracted, replace them with a consumer-owned resolved-parameters projection.
 - `domains/ecs` → `domains/ecs/taskdefinition` (owner bypass): `tags.go` reads and writes the revision record through its generic tag registry; this clears when ECS tagging invokes an owner-provided tag mutation capability, and moving `tags.go` to `domains/ecs/awsapi` alone does not clear it. `taskDefinitionResolver` returns the persisted `Record` to task and service launch code; when the task and service owners are extracted, replace the record crossing that boundary with a consumer-owned resolved task-definition (launch-spec) projection. Moving packages has not made the record private.
 - `domains/ecs/taskdefinition` (deliberate duplication, not debt): private `getJSON`, `putJSON` and `keysWithPrefix` copy the raw-KV helpers in `domains/ecs` so the extraction keeps ECS's get/put/list error and recovery semantics. Revisit only if a later, tested state abstraction proves it preserves those semantics; do not migrate ECS to `kvstore` merely to remove them.
- `domains/eks/cluster` → `domains/eks/access` (legacy edges): `ResolveTokenReview` opens the EKS bucket in the gateway process and reads records through the exported `access.Get`; this clears when token review asks the owner across a contract rather than opening the bucket itself. `DeleteClusterPrefix` erases access-entry keys as part of its blanket cluster sweep; this clears when cluster delete invokes an owner-provided removal of the cluster's entries. The owner returns the persisted `Record` to the AWS adapter and token review; replace it with consumer-owned projections when those consumers are extracted.
- `domains/eks/cluster` → `domains/eks/addon` (legacy edges): `DeleteClusterPrefix` erases add-on records and manifests in its blanket cluster sweep; this clears when cluster delete invokes an add-on-owned teardown capability. `stageGPUDeviceAddon` is a temporary node-group-to-add-on orchestration edge: best-effort, errors logged and swallowed, never undone on node-group delete; it clears when the node-group owner is extracted and the dependency is designed. `ClusterReconciler` hosts the add-on status subscription; it clears when add-on delivery observation has its own host under the lifecycle correction. Cluster existence is checked in the handler, not through an owner-declared capability; it moves into the owner when cluster admission is defined (today status is ignored). Guest wire types (`AddonStatusReport`, `AddonDeliveryPhase`, `AddonStatusSubject`, `StagedAddonManifest`, `InternalAddonsResponse`) are owned by `contracts/eks/v1`, which `domains/eks/cluster`, `gateway/eks` and `agents/eks/gatewayfetch` import directly.
+- `domains/eks/cluster` → `domains/eks/access` (legacy edges): `ResolveTokenReview` opens the EKS bucket in the gateway process and reads records through the exported `access.Get`; this clears when token review asks the owner across a contract rather than opening the bucket itself. `DeleteClusterChildren` erases access-entry keys as part of its blanket cluster sweep; this clears when cluster delete invokes an owner-provided removal of the cluster's entries. The owner returns the persisted `Record` to the AWS adapter and token review; replace it with consumer-owned projections when those consumers are extracted.
+- `domains/eks/cluster` → `domains/eks/addon` (legacy edges): `DeleteClusterChildren` erases add-on records and manifests in its blanket cluster sweep; this clears when cluster delete invokes an add-on-owned teardown capability. `stageGPUDeviceAddon` is a temporary node-group-to-add-on orchestration edge: best-effort, errors logged and swallowed, never undone on node-group delete; it clears when the node-group owner is extracted and the dependency is designed. `ClusterReconciler` hosts the add-on status subscription; it clears when add-on delivery observation has its own host under the lifecycle correction. Cluster existence is checked in the handler, not through an owner-declared capability; it moves into the owner when cluster admission is defined (today status is ignored). Guest wire types (`AddonStatusReport`, `AddonDeliveryPhase`, `AddonStatusSubject`, `StagedAddonManifest`, `InternalAddonsResponse`) are owned by `contracts/eks/v1`, which `domains/eks/cluster`, `gateway/eks` and `agents/eks/gatewayfetch` import directly.
 - `domains/eks/cluster` and `domains/eks/awsapi` → `domains/eks/addon` (further edges): the AWS adapter reads the persisted `Record` through `addon.Get`/`addon.List`; this clears when the owner returns a view and the add-on adapter moves to `domains/eks/awsapi`. The guest route reaches the owner through the `handlers_eks` NATS RPC (`ListStagedAddonManifests`); this clears with per-resource commands behind EKS registration. The add-on ARN is derived in both the owner and `domains/eks/awsapi/authz.go`; this clears when authorization obtains the ARN from the owner, which an incarnation suffix would require. The owner's `Manifest`/`Report` beside the `contracts/eks/v1` types is deliberate duplication, not debt.
 - `agents/rds/agent` (dependency, narrowed): engine metadata now arrives through the agent-declared `EngineCatalog`, but the agent still imports `domains/rds/instance` for the command protocol, wire and record types (`Command*`, `Parameter`, `EngineHealth*`, `BootstrapMode*`) and API output types. This clears when those move behind a guest/controller wire contract.
 - `domains/rds/awsapi` → `domains/rds/instance` (`InstanceSizing`): the gateway obtains its `Sizing` through the RDS handler package's adapter rather than its own composition root. This clears when the gateway's RDS composition moves to `runtime/roles/awsgw` and builds `Sizing` there.
@@ -236,12 +236,12 @@ Each stage runs to empty before the next begins, so nothing is deleted while som
 A large account takes minutes to tear down. The API does not hold the connection open for it: `DeleteAccount` starts the job, returns a `deletionId` immediately, and progress is read from `DescribeAccountDeletion`.

 ```bash
-AWS_PROFILE=spinifex-operator spx admin account delete 000000000042 --remote \
+AWS_PROFILE=spinifex-operator spx admin account delete 000000000042 \
  --name customer@example.com --yes \
  --endpoint https://node1.example.com:9999 --region us-west-1
 ```

-`spx --remote` starts the job and then polls, printing each stage as it lands. Your own client can do the same, or start the job and check back later — the deletion record outlives the account, so `DescribeAccountDeletion` stays answerable after the account is gone.
+`spx admin account delete` always goes through the admin API, so it signs with the standard AWS credential chain. On a cluster node that means `AWS_PROFILE=spinifex` (there is no default profile, so without it the command cannot find credentials), or equivalent credentials for an IAM user holding `spinifex:DeleteAccount` and `spinifex:DescribeAccountDeletion`, and a reachable gateway. The older `--remote` flag is accepted and ignored. The command starts the job and then polls, printing each stage as it lands. Your own client can do the same, or start the job and check back later — the deletion record outlives the account, so `DescribeAccountDeletion` stays answerable after the account is gone.

 One job runs per account. A second `DeleteAccount` with a different token while one is running returns `OperationInProgress`; with the *same* token it returns the running job, which is what makes a retry safe.

@@ -340,7 +340,7 @@ spx get vms
 Spinifex ships with a built-in web console — an alternative to the AWS CLI, SDKs, and Terraform, analogous to the AWS Management Console. Every action in the UI is a standard AWS SigV4 API call, so the same IAM policies and audit behaviour apply.

 <p align="center">
-  <img src="../../../.github/assets/spinifex-ui.jpg" alt="Spinifex Web UI" width="900">
+  <img src="../../../.github/assets/dashboard.webp" alt="Spinifex Web UI" width="900">
 </p>

 ### Open the Console
@@ -23,6 +23,7 @@ resources:
 - [Overview](#overview)
 - [Launch](#launch)
 - [Manage](#manage)
+- [Stopping and Data Volumes](#stopping-and-data-volumes)
 - [Modify Instance Attributes](#modify-instance-attributes)
 - [Instance Metadata Options](#instance-metadata-options)
 - [Spot Instances](#spot-instances)
@@ -73,6 +74,31 @@ aws ec2 terminate-instances --instance-ids $INSTANCE_ID
 aws ec2 reboot-instances --instance-ids $INSTANCE_ID
 ```

+## Stopping and Data Volumes
+
+A guest that powers itself off (`poweroff`, `shutdown -h`) goes `stopping` then `stopped`, the same as `stop-instances`.
+It is not restarted automatically.
+`InstanceInitiatedShutdownBehavior=terminate` is not honoured yet: a guest shutdown always stops the instance.
+
+Data volumes stay `in-use` and attached to a stopped instance, and `start-instances` mounts them again.
+While the instance is stopped, `attach-volume` of one of its data volumes to another instance returns `VolumeInUse`, and `delete-volume` is refused.
+
+`detach-volume` works on a stopped instance, with or without `--instance-id`.
+The volume becomes `available` and the next start does not attach it.
+The root volume of a stopped instance can be detached too, as on AWS.
+The internal EFI variable store is not a customer volume: it never appears in `describe-volumes` or `BlockDeviceMappings`, and detaching it is refused with `OperationNotPermitted`.
+The instance then cannot start: `start-instances` returns `InvalidParameterValue` with "Instance does not have a volume attached at root (/dev/sda1)".
+
+`attach-volume` works on a stopped instance too.
+The volume must be `available` and in the instance's availability zone, otherwise the call returns `VolumeInUse` or `InvalidVolume.ZoneMismatch`.
+The attachment is recorded on the stopped instance and the volume becomes `in-use` straight away.
+Attaching at the root device name (for example `/dev/sda1`) makes the volume the root again, and the next `start-instances` boots from it.
+
+An instance stopped by an earlier build had its data volumes released to `available` while its record still listed them.
+On start, a listed data volume that was deleted or is now attached to another instance is dropped with a warning in the daemon log, and the start continues.
+A listed volume that is still `available` and unattached is attached again.
+A missing root volume fails the start with the same `InvalidParameterValue` answer.
+
 ## Modify Instance Attributes

 Change instance type, user data, or termination protection. Instance type and user data require the instance to be **stopped** first.
@@ -135,6 +161,30 @@ aws ec2 get-console-output --instance-id $INSTANCE_ID \
  --query 'Output' --output text | base64 -d
 ```

+Like AWS, the call returns the most recent 64 KiB of the console.
+While an instance is shutting down after a terminate request, the call reads the renamed file described below.
+Once the instance has left the node's running set, which includes a stopped or terminated instance, the node no longer answers and the call returns `InvalidInstanceID.NotFound`.
+AWS still lists a terminated instance for a while after termination, so a caller may expect this call to work then; it does not here, a known gap.
+Whether the API should answer for a stopped instance is not settled here; only the file on the host is kept.
+
+### Where console logs live and how big they get
+
+The serial console of each instance is written on its host to `/run/spinifex/console-<instance-id>.log`.
+`/run` is memory-backed, so these files are cleared when the host reboots, including any left by older releases.
+The daemon keeps them bounded.
+A file over 4 MiB is cut back in place to its newest 1 MiB, and the guest keeps writing while it happens.
+The console is opened in append mode, so a reboot or auto-restart of the guest adds to the same file instead of starting over.
+A live `console-<instance-id>.log` is deleted only when it has not changed for an hour and no QEMU process on the host has it open, so a stopped instance keeps its console for an hour and a quiet running guest keeps it indefinitely.
+If the daemon cannot read the open files of every QEMU process on the host, it deletes nothing on that pass and logs a warning.
+This also clears the file of an instance that was stopped and then terminated, or that was recovered on another node, within about an hour of its last write.
+When an instance is terminated, including a failed first launch and the system instances behind load balancers, queues and EKS control planes, its file is renamed to `console-<instance-id>.log.terminated`.
+The daemon removes a `.terminated` file an hour after its last write.
+
+Known limits:
+
+- The log shipper follows only `console-*.log`, so it stops following a file when it is renamed on terminate, and the last few seconds of output that it had not yet read can be lost.
+- The instance password retrieval reads only the live file name, so it does not find a console renamed after terminate.
+
 ## Instance Types

 List instance types available on the current host. The catalog is generated from the host CPU (Intel, AMD, or ARM) and includes burstable (t-family), general purpose (m-family), compute optimised (c-family), and memory optimised (r-family) types.
@@ -315,7 +315,7 @@ prefix_len = 24

 ### Host access to instances (jumpbox pattern)

-The spinifex host automatically reaches every instance's **private IP**: IGW attach installs a host route into OVN (`<vpc-cidr> via <gateway-transit-ip> dev spx-nat-host`) and exempts the transit net from SNAT, so replies to host-initiated connections come back un-NATted. No per-instance setup.
+The spinifex host automatically reaches every instance's **private IP**: each node converges a host route into OVN (`<vpc-cidr> via <gateway-transit-ip> dev spx-nat-host`) for every attached VPC, and the IGW attach exempts the transit net from SNAT, so replies to host-initiated connections come back un-NATted. No per-instance setup. This only exists in routed NAT mode; other modes install no host routes. Every node runs this loop itself, on each intent change, on a nudge from the local IGW hooks and every 30 seconds, so a node that missed the attach event still gets the route. A route is removed on every node when no attached VPC holds its CIDR any more, such as after a detach. When several attached VPCs share one CIDR, the bootstrap VPC wins, then the lowest VPC ID, so detaching one gateway repoints the route to another attached VPC instead of removing it.

 Security groups still apply and the default SG is closed to the host, same as AWS — open SSH/ICMP from the transit net first:

@@ -58,7 +58,10 @@ Authentication uses **EKS access entries** (the API authentication mode) — IAM
 **Spinifex specifics**

 - **Access entries are the only auth path.** `authentication_mode` accepts `API` (the default when unset) and `API_AND_CONFIG_MAP`; bare `CONFIG_MAP` is rejected. Spinifex has no `aws-auth` ConfigMap path, so `API_AND_CONFIG_MAP` behaves as `API`: grant access with access entries.
- **Security groups are auto-managed.** Spinifex creates the cluster and node-group security groups deterministically (e.g. `eks-cluster-<name>-nodegroup-sg`); `vpc_config.security_group_ids` is ignored. To expose a workload you add an ingress rule to the auto-managed node-group SG.
+- **The cluster security group is auto-managed.** As on EKS, Spinifex creates one `eks-cluster-sg-<name>-<id>` in your VPC, tagged `kubernetes.io/cluster/<name>=owned` and `aws:eks:cluster-name`, allowing all traffic from itself. It is attached to the workers and the private-endpoint network interface, and `DescribeCluster` reports it as `clusterSecurityGroupId`. A launch template that names security groups replaces it on the workers, as on AWS. Clusters created before this change keep their `-control-plane-sg` and `-nodegroup-sg` until you recreate them. Deleting a cluster deletes the cluster security group and removes only the rules on your other security groups that reference it. `vpc_config.security_group_ids` is ignored.
+- **Node group deletes are asynchronous.** `DeleteNodegroup` returns at once with the group `DELETING`, and `DescribeNodegroup` answers `ResourceNotFoundException` once the workers and the group's instance profile are gone. Deleting a cluster or creating a group of the same name while a group is `DELETING` returns `ResourceInUseException`, as does a second delete of a `DELETING` group.
+- **If a node group delete cannot finish,** the group moves to `DELETE_FAILED` after 30 minutes, with a `NODE_TERMINATION_FAILURE` health issue naming the workers that are still running. Run `DeleteNodegroup` again to resume it.
+- **Each node group owns an EKS-managed instance profile.** Spinifex creates it, tags it `spinifex:eks-cluster`, `spinifex:eks-nodegroup` and `spinifex:eks-profile-id`, and puts the node role in it. It is removed with the group, so you can delete the node role afterwards with no extra steps. Spinifex never deletes an instance profile it did not create, or one whose tags, name or role no longer match what it recorded; it leaves the profile and logs a warning with its name.
 - **The worker image is fixed.** Node groups always boot Spinifex's `eks-node` image. `ami_type` is recorded but does not select the image, and the image must be registered on the cluster before you create a node group.
 - **Workers launch with IMDSv2 required and an IMDS hop limit of 2.**
  This matches AWS managed node groups and lets a pod without IRSA reach the node role through the SDK default credential chain.
@@ -96,6 +99,72 @@ Limits:
  The load balancer VM reports target health through its own host's gateway, so while that gateway is down the unhealthy control plane stays in rotation.
 - Not measured: more than one control plane per host, a physical-node failure instead of a stopped gateway, and the target health state as `DescribeTargetHealth` reports it, because the load balancer and control-plane VMs belong to the system account.

+### IAM roles for service accounts in your workloads
+
+A pod gets an IAM role by running under a ServiceAccount that carries the role ARN in an annotation.
+No pod-spec wiring is needed.
+
+```yaml
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: app
+  annotations:
+    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/app-role
+```
+
+Every control plane runs a mutating webhook that acts on pod creation.
+When the pod's ServiceAccount has the annotation, the webhook adds these to each container and init container:
+
+- `AWS_ROLE_ARN` and `AWS_WEB_IDENTITY_TOKEN_FILE`, with a projected token volume named `aws-iam-token` mounted at `/var/run/secrets/eks.amazonaws.com/serviceaccount`.
+- `AWS_REGION`, `AWS_DEFAULT_REGION` and `AWS_STS_REGIONAL_ENDPOINTS`.
+- `AWS_ENDPOINT_URL_STS`, so the SDK calls the Spinifex STS endpoint.
+- `AWS_CA_BUNDLE`, pointing at `/etc/spinifex/ca-bundle/ca.pem`, a ConfigMap named `spinifex-ca-bundle` that is created in every namespace and holds the system roots plus the gateway CA.
+- `SSL_CERT_FILE` with the same path, only when the container does not set it. Clients such as the AWS Load Balancer Controller read this variable instead of `AWS_CA_BUNDLE`.
+
+Your workload still sets `AWS_ENDPOINT_URL` (or the per-service variable) for every service other than STS.
+
+The token audience defaults to `sts.amazonaws.com`.
+The expiry defaults to a value between 79200 and 86400 seconds and is clamped to 600 to 86400 seconds.
+Set them with the `eks.amazonaws.com/audience` and `eks.amazonaws.com/token-expiration` annotations on the ServiceAccount.
+The `token-expiration` annotation on the pod wins over the ServiceAccount.
+
+A container is left alone when it already has the role, region and regional STS settings, which is how the managed add-ons and your own explicit wiring are skipped.
+When some are present, the missing ones and the mount are added.
+Containers named in the pod's `eks.amazonaws.com/skip-containers` annotation (comma separated) are not touched.
+A pod whose ServiceAccount has no role annotation is not changed.
+
+The webhook fails open: if it cannot decide, the pod is created without injection.
+Each injection, skip and failure is logged with the namespace, pod and reason in the control-plane console log.
+Because it fails open, a pod that needs a role but starts without credentials is the symptom to look for, and the console log says why.
+
+The webhook listens on `127.0.0.1:8445` of each control plane, and each API server calls its own.
+Port 8444 is a different listener, the readiness check described above.
+
+Existing clusters get the webhook only when their control planes are replaced by the current `eks-node` image.
+No user-facing operation replaces a control plane, and automatic replacement does not currently work when the replacement reuses the failed member's IP.
+Recreate the cluster to adopt it.
+User-workload injection through a cluster created with an older image is not measured.
+
+### Add-on versions
+
+Add-on versions use the AWS form `v<x.y.z>-eksbuild.<n>`, for example `v1.40.1-eksbuild.1`.
+`DescribeAddonVersions` returns versions in this form, newest first, and every other add-on response reports the version the same way.
+Spinifex ships one EKS build of each upstream version, so `<n>` is `1` today.
+
+`DescribeAddonVersions` honours `addonName`, `kubernetesVersion`, `types`, `publishers`, `owners`, `maxResults` and `nextToken`.
+Each version carries `compatibilities` with the `clusterVersion`, the `platformVersions`, and a `defaultVersion` flag that is true for exactly one version per add-on and cluster version.
+A `kubernetesVersion` that Spinifex does not run returns no add-ons.
+
+`CreateAddon` and `UpdateAddon` accept the AWS form of a version in the catalogue.
+They also accept the bare upstream form, such as `1.40.1`, and Spinifex logs each use at Info.
+This is a Spinifex leniency that AWS does not offer, and it exists so that older scripts keep working.
+Any other value is refused with a message that names `addonVersion` and lists the supported versions, for example `v1.40.1-eksbuild.2`, `1.40` or `v1.40.1`.
+
+In Terraform, pass the AWS form to `aws_eks_addon.addon_version`, either as a literal or from `data.aws_eks_addon_version`.
+The AWS provider validates the format before it calls the API, so it rejects the bare form.
+Leaving `addon_version` unset still installs the catalogue default.
+
 ## Prerequisites

 > [!IMPORTANT]
@@ -178,12 +247,11 @@ The list shows the node group workers only. The control-plane VMs never appear a

 ### 4. Expose a workload

-Workers join an auto-managed security group that only allows intra-cluster traffic. To reach a NodePort service, add one ingress rule to that SG (look it up by its deterministic name):
+Workers join the cluster security group, which only allows intra-cluster traffic. To reach a NodePort service, add one ingress rule to it (`DescribeCluster` reports its id):

 ```bash
-SG=$(aws ec2 describe-security-groups \
-  --filters Name=group-name,Values=eks-cluster-demo-nodegroup-sg \
-  --query 'SecurityGroups[0].GroupId' --output text)
+SG=$(aws eks describe-cluster --name demo \
+  --query 'cluster.resourcesVpcConfig.clusterSecurityGroupId' --output text)

 aws ec2 authorize-security-group-ingress \
  --group-id "$SG" --protocol tcp --port 30080 --cidr 0.0.0.0/0
@@ -225,7 +293,7 @@ The form notes that the authentication mode is API and that the legacy aws-auth

 - The **Access** tab lists IAM principals and their cluster-access policies — add an access entry to let another user or role reach the cluster.
 - The **Addons** tab installs and shows the health of managed addons.
- The **Networking** tab shows the cluster's VPC, subnets, endpoint access, and the auto-managed security groups.
+- The **Networking** tab shows the cluster's VPC, subnets, endpoint access, and the auto-managed cluster security group.

 To run `kubectl` against the cluster, follow the AWS CLI tab's `update-kubeconfig` step.

@@ -308,11 +376,20 @@ Control plane readiness and kubectl authentication above lists which clusters re
 - Confirm the workers can pull images — a default route to an Internet Gateway (public) or NAT Gateway (private).
 - Confirm the node role has `AmazonEKSWorkerNodePolicy`, `AmazonEKS_CNI_Policy`, and `AmazonEC2ContainerRegistryReadOnly`.

-**Workload unreachable from outside the cluster.** The node-group security group only allows intra-cluster traffic by default. Add an ingress rule for your NodePort to `eks-cluster-<name>-nodegroup-sg`, as shown in the CLI tab.
+**Workload unreachable from outside the cluster.** The cluster security group only allows intra-cluster traffic by default. Add an ingress rule for your NodePort to it, as shown in the CLI tab. An Ingress served by the AWS Load Balancer Controller needs no rule: the controller adds it to the cluster security group.

 **`create-cluster` rejected with an authentication-mode error.** `CONFIG_MAP` is the one mode Spinifex rejects, because it has no `aws-auth` ConfigMap path. Set `authentication_mode` to `API` or `API_AND_CONFIG_MAP`, or leave it unset, and grant access with access entries.

-**Addon install fails.** Only addons bundled into the registered `eks-node` image install successfully. If you pin an addon version in Terraform, leave `addon_version` unset so Spinifex selects the catalog default.
+**`delete-role` fails with `DeleteConflict` after deleting a node group.** Node groups created before per-group instance profiles left an instance profile named after the role, which Spinifex no longer removes because it cannot prove it created it. Remove it yourself:
+
+```bash
+aws iam remove-role-from-instance-profile --instance-profile-name <role-name> --role-name <role-name>
+aws iam delete-instance-profile --instance-profile-name <role-name>
+```
+
+**A node group is `DELETE_FAILED`.** `aws eks describe-nodegroup` lists the workers that did not stop under `health.issues`. Once they are gone, or after you fix the cause, run `aws eks delete-nodegroup` again.
+
+**Addon install fails.** Only addons bundled into the registered `eks-node` image install successfully. If you pin an addon version in Terraform, use the AWS form from `DescribeAddonVersions` (see Add-on versions above), or leave `addon_version` unset so Spinifex selects the catalog default.

 ## Reference Architectures

@@ -17,7 +17,7 @@ tags:

 ## Overview

-The platform serves **405 operations** across the AWS APIs below. Every page names the operations its service implements from the pinned model, generated from the dispatch tables on each build rather than written by hand.
+The platform serves **406 operations** across the AWS APIs below. Every page names the operations its service implements from the pinned model, generated from the dispatch tables on each build rather than written by hand.

 | Service | Operations |
 |---|---:|
@@ -27,9 +27,9 @@ The platform serves **405 operations** across the AWS APIs below. Every page nam
 | [ECS](/coverage/ecs) | 31 |
 | [EKS](/coverage/eks) | 27 |
 | [ELBv2](/coverage/elbv2) | 36 |
-| [IAM](/coverage/iam) | 81 |
+| [IAM](/coverage/iam) | 82 |
 | [RDS](/coverage/rds) | 27 |
 | [Route 53](/coverage/route53) | 13 |
 | [S3](/coverage/s3) | 26 |
 | [STS](/coverage/sts) | 5 |
-| **Total** | **405** |
+| **Total** | **406** |
@@ -2413,6 +2413,7 @@
          "ListAssociatedAccessPolicies",
          "ListClusters",
          "ListInternalAddons",
+          "ListInternalWorkers",
          "ListNodegroups",
          "ListTagsForResource",
          "PublishInternal",
@@ -2502,6 +2503,7 @@
        "extra": [
          "GetRecoveryDirective",
          "ListInternalAddons",
+          "ListInternalWorkers",
          "PublishInternal",
          "WebhookTokenReview"
        ]
@@ -2938,7 +2940,8 @@
          "UpdateAccessKey",
          "UpdateAssumeRolePolicy",
          "UpdateRole",
-          "UpdateRoleDescription"
+          "UpdateRoleDescription",
+          "UpdateUser"
        ],
        "implemented": [
          "AddRoleToInstanceProfile",
@@ -3021,7 +3024,8 @@
          "UpdateAccessKey",
          "UpdateAssumeRolePolicy",
          "UpdateRole",
-          "UpdateRoleDescription"
+          "UpdateRoleDescription",
+          "UpdateUser"
        ],
        "stubbed": [],
        "unsupported": [],
@@ -3121,7 +3125,6 @@
          "UpdateServerCertificate",
          "UpdateServiceSpecificCredential",
          "UpdateSigningCertificate",
-          "UpdateUser",
          "UploadSSHPublicKey",
          "UploadServerCertificate",
          "UploadSigningCertificate"
@@ -3978,6 +3981,86 @@
        "extra": []
      }
    },
+    {
+      "service": "sqs",
+      "api_version": "2012-11-05",
+      "opaque": false,
+      "note": "",
+      "operations": {
+        "modelled": [
+          "AddPermission",
+          "CancelMessageMoveTask",
+          "ChangeMessageVisibility",
+          "ChangeMessageVisibilityBatch",
+          "CreateQueue",
+          "DeleteMessage",
+          "DeleteMessageBatch",
+          "DeleteQueue",
+          "GetQueueAttributes",
+          "GetQueueUrl",
+          "ListDeadLetterSourceQueues",
+          "ListMessageMoveTasks",
+          "ListQueueTags",
+          "ListQueues",
+          "PurgeQueue",
+          "ReceiveMessage",
+          "RemovePermission",
+          "SendMessage",
+          "SendMessageBatch",
+          "SetQueueAttributes",
+          "StartMessageMoveTask",
+          "TagQueue",
+          "UntagQueue"
+        ],
+        "registered": [
+          "ChangeMessageVisibility",
+          "ChangeMessageVisibilityBatch",
+          "CreateQueue",
+          "DeleteMessage",
+          "DeleteMessageBatch",
+          "GetQueueAttributes",
+          "GetQueueUrl",
+          "ListQueueTags",
+          "ListQueues",
+          "ReceiveMessage",
+          "SendMessage",
+          "SendMessageBatch",
+          "SetQueueAttributes",
+          "TagQueue",
+          "UntagQueue"
+        ],
+        "implemented": [
+          "ChangeMessageVisibility",
+          "ChangeMessageVisibilityBatch",
+          "CreateQueue",
+          "DeleteMessage",
+          "DeleteMessageBatch",
+          "GetQueueAttributes",
+          "GetQueueUrl",
+          "ListQueueTags",
+          "ListQueues",
+          "ReceiveMessage",
+          "SendMessage",
+          "SendMessageBatch",
+          "SetQueueAttributes",
+          "TagQueue",
+          "UntagQueue"
+        ],
+        "stubbed": [],
+        "unsupported": [],
+        "missing": [
+          "AddPermission",
+          "CancelMessageMoveTask",
+          "DeleteQueue",
+          "ListDeadLetterSourceQueues",
+          "ListMessageMoveTasks",
+          "PurgeQueue",
+          "RemovePermission",
+          "StartMessageMoveTask"
+        ],
+        "extra": []
+      }
+    },
    {
      "service": "sts",
      "api_version": "2011-06-15",
@@ -43,6 +43,21 @@ A missing `Name`, `BlockDeviceMappings` or root snapshot answers `MissingParamet
 Registering with `--ena-support`, `--no-ena-support` and neither, copying the image, and three refusals were run on a three-node cluster with the AWS CLI.
 Launching an instance from an `EnaSupport` image was not run, so the launch path is unchanged by construction and not by test.

+### CreateSnapshot Error Mapping
+
+`CreateSnapshot` drains and snapshots the volume inside the EBS provider.
+A provider that cannot snapshot a volume because it is in use, as the `qemunbd` provider cannot for an attached volume, answers `ErrVolumeInUse`, which becomes `IncorrectState` with the message `The volume 'vol-...' is in use. Detach it before creating a snapshot.`
+The `viperblockd` provider drains a mounted export and snapshots it, so an attached volume succeeds there.
+A provider that cannot answer now, for example because another node holds the volume, answers `ServiceUnavailable` so the caller retries.
+Any other provider failure is `ServerInternal`.
+The snapshot taken for `CreateImage` maps these errors the same way.
+
+### EBS Encryption by Default Follows the Platform
+
+Whether new volumes are encrypted is decided by the platform's encryption key, not by the account.
+`GetEbsEncryptionByDefault` and `EnableEbsEncryptionByDefault` report `true` when the platform has an encryption key configured and `false` otherwise, and `EnableEbsEncryptionByDefault` does not change that.
+`DisableEbsEncryptionByDefault` is refused with `OperationNotPermitted` and the message `EBS encryption by default cannot be disabled: this platform encrypts every volume` while the platform encrypts, and answers `false` where it does not.
+
 ### Operations

 | Operation |
@@ -18,7 +18,7 @@ tags:

 ## Overview

-Spinifex implements **81 operations** in the IAM `2010-05-08` API model.
+Spinifex implements **82 operations** in the IAM `2010-05-08` API model.

 ### Scope

@@ -109,3 +109,4 @@ All IAM operations are account-scoped. The root user of an account bypasses poli
 | `UpdateAssumeRolePolicy` |
 | `UpdateRole` |
 | `UpdateRoleDescription` |
+| `UpdateUser` |
@@ -903,7 +903,7 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,

 ## eks

-`2017-11-01` — 27 of 70 modelled operations implemented (38.6%), 0 stubbed, 0 not applicable, 4 registered outside the pinned model.
+`2017-11-01` — 27 of 70 modelled operations implemented (38.6%), 0 stubbed, 0 not applicable, 5 registered outside the pinned model.

 ### Not implemented

@@ -955,6 +955,7 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,

 - `GetRecoveryDirective`
 - `ListInternalAddons`
+- `ListInternalWorkers`
 - `PublishInternal`
 - `WebhookTokenReview`

@@ -987,7 +988,7 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,

 ## iam

-`2010-05-08` — 81 of 180 modelled operations implemented (45.0%), 0 stubbed, 19 not applicable, 0 registered outside the pinned model.
+`2010-05-08` — 82 of 180 modelled operations implemented (45.6%), 0 stubbed, 19 not applicable, 0 registered outside the pinned model.

 ### Not implemented

@@ -1069,7 +1070,6 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
 - `UpdateOpenIDConnectProviderThumbprint`
 - `UpdateSAMLProvider`
 - `UpdateServerCertificate`
- `UpdateUser`
 - `UploadServerCertificate`

 ### Not applicable — do not build
@@ -1441,6 +1441,21 @@ request-payment: Requester Pays shifts transfer charges to the caller, which nee
 - `GetBucketRequestPayment`
 - `PutBucketRequestPayment`

+## sqs
+
+`2012-11-05` — 15 of 23 modelled operations implemented (65.2%), 0 stubbed, 0 not applicable, 0 registered outside the pinned model.
+
+### Not implemented
+
+- `AddPermission`
+- `CancelMessageMoveTask`
+- `DeleteQueue`
+- `ListDeadLetterSourceQueues`
+- `ListMessageMoveTasks`
+- `PurgeQueue`
+- `RemovePermission`
+- `StartMessageMoveTask`
+
 ## sts

 `2011-06-15` — 5 of 11 modelled operations implemented (45.5%), 0 stubbed, 0 not applicable, 0 registered outside the pinned model.
@@ -1,7 +1,7 @@
 ---
 title: "Oracle Linux Hosts"
 seoTitle: "Install Spinifex on Oracle Linux 10 — Spinifex Docs"
-description: "Prepare Oracle Linux 10 hosts for Spinifex: UEK R8, SELinux, a NetworkManager br-wan bridge, the NFV SIG repository, IPsec on libreswan and host updates."
+description: "Install Spinifex on Oracle Linux 10: prepare the host with UEK R8, SELinux permissive and a NetworkManager br-wan bridge, then run the single-node install."
 category: "Install"
 tags:
  - install
@@ -17,55 +17,42 @@ resources:

 # Oracle Linux Hosts

-> What differs when the Spinifex host runs Oracle Linux 10 instead of Debian or Ubuntu.
+> Install Spinifex on Oracle Linux 10.

 ## Table of Contents

 - [Overview](#overview)
- [Prerequisites](#prerequisites)
 - [Instructions](#instructions)
+- [What the Installer Changes](#what-the-installer-changes)
 - [Troubleshooting](#troubleshooting)

 ---

 ## Overview

-Spinifex installs on Oracle Linux 10.2 or a later 10.x release, running the UEK R8 kernel, on x86_64.
-The installer, the release tarball and the install steps are the same as on Debian; this page covers what the host needs first and what the installer changes on it.
-Follow it alongside [Single-Node Install](/docs/install).
+Spinifex runs on Oracle Linux 10.2, on the UEK R8 kernel, on x86_64 with an x86-64-v3 CPU (Intel Haswell, AMD Excavator or later).

-**Supported on Oracle Linux:**
+The host must be a dedicated Spinifex host: Spinifex claims every `tap*` device and owns libreswan's `ipsec.service`.

- A single node, or a cluster in which every node runs Oracle Linux; servers beyond the third join as compute nodes, as on Debian.
- Guests, VPCs, routed NAT, Elastic IPs, and Application and Network Load Balancers.
- Whole-GPU passthrough, on x86_64 GPU hosts (see [GPU Passthrough](/docs/gpu-passthrough)).
+**Not supported yet:** clusters mixing Oracle Linux with Debian or Ubuntu, SELinux enforcing, aarch64, Oracle Linux 9 or earlier, the Red Hat Compatible Kernel, a ZFS root, NVIDIA MIG and air-gapped installs.

-**Not supported on Oracle Linux yet:**
+## Instructions

- Clusters that mix Oracle Linux with Debian or Ubuntu nodes, on any release.
- SELinux enforcing, aarch64, Oracle Linux 9 or earlier, the Red Hat Compatible Kernel, a ZFS root, and NVIDIA MIG.
+Work through both tabs in order.

-Open vSwitch and OVN come from the CentOS NFV SIG, because Oracle ships neither for Oracle Linux 10.
-The NFV SIG is a community repository, and Oracle does not support the packages it publishes.
-Spinifex installs `openvswitch3.5` and `ovn25.03`, the versions Debian 13 ships, and its repository file lets the SIG supply nothing else.
-QEMU comes from Oracle's KVM Utilities repository, because only that build has the `microvm` machine type Spinifex load balancers boot on.
+:::tabs
+@tab 1. Prepare Oracle Linux

-An Oracle Linux node must be a dedicated Spinifex host.
-Spinifex claims every `tap*` device and owns libreswan's `ipsec.service`, which it keeps stopped on a single node and runs for its own tunnels on a cluster node, so VPNs, hand-run QEMU guests and libreswan tunnels of your own cannot share the host.
-Crypto policies other than `DEFAULT` and `FIPS` are untested, and so is Secure Boot.
+### Update and check the host

-## Prerequisites
-
-### Operating system and kernel
-
-Install Oracle Linux 10 with Oracle's ISO or a KVM template, then update it to the current minor release and reboot:
+Update to the current minor release and reboot:

 ```bash
 sudo dnf update -y
 sudo reboot
 ```

-After the reboot, check the release, the kernel and the CPU:
+Then check the release, the running kernel and the CPU:

 ```bash
 cat /etc/oracle-release        # Oracle Linux Server release 10.2 or later
@@ -73,17 +60,7 @@ uname -r                       # 6.12.0-...el10uek.x86_64
 grep -c -w avx2 /proc/cpuinfo  # non-zero
 ```

-The kernel must be UEK R8 (`6.12.0-*uek`), and it must be the running kernel: an installed but not booted `kernel-uek` is refused.
-The CPU must be x86-64-v3 or newer (Intel Haswell, AMD Excavator or later), since the NFV SIG and KVM Utilities builds target it.
-The installer names any missing CPU flag.
-
-The installer adds `kernel-uek-modules` for the running kernel, which carries `openvswitch.ko` and pulls in `linux-firmware`, about 800 MB.
-Oracle publishes modules for each kernel separately, so a host running an old kernel may find its modules gone from the repository; the installer then stops and asks for a kernel update and a reboot.
-
-### SELinux
-
-Spinifex ships no SELinux policy yet, so SELinux must be permissive or disabled, both now and at the next boot.
-Oracle's images boot enforcing.
+### Set SELinux permissive

 ```bash
 sudo setenforce 0
@@ -91,14 +68,9 @@ sudo sed -i 's/^SELINUX=.*/SELINUX=permissive/' /etc/selinux/config
 getenforce                     # Permissive
 ```

-The installer and `spx admin preflight` refuse a host that is enforcing in either place.
+### Create the br-wan bridge

-### WAN bridge with NetworkManager
-
-As on every OS, the host's uplink must already sit on a Linux bridge named `br-wan`, with the host address and default route on the bridge (see [Single-Node Install](/docs/install#prerequisites)).
-On Oracle Linux, NetworkManager owns `br-wan`; Spinifex leaves the bridge and its NIC to it and marks the links it creates itself as unmanaged.
-
-Create the bridge from the console, not over SSH on the NIC being moved, since the host loses its address while the profile changes.
+Run this from the console, not over SSH on the NIC being moved.
 Replace `eno1` with the uplink NIC and `"System eno1"` with its current profile (`nmcli connection show`):

 ```bash
@@ -109,118 +81,88 @@ sudo nmcli connection down "System eno1"
 sudo nmcli connection up br-wan
 ```

-For a static address, give the bridge `ipv4.method manual` with `ipv4.addresses`, `ipv4.gateway` and `ipv4.dns` instead of `ipv4.method auto`.
-
-`bridge.stp no` matters: a NetworkManager bridge runs STP by default, which holds the port in its forwarding delay for about 15 seconds after every link up, long enough for DHCP on the bridge to time out.
-Then verify, as on Debian:
+For a static address, use `ipv4.method manual` with `ipv4.addresses`, `ipv4.gateway` and `ipv4.dns`.
+Keep `bridge.stp no`, or DHCP on the bridge can time out.

 ```bash
 ip -br link show br-wan        # UP
 ip route                       # default via ... dev br-wan
 ```

-### Network access during install
+@tab 2. Install Single Node

-Besides `install.mulgadc.com` and the AWS CLI download, the install reaches `yum.oracle.com` for the Oracle Linux repositories, `mirror.stream.centos.org` for the NFV SIG packages, and `www.centos.org` for the NFV SIG signing key.
-An air-gapped install on Oracle Linux is not documented yet.
-
-## Instructions
-
-## Step 1. Install Spinifex
-
-Run the installer as a user with `sudo`, exactly as on Debian:
+### Install Spinifex

 ```bash
 curl -fsSL https://install.mulgadc.com | bash
 ```

-For a single node, continue with [Single-Node Install](/docs/install) from Step 2: `setup-ovn.sh --management`, `spx admin init` and `systemctl start spinifex.target`.
-IPsec stays off on a single node, because there is no peer to encrypt traffic to.
+The installer stops and masks firewalld; see [firewalld](#firewalld) before relying on any rule it held.

-For a cluster, prepare and install every node this way, then continue with [Multi-Node Install](/docs/install-multi-node) from Step 2.
-
-### Clusters
-
-Every node in an Oracle Linux cluster runs Oracle Linux.
-`install-node.sh` refuses a host list that mixes Oracle Linux with Debian or Ubuntu, but `spx admin join` does not check, so keep to one OS yourself when joining by hand.
-
-IPsec encrypts the Geneve tunnels between nodes and is on by default, as on Debian.
-On Oracle Linux it runs on libreswan, the IKE daemon Oracle ships, rather than strongSwan.
-Once the node's IPsec certificate exists, the daemon unmasks and starts `ipsec.service` and `openvswitch-ipsec.service`.
-`ovs-monitor-ipsec` then loads the node certificate, its key and the cluster CA into libreswan's NSS database at `/var/lib/ipsec/nss`, under nicknames that start with `ovs_cert`.
-
-The tunnels are plain ESP (IP protocol 50), as on Debian, negotiated over UDP 500.
-Open UDP 500, UDP 4500 and ESP between nodes: libreswan moves to UDP 4500 if it detects NAT between two nodes.
-The Spinifex host firewall opens all three to cluster peers; a firewall between the nodes must allow them too.
-While IKE to a peer fails, the node drops its Geneve traffic to that peer rather than send it in clear.
-The nft table `inet spinifex_ipsec_guard` drops outbound UDP 6081 from the encap address whose route carries no SA, which also covers the moments pluto flushes the xfrm policy table as it starts and stops.
-It is not persisted: `ipsec.service` puts it back at boot before ovn-controller builds the tunnels, and the daemon re-syncs it every 60 seconds.
-`spx admin init --ipsec=false` leaves the tunnels unencrypted, as on any OS.
-
-## Step 2. Know What the Installer Changed
-
-On Oracle Linux the installer makes these changes, which Debian hosts do not need:
-
- **Repositories.** It enables `ol10_UEKR8` and `ol10_kvm_utils`, and adds `qemu-kvm*` and `qemu-img*` to the `excludepkgs` of `ol10_appstream`, keeping any excludes already set, so a later AppStream build can never replace the KVM Utilities QEMU.
- **NFV SIG.** It writes `/etc/yum.repos.d/spinifex-nfv-sig.repo` with an `includepkgs=` list of the OVS and OVN packages, and imports the SIG key to `/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-NFV` only when its fingerprint is `35154228174901BEFA8E69A621465E289D2A76A7`.
- **OVS and OVN user.** Both run as root, as on Debian, through `OVS_USER_ID` and `OVN_USER_ID` in `/etc/sysconfig/openvswitch` and `/etc/sysconfig/ovn`.
- **DNS.** Where systemd-resolved is installed but not running, it saves `/etc/resolv.conf` to `/var/lib/spinifex-reset/resolv.conf.orig`, starts resolved behind NetworkManager and links `/etc/resolv.conf` to the resolved stub. It then checks that names still resolve and stops, printing the commands that put DNS back, if they do not.
- **chrony.** It appends `confdir /etc/chrony.d` to `/etc/chrony.conf` once, and writes its step policy to `/etc/chrony.d/spinifex.conf`.
- **`spx` under `sudo`.** Oracle Linux's `sudo` path leaves out `/usr/local/bin`, so the installer links `/usr/bin/spx` to `/usr/local/bin/spx`.
-
-The kernel modules list (`/etc/modules-load.d/spinifex.conf`) and the host firewall below apply on every OS.
-
-## Step 3. Replace firewalld Rules You Depend On
-
-The installer stops and masks firewalld, because with it running a packet must pass both firewalld's table and the Spinifex one, which silently breaks routed NAT, Elastic IPs and instance metadata.
-When firewalld was active, the installer turns on the Spinifex nftables policy in its place; pass `--firewall=on` or `--firewall=off` to choose yourself.
-
-firewalld's rules are not saved or migrated.
-The Spinifex policy filters inbound traffic only and opens 53, 443, 3000, 8443 and 9999 to any source.
-It accepts SSH from every IPv4 source until `$trusted_ssh_peers` in `/etc/spinifex/firewall/custom.nft` narrows it.
-Restate any other restriction you had, such as SSH limited to a management subnet, in that file; [Host Firewall](/docs/host-firewall) shows how.
-
-From the moment firewalld stops until the daemon first loads the Spinifex policy, the host accepts every connection.
-Uninstalling Spinifex leaves firewalld unmasked but disabled; it does not turn it back on.
-
-## Step 4. Keep the Host Updated
-
-Do not enable `dnf-automatic`.
-An Open vSwitch package upgrade restarts OVS, which interrupts guest networking, so update `openvswitch3.5`, `ovn25.03*` and `qemu-kvm*` in a maintenance window.
-Stay on the current Oracle Linux 10 minor release, which is what Spinifex is tested on.
-
-Before rebooting into a new kernel, check that its modules are installed, or OVS cannot start after the reboot:
+### Set up OVN networking

 ```bash
-rpm -q kernel-uek-core kernel-uek-modules
+sudo /usr/local/share/spinifex/setup-ovn.sh --management
 ```

-Each `kernel-uek-core` version needs a `kernel-uek-modules` of the same version; install a missing one with `sudo dnf install kernel-uek-modules-<version>`.
+### Initialize

-Track Oracle Linux security advisories (ELSA) for the host, the kernel and QEMU, for example with `dnf updateinfo list --security`.
-Those advisories do not cover the NFV SIG packages, so follow Open vSwitch and OVN security announcements for those.
+```bash
+sudo spx admin init --node node1 --nodes 1
+```

-## Step 5. Uninstall
+### Start services

-`/usr/local/share/spinifex/uninstall-spx.sh` removes Spinifex as on Debian.
-On a cluster node it also stops `openvswitch-ipsec.service` and deletes the `ovs_cert*` entries from libreswan's NSS database, so the node can join a different cluster; other NSS identities and your connection files in `/etc/ipsec.d` stay.
-It lists what it leaves in place on Oracle Linux: the repository changes and the SIG key, `OVS_USER_ID` and `OVN_USER_ID`, the OVS and OVN logrotate edits, the chrony include, `/etc/modules-load.d/spinifex.conf`, the SELinux mode, a masked `ipsec.service`, firewalld disabled, and any GPU passthrough configuration.
-`--purge-deps` is refused on Oracle Linux, because dnf cannot tell the packages Spinifex installed from the host's own; remove them with `dnf remove` yourself.
+```bash
+sudo systemctl start spinifex.target
+```
+
+### Verify
+
+```bash
+export AWS_PROFILE=spinifex
+aws ec2 describe-instance-types
+```
+
+If this returns a list of instance types, Spinifex is installed.
+Continue to [Setting Up Your Cluster](/docs/setting-up-your-cluster) to import an AMI and launch your first instance.
+:::
+
+**For a cluster**, prepare and install every node with the steps above, then continue with [Multi-Node Install](/docs/install-multi-node) from Step 2.
+Every node must run Oracle Linux.
+IPsec between nodes runs on libreswan, so open UDP 500, UDP 4500 and ESP between nodes on any firewall between them.
+
+## What the Installer Changes
+
+On Oracle Linux the installer also:
+
+- Enables `ol10_UEKR8` and `ol10_kvm_utils`, and excludes `qemu-kvm*` and `qemu-img*` from `ol10_appstream`.
+- Adds the NFV SIG repository as `/etc/yum.repos.d/spinifex-nfv-sig.repo`, limited to the OVS and OVN packages.
+- Installs `kernel-uek-modules` for the running kernel, about 800 MB with `linux-firmware`.
+- Switches DNS to systemd-resolved behind NetworkManager where resolved is installed but not running.
+- Links `/usr/bin/spx` to `/usr/local/bin/spx`, so `sudo spx` works.
+
+### firewalld
+
+The installer stops and masks firewalld and turns on the Spinifex nftables policy in its place; firewalld's rules are not migrated.
+The Spinifex policy opens 53, 443, 3000, 8443, 9999 and SSH to any source.
+Restate any restriction you relied on, such as SSH limited to a management subnet, in `/etc/spinifex/firewall/custom.nft`; see [Host Firewall](/docs/host-firewall).
+
+### Updates
+
+Do not enable `dnf-automatic`: an Open vSwitch upgrade restarts OVS and interrupts guest networking, so update `openvswitch3.5`, `ovn25.03*` and `qemu-kvm*` in a maintenance window.
+Before rebooting into a new kernel, check `rpm -q kernel-uek-core kernel-uek-modules` shows matching versions, or OVS cannot start.
+
+### Uninstall
+
+`/usr/local/share/spinifex/uninstall-spx.sh` removes Spinifex and lists what it leaves in place.
+`--purge-deps` is refused on Oracle Linux; remove packages with `dnf remove` yourself.

 ## Troubleshooting

-### Installer stops: SELinux is enforcing
+### Installer stops on SELinux, the kernel or kernel modules

-Set SELinux permissive now and in `/etc/selinux/config`, as in [SELinux](#selinux), then run the installer again.
-Fixing only one of the two is refused too.
-
-### Installer stops: the running kernel is not UEK R8
-
-Install `kernel-uek`, reboot into it, check `uname -r`, then run the installer again.
-
-### Installer stops: kernel-uek-modules is no longer in the repositories
-
-The running kernel is older than anything the repository still carries modules for.
+Set SELinux permissive both now and in `/etc/selinux/config`, or update and boot into the current UEK R8 kernel, then run the installer again:

 ```bash
 sudo dnf update -y kernel-uek
@@ -229,41 +171,30 @@ sudo reboot

 ### Installer stops: a repository is unreachable

-The message names the repository.
-Check the host can reach `yum.oracle.com` and `mirror.stream.centos.org` over HTTPS, including through any proxy dnf uses, then run the installer again.
+Check the host can reach `yum.oracle.com` and `mirror.stream.centos.org` over HTTPS, including through any proxy dnf uses.

 ### DHCP on br-wan times out

-Check STP is off on the bridge:
+STP is on. Turn it off:

 ```bash
-nmcli -g bridge.stp connection show br-wan
+sudo nmcli connection modify br-wan bridge.stp no
+sudo nmcli connection up br-wan
 ```

-If it prints `yes`, run `sudo nmcli connection modify br-wan bridge.stp no` and `sudo nmcli connection up br-wan`.
-
-### `sudo spx` finds no command
-
-Check that `/usr/bin/spx` links to `/usr/local/bin/spx`; run the installer again if it does not, or call `sudo /usr/local/bin/spx` meanwhile.
-
 ### Cluster nodes form no IPsec tunnels

-Check that both IPsec services run, and that libreswan holds the node's certificate and the cluster CA:
-
 ```bash
 systemctl is-active ipsec.service openvswitch-ipsec.service
-sudo certutil -L -d sql:/var/lib/ipsec/nss     # ovs_certkey_<node>, ovs_cert_cacert
 sudo ipsec trafficstatus                       # one line per peer tunnel
 ```

-If the services are inactive, check the daemon log with `journalctl -u spinifex-daemon` for the IPsec reconcile.
-If the certificates are present but no tunnel is listed, check that UDP 500 and ESP pass between the nodes in both directions, and look for `SEC_ERROR_UNKNOWN_ISSUER` in `journalctl -u ipsec`, which means a peer's certificate is from another CA.
-Until the tunnel forms, guest traffic between those two nodes is dropped, which is the guard in [Clusters](#clusters) working.
+If the services are inactive, check `journalctl -u spinifex-daemon`.
+If no tunnel is listed, check UDP 500 and ESP pass between the nodes in both directions.
+Guest traffic between two nodes is dropped until their tunnel forms.

 ### CA certificate not trusted

-`sudo spx admin init` installs the CA automatically. To install it again by hand on Oracle Linux:
-
 ```bash
 sudo cp /etc/spinifex/ca.pem /etc/pki/ca-trust/source/anchors/spinifex-ca.crt
 sudo update-ca-trust
@@ -633,7 +633,7 @@ ssh -i ~/.ssh/oci-spx ubuntu@"$(head -1 .validate-vm-multi/hosts)" \
 If this returns a list of available instance types, your installation is working.

 <p align="center">
-  <img src="../../.github/assets/spinifex-ui.jpg" alt="Spinifex Web UI" width="900">
+  <img src="../../.github/assets/dashboard.webp" alt="Spinifex Web UI" width="900">
 </p>


@@ -16,16 +16,16 @@ Create, update, delete and policy association complete synchronously within the
 |---|---|---|
 | Identity | Cluster name plus principal ARN. Key and ARN suffix use `sha256hex(principalARN)` (`domains/eks/access/store.go` `PrincipalARNHash`). ARN `arn:aws:eks:{region}:{account}:access-entry/{cluster}/{hash}` (`foundation/aws/arn/eks.go`); the AWS ARN shape differs and is not verified. | `handlers/eks/service_impl_test.go` `TestAccessEntry_CreateDescribeListDelete`, `domains/eks/access/store_test.go` `TestKeyPaths`, `gateway/eks/authz_test.go` `TestResourceARNsAccessEntryMatchesHandler` |
 | Scope | Account and Region, within one cluster: keys live under `clusters/{cluster}/access-entries/` in the `eks-account-{account}` bucket. | `handlers/eks/access_lifecycle_test.go` `TestAccessEntryPersistedLayout_TopLevelAndNestedFieldNames`, `TestAccessEntryRecord_DecodesHandWrittenJSON`, `handlers/eks/create_cluster_test.go` `TestCreateCluster_SeedsCreatorAdminAccessEntry` |
-| Durable owner and record | `domains/eks/access` (`Owner`) is the only writer of `Record` with nested `AssociatedPolicy` and `Scope`: Create, SeedCreatorAdmin (cluster launch), Update, Associate, Disassociate and Delete. Keys and field names are unchanged by the move. `DeleteClusterPrefix` (`handlers/eks/cluster_state.go`) still erases entries with the cluster, and the gateway process reads records through the exported `access.Get` in `ResolveTokenReview` (`handlers/eks/token_review.go`). | `handlers/eks/access_lifecycle_test.go` `TestAccessEntryPersistedLayout_TopLevelAndNestedFieldNames`, `TestAccessEntryRecord_DecodesHandWrittenJSON`, `domains/eks/access/owner_test.go`, `domains/eks/access/store_test.go`, `handlers/eks/create_cluster_test.go` `TestCreateCluster_SeedsCreatorAdminAccessEntry` |
+| Durable owner and record | `domains/eks/access` (`Owner`) is the only writer of `Record` with nested `AssociatedPolicy` and `Scope`: Create, SeedCreatorAdmin (cluster launch), Update, Associate, Disassociate and Delete. Keys and field names are unchanged by the move. `DeleteClusterChildren` (`domains/eks/cluster/cluster_state.go`) still erases entries with the cluster, and the gateway process reads records through the exported `access.Get` in `ResolveTokenReview` (`handlers/eks/token_review.go`). | `handlers/eks/access_lifecycle_test.go` `TestAccessEntryPersistedLayout_TopLevelAndNestedFieldNames`, `TestAccessEntryRecord_DecodesHandWrittenJSON`, `domains/eks/access/owner_test.go`, `domains/eks/access/store_test.go`, `handlers/eks/create_cluster_test.go` `TestCreateCluster_SeedsCreatorAdminAccessEntry` |
 | AWS adapter | `handlers/eks` keeps request validation (entry type, supported policy, scope), the cluster existence check (`acctKVForCluster`), awserrors mapping and the AWS projection. `gateway/eks/authz.go` uses `access.PrincipalARNHash`. | `handlers/eks/service_impl_test.go` access cases, `gateway/eks/authz_test.go` `TestResourceARNsAccessEntryMatchesHandler` |
 | Desired state | Principal, Kubernetes username and groups, type (`STANDARD` only), tags (set at create only) and cluster-scoped access policy associations. | `handlers/eks/service_impl_test.go` `TestAccessEntry_RejectsNodeType`, `TestAccessPolicy_AssociateRejectsUnsupportedPolicyAndScope`, `TestAccessPolicy_AssociateRejectsNamespaceScope`, `handlers/eks/access_lifecycle_test.go` `TestAccessEntry_UpdateWithNoFieldsKeepsUsernameAndGroups` |
 | Generation | None. Update, associate and disassociate retry against the KV revision; create and delete do not. | |
 | Observed state | Not tracked; the record is the only state. | |
 | Readiness | Usable by token review as soon as the write returns. | `handlers/eks/token_review_test.go` `TestResolveTokenReview_AuthenticatesViaVerifyAndKV`, `TestResolveTokenReview_NoAccessEntryDenies`, `handlers/eks/service_impl_test.go` `TestAccessPolicy_DisassociateRemovesProjectedGroup` |
 | Idempotency | `clientRequestToken` is ignored. Update always advances `modifiedAt`, even when nothing changes. A duplicate create returns `ResourceInUseException`; a repeated delete returns `ResourceNotFoundException`; disassociating an unbound policy succeeds without change. Internal NATS redelivery of create returns `ResourceInUseException` to the redelivered request. | `handlers/eks/service_impl_test.go` `TestAccessEntry_CreateDescribeListDelete`, `TestAccessEntry_DescribeDeleteMissingIsNotFound`, `handlers/eks/access_lifecycle_test.go` `TestAccessPolicy_DisassociateUnassociatedLeavesRecordUnchanged`, `TestAccessPolicy_AssociateWithNoEntryIsNotFound` |
-| Dependencies | Every action requires only that the cluster's meta record exists (`acctKVForCluster`); cluster status is not read, so actions succeed on a `CREATING`, `FAILED` or `DELETING` cluster. Token review does not check the cluster at all. The principal is not checked against IAM. | `handlers/eks/service_impl_test.go` `TestAccessEntry_UnknownClusterIsNotFound`, `handlers/eks/access_lifecycle_test.go` `TestAccessEntryActionsIgnoreClusterLifecycleStatus`, `TestResolveTokenReview_AuthenticatesWithoutClusterMeta` |
+| Dependencies | Every action requires that the cluster's meta record exists (`acctKVForCluster`). `CreateAccessEntry` also refuses a `DELETING` cluster with `ResourceInUseException`, since a recorded delete admits no new dependant; no other status is read, so actions succeed on a `CREATING` or `FAILED` cluster. Token review does not check the cluster at all. The principal is not checked against IAM. | `handlers/eks/service_impl_test.go` `TestAccessEntry_UnknownClusterIsNotFound`, `handlers/eks/access_lifecycle_test.go` `TestAccessEntryActionsIgnoreFailedClusterStatus`, `delete_cluster_async_test.go` `TestS2_EKSDeletingClusterAdmitsNoNewDependants`, `TestResolveTokenReview_AuthenticatesWithoutClusterMeta` |
 | Authorization | Gateway authz derives the access-entry ARN with `PrincipalARNHash` (`gateway/eks/authz.go`). No `iam:PassRole`. The STS step of token review is the existing `eks.VerifyToken` NATS contract served by awsgw. | `gateway/eks/authz_test.go` `TestResourceARNsAccessEntryMatchesHandler` |
-| Deletion ordering | Delete is get-then-delete, not conditional. Cluster delete erases entries in the prefix purge only after all infrastructure teardown succeeds; on failure they survive while the cluster stays `DELETING`. | `handlers/eks/access_lifecycle_test.go` `TestDeleteClusterPrefix_SweepsAccessEntriesScopedToCluster` |
+| Deletion ordering | Delete is get-then-delete, not conditional. Cluster delete erases entries with the cluster's other child records, before the meta, only after all infrastructure teardown succeeds; on failure they survive while the cluster stays `DELETING`. | `handlers/eks/access_lifecycle_test.go` `TestDeleteClusterChildren_SweepsAccessEntriesScopedToCluster` |
 | Restart and interruption | The prefix purge deletes keys in listing order; an interruption after the meta key is gone leaves entries that a new cluster of the same name inherits. Reclaiming a `FAILED` cluster and a failed launch purge with the meta kept, so the recreated cluster inherits the failed attempt's entries. | none |
 | End-to-end | `tests/e2e/eks/eks_test.go` `AccessEntry` and `GetToken`; `tests/integration/cross_tenant_test.go`. No Terraform apply/destroy evidence for `aws_eks_access_entry` or `aws_eks_access_policy_association`. | |

@@ -33,7 +33,7 @@ Create, update, delete and policy association complete synchronously within the

 1. Create is get-then-put with no exclusive create, so concurrent creates for one principal both succeed and the later write wins.
 2. The creator-admin seed at cluster launch is an unconditional put after the cluster meta is final, so it replaces an entry for the same principal created while the cluster was `CREATING`, including its policy associations.
-3. Access actions do not check cluster status (see dependencies row), and opening the bucket creates it for an account that has none.
+3. Access actions other than `CreateAccessEntry` do not check cluster status (see dependencies row), and opening the bucket creates it for an account that has none.
 4. Entries can outlive their cluster (interrupted prefix purge) or carry over to a recreated cluster (reclaim and failed launch keep the meta and do not purge entries separately).
 5. Token review authenticates against an entry without checking the cluster exists or its status.
 6. `ResolveTokenReview` opens the EKS bucket and reads the record in the gateway process through the exported `access.Get`, outside the owner.
@@ -16,11 +16,11 @@ There is no generation, so the record mixes desired fields with the latest obser
 | Item | Present behaviour | Evidence |
 |---|---|---|
 | Identity | Cluster name plus add-on name; one add-on per name per cluster. ARN `arn:aws:eks:{region}:{account}:addon/{cluster}/{addon}` (`foundation/aws/arn/eks.go` `FormatEKSAddon`), deterministic by name, with no per-incarnation suffix. The gateway authorizer derives the same ARN (`domains/eks/awsapi/authz.go` `addonARN`). | `handlers/eks/addon_lifecycle_test.go` `TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames`, `domains/eks/awsapi/authz_test.go` `TestResourceARNsFidelity` |
-| Scope | Account and Region, within one cluster: keys live under `clusters/{cluster}/addons/` in the `eks-account-{account}` bucket. | `TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames`, `TestDeleteClusterPrefix_SweepsAddonRecordsAndManifestsScopedToCluster` |
+| Scope | Account and Region, within one cluster: keys live under `clusters/{cluster}/addons/` in the `eks-account-{account}` bucket. | `TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames`, `TestDeleteClusterChildren_SweepsAddonRecordsAndManifestsScopedToCluster` |
 | Durable record | `clusters/{cluster}/addons/{addon}` holds `Record` (`domains/eks/addon/record.go`): `addonName`, `addonVersion`, `status`, `serviceAccountRoleArn`, `configurationValues`, `health`, `arn`, `tags`, `createdAt`, `modifiedAt` (the optional ones `omitempty`). `clusters/{cluster}/addons/{addon}/manifest` holds `Manifest` (`domains/eks/addon/record.go`): `addonName`, `addonVersion`, `serviceAccountRoleArn`, `configurationValues`. Listing skips the manifest sub-key by segment count. | `TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames`, `TestAddonRecord_DecodesHandWrittenJSON`, `domains/eks/addon/store_test.go` `TestRecord_RoundTrip`, `TestList_SkipsManifestSubKeys` |
-| Writers | Record: the owner in `domains/eks/addon/owner.go`, reached from the `CreateAddon`, `UpdateAddon` and `DeleteAddon` handlers, the cluster reconciler's report relay and the GPU node group launch: `Owner.Create` (unconditional put), `Owner.Update` and its failure marking (revision CAS), `Owner.Delete` (get then delete), `Owner.ApplyReport` (revision CAS), `Owner.EnsureGPUDevicePlugin` through `Owner.Create`; and the cluster prefix purge (`DeleteClusterPrefix`). Manifest: `StagingInstaller.Install` (unconditional put on create and update), `StagingInstaller.Uninstall` (`domains/eks/addon/delivery.go`), `Owner.Delete` and the prefix purge. | `TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames`, `TestUpdateAddon_ReportsSuccessfulWhileRecordIsUpdating`, `TestDeleteAddon_ReturnsDeletingAfterErasingRecordAndManifest`, `TestDeleteClusterPrefix_SweepsAddonRecordsAndManifestsScopedToCluster`, `domains/eks/addon/store_test.go` `TestCasUpdate`, `TestDeleteRecord_RemovesRecordAndManifest` |
+| Writers | Record: the owner in `domains/eks/addon/owner.go`, reached from the `CreateAddon`, `UpdateAddon` and `DeleteAddon` handlers, the cluster reconciler's report relay and the GPU node group launch: `Owner.Create` (unconditional put), `Owner.Update` and its failure marking (revision CAS), `Owner.Delete` (get then delete), `Owner.ApplyReport` (revision CAS), `Owner.EnsureGPUDevicePlugin` through `Owner.Create`; and the cluster prefix purge (`DeleteClusterChildren`). Manifest: `StagingInstaller.Install` (unconditional put on create and update), `StagingInstaller.Uninstall` (`domains/eks/addon/delivery.go`), `Owner.Delete` and the prefix purge. | `TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames`, `TestUpdateAddon_ReportsSuccessfulWhileRecordIsUpdating`, `TestDeleteAddon_ReturnsDeletingAfterErasingRecordAndManifest`, `TestDeleteClusterChildren_SweepsAddonRecordsAndManifestsScopedToCluster`, `domains/eks/addon/store_test.go` `TestCasUpdate`, `TestDeleteRecord_RemovesRecordAndManifest` |
 | Readers | `ListAddons`, `DescribeAddon`, `CreateAddon` (existence check) and the reconciler read the record. `ListStagedAddonManifests` reads every `/manifest` key for the cluster (`addon.ListManifests`); it is served on `eks.ListStagedAddonManifests` (`daemon/daemon.go`) and reached only through the gateway's internal-addons route. | `handlers/eks/addons_test.go` `TestAddon_DescribeListDelete`, `TestListStagedAddonManifests` |
-| Catalogue | Static in-binary catalogue (`domains/eks/addon/catalog.go`): `aws-load-balancer-controller`, `argocd`, `aws-ebs-csi-driver`, and hidden-but-creatable `nvidia-device-plugin` and `spinifex-noop`; one version each, the first being the default. Create and update reject unknown names and versions with `InvalidParameterValue`. | `domains/eks/addon/catalog_test.go`, `handlers/eks/addons_test.go` `TestDescribeAddonVersions_ReturnsCatalog`, `TestCreateAddon_UnknownAddonRejected`, `TestCreateAddon_UnknownVersionRejected`, `TestUpdateAddon_UnknownVersionRejected` |
+| Catalogue | Static in-binary catalogue (`domains/eks/addon/catalog.go`): `aws-load-balancer-controller`, `argocd`, `aws-ebs-csi-driver`, and hidden-but-creatable `nvidia-device-plugin` and `spinifex-noop`; one version each, the first being the default. Versions are stored and used internally in the bare upstream form, and API responses show the AWS form `v<x.y.z>-eksbuild.<n>` (`Spec.AWSVersion`). Create and update accept the AWS form or the exact bare form and reject unknown names and other version strings with `InvalidParameterValue`, which the gateway returns as `InvalidParameterException`. | `domains/eks/addon/catalog_test.go`, `handlers/eks/addons_test.go` `TestDescribeAddonVersions_ReturnsCatalog`, `TestCreateAddon_UnknownAddonRejected`, `TestCreateAddon_UnknownVersionRejected`, `TestUpdateAddon_UnknownVersionRejected` |
 | Desired state | Version, `serviceAccountRoleArn`, `configurationValues` and create-time tags. `resolveConflicts`, `podIdentityAssociations` and `clientRequestToken` are accepted and dropped; `preserve` on delete is ignored. `configurationValues` reaches the guest but the agent discards it (`mulga-eks-addon-sync.sh` `sync_once`), so it has no effect. | `TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames`, `TestDeleteAddon_ReturnsDeletingAfterErasingRecordAndManifest` |
 | Observed state | `status` and `health` are overwritten by guest reports on the same record as the desired fields. `health` is the latest failure message, projected as one `AddonIssue` with only `message`. Statuses held: `CREATING`, `ACTIVE`, `UPDATING`, `DEGRADED`, `CREATE_FAILED`; `DELETING` is defined but never stored. | `TestAddonRecord_DecodesHandWrittenJSON`, `handlers/eks/addon_status_reconciler_test.go` `TestApplyAddonStatusReport` |
 | Generation | None. Reports carry `version` and `ts`, and neither is read: any report for the add-on name applies to whatever is currently desired. | `TestAddonStatusReport_MismatchedVersionIsAccepted` |
@@ -37,9 +37,9 @@ There is no generation, so the record mixes desired fields with the latest obser
 | Cross-resource dependency | A GPU node group's launch calls `stageGPUDeviceAddon`, which checks the cluster exists and calls `Owner.EnsureGPUDevicePlugin` for `nvidia-device-plugin`; an existing record is a no-op and every other error is logged only. Deleting the node group never unstages it, and an existing record in any status, including `CREATE_FAILED`, blocks re-staging. | `handlers/eks/nodegroup_test.go` `TestCreateNodegroup_GPUNodegroupStagesDevicePluginAddon`, `domains/eks/addon/owner_test.go` `TestOwnerEnsureGPUDevicePlugin_StagesDefaultVersionOnceAndKeepsAnyExistingRecord`, `TestCreateNodegroup_NonGPUNodegroupDoesNotStageDevicePluginAddon`, `tests/e2e/gpu/eks_gpu_test.go` `TestEKSGPUPodExposure` |
 | Idempotency | `clientRequestToken` is ignored. A duplicate create, including an SDK retry with the same token, returns `ResourceInUseException`. A repeated update re-stages and returns another `Successful` with the same id. A repeated delete returns `ResourceNotFoundException`. | `TestCreateAddon_RepeatedClientRequestTokenIsResourceInUse`, `TestUpdateAddon_ReportsSuccessfulWhileRecordIsUpdating`, `TestDeleteAddon_ReturnsDeletingAfterErasingRecordAndManifest`, `handlers/eks/addons_test.go` `TestCreateAddon_CreatesStagingAndDefaultsVersion` |
 | Absent resource | An unknown cluster returns `ResourceNotFoundException` for every action, including the internal manifest listing. A missing add-on returns `ResourceNotFoundException` from describe, update and delete. | `handlers/eks/addons_test.go` `TestCreateAddon_UnknownClusterIsNotFound`, `TestDescribeDeleteAddon_MissingIsNotFound`, `TestUpdateAddon_MissingIsNotFound`, `TestListStagedAddonManifests` |
-| Dependencies | Only the cluster meta's existence is checked; cluster status is not read, so add-ons can be created on a `DELETING` cluster. | `TestAddonActionsIgnoreClusterLifecycleStatus` |
+| Dependencies | `CreateAddon` checks the cluster meta exists and refuses a `DELETING` cluster with `ResourceInUseException`, since a recorded delete admits no new dependant; other statuses are not read. | `TestCreateAddonRefusesOnlyADeletingCluster` |
 | Authorization | Gateway policy uses the add-on ARN above. `serviceAccountRoleArn` requires `iam:PassRole` on `CreateAddon` and `UpdateAddon` (`domains/eks/awsapi/passrole.go`), the same check `CreateCluster`/`CreateNodegroup` apply, including that the role exists. Tag actions on an add-on ARN return `NotImplemented`. | `domains/eks/awsapi/gateway_wire_test.go` `TestPassedRoleARNs_AddonServiceAccountRoleIsChecked`, `domains/eks/awsapi/gateway_passrole_test.go` `TestEKSCreateAddon_PassRole`, `TestEKSUpdateAddon_PassRole`, `domains/eks/awsapi/authz_test.go` `TestResourceARNsFidelity` |
-| Cluster deletion | Add-on records and manifests are erased only by the cluster prefix purge, after the cluster's infrastructure teardown succeeds; there is no per-add-on delete. Reclaiming a failed cluster keeps the meta and does not purge, so the recreated cluster inherits the add-ons. | `TestDeleteClusterPrefix_SweepsAddonRecordsAndManifestsScopedToCluster`, `handlers/eks/cluster_state_test.go` `TestDeleteClusterPrefix_SweepsEveryKey` |
+| Cluster deletion | Add-on records and manifests are erased only by the cluster prefix purge, after the cluster's infrastructure teardown succeeds; there is no per-add-on delete. Reclaiming a failed cluster keeps the meta and does not purge, so the recreated cluster inherits the add-ons. | `TestDeleteClusterChildren_SweepsAddonRecordsAndManifestsScopedToCluster`, `handlers/eks/cluster_state_test.go` `TestDeleteClusterChildren_SweepsEveryKey` |
 | Restart and interruption | Create interrupted between the record put and staging leaves a `CREATING` record with no manifest that nothing re-stages and that blocks a retry. Delete interrupted after unstaging leaves a record in its previous status while the guest removes the software. An update interrupted after the CAS leaves `UPDATING` with the previous manifest. A daemon restart loses nothing durable, but reports sent while no reconciler runs are lost until the next tick. | none |
 | End-to-end | `tests/e2e/eks/eks_test.go` `TestEKS/AddonDelivery` (create to `ACTIVE`, objects present, delete removes the namespace) and `tests/e2e/gpu/eks_gpu_test.go` `TestEKSGPUPodExposure`; `tests/integration/cross_tenant_test.go` `TestCrossTenant_EveryScopedActionIsAccountedFor`. No E2E for update, failure, configuration or the role. No Terraform apply/destroy evidence for `aws_eks_addon`. | |

@@ -0,0 +1,42 @@
+# EKS cluster: delete lifecycle contract
+
+Contract under ADR-0003 S1 for the delete half of an EKS cluster's lifecycle, the half ADR-0003 S8 constrains.
+Create, update and the control-plane reconciler are out of its scope and keep the observed gaps recorded elsewhere.
+Paths are relative to `spinifex/` unless they start with `tests/`, which is relative to the spinifex repo root.
+Test names without a path are in `domains/eks/cluster/`.
+
+## Scope
+
+A cluster is a composite resource: its record at `clusters/{name}/meta` in the `eks-account-{account}` bucket names the dependants a delete must remove.
+In the system account those are the control-plane VMs and their root volumes, the endpoint load balancer with its target groups, the managed control-plane VPC and its network, and the spread placement group.
+In the customer account they are the egress address, the private-endpoint interface and security group, the cluster security groups, a cluster-owned internet gateway, and load balancers the in-cluster controller created.
+Node group workers and CSI-provisioned EBS volumes are never removed by a cluster delete.
+
+## Contract
+
+| Item | Behaviour | Evidence |
+|---|---|---|
+| Request | `DeleteCluster` validates, answers `ResourceNotFoundException` for an absent cluster, answers a `DELETING` cluster from its record, refuses a cluster with node groups with `ResourceInUseException`, records `DELETING` and returns. It waits for no teardown step. | `TestS8_EKSDeleteClusterReturnsBeforeTheTeardownCompletes`, `TestRLC1_EKSDeleteClusterNotFoundOnAbsent`, `TestS8_EKSDeleteClusterRefusesWhileNodegroupsExist`, `tests/e2e/eks` `TestEKS/DeleteCluster` |
+| Obligation | `DELETING` with `DeletingSince` on the record is the tenant's decision. No status writer moves a record off it, and a create launch still running adds its refs to it instead of overwriting it. | `TestS2_EKSDeletingIsStickyForStatusWriters`, `TestS2_EKSLaunchWriteAfterDeleteNeitherRevertsNorRecreates` |
+| Retry | A repeat delete answers `DELETING` from the record and starts nothing. | `TestS8_EKSSecondDeleteAnswersFromTheRecordWithoutASecondTeardown` |
+| Convergence | `convergeClusterDeletion` (`domains/eks/cluster/cluster_teardown.go`), started by the request and backed by `EKSDeletingReaper`, runs under a per-cluster in-process gate and the cross-node teardown lease, renewed on the service context. It acts only on a record that is `DELETING` when re-read. | `TestS8_EKSKickAndSameNodeSweepRunOneTeardown`, `TestS8_EKSTeardownLeaseOutlivesTheRequest`, `TestS7_EKSTeardownActsOnlyOnARecordedDelete` |
+| Retained cleanup | A failed step keeps the record `DELETING` with its refs and the error. The record is removed last, after its child records, and only if no ref appeared that the pass did not tear down. A record with a node group is never removed; a record that is not `DELETING` is refused before anything else is checked. | `TestRLC2_EKSBillableTeardownBeforeKVSweep`, `TestS2_EKSTeardownRemovesTheMetaLast`, `TestS2_EKSTeardownDoesNotSweepARecordChangedDuringTeardown`, `TestS7_EKSTeardownNeverRemovesAClusterWithNodegroupWorkers` |
+| Recovery | A teardown interrupted by a crash or a lost lease is finished by the reaper on any node. Failed passes back off from 90 s, doubling to a 30 min cap, and are never abandoned; `DeleteReapExhausted` only flags a long failure for an operator. | `TestS2_EKSTeardownResumesAfterAnInterruptedConverge`, `TestRLC4_DeletingReaperReDrivesWedgedTeardown`, `TestDeletingReaper_KeepsRetryingPastTheOldAttemptLimit`, `TestDeletingReaper_ResumesAnAlreadyExhaustedRecord` |
+| Visibility | `DescribeCluster` and `ListClusters` show the cluster `DELETING` until the record is removed, then `ResourceNotFoundException`. A failed pass shows one `INTERNAL_FAILURE` health issue naming the step, never the internal error. | `TestDescribeCluster_StuckDeleteShowsTheStepNotTheError`, `TestRLC2_EKSBillableTeardownBeforeKVSweep` |
+| New dependants | `CreateNodegroup` requires `ACTIVE`; `CreateAddon` and `CreateAccessEntry` refuse a `DELETING` cluster with `ResourceInUseException`. | `TestCreateAddonRefusesOnlyADeletingCluster`, `TestS2_EKSDeletingClusterAdmitsNoNewDependants` |
+| Subject budgets | Every `eks.*` request subject is classified by its handler's work in `eksSubjects` (`domains/eks/cluster/service_nats.go`) and keeps the 30 s budget; `eks.DeleteCluster` is request-scaled. A subject the gateway sends or the daemon serves that is not classified fails the table. | `TestS8_EKSSubjectBudgets` |
+| Late child records | No child record outlives the cluster or leaks into a later one of the same name. The teardown sweeps child keys again after removing the meta and stops the cluster's bootstrap subscriber, bootstrap refuses to write into a record that is gone or `DELETING`, the launch re-checks its record around seeding the creator's access entry, and a fresh claim clears whatever an earlier cluster left under the name. | `TestS2_EKSLaunchOvertakenByDeleteLeavesNoChildRecords`, `TestS2_EKSBootstrapWritesNothingIntoADeletingCluster`, `TestS2_EKSFreshClaimInheritsNoStaleChildRecords` |
+| Launch generation | The create launch's writes are fenced on the record's generation (`CreatedAt` and ARN). An overtaken launch never writes a newer cluster of the same name and releases only the instances it started, by ID. One whose record was swept recreates it as `DELETING` with its refs, so the teardown and reaper own the release. One whose record was marked `FAILED` merges its refs and releases what it built. | `TestS2_EKSOvertakenLaunchNeverWritesANewerClusterOfTheSameName`, `TestS2_EKSOvertakenLaunchHandsItsInfraToTheTeardown`, `TestS2_EKSLaunchMarkedFailedReleasesWhatItBuilt` |
+| Running launch | A create launch heartbeats its owner and time on the record (`LaunchOwner`, `LaunchHeartbeat`). The teardown keeps the record while that heartbeat is fresh, so the cluster reads `DELETING` throughout and never `ResourceNotFoundException` then `DELETING` again; the launch hands its refs over at its next write and starts the teardown when it returns. A heartbeat older than 90 s is taken as a crashed launch and no longer waited for, and recreating a swept record stays as the backstop. The launch's OIDC keypair writes are fenced on the record's generation. | `TestS2_EKSDeletedClusterNeverReadsMissingWhileItsLaunchRuns`, `TestS2_EKSCrashedLaunchDoesNotPinTheDelete`, `TestS2_EKSOvertakenLaunchCannotReplaceANewerClustersSigningKey` |
+| Bounded passes | One converger run is bounded at 15 minutes, so a hung step cannot hold the gate and lease; the timed-out pass is recorded and backed off. Waiting on node groups is not counted as a failed pass, and deleting the last node group resumes the delete. | `TestS8_EKSHungTeardownStepReleasesTheGate`, `TestS2_EKSLastNodegroupDeletedResumesTheClusterDelete` |
+| FAILED reclaim | Re-creating a `FAILED` cluster's name tears the old attempt down under the same gate and lease as a delete, and answers `ResourceInUseException` while another holder has it; the per-node teardown bound does not apply to it. | `TestS2_EKSFailedClusterReclaimRunsUnderTheTeardownLease`, `TestS2_EKSFailedClusterReclaimIsNotRefusedForCapacity` |
+| Tenant data | The teardown never deletes a CSI-provisioned volume or terminates a node group worker. | `TestS7_EKSTeardownNeverDeletesCSIVolumesOrWorkers`, `TestS7_EKSTeardownNeverRemovesAClusterWithNodegroupWorkers` |
+
+## Gaps
+
+1. Three `eks.*` handlers still hold their request for work that grows with the resource, and the subject table names them as not conformant: `CreateCluster` (reclaiming a `FAILED` cluster of the same name tears its infra down inline), `UpdateNodegroupConfig` (one awaited launch or terminate per worker added or removed) and `RestoreSnapshot` (launches a control-plane VM and fences the old one inline).
+2. AWS's answer to a repeat delete on a `DELETING` cluster has not been observed; `DELETING` from the record is the chosen behaviour.
+3. Control-plane, load balancer and VPC resources in the system account are keyed by the bare cluster name, so a teardown by name can reach another tenant's same-named cluster.
+4. The teardown deletes load balancers the in-cluster controller created in the customer account, which AWS leaves behind.
+5. An overtaken launch's late write can still land under a newer cluster of the same name in the instant between that cluster's claim sweep and the old launch's generation check; it is bounded by one write, since the old launch stops at its next fenced write.
+6. The `FAILED`-name reclaim purge still runs inside the `CreateCluster` request (gap 1).
--- a/Show More
+++ b/Show More