feat(ui): console feedback v1 and the admin Nodes view (#1454)

* feat(ui): console feedback round: state tones, inline tags, filter bar and form fixes

Settled states go green and transitions yellow with a spinner, status text is
lowercase, tags are edited inline row by row, every list filters through one
filter bar, long cells truncate with a tooltip and times are relative.

EC2 launch takes a Name and offers ICMP echo, and leaves Placement Group out
when no group is available. RDS create is reordered with engine tiles and
window pickers. Route 53 records get bulk delete, a TTL select, value labels
that follow the type, and NS. Tooltips use the popover tokens, and the console
is sized over a 110% root.

* feat(ui): size RDS instance classes and give each record value its own input

An RDS class is a db.-prefixed EC2 type, so the class picker shows its vCPUs
and memory in every option and under the field, as the EC2 launch form does,
on create, modify and restore.

A Route 53 record's values were one textarea, which let a line break into a
value. Each value is now a single-line input with Add value; Enter starts the
next one, a pasted list splits into rows, and CNAME takes exactly one.

* fix(ui): open a DNS record for editing from the keyboard

* feat: admin node inventory, topology, live host metrics and network test

Each node collects its hardware facts as root at daemon start (dmidecode,
ethtool, networkctl LLDP where available) into /etc/spinifex/node-facts, and
the daemon serves them with its plane wiring over NATS.

New admin-only actions, refused for every account but the system admin
account regardless of policy:

- GetNodeFacts: hardware, NICs, LLDP neighbours and planes per node
- GetNodeMetrics: live CPU, memory and per-plane traffic for one node
- StartNetworkTest / DescribeNetworkTests: iperf2 between nodes over a
  chosen plane, both directions, one test cluster-wide at a time, recorded
  in KV and converged asynchronously

The iperf listener runs only for the duration of a run, without
capabilities, on TCP 5001, which the peers firewall rule now allows. A run
refuses any target outside the node's own plane subnet.

The console gains a node panel (hardware, network, metrics, network test
with a dial scaled to the port's line rate) and a topology view built with
@xyflow/react. setup.sh installs dmidecode and iperf.

* test: cover the node insight handlers and the facts file mode

* feat: report every CPU flag in node facts

* feat(ui): node table on the dashboard, every CPU flag and a readable topology

The dashboard's nodes section is the Nodes page table, open by default, and a
row opens that node. Ready reads green, capacity sorts by most available, and
the processor's features tip lists every flag. Topology draws one row per
port with switches in a muted card, highlights a node's links on hover, and
offers a left-to-right layout beside top-down.

* chore(ui): rebuild dist

* fix: align the CPU facts fields

* fix: bound the network test server start and read ARM CPU features and cores

* chore(ui): rebuild dist

* test(ui): pin the instances baseline clock and take the screenshots from CI's Playwright image

* fix: retry a network test claim whose holder released it mid-check

.gitattributes

@@ -0,0 +1,3 @@
+# The committed bundle carries the AWS SDK's template strings verbatim,
+# including trailing spaces that are part of their values.
+spinifex/runtime/roles/spinifexui/frontend/dist/** -whitespace

.github/assets/diagrams/tf-nginx-alb.svg

@@ -19,8 +19,8 @@
   </defs>
   <rect width="900" height="480" fill="url(#bg)" rx="12"/>
 
-  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">NGINX + ALB · VPC WITH PUBLIC AND PRIVATE SUBNETS</text>
-  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">10.20.0.0/16 · multi-AZ</text>
+  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">NGINX + ALB · PUBLIC SUBNETS · SPREAD ACROSS NODES</text>
+  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">10.20.0.0/16 · server_count = 3</text>
 
   <!-- Internet -->
   <rect x="350" y="54" width="200" height="40" rx="8" fill="url(#inet-grad)" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.55"/>
@@ -41,58 +41,45 @@
   <line x1="230" y1="184" x2="230" y2="202" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
   <line x1="670" y1="184" x2="670" y2="202" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
 
-  <!-- public_a -->
   <g transform="translate(108, 202)">
-    <rect width="244" height="104" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
+    <rect width="244" height="150" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
     <text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">public_a</text>
-    <text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">az-a</text>
+    <text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">10.20.1.0/24</text>
     <line x1="16" y1="32" x2="228" y2="32" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.5"/>
-
-    <rect x="20" y="42" width="200" height="22" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="120" y="57" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">ALB ENI</text>
-
-    <rect x="20" y="70" width="200" height="22" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="120" y="85" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">NAT GW</text>
+    <rect x="22" y="42" width="200" height="22" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="122" y="57" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">ALB ENI</text>
+    <rect x="22" y="70" width="200" height="36" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="122" y="86" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">nginx-alb-1</text>
+    <text x="122" y="100" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">public IP · Ubuntu · cloud-init</text>
+    <rect x="22" y="110" width="200" height="36" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="122" y="126" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">nginx-alb-3</text>
+    <text x="122" y="140" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">public IP · Ubuntu · cloud-init</text>
   </g>
 
-  <!-- public_b -->
   <g transform="translate(548, 202)">
-    <rect width="244" height="104" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
+    <rect width="244" height="150" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
     <text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">public_b</text>
-    <text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">az-b</text>
+    <text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">10.20.2.0/24</text>
     <line x1="16" y1="32" x2="228" y2="32" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.5"/>
-
-    <rect x="20" y="56" width="200" height="22" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="120" y="71" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">ALB ENI</text>
+    <rect x="22" y="42" width="200" height="22" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="122" y="57" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">ALB ENI</text>
+    <rect x="22" y="70" width="200" height="36" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="122" y="86" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">nginx-alb-2</text>
+    <text x="122" y="100" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">public IP · Ubuntu · cloud-init</text>
   </g>
 
-  <!-- SNAT arrow from public_a NAT → private row -->
-  <line x1="230" y1="306" x2="230" y2="340" stroke="#F87171" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.65"/>
-  <polygon points="226,336 234,336 230,344" fill="#F87171" opacity="0.8"/>
-  <rect x="256" y="314" width="240" height="20" rx="10" fill="#0a0a0a" stroke="#F87171" stroke-width="0.6" stroke-opacity="0.55"/>
-  <text x="376" y="328" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#FCA5A5" text-anchor="middle">SNAT for private subnets (egress)</text>
-
-  <!-- private_a -->
-  <g transform="translate(108, 350)">
-    <rect width="244" height="104" rx="8" fill="url(#priv-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.55"/>
-    <text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#c084fc" font-weight="700">private_a</text>
-    <text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#d8b4fe" text-anchor="end">az-a</text>
-    <line x1="16" y1="32" x2="228" y2="32" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.5"/>
-
-    <rect x="20" y="44" width="200" height="48" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="120" y="64" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#d8b4fe" text-anchor="middle" font-weight="700">nginx_1</text>
-    <text x="120" y="80" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">Debian 12 · cloud-init</text>
-  </g>
-
-  <!-- private_b -->
-  <g transform="translate(548, 350)">
-    <rect width="244" height="104" rx="8" fill="url(#priv-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.55"/>
-    <text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#c084fc" font-weight="700">private_b</text>
-    <text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#d8b4fe" text-anchor="end">az-b</text>
-    <line x1="16" y1="32" x2="228" y2="32" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.5"/>
-
-    <rect x="20" y="44" width="200" height="48" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="120" y="64" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#d8b4fe" text-anchor="middle" font-weight="700">nginx_2</text>
-    <text x="120" y="80" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">Debian 12 · cloud-init</text>
-  </g>
+  <!-- spread placement group -->
+  <rect x="108" y="372" width="684" height="88" rx="8" fill="url(#priv-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.55"/>
+  <text x="128" y="394" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#c084fc" font-weight="700">nginx-alb-spread</text>
+  <text x="772" y="394" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#d8b4fe" text-anchor="end">placement group · strategy spread · one server per node</text>
+  <line x1="124" y1="402" x2="776" y2="402" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.5"/>
+  <rect x="128" y="412" width="204" height="36" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
+  <text x="230" y="428" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#d8b4fe" text-anchor="middle" font-weight="700">node1</text>
+  <text x="230" y="442" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">one web server</text>
+  <rect x="348" y="412" width="204" height="36" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
+  <text x="450" y="428" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#d8b4fe" text-anchor="middle" font-weight="700">node2</text>
+  <text x="450" y="442" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">one web server</text>
+  <rect x="568" y="412" width="204" height="36" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
+  <text x="670" y="428" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#d8b4fe" text-anchor="middle" font-weight="700">node3</text>
+  <text x="670" y="442" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">one web server</text>
 </svg>

.github/assets/diagrams/vultr-deployment.svg

@@ -0,0 +1,110 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="580" viewBox="0 0 900 580">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+    <linearGradient id="vpc-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#58a6ff14"/>
+      <stop offset="100%" style="stop-color:#58a6ff05"/>
+    </linearGradient>
+    <linearGradient id="node-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#3fb9501f"/>
+      <stop offset="100%" style="stop-color:#3fb95008"/>
+    </linearGradient>
+    <linearGradient id="svc-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#a855f71f"/>
+      <stop offset="100%" style="stop-color:#a855f708"/>
+    </linearGradient>
+    <marker id="arrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
+      <path d="M0,0 L10,5 L0,10 z" fill="#71717a"/>
+    </marker>
+  </defs>
+  <rect width="900" height="580" fill="url(#bg)" rx="12"/>
+  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">SPINIFEX ON VULTR · ONE NODE OR THREE</text>
+  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">--nodes 1 or 3 · vx1-g-16c-64g or bare metal</text>
+  <rect x="30" y="52" width="250" height="56" rx="8" fill="#e4e4e70d" stroke="#e4e4e7" stroke-width="1.1" stroke-opacity="0.6"/>
+  <text x="155.0" y="76" font-family="'SF Mono', ui-monospace, monospace" font-size="12.5" fill="#e4e4e7" text-anchor="middle" font-weight="700">Your machine</text>
+  <text x="155.0" y="94" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">deploy-vultr.sh · OpenTofu or Terraform</text>
+  <rect x="325" y="52" width="250" height="56" rx="8" fill="#3fb9500d" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.6"/>
+  <text x="450.0" y="76" font-family="'SF Mono', ui-monospace, monospace" font-size="12.5" fill="#3fb950" text-anchor="middle" font-weight="700">Internet</text>
+  <text x="450.0" y="94" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#7ee787" text-anchor="middle">customers reach your EC2 and load balancers</text>
+  <rect x="620" y="52" width="250" height="56" rx="8" fill="#58a6ff12" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
+  <text x="745.0" y="76" font-family="'SF Mono', ui-monospace, monospace" font-size="12.5" fill="#79c0ff" text-anchor="middle" font-weight="700">Vultr API</text>
+  <text x="745.0" y="94" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a5d6ff" text-anchor="middle">VPC · servers · reserved IPs</text>
+  <rect x="30" y="160" width="840" height="400" rx="10" fill="url(#vpc-grad)" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.55" stroke-dasharray="6,4"/>
+  <text x="50" y="184" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#79c0ff" font-weight="700">Vultr VPC</text>
+  <text x="850" y="184" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#58a6ff" text-anchor="end">private network 10.10.8.0/24</text>
+  <line x1="155" y1="108" x2="155" y2="158" stroke="#71717a" stroke-width="1.2" opacity="0.8" marker-end="url(#arrow)"/>
+  <text x="165" y="138" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" text-anchor="start">Terraform builds · SSH installs</text>
+  <line x1="450" y1="108" x2="450" y2="158" stroke="#71717a" stroke-width="1.2" opacity="0.8" marker-end="url(#arrow)"/>
+  <text x="460" y="138" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" text-anchor="start">reserved IPs</text>
+  <line x1="745" y1="158" x2="745" y2="110" stroke="#71717a" stroke-width="1.2" opacity="0.8" marker-end="url(#arrow)"/>
+  <text x="735" y="138" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" text-anchor="end">Spinifex calls the API</text>
+  <rect x="60" y="205" width="250" height="118" rx="8" fill="url(#node-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.6"/>
+  <text x="78" y="229" font-family="'SF Mono', ui-monospace, monospace" font-size="12.5" fill="#3fb950" font-weight="700">node1</text>
+  <text x="292" y="229" font-family="'Segoe UI', system-ui, sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">Ubuntu 24.04 · Spinifex</text>
+  <line x1="74" y1="237" x2="296" y2="237" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.6"/>
+  <rect x="76" y="250" width="74" height="24" rx="5" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
+  <text x="113.0" y="266" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle">AWS API</text>
+  <rect x="160" y="250" width="60" height="24" rx="5" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
+  <text x="190.0" y="266" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle">EBS</text>
+  <rect x="230" y="250" width="60" height="24" rx="5" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
+  <text x="260.0" y="266" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle">S3</text>
+  <text x="185" y="300" font-family="'Segoe UI', system-ui, sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">VPC networking · your instances</text>
+  <rect x="325" y="205" width="250" height="118" rx="8" fill="url(#node-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.6" stroke-dasharray="5,3"/>
+  <text x="343" y="229" font-family="'SF Mono', ui-monospace, monospace" font-size="12.5" fill="#3fb950" font-weight="700">node2</text>
+  <text x="557" y="229" font-family="'Segoe UI', system-ui, sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">with --nodes 3</text>
+  <line x1="339" y1="237" x2="561" y2="237" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.6"/>
+  <rect x="341" y="250" width="74" height="24" rx="5" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
+  <text x="378.0" y="266" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle">AWS API</text>
+  <rect x="425" y="250" width="60" height="24" rx="5" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
+  <text x="455.0" y="266" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle">EBS</text>
+  <rect x="495" y="250" width="60" height="24" rx="5" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
+  <text x="525.0" y="266" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle">S3</text>
+  <text x="450" y="300" font-family="'Segoe UI', system-ui, sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">VPC networking · your instances</text>
+  <rect x="590" y="205" width="250" height="118" rx="8" fill="url(#node-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.6" stroke-dasharray="5,3"/>
+  <text x="608" y="229" font-family="'SF Mono', ui-monospace, monospace" font-size="12.5" fill="#3fb950" font-weight="700">node3</text>
+  <text x="822" y="229" font-family="'Segoe UI', system-ui, sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">with --nodes 3</text>
+  <line x1="604" y1="237" x2="826" y2="237" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.6"/>
+  <rect x="606" y="250" width="74" height="24" rx="5" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
+  <text x="643.0" y="266" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle">AWS API</text>
+  <rect x="690" y="250" width="60" height="24" rx="5" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
+  <text x="720.0" y="266" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle">EBS</text>
+  <rect x="760" y="250" width="60" height="24" rx="5" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
+  <text x="790.0" y="266" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle">S3</text>
+  <text x="715" y="300" font-family="'Segoe UI', system-ui, sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">VPC networking · your instances</text>
+  <line x1="310" y1="264" x2="325" y2="264" stroke="#58a6ff" stroke-width="1.6" stroke-dasharray="3,3" opacity="0.9"/>
+  <line x1="575" y1="264" x2="590" y2="264" stroke="#58a6ff" stroke-width="1.6" stroke-dasharray="3,3" opacity="0.9"/>
+  <text x="450" y="348" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#79c0ff" text-anchor="middle">nodes cluster over the private network: control plane, EBS replicas and S3 shards</text>
+  <rect x="60" y="370" width="780" height="166" rx="8" fill="url(#svc-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.55"/>
+  <text x="80" y="394" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#c084fc" font-weight="700">AWS services across the fleet</text>
+  <text x="820" y="394" font-family="'Segoe UI', system-ui, sans-serif" font-size="9.5" fill="#d8b4fe" text-anchor="end">spread across the nodes; with three, moved to a survivor if one fails</text>
+  <line x1="76" y1="402" x2="824" y2="402" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.6"/>
+  <rect x="76" y="418" width="86" height="52" rx="6" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
+  <text x="119.0" y="440" font-family="'SF Mono', ui-monospace, monospace" font-size="11.5" fill="#d8b4fe" text-anchor="middle" font-weight="700">EC2</text>
+  <text x="119.0" y="457" font-family="'Segoe UI', system-ui, sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">instances</text>
+  <rect x="169" y="418" width="86" height="52" rx="6" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
+  <text x="212.0" y="440" font-family="'SF Mono', ui-monospace, monospace" font-size="11.5" fill="#d8b4fe" text-anchor="middle" font-weight="700">ALB / NLB</text>
+  <text x="212.0" y="457" font-family="'Segoe UI', system-ui, sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">load balancers</text>
+  <rect x="262" y="418" width="86" height="52" rx="6" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
+  <text x="305.0" y="440" font-family="'SF Mono', ui-monospace, monospace" font-size="11.5" fill="#d8b4fe" text-anchor="middle" font-weight="700">EKS</text>
+  <text x="305.0" y="457" font-family="'Segoe UI', system-ui, sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">Kubernetes</text>
+  <rect x="355" y="418" width="86" height="52" rx="6" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
+  <text x="398.0" y="440" font-family="'SF Mono', ui-monospace, monospace" font-size="11.5" fill="#d8b4fe" text-anchor="middle" font-weight="700">ECS</text>
+  <text x="398.0" y="457" font-family="'Segoe UI', system-ui, sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">containers</text>
+  <rect x="448" y="418" width="86" height="52" rx="6" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
+  <text x="491.0" y="440" font-family="'SF Mono', ui-monospace, monospace" font-size="11.5" fill="#d8b4fe" text-anchor="middle" font-weight="700">RDS</text>
+  <text x="491.0" y="457" font-family="'Segoe UI', system-ui, sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">databases</text>
+  <rect x="541" y="418" width="86" height="52" rx="6" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
+  <text x="584.0" y="440" font-family="'SF Mono', ui-monospace, monospace" font-size="11.5" fill="#d8b4fe" text-anchor="middle" font-weight="700">S3</text>
+  <text x="584.0" y="457" font-family="'Segoe UI', system-ui, sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">object storage</text>
+  <rect x="634" y="418" width="86" height="52" rx="6" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
+  <text x="677.0" y="440" font-family="'SF Mono', ui-monospace, monospace" font-size="11.5" fill="#d8b4fe" text-anchor="middle" font-weight="700">EBS</text>
+  <text x="677.0" y="457" font-family="'Segoe UI', system-ui, sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">volumes</text>
+  <rect x="727" y="418" width="86" height="52" rx="6" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
+  <text x="770.0" y="440" font-family="'SF Mono', ui-monospace, monospace" font-size="11.5" fill="#d8b4fe" text-anchor="middle" font-weight="700">IAM</text>
+  <text x="770.0" y="457" font-family="'Segoe UI', system-ui, sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">access</text>
+  <text x="450" y="500" font-family="'Segoe UI', system-ui, sans-serif" font-size="10.5" fill="#a1a1aa" text-anchor="middle">Use the AWS CLI, SDKs and Terraform against your own endpoint, unmodified.</text>
+  <text x="450" y="520" font-family="'Segoe UI', system-ui, sans-serif" font-size="9.5" fill="#71717a" text-anchor="middle">Public addresses are Vultr reserved IPs, attached to the node each resource runs on.</text>
+</svg>

.github/e2e-cells.json

@@ -8,7 +8,8 @@
     "arch": "x86_64",
     "runner": "ci-single",
     "extra": "",
-    "suite": ""
+    "suite": "",
+    "timeout_minutes": 60
   },
   {
     "cell": 2,
@@ -19,7 +20,8 @@
     "arch": "x86_64",
     "runner": "ci-single",
     "extra": "",
-    "suite": ""
+    "suite": "",
+    "timeout_minutes": 60
   },
   {
     "cell": 3,
@@ -30,7 +32,8 @@
     "arch": "x86_64",
     "runner": "ci-single",
     "extra": "",
-    "suite": ""
+    "suite": "",
+    "timeout_minutes": 60
   },
   {
     "cell": 8,
@@ -165,7 +168,7 @@
     "runner": "ci-multi",
     "extra": "",
     "suite": "instancerecovery",
-    "timeout_minutes": 105
+    "timeout_minutes": 125
   },
   {
     "cell": 32,
@@ -190,5 +193,61 @@
     "extra": "",
     "suite": "kvquorum",
     "timeout_minutes": 55
+  },
+  {
+    "cell": 34,
+    "install": "binary",
+    "topo": "single",
+    "net": "dhcp",
+    "host": "oracle-10",
+    "arch": "x86_64",
+    "runner": "ci-single",
+    "extra": "",
+    "suite": "",
+    "timeout_minutes": 60
+  },
+  {
+    "cell": 35,
+    "install": "binary",
+    "topo": "single",
+    "net": "static",
+    "host": "oracle-10",
+    "arch": "x86_64",
+    "runner": "ci-single",
+    "extra": "reboot-single",
+    "suite": ""
+  },
+  {
+    "cell": 36,
+    "install": "binary",
+    "topo": "single",
+    "net": "nat",
+    "host": "oracle-10",
+    "arch": "x86_64",
+    "runner": "ci-single",
+    "extra": "nat-single",
+    "suite": ""
+  },
+  {
+    "cell": 37,
+    "install": "binary",
+    "topo": "real-multi",
+    "net": "dhcp",
+    "host": "oracle-10",
+    "arch": "x86_64",
+    "runner": "ci-multi",
+    "extra": "",
+    "suite": ""
+  },
+  {
+    "cell": 38,
+    "install": "binary",
+    "topo": "real-multi",
+    "net": "nat",
+    "host": "oracle-10",
+    "arch": "x86_64",
+    "runner": "ci-multi",
+    "extra": "nat-multi",
+    "suite": ""
   }
 ]

.github/pull_request_template.md

@@ -16,6 +16,10 @@
 - Evidence added or rechecked:
 - ADR-0008 lifecycle register / ADR-0001 migration-record / bead updates:
 
+## Mulga design
+
+<!-- One line, so a reviewer knows whether the design records in the private mulga repository need to change. Write either "Updated: <mulga PR or bead id>" or "Not needed: <reason>". This is a reminder, not a gate. -->
+
 ## Community Contribution License
 
 - [ ] All community contributions in this pull request are licensed to Mulga under the terms of the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0). By creating this pull request I represent that I have the right to license these contributions to Mulga under the Apache License 2.0.

.github/scripts/hypervisor-disk-preflight.sh

@@ -0,0 +1,66 @@
+#!/usr/bin/env bash
+# Usage: hypervisor-disk-preflight.sh <env.tfvars>
+#
+# Fails fast when a hypervisor named in the tofu env is short of disk, before
+# `tofu apply` tries to define a domain there. libvirt writes each domain's XML
+# under /etc/libvirt/qemu on the root filesystem and its disks into the pool, and
+# a full one fails the apply minutes in as "Domain Creation Failed".
+#
+# Hosts are reached the way the libvirt provider reaches them (ssh <user>@<name>,
+# resolved through ~/.ssh/config). A host that cannot be reached is reported and
+# skipped: tofu's own error for it is clearer than anything said here.
+#
+# Env: HV_SSH_USER (tf-user), HV_MIN_ROOT_FREE_MB (1024), HV_MIN_POOL_FREE_GB (20).
+set -uo pipefail
+
+tfvars="${1:?usage: $0 <env.tfvars>}"
+user="${HV_SSH_USER:-tf-user}"
+min_root_kb=$(( ${HV_MIN_ROOT_FREE_MB:-1024} * 1024 ))
+min_pool_kb=$(( ${HV_MIN_POOL_FREE_GB:-20} * 1024 * 1024 ))
+
+if [ ! -r "$tfvars" ]; then
+  echo "::error::hypervisor disk preflight: cannot read $tfvars" >&2
+  exit 1
+fi
+
+# One node per line in every env file: { name = "...", ..., storage_pool = "..." }.
+pairs=$(grep -v '^[[:space:]]*#' "$tfvars" |
+  sed -nE 's/.*name[[:space:]]*=[[:space:]]*"([^"]+)".*storage_pool[[:space:]]*=[[:space:]]*"([^"]+)".*/\1 \2/p')
+if [ -z "$pairs" ]; then
+  echo "::warning::hypervisor disk preflight: no nodes with a storage_pool in $tfvars; nothing checked"
+  exit 0
+fi
+
+# Prints "<root-avail-kb> <pool-avail-kb> <pool-path>" for the pool named in $1.
+# shellcheck disable=SC2016 # expanded on the hypervisor, not here
+remote='set -u
+root=$(df -Pk /etc/libvirt/qemu | awk "NR==2{print \$4}")
+path=$(virsh -c qemu:///system pool-dumpxml "$1" 2>/dev/null | sed -n "s:.*<path>\(.*\)</path>.*:\1:p" | head -n1)
+pool=-
+[ -n "$path" ] && pool=$(df -Pk "$path" | awk "NR==2{print \$4}")
+echo "$root $pool ${path:--}"'
+
+fail=0
+while read -r host pool; do
+  [ -n "$host" ] || continue
+  if ! out=$(ssh -o BatchMode=yes -o ConnectTimeout=10 "${user}@${host}" bash -s -- "$pool" <<<"$remote" 2>&1); then
+    echo "::warning::hypervisor disk preflight: cannot reach $host ($out); its disk is unchecked"
+    continue
+  fi
+  read -r root_kb pool_kb pool_path <<<"$(tail -n1 <<<"$out")"
+  if ! [[ "$root_kb" =~ ^[0-9]+$ ]]; then
+    echo "::warning::hypervisor disk preflight: unexpected answer from $host: $out"
+    continue
+  fi
+  echo "$host: root fs $((root_kb / 1024)) MiB free; pool $pool ($pool_path) $([[ "$pool_kb" =~ ^[0-9]+$ ]] && echo "$((pool_kb / 1024 / 1024)) GiB free" || echo "not found")"
+  if [ "$root_kb" -lt "$min_root_kb" ]; then
+    echo "::error::hypervisor $host has $((root_kb / 1024)) MiB free on the filesystem holding /etc/libvirt/qemu (need ${HV_MIN_ROOT_FREE_MB:-1024} MiB): libvirt cannot write a domain definition there. Free space on $host before re-running." >&2
+    fail=1
+  fi
+  if [[ "$pool_kb" =~ ^[0-9]+$ ]] && [ "$pool_kb" -lt "$min_pool_kb" ]; then
+    echo "::error::hypervisor $host has $((pool_kb / 1024 / 1024)) GiB free in storage pool $pool at $pool_path (need ${HV_MIN_POOL_FREE_GB:-20} GiB): the node's disk cannot grow. Free space on $host before re-running." >&2
+    fail=1
+  fi
+done <<<"$pairs"
+
+exit "$fail"

.github/scripts/hypervisor-disk-preflight_test.sh

@@ -0,0 +1,87 @@
+#!/bin/sh
+# hypervisor-disk-preflight_test.sh — drives hypervisor-disk-preflight.sh against
+# a stub ssh that answers for each hypervisor from a canned file, so every
+# verdict is checked without a real host.
+set -eu
+
+SCRIPT_DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
+PREFLIGHT="$SCRIPT_DIR/hypervisor-disk-preflight.sh"
+TMP=$(mktemp -d)
+trap 'rm -rf "$TMP"' EXIT
+
+fails=0
+pass() { echo "  ok   - $1"; }
+fail() { echo "  FAIL - $1"; fails=$((fails + 1)); }
+
+# The stub answers "<root-kb> <pool-kb> <path>" from $FAKE/<host>, or exits 255
+# when that host has no file, the way ssh does for a host it cannot reach.
+mkdir -p "$TMP/bin" "$TMP/hosts"
+cat > "$TMP/bin/ssh" <<'STUB'
+#!/bin/sh
+for a in "$@"; do case $a in *@*) host=${a#*@} ;; esac; done
+cat >/dev/null
+if [ -f "$FAKE/$host" ]; then cat "$FAKE/$host"; exit 0; fi
+echo "ssh: Could not resolve hostname $host" >&2
+exit 255
+STUB
+chmod +x "$TMP/bin/ssh"
+PATH="$TMP/bin:$PATH"
+FAKE="$TMP/hosts"
+export PATH FAKE
+
+cat > "$TMP/env.tfvars" <<'TFVARS'
+nodes = [
+  { name = "wattle", wan_ip = "192.0.2.1", storage_pool = "drive1-pool", base_image = "/x.qcow2" },
+  # { name = "retired", wan_ip = "192.0.2.9", storage_pool = "drive1-pool", base_image = "/x.qcow2" },
+  { name = "banksia", wan_ip = "192.0.2.2", storage_pool = "nvme1-pool", base_image = "/x.qcow2" },
+]
+TFVARS
+
+GiB=1048576
+plenty="$((50 * GiB)) $((200 * GiB)) /mnt/drive1/pool"
+
+# run <label> <want-rc> <want-text>
+run() {
+    rc=0
+    out=$(bash "$PREFLIGHT" "$TMP/env.tfvars" 2>&1) || rc=$?
+    if [ "$rc" -ne "$2" ]; then
+        fail "$1: exit $rc, want $2"
+        echo "$out" | sed 's/^/        /'
+        return
+    fi
+    case $out in
+        *"$3"*) pass "$1" ;;
+        *) fail "$1: output lacks '$3'"; echo "$out" | sed 's/^/        /' ;;
+    esac
+}
+
+echo "-- hypervisor-disk-preflight.sh"
+
+echo "$plenty" > "$FAKE/wattle"
+echo "$plenty" > "$FAKE/banksia"
+run "every host has room" 0 "banksia: root fs 51200 MiB free"
+
+echo "0 $((200 * GiB)) /mnt/drive1/pool" > "$FAKE/wattle"
+run "a full root filesystem fails fast" 1 "::error::hypervisor wattle has 0 MiB free on the filesystem holding /etc/libvirt/qemu"
+
+echo "$plenty" > "$FAKE/wattle"
+echo "$((50 * GiB)) $((3 * GiB)) /mnt/nvme1/pool" > "$FAKE/banksia"
+run "a nearly full pool fails fast" 1 "::error::hypervisor banksia has 3 GiB free in storage pool nvme1-pool"
+
+echo "$plenty" > "$FAKE/banksia"
+rm -f "$FAKE/wattle"
+run "an unreachable host is reported, not failed" 0 "::warning::hypervisor disk preflight: cannot reach wattle"
+
+echo "$plenty" > "$FAKE/wattle"
+echo "$((50 * GiB)) - -" > "$FAKE/banksia"
+run "a pool libvirt does not know is reported, not failed" 0 "pool nvme1-pool (-) not found"
+
+echo "0 0 /x" > "$FAKE/retired"
+echo "$plenty" > "$FAKE/banksia"
+run "a commented-out node is never checked" 0 "wattle: root fs"
+
+if [ "$fails" -ne 0 ]; then
+    echo "$fails failure(s)"
+    exit 1
+fi
+echo "  hypervisor-disk-preflight ok"

.github/scripts/mulga_design_check.py

@@ -0,0 +1,36 @@
+#!/usr/bin/env python3
+"""Warn when a PR body has no filled "Mulga design" line. Reads PR_BODY; never fails the job."""
+
+import os
+import re
+
+COMMENT_RE = re.compile(r"<!--.*?-->", re.S)
+SECTION_RE = re.compile(r"^#{1,6}\s*Mulga design\s*$", re.I)
+LINE_RE = re.compile(r"^\s*(Updated|Not needed)\s*:\s*(.*?)\s*$", re.I)
+PLACEHOLDER_RE = re.compile(r"<[^<>]*>")
+
+
+def filled(body):
+    inside = False
+    for line in COMMENT_RE.sub("", body or "").splitlines():
+        if SECTION_RE.match(line):
+            inside = True
+        elif inside and line.lstrip().startswith("#"):
+            return False
+        elif inside:
+            m = LINE_RE.match(line)
+            if m and PLACEHOLDER_RE.sub("", m.group(2)).strip():
+                return True
+    return False
+
+
+def main():
+    if filled(os.environ.get("PR_BODY", "")):
+        print("The Mulga design line is filled in.")
+    else:
+        print('::warning title=Mulga design::Add a line under "## Mulga design": "Updated: <mulga PR or bead id>" or "Not needed: <reason>".')
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

.github/scripts/suite-sets.sh

@@ -117,8 +117,9 @@ e2e_suite_timeout() {
     # storage case waits for a pause measured at 4m47s and then holds four
     # minutes to prove nothing moved. Three tests, each taking a node away and
     # bringing it back, and the partition case adds a lease validity, a lease
-    # TTL and a boot on top of everything the other two wait for.
-    instancerecovery) echo "90m" ;;
+    # TTL and a boot on top of everything the other two wait for. The data
+    # test lets its guest age past the pending watchdog twice, about 20 more.
+    instancerecovery) echo "110m" ;;
     # lbrecovery pays for a load balancer twice: once to stand one up with two
     # backends placed around it, and once for the recovery itself — a stale
     # owner, a lease, a boot, HAProxy's own start, and its health checks against
@@ -129,6 +130,10 @@ e2e_suite_timeout() {
     # budget on a node stopping and starting: seven units down, a raft election,
     # and the whole service stack back up with every stream caught up.
     kvquorum) echo "40m" ;;
+    # single is the broadest suite and finishes, it does not hang: passing runs
+    # took 24-29m, and an Oracle host ran out at 30m inside the last test's
+    # cleanup. The default left it no headroom at all.
+    single) echo "40m" ;;
     *) echo "30m" ;;
   esac
 }

.github/scripts/test_mulga_design_check.py

@@ -0,0 +1,23 @@
+import unittest
+
+from mulga_design_check import filled
+
+
+class FilledTest(unittest.TestCase):
+    def test_accepts_either_form(self):
+        self.assertTrue(filled("## Mulga design\nUpdated: mulga#275"))
+        self.assertTrue(filled("## Mulga design\n\nnot needed: internal refactor\n## Next"))
+
+    def test_rejects_unanswered(self):
+        self.assertFalse(filled(""))
+        self.assertFalse(filled("## Mulga design\n<!-- Updated: x -->"))
+        self.assertFalse(filled("## Mulga design\nUpdated: <mulga PR or bead id>"))
+        self.assertFalse(filled("## Mulga design\nUpdated:"))
+
+    def test_scoped_to_the_section(self):
+        self.assertFalse(filled("Updated: mulga#1"))
+        self.assertFalse(filled("## Mulga design\n## Other\nUpdated: mulga#1"))
+
+
+if __name__ == "__main__":
+    unittest.main()

.github/workflows/actions.lock

@@ -18,6 +18,11 @@ workflows:
         - 'actions/create-github-app-token@v3.3.0'
         - 'actions/upload-artifact@v7.0.2'
         - 'hashicorp/setup-terraform@v4.0.1'
+    '.github/workflows/e2e-cloudvendor-vultr.yml':
+        - 'actions/checkout@v7.0.1'
+        - 'actions/create-github-app-token@v3.3.0'
+        - 'actions/upload-artifact@v7.0.2'
+        - 'hashicorp/setup-terraform@v4.0.1'
     '.github/workflows/e2e-ddil.yml':
         - 'actions/checkout@v7.0.1'
         - 'actions/create-github-app-token@v3.3.0'
@@ -45,6 +50,8 @@ workflows:
     '.github/workflows/image-scripts.yml':
         - 'actions/checkout@v7.0.1'
         - 'actions/setup-go@v7.0.0'
+    '.github/workflows/mulga-design.yml':
+        - 'actions/checkout@v7.0.1'
     '.github/workflows/perf-3node.yml':
         - 'actions/checkout@v7.0.1'
         - 'actions/create-github-app-token@v3.3.0'
@@ -73,6 +80,7 @@ workflows:
     '.github/workflows/ui.yml':
         - 'actions/checkout@v7.0.1'
         - 'actions/setup-node@v7.1.0'
+        - 'actions/upload-artifact@v7.0.2'
         - 'pnpm/action-setup@v6.1.0'
 dependencies:
     'actions/cache@v6.1.0':

.github/workflows/apt-packages.yml

@@ -2,7 +2,8 @@ name: Apt Packages
 
 # Path-filtered like image-scripts.yml: this resolves setup.sh's runtime package
 # list against each supported suite's real indices, so a renamed, dropped, or
-# ambiguously virtual package fails here rather than on a user's machine.
+# ambiguously virtual package fails here rather than on a user's machine. The
+# dnf job does the same for Oracle Linux, through setup.sh's own repo setup.
 on:
   push:
     branches:
@@ -11,6 +12,7 @@ on:
     paths:
       - scripts/setup.sh
       - scripts/lint-apt-packages.sh
+      - scripts/lint-dnf-packages.sh
       - .github/workflows/apt-packages.yml
   pull_request:
     branches:
@@ -19,6 +21,7 @@ on:
     paths:
       - scripts/setup.sh
       - scripts/lint-apt-packages.sh
+      - scripts/lint-dnf-packages.sh
       - .github/workflows/apt-packages.yml
   workflow_dispatch:
 
@@ -44,3 +47,14 @@ jobs:
 
       - name: Resolve runtime package list
         run: ./scripts/lint-apt-packages.sh ${{ matrix.image }}
+
+  dnf_packages:
+    name: Resolve dnf list (oraclelinux:10)
+    runs-on: ubuntu-26.04
+
+    steps:
+      - name: Checkout Code
+        uses: actions/checkout@v7.0.1
+
+      - name: Resolve runtime package list
+        run: ./scripts/lint-dnf-packages.sh oraclelinux:10

.github/workflows/dev-release.yml

@@ -102,9 +102,9 @@ jobs:
         env:
           GH_TOKEN: ${{ github.token }}
 
-      - name: Build microVM artifacts (lb-agent + kernel + initramfs)
+      - name: Build microVM artifacts (lb-agent + kernel + initramfs + qboot + SQS store and worker images)
         if: steps.version.outputs.exists != 'true'
-        run: cd spinifex && make build-microvm-image
+        run: cd spinifex && make build-microvm-image build-sqs-worker-microvm-image
 
       - name: Build amd64 Tarball
         if: steps.version.outputs.exists != 'true'
@@ -144,6 +144,7 @@ jobs:
               --notes-file "$RUNNER_TEMP/notes.md" \
               "dist/spinifex-$TAG-linux-amd64.tar.gz" \
               "dist/spinifex-$TAG-linux-amd64.tar.gz.sha256" \
+              dist/qboot-*.tar.gz \
               "scripts/setup.sh"
 
       # The assertion that matters. A prerelease must never become the latest

.github/workflows/e2e-cloudvendor-vultr.yml

@@ -0,0 +1,142 @@
+name: E2E Cloud Vendor Vultr
+
+# On demand only. Deploys a ref to Vultr with deploy-vultr.sh, runs the same
+# published Terraform workbooks as the hypervisor nightly and OCI, and destroys
+# the deployment, failing if anything is left billing.
+on:
+  workflow_dispatch:
+    inputs:
+      nodes:
+        description: "1 or 3 nodes"
+        required: false
+        default: "1"
+        type: choice
+        options: ["1", "3"]
+      plan:
+        description: "Vultr plan: vx1-g-16c-64g (VM) or vbm-8c-64gb-amd (bare metal)"
+        required: false
+        default: vx1-g-16c-64g
+      region:
+        description: "Vultr region"
+        required: false
+        default: ord
+      workbooks:
+        description: "Space-separated workbooks (blank = the driver's full list)"
+        required: false
+      source:
+        description: "tree (build this ref) or release (the published installer)"
+        required: false
+        default: tree
+        type: choice
+        options: [tree, release]
+      channel:
+        description: "With source=release: latest, or dev for the newest prerelease"
+        required: false
+        default: latest
+      keep_on_fail:
+        description: "Leave a failed deployment up to inspect. It keeps billing."
+        required: false
+        type: boolean
+        default: false
+      runner:
+        description: "Runner label (blank = the VULTR_E2E_RUNNER variable, else ubuntu-26.04)"
+        required: false
+
+permissions:
+  contents: read
+
+# Deployments are named per run, but the account's reserved-IP quota is shared,
+# so a second dispatch queues rather than competing with the first.
+concurrency:
+  group: e2e-vultr
+  cancel-in-progress: false
+
+jobs:
+  vultr:
+    # Hosted by default: the job only drives Terraform and SSH. The API key's
+    # access control must admit this runner, so a hosted runner needs a key
+    # whose list holds 0.0.0.0/0; a self-hosted runner can use a fixed address.
+    runs-on: ${{ inputs.runner || vars.VULTR_E2E_RUNNER || 'ubuntu-26.04' }}
+    timeout-minutes: 240
+    env:
+      VULTR_NAME: e2e-${{ github.run_id }}
+      VULTR_NODES: ${{ inputs.nodes }}
+      VULTR_PLAN: ${{ inputs.plan }}
+      VULTR_REGION: ${{ inputs.region }}
+      VULTR_SOURCE: ${{ inputs.source }}
+      VULTR_CHANNEL: ${{ inputs.channel }}
+      VULTR_ARTIFACT_DIR: ${{ github.workspace }}/e2e-vultr
+      SPX_REF: ${{ github.ref_name }}
+    steps:
+      - uses: actions/checkout@v7.0.1
+        with:
+          path: spinifex
+          fetch-depth: 0
+
+      - uses: actions/create-github-app-token@v3.3.0
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      # build-install-artifacts.sh lives in mulga; the workbooks and their driver
+      # are in spinifex, so only the one directory is needed here.
+      - uses: actions/checkout@v7.0.1
+        with:
+          repository: mulgadc/mulga
+          path: mulga
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: scripts/tofu-cluster
+
+      - uses: hashicorp/setup-terraform@v4.0.1
+        with:
+          terraform_wrapper: false
+
+      - name: Run Vultr
+        working-directory: spinifex/scripts/terraform/vultr-spx
+        env:
+          VULTR_API_KEY: ${{ secrets.VULTR_API_KEY }}
+          MULGA_ROOT: ${{ github.workspace }}/mulga
+          # Through the environment, never interpolated into the script: a
+          # dispatch input is untrusted text to the shell.
+          WORKBOOKS_INPUT: ${{ inputs.workbooks }}
+          VULTR_KEEP_ON_FAIL: ${{ inputs.keep_on_fail && '1' || '0' }}
+        run: |
+          set -uo pipefail
+          if [ -n "$WORKBOOKS_INPUT" ]; then export WORKBOOKS="$WORKBOOKS_INPUT"; fi
+          ./e2e-vultr.sh 2>&1 | tee "$RUNNER_TEMP/e2e-vultr.log"
+          exit "${PIPESTATUS[0]}"
+
+      # always(): a cancelled or timed-out job is the one that would leave
+      # servers and reserved IPs billing. destroy works from this run's state.
+      - name: Sweep anything left behind
+        if: always() && !inputs.keep_on_fail
+        working-directory: spinifex/scripts/terraform/vultr-spx
+        env:
+          VULTR_API_KEY: ${{ secrets.VULTR_API_KEY }}
+        run: ./e2e-vultr.sh --sweep-only
+
+      - name: Publish the verdict
+        if: always()
+        run: |
+          {
+            echo "### Vultr: ${VULTR_NODES} x ${VULTR_PLAN} in ${VULTR_REGION}, ${VULTR_SOURCE} build of ${SPX_REF}"
+            echo '```'
+            cat "$VULTR_ARTIFACT_DIR/verdict.txt" 2>/dev/null || echo "no verdict written"
+            echo '```'
+          } >> "$GITHUB_STEP_SUMMARY"
+
+      # Logs and the verdict only. The throwaway SSH key and the Terraform state
+      # under ~/.spinifex never reach an artifact.
+      - name: Upload the run
+        if: always()
+        uses: actions/upload-artifact@v7.0.2
+        with:
+          name: e2e-vultr-${{ github.run_id }}
+          path: |
+            ${{ runner.temp }}/e2e-vultr.log
+            ${{ github.workspace }}/e2e-vultr/*.log
+            ${{ github.workspace }}/e2e-vultr/verdict.txt
+            ${{ github.workspace }}/e2e-vultr/node*/*.log
+          retention-days: 14

.github/workflows/e2e-nightly.yml

@@ -10,7 +10,7 @@ on:
   workflow_dispatch:
     inputs:
       cells:
-        description: "Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection, 30 routed NAT on three nodes, 31 instance recovery on three nodes, 32 load balancer recovery on three nodes, 33 KV bucket quorum under node loss on three nodes"
+        description: "Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 34-36 Oracle Linux single node (basic, reboot, routed NAT), 37-38 Oracle Linux on three nodes (multi-node subset with IPsec, routed NAT), 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection, 30 routed NAT on three nodes, 31 instance recovery on three nodes, 32 load balancer recovery on three nodes, 33 KV bucket quorum under node loss on three nodes"
         required: false
 
 permissions:
@@ -33,6 +33,7 @@ concurrency:
 env:
   GIT_BRANCH: ${{ github.ref_name }}
   UBUNTU_GOLD_IMAGE: /mnt/disk1/gold-images/spinifex-image-builder-ubuntu-26.04.qcow2
+  ORACLE_GOLD_IMAGE: /mnt/disk1/gold-images/spinifex-image-builder-oracle-10.qcow2
   SPINIFEX_OTLP_ENDPOINT: ${{ vars.SPINIFEX_OTLP_ENDPOINT || 'http://192.168.1.13:8200' }}
   SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }}
 
@@ -183,6 +184,14 @@ jobs:
             echo "running qemu: $(pgrep -c qemu-system-x86 || echo 0)"
           } >> "${ARTIFACT_DIR}/hypervisor-load.txt"
 
+      # A hypervisor out of disk fails the apply minutes in, as "Domain Creation
+      # Failed" from libvirt writing the domain XML. Say so before starting.
+      - name: Preflight hypervisor disk
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          "${GITHUB_WORKSPACE}/spinifex/.github/scripts/hypervisor-disk-preflight.sh" \
+            "envs/${SPINIFEX_CI_ENV}.tfvars"
+
       - name: Provision VMs
         working-directory: mulga/scripts/tofu-cluster
         env:
@@ -195,7 +204,8 @@ jobs:
           # back onto the debian gold.
           TOFU_VAR_ARGS=""
           case "$MATRIX_HOST" in
-            ubuntu-*) TOFU_VAR_ARGS="-var=host_image_override=${UBUNTU_GOLD_IMAGE}" ;;
+            ubuntu-*) TOFU_VAR_ARGS="-var=host_image_override=${UBUNTU_GOLD_IMAGE} -var=host_os=ubuntu" ;;
+            oracle-*) TOFU_VAR_ARGS="-var=host_image_override=${ORACLE_GOLD_IMAGE} -var=host_os=oracle" ;;
           esac
 
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
@@ -239,6 +249,11 @@ jobs:
           . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
           e2e_suites_assert_subset "e2e-nightly single" "${E2E_SUITES_NIGHTLY_SINGLE}" "${E2E_SUITES_SINGLE}"
 
+          # The select job sized this cell's budget from these same timeouts, so
+          # resolve them here and carry the pairs into the remote loop.
+          SUITE_TMOS=""
+          for s in ${E2E_SUITES_NIGHTLY_SINGLE}; do SUITE_TMOS="${SUITE_TMOS}${s}:$(e2e_suite_timeout "${s}") "; done
+
           INVENTORY=$(tofu output -json inventory)
           WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
@@ -248,9 +263,10 @@ jobs:
              for SUITE in ${E2E_SUITES_NIGHTLY_SINGLE}; do \
                echo \"::group::suite \$SUITE\"; \
                date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
+               TMO=30m; for kv in ${SUITE_TMOS}; do case \$kv in \$SUITE:*) TMO=\${kv#*:} ;; esac; done; \
                GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
                  ARTIFACT_DIR='${ARTIFACT_DIR}' \
-                 ./\$SUITE.test -test.v -test.timeout=30m \
+                 ./\$SUITE.test -test.v -test.timeout=\$TMO \
                  2>&1 | tee '${ARTIFACT_DIR}/test-'\$SUITE'.log'; \
                rc=\${PIPESTATUS[0]}; echo \"::endgroup::\"; \
                if [ \$rc -ne 0 ]; then echo \"\$SUITE: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; FAIL=1; fi; \
@@ -322,16 +338,20 @@ jobs:
             NODES_CSV="${NODES_CSV:+$NODES_CSV,}$ip"
           done
 
+          SUITE_TMOS=""
+          for s in ${E2E_SUITES_NIGHTLY_MULTI}; do SUITE_TMOS="${SUITE_TMOS}${s}:$(e2e_suite_timeout "${s}") "; done
+
           ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
             "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; FAIL=0; \
              for SUITE in ${E2E_SUITES_NIGHTLY_MULTI}; do \
                echo \"::group::suite \$SUITE\"; \
                date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
+               TMO=30m; for kv in ${SUITE_TMOS}; do case \$kv in \$SUITE:*) TMO=\${kv#*:} ;; esac; done; \
                GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
                  SPINIFEX_NODES='${NODES_CSV}' SPINIFEX_NODE_IPS='${NODES_CSV}' \
                  SPINIFEX_SSH_USER=tf-user SPINIFEX_SSH_KEY=\$HOME/.ssh/tf-user-ap-southeast-2 \
                  ARTIFACT_DIR='${ARTIFACT_DIR}' \
-                 ./\$SUITE.test -test.v -test.timeout=30m \
+                 ./\$SUITE.test -test.v -test.timeout=\$TMO \
                  2>&1 | tee '${ARTIFACT_DIR}/test-'\$SUITE'.log'; \
                rc=\${PIPESTATUS[0]}; echo \"::endgroup::\"; \
                if [ \$rc -ne 0 ]; then echo \"\$SUITE: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; FAIL=1; fi; \
@@ -687,7 +707,7 @@ jobs:
               'echo "host: $(hostname)"; echo "loadavg: $(cat /proc/loadavg)";
                grep -E "^(MemTotal|MemAvailable|SwapFree):" /proc/meminfo;
                echo "cpu(steal is field 8): $(grep "^cpu " /proc/stat)";
-               echo "running qemu: $(pgrep -c qemu-system-x86 || echo 0)"' 2>&1
+               echo "running qemu: $(pgrep -c "qemu-system-x86|qemu-kvm" || echo 0)"' 2>&1
           } >> "${ARTIFACT_DIR}/node-load.txt"
           exit 0
 
@@ -858,7 +878,8 @@ jobs:
           # different image path than apply did.
           TOFU_VAR_ARGS=""
           case "$MATRIX_HOST" in
-            ubuntu-*) TOFU_VAR_ARGS="-var=host_image_override=${UBUNTU_GOLD_IMAGE}" ;;
+            ubuntu-*) TOFU_VAR_ARGS="-var=host_image_override=${UBUNTU_GOLD_IMAGE} -var=host_os=ubuntu" ;;
+            oracle-*) TOFU_VAR_ARGS="-var=host_image_override=${ORACLE_GOLD_IMAGE} -var=host_os=oracle" ;;
           esac
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
@@ -1182,6 +1203,16 @@ jobs:
           sleep 2
           rm -f "$TMPDIR"/tmp.*.qcow2 "$TMPDIR"/tmp.*.log "$TMPDIR"/tmp.*-node.log
 
+      # The ISO build runs as root inside the workspace, and the next job on
+      # this runner, from any workflow, cannot remove what it leaves there.
+      - name: Release the workspace
+        if: always()
+        run: |
+          sudo umount -R -l "$GITHUB_WORKSPACE"/mulga/scripts/iso-builder/.cache/rootfs 2>/dev/null || true
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/mulga/scripts/iso-builder/.cache \
+              "$GITHUB_WORKSPACE"/mulga/scripts/iso-builder/dist
+          sudo chown -R "$(id -u):$(id -g)" "$GITHUB_WORKSPACE"
+
   iso-multi:
     name: cell-22 (iso/dhcp/three-node)
     # After cell-21 rather than beside it: same concurrency group, and running
@@ -1333,6 +1364,16 @@ jobs:
           rm -rf /tmp/cell22-artifacts-$GITHUB_RUN_ID /tmp/cell22-$GITHUB_RUN_ID.log \
                  "$WORK_DIR"
 
+      # The ISO build runs as root inside the workspace, and the next job on
+      # this runner, from any workflow, cannot remove what it leaves there.
+      - name: Release the workspace
+        if: always()
+        run: |
+          sudo umount -R -l "$GITHUB_WORKSPACE"/mulga/scripts/iso-builder/.cache/rootfs 2>/dev/null || true
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/mulga/scripts/iso-builder/.cache \
+              "$GITHUB_WORKSPACE"/mulga/scripts/iso-builder/dist
+          sudo chown -R "$(id -u):$(id -g)" "$GITHUB_WORKSPACE"
+
   # Predastore throughput ("cell 23"). Predastore has no runners of its own and
   # spinifex depends on it, so the benchmark lives here and checks predastore
   # out as a sibling.

.github/workflows/e2e.yml

@@ -536,6 +536,13 @@ jobs:
           log-dir: ${{ env.ARTIFACT_DIR }}
           title: "E2E failure analysis"
 
+      # The WAN capture is most of the artifact and only matters when a run
+      # failed. Anything short of a clean pass (suite rc, a failed setup or
+      # collect step, a missing rc) keeps it.
+      - name: Drop the packet capture from a passing single-node run
+        if: always() && steps.e2e.outputs.rc == '0' && job.status == 'success'
+        run: rm -f "${ARTIFACT_DIR}"/net-diag/wan.pcap
+
       - uses: actions/upload-artifact@v7.0.2
         if: always()
         with:
@@ -990,6 +997,13 @@ jobs:
           log-dir: ${{ env.ARTIFACT_DIR }}
           title: "Multi-Node E2E failure analysis"
 
+      # The WAN captures are 90% of the artifact and only matter when a run
+      # failed. Anything short of a clean pass (suite rc, a failed setup or
+      # collect step, a missing rc) keeps them.
+      - name: Drop the packet captures from a passing multi-node run
+        if: always() && steps.e2e.outputs.rc == '0' && job.status == 'success'
+        run: rm -f "${ARTIFACT_DIR}"/net-diag-node*/wan.pcap
+
       - uses: actions/upload-artifact@v7.0.2
         if: always()
         with:

.github/workflows/mulga-design.yml

@@ -0,0 +1,33 @@
+name: Mulga design
+
+# Reminds the author to say whether the Mulga design records need to change.
+# It reads only the pull request body, uses no secrets, and warns without ever
+# failing: the check is a prompt, not a gate.
+
+on:
+  pull_request:
+    branches:
+      - dev
+    types: [opened, edited, reopened]
+
+permissions:
+  contents: read
+
+jobs:
+  mulga_design:
+    name: Mulga design line
+    if: ${{ !endsWith(github.actor, '[bot]') }}
+    runs-on: ubuntu-26.04
+    steps:
+      - name: Checkout Code
+        uses: actions/checkout@v7.0.1
+        with:
+          sparse-checkout: .github/scripts
+
+      - name: Check the Mulga design line
+        env:
+          PR_BODY: ${{ github.event.pull_request.body }}
+        working-directory: .github/scripts
+        run: |
+          python3 -m unittest -q test_mulga_design_check
+          python3 mulga_design_check.py

.github/workflows/release.yml

@@ -57,8 +57,8 @@ jobs:
           go-version-file: spinifex/go.mod
           cache: false
 
-      - name: Build microVM artifacts (lb-agent + kernel + initramfs)
-        run: cd spinifex && make build-microvm-image
+      - name: Build microVM artifacts (lb-agent + kernel + initramfs + qboot + SQS store and worker images)
+        run: cd spinifex && make build-microvm-image build-sqs-worker-microvm-image
 
       - name: Build amd64 Tarball
         env:
@@ -72,6 +72,7 @@ jobs:
           path: |
             spinifex/dist/spinifex-${{ github.ref_name }}-linux-amd64.tar.gz
             spinifex/dist/spinifex-${{ github.ref_name }}-linux-amd64.tar.gz.sha256
+            spinifex/dist/qboot-*.tar.gz
             spinifex/scripts/setup.sh
 
   distro-arm64:

.github/workflows/spinifex.yml

@@ -119,9 +119,6 @@ jobs:
       - name: Build Spinifex
         run: make -C spinifex build
 
-      - name: Manifest check + drift lint
-        run: make -C spinifex manifest-check manifest-lint
-
       - name: Run Tests with Coverage
         env:
           SPINIFEX_REQUIRE_CONFORMANCE_TOOLS: "1"

.github/workflows/ui.yml

@@ -7,6 +7,11 @@ on:
       - dev
     paths:
       - spinifex/runtime/roles/spinifexui/frontend/**
+      - spinifex/runtime/roles/spinifexui/*.go
+      - spinifex/operator/admin/instance_metrics*.go
+      - spinifex/operator/admin/live_streams*.go
+      - spinifex/runtime/roles/qmpcollector/**
+      - contracts/telemetry/**
       - .github/workflows/ui.yml
   pull_request:
     branches:
@@ -14,6 +19,11 @@ on:
       - dev
     paths:
       - spinifex/runtime/roles/spinifexui/frontend/**
+      - spinifex/runtime/roles/spinifexui/*.go
+      - spinifex/operator/admin/instance_metrics*.go
+      - spinifex/operator/admin/live_streams*.go
+      - spinifex/runtime/roles/qmpcollector/**
+      - contracts/telemetry/**
       - .github/workflows/ui.yml
   workflow_dispatch:
 
@@ -69,3 +79,85 @@ jobs:
             echo "$drift"
             exit 1
           fi
+
+  # The pinned image lives in one file that scripts/ui-playwright.sh also reads,
+  # so make ui-baselines and the job below draw screenshots in the same image.
+  ui_e2e_image:
+    name: UI End-to-End Image
+    runs-on: ubuntu-26.04
+    outputs:
+      image: ${{ steps.read.outputs.image }}
+
+    steps:
+      - name: Checkout Code
+        uses: actions/checkout@v7.0.1
+        with:
+          sparse-checkout: spinifex/runtime/roles/spinifexui/frontend/playwright.image
+          sparse-checkout-cone-mode: false
+
+      - name: Read Playwright Image
+        id: read
+        run: echo "image=$(tr -d '[:space:]' < spinifex/runtime/roles/spinifexui/frontend/playwright.image)" >> "$GITHUB_OUTPUT"
+
+  # The browser suite: the console against a faked API on Vite dev, plus the
+  # built dist under the production CSP. It runs in Playwright's own image, which
+  # carries the browsers, so screenshots rasterise as they did for the baselines.
+  ui_e2e:
+    name: UI End-to-End
+    needs: ui_e2e_image
+    runs-on: ubuntu-26.04
+    timeout-minutes: 20
+    container:
+      image: ${{ needs.ui_e2e_image.outputs.image }}
+      options: --ipc=host
+    defaults:
+      run:
+        working-directory: spinifex/runtime/roles/spinifexui/frontend
+
+    steps:
+      - name: Checkout Code
+        uses: actions/checkout@v7.0.1
+        with:
+          sparse-checkout: spinifex/runtime/roles/spinifexui/frontend
+
+      - name: Setup pnpm
+        uses: pnpm/action-setup@v6.1.0
+        with:
+          package_json_file: spinifex/runtime/roles/spinifexui/frontend/package.json
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v7.1.0
+        with:
+          cache: "pnpm"
+          cache-dependency-path: spinifex/runtime/roles/spinifexui/frontend/pnpm-lock.yaml
+          node-version-file: spinifex/runtime/roles/spinifexui/frontend/.nvmrc
+
+      - name: Install Dependencies
+        run: pnpm install --frozen-lockfile
+
+      # The image's browsers belong to one Playwright release, so a lockfile
+      # bump without an image bump would otherwise fail on a missing executable.
+      - name: Check Playwright Matches the Image
+        env:
+          IMAGE: ${{ needs.ui_e2e_image.outputs.image }}
+        run: |
+          version=$(pnpm exec playwright --version | awk '{print $2}')
+          case "$IMAGE" in
+            *":v$version-"*) ;;
+            *) echo "::error::Lockfile has Playwright $version but the image is $IMAGE. Update playwright.image."; exit 1 ;;
+          esac
+
+      - name: Build Frontend
+        run: pnpm build
+
+      - name: Run End-to-End Suite
+        run: pnpm e2e
+
+      - name: Upload Screenshot Diffs
+        if: failure()
+        uses: actions/upload-artifact@v7.0.2
+        with:
+          name: ui-e2e-test-results
+          path: spinifex/runtime/roles/spinifexui/frontend/test-results/
+          if-no-files-found: ignore
+          retention-days: 14

.gitignore

@@ -19,6 +19,15 @@ bin/
 dist/
 build/microvm/vmlinuz
 build/microvm/initramfs.cpio.gz
+build/microvm/qboot.rom
+build/microvm/qboot.LICENSE
+build/microvm/sqs-worker/vmlinuz
+build/microvm/sqs-worker/initramfs.cpio.gz
+build/microvm/sqs-worker/.stage.*/
+build/microvm/lambda-runtime/vmlinuz
+build/microvm/lambda-runtime/initramfs.cpio.gz
+build/microvm/lambda-runtime/.stage.*/
+build/qboot/
 !spinifex/runtime/roles/spinifexui/frontend/dist/
 tmp/
 *.test

Makefile

@@ -56,6 +56,19 @@ build-ui:
 	@echo -e "\n....Building spinifex-ui frontend...."
 	cd spinifex/runtime/roles/spinifexui/frontend && pnpm build
 
+# Playwright suite for the console, against Vite dev and the built dist under
+# the console's CSP. Not part of preflight; needs `pnpm exec playwright install chromium`.
+test-ui-e2e: build-ui
+	cd spinifex/runtime/roles/spinifexui/frontend && pnpm e2e
+
+# The same suite inside the pinned Playwright image CI uses (podman or docker),
+# so screenshots compare as they will on the runner. ui-baselines redraws them.
+ui-e2e-container:
+	scripts/ui-playwright.sh compare
+
+ui-baselines:
+	scripts/ui-playwright.sh baselines
+
 # GO commands
 VERSION ?= $(shell git describe --tags --always --dirty --exclude '*-dev')
 COMMIT  ?= $(shell git rev-parse --short HEAD)
@@ -140,22 +153,79 @@ $(MICROVM_ARTIFACTS) &: $(MICROVM_INPUTS)
 bin/lb-agent:
 	$(MAKE) build-lb-agent
 
-build-microvm-image: $(MICROVM_ARTIFACTS) ## Build microVM kernel + initramfs (incremental — skips when up to date)
+# The dedicated SQS server, built from the go.mod-pinned module so the image
+# ships exactly the reviewed version. Static for the musl guest.
+bin/nats-server: go.mod go.sum
+	@mkdir -p bin
+	GOWORK=off CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags '-s -w' -o $@ github.com/nats-io/nats-server/v2
 
-install-microvm: $(MICROVM_ARTIFACTS) ## Install microVM artifacts to /usr/share/spinifex/microvm/
+# go_local_srcs lists the main module package dirs and files guest package $(1)
+# links, scanned with the guest build settings; a dir's mtime catches a deleted
+# or renamed file. A failed scan names a missing prerequisite, so make stops.
+go_local_srcs = $(shell GOWORK=off CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go list -deps -f '{{if and .Module .Module.Main}}{{.Dir}} {{range .GoFiles}}{{$$.Dir}}/{{.}} {{end}}{{range .EmbedFiles}}{{$$.Dir}}/{{.}} {{end}}{{end}}' $(1) || echo go-list-failed-for-$(1))
+.SECONDEXPANSION:
+
+# The SQS microVM image: one guest running its own SQS nats-server and the SQS
+# worker as separate processes. A separate artifact from the LB image above.
+SQS_WORKER_MICROVM_OUT_DIR := $(MICROVM_OUT_DIR)/sqs-worker
+SQS_WORKER_MICROVM_ARTIFACTS := $(SQS_WORKER_MICROVM_OUT_DIR)/vmlinuz $(SQS_WORKER_MICROVM_OUT_DIR)/initramfs.cpio.gz
+SQS_WORKER_MICROVM_INPUTS := scripts/build-sqs-store-microvm-image.sh scripts/lib/microvm-image.sh scripts/lib/sqs-store-microvm-publish.sh $(SQS_WORKER_MICROVM_OUT_DIR)/init.sh $(SQS_WORKER_MICROVM_OUT_DIR)/inittab $(addprefix $(SQS_WORKER_MICROVM_OUT_DIR)/,sqs-worker-lib.sh sqs-worker-prepare sqs-worker-store sqs-worker-ready sqs-worker-stop) bin/nats-server bin/sqs-worker
+
+$(SQS_WORKER_MICROVM_ARTIFACTS) &: $(SQS_WORKER_MICROVM_INPUTS)
+	./scripts/build-sqs-store-microvm-image.sh
+
+bin/sqs-worker: go.mod go.sum $$(call go_local_srcs,./cmd/sqs-worker)
+	@mkdir -p bin
+	GOWORK=off CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags '-s -w' -o $@ ./cmd/sqs-worker
+
+build-sqs-worker-microvm-image: $(SQS_WORKER_MICROVM_ARTIFACTS) ## Build the SQS microVM kernel and initramfs
+
+# The Lambda runtime microVM image. Built on demand; not shipped or installed.
+LAMBDA_RUNTIME_MICROVM_OUT_DIR := $(MICROVM_OUT_DIR)/lambda-runtime
+LAMBDA_RUNTIME_MICROVM_ARTIFACTS := $(LAMBDA_RUNTIME_MICROVM_OUT_DIR)/vmlinuz $(LAMBDA_RUNTIME_MICROVM_OUT_DIR)/initramfs.cpio.gz
+LAMBDA_RUNTIME_MICROVM_INPUTS := scripts/build-lambda-runtime-microvm-image.sh scripts/lib/microvm-image.sh $(addprefix $(LAMBDA_RUNTIME_MICROVM_OUT_DIR)/,init.sh inittab run-agent) bin/lambda-agent
+
+$(LAMBDA_RUNTIME_MICROVM_ARTIFACTS) &: $(LAMBDA_RUNTIME_MICROVM_INPUTS)
+	./scripts/build-lambda-runtime-microvm-image.sh
+
+bin/lambda-agent: go.mod go.sum $$(call go_local_srcs,./cmd/lambda-agent)
+	@mkdir -p bin
+	GOWORK=off CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags '-s -w' -o $@ ./cmd/lambda-agent
+
+build-lambda-runtime-microvm-image: $(LAMBDA_RUNTIME_MICROVM_ARTIFACTS) ## Build the Lambda runtime microVM kernel and initramfs (not shipped)
+
+# qboot is the microvm firmware. The source tarball ships with each release
+# because qboot is GPL-2.0.
+QBOOT_COMMIT := $(shell sed -n 's/^QBOOT_COMMIT="\(.*\)"$$/\1/p' scripts/build-qboot.sh)
+QBOOT_SOURCE := build/qboot/qboot-$(QBOOT_COMMIT).tar.gz
+QBOOT_ARTIFACTS := $(MICROVM_OUT_DIR)/qboot.rom $(MICROVM_OUT_DIR)/qboot.LICENSE $(QBOOT_SOURCE)
+
+$(QBOOT_ARTIFACTS) &: scripts/build-qboot.sh
+	./scripts/build-qboot.sh
+
+build-microvm-image: $(MICROVM_ARTIFACTS) $(QBOOT_ARTIFACTS) ## Build microVM kernel, initramfs and qboot firmware (incremental — skips when up to date)
+
+install-sqs-worker-microvm: $(SQS_WORKER_MICROVM_ARTIFACTS) ## Install the SQS microVM image to /usr/share/spinifex/microvm/sqs-worker/
+	sudo install -d /usr/share/spinifex/microvm/sqs-worker
+	sudo rm -f /usr/share/spinifex/microvm/sqs-worker/initramfs.cpio.gz
+	sudo install -m 0644 $(SQS_WORKER_MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/sqs-worker/vmlinuz
+	sudo install -m 0644 $(SQS_WORKER_MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/sqs-worker/initramfs.cpio.gz
+
+install-microvm: $(MICROVM_ARTIFACTS) $(QBOOT_ARTIFACTS) ## Install microVM artifacts to /usr/share/spinifex/microvm/
 	sudo install -d /usr/share/spinifex/microvm
 	sudo install -m 0644 $(MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/vmlinuz
 	sudo install -m 0644 $(MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/initramfs.cpio.gz
+	sudo install -m 0644 $(MICROVM_OUT_DIR)/qboot.rom /usr/share/spinifex/microvm/qboot.rom
+	sudo install -m 0644 $(MICROVM_OUT_DIR)/qboot.LICENSE /usr/share/spinifex/microvm/qboot.LICENSE
 
-# the pre-commit gate: manifest checks, lint, vuln, and the unit and e2e-harness tiers.
+# the pre-commit gate: lint, vuln, and the unit and e2e-harness tiers.
 # integration and race tests skipped to keep quick, they run in CI.
 #
 # -j2 overlaps the only two heavy gates, lint and test-cover: alone each leaves
 # most of the machine idle, so pairing them cuts a cold run by about a quarter.
 # -Otarget keeps each gate's output in one block instead of interleaving them.
-# manifest-check manifest-lint test-package-check removed, can add back later
 preflight:
-	@$(MAKE) --no-print-directory -j2 -Otarget QUIET=1 lint govulncheck test-cover diff-coverage test-harness test-build-scripts manifest-check manifest-lint
+	@$(MAKE) --no-print-directory -j2 -Otarget QUIET=1 lint govulncheck test-cover diff-coverage test-harness test-build-scripts
 	@echo -e "\n ✅ Preflight passed — safe to commit."
 
 # Shell suites + shellcheck for build/scripts/, the systemd-unit helpers that
@@ -167,7 +237,7 @@ preflight:
 # reason: between them they install the software, wire OVN, erase the state and
 # remove the install, so a shellcheck warning in one is an outage or a wrong
 # deletion, not a style note.
-NODE_SCRIPTS := scripts/setup.sh scripts/setup-ovn.sh scripts/install-node.sh scripts/smoke-test.sh scripts/node-reset.sh scripts/uninstall-spx.sh
+NODE_SCRIPTS := scripts/setup.sh scripts/setup-ovn.sh scripts/install-node.sh scripts/smoke-test.sh scripts/node-reset.sh scripts/uninstall-spx.sh scripts/lib/distro.sh scripts/lib/stage-sqs-store-microvm.sh
 
 test-build-scripts:
 	@echo -e "\n....Running build/scripts/**/*_test.sh...."
@@ -175,13 +245,20 @@ test-build-scripts:
 		echo "-- $$t"; \
 		sh "$$t" || exit 1; \
 	done
-	@echo -e "\n....Running scripts/*_test.sh...."
-	@for t in $$(find scripts -maxdepth 1 -name '*_test.sh' | sort); do \
+	@echo -e "\n....Running scripts/*_test.sh, scripts/lib/*_test.sh and scripts/appliance/*_test.sh...."
+	@for t in $$( (find scripts -maxdepth 1 -name '*_test.sh'; find scripts/lib scripts/appliance -name '*_test.sh') | sort); do \
 		echo "-- $$t"; \
 		sh "$$t" || exit 1; \
 	done
+	@echo -e "\n....Running .github/scripts/*_test.sh...."
+	@for t in $$(find .github/scripts -maxdepth 1 -name '*_test.sh' | sort); do \
+		echo "-- $$t"; \
+		sh "$$t" || exit 1; \
+	done
+	shellcheck -S warning .github/scripts/hypervisor-disk-preflight.sh .github/scripts/hypervisor-disk-preflight_test.sh
 	@echo -e "\n....Running shellcheck over build/scripts/**/*.sh + the node scripts...."
 	shellcheck -S warning $$(find build/scripts -name '*.sh' | sort) $(NODE_SCRIPTS)
+	shellcheck -S warning -x scripts/build-oracle-appliance.sh $$(find scripts/appliance -name '*.sh' | sort)
 	@echo "  test-build-scripts ok"
 
 # E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
@@ -228,23 +305,6 @@ test-segscan-oracle:
 	@echo -e "\n....Running segscan storage oracle test...."
 	$(_Q)LOG_IGNORE=1 go test -tags=integration,segscanoracle -timeout 120s ./tests/integration/... -run TestSegscanOracle $(_RACEQ)
 
-# Validate docs/service-interfaces.yaml. Schema check + cross-reference
-# of services/suites/fixtures + on-disk path existence.
-manifest-check:
-	@echo -e "\n....Checking service-interfaces.yaml...."
-	@go run ./tests/e2e/manifest-check/cmd/manifest-check -repo-root . -manifest docs/service-interfaces.yaml
-
-# Drift guards: direct-create fixture lint + NATS subject lint,
-# ratcheted against tests/e2e/manifest-lint/baseline.txt. Fails only on NEW
-# drift beyond the baseline.
-manifest-lint:
-	@echo -e "\n....Linting manifest drift (fixtures + subjects)...."
-	@go run ./tests/e2e/manifest-lint/cmd/manifest-lint -repo-root .
-
-# Accept current drift into the baseline. Run after an intentional change.
-manifest-lint-update:
-	@go run ./tests/e2e/manifest-lint/cmd/manifest-lint -repo-root . -update
-
 # Run unit tests
 test:
 	@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
@@ -265,7 +325,7 @@ test-cover:
 # Run unit tests with race detector
 test-race:
 	@echo -e "\n....Running tests with race detector for $(GO_PROJECT_NAME)...."
-	$(_Q)LOG_IGNORE=1 go test $(GOTESTFLAGS) -race -timeout 300s ./spinifex/... ./cmd/... ./internal/... ./contracts/... ./architecture/... $(_RACEQ)
+	$(_Q)LOG_IGNORE=1 SPINIFEX_SQS_R3=1 go test $(GOTESTFLAGS) -race -timeout 300s ./spinifex/... ./cmd/... ./internal/... ./contracts/... ./architecture/... $(_RACEQ)
 
 # Unit tests for in-repo GitHub Actions (e.g. .github/actions/e2e-analyze).
 # Kept out of `test-cover` so coverage % isn't diluted by CI-only tooling.
@@ -333,14 +393,20 @@ fuzz:
 # Microvm artifacts are reinstalled when they already exist on disk — the rule's
 # input timestamps drive a rebuild only if anything actually changed. On a fresh
 # checkout (no build/microvm/vmlinuz yet) the install step is skipped; run
-# `make install-microvm` explicitly the first time.
+# `make install-microvm` explicitly the first time. They go in before the
+# preflight, which checks the microvm firmware.
 deploy: build
 	sudo install -m 755 bin/spx /usr/local/bin/spx
+	@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ]; then \
+		$(MAKE) install-microvm; \
+	else \
+		echo "[deploy] microvm artifacts absent — run 'make install-microvm' for first-time setup"; \
+	fi
 	@if [ "$${SKIP_ASSET_PREFLIGHT:-}" != "1" ]; then \
 		if ! sudo /usr/local/bin/spx admin preflight; then \
 			echo ""; \
-			echo "[deploy] host-asset preflight failed: this node's helper scripts or sudoers"; \
-			echo "  grants don't match the binary just installed. Remediate with:"; \
+			echo "[deploy] host-asset preflight failed: this node's helper scripts, sudoers"; \
+			echo "  grants, QEMU or microvm firmware don't match the binary just installed. Remediate with:"; \
 			echo "    scripts/update-nodes.sh   (from the mulga root)"; \
 			echo "  or:"; \
 			echo "    make reinstall"; \
@@ -350,11 +416,6 @@ deploy: build
 	else \
 		echo "[deploy] SKIP_ASSET_PREFLIGHT=1 — skipping host-asset preflight"; \
 	fi
-	@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ]; then \
-		$(MAKE) install-microvm; \
-	else \
-		echo "[deploy] microvm artifacts absent — run 'make install-microvm' for first-time setup"; \
-	fi
 	sudo systemctl daemon-reload
 	sudo systemctl restart spinifex.target
 
@@ -367,16 +428,25 @@ clean:
 	rm -f ./bin/$(GO_PROJECT_NAME)
 	rm -rf spinifex/runtime/roles/spinifexui/frontend/dist
 
+# On dnf hosts the runtime packages and repositories come from setup.sh, so the
+# two cannot drift; nbdkit-devel, for building the plugin, is only in CodeReady.
 install-system:
 	@echo -e "\n....Installing system dependencies for $(ARCH)...."
-	sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
-		-o Dpkg::Options::="--force-confdef" \
-		-o Dpkg::Options::="--force-confold" \
-		nbdkit nbdkit-plugin-dev pkg-config qemu-system-x86 qemu-system-arm qemu-utils \
-		ovmf qemu-efi-aarch64 \
-		libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
-		iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
-		ovn-central ovn-host openvswitch-switch dhcpcd-base
+	@pm=$$(INSTALL_SPINIFEX_LIB_ONLY=1 bash -c '. scripts/setup.sh && distro_pkg_manager') || exit 1; \
+	if [ "$$pm" = dnf ]; then \
+		INSTALL_SPINIFEX_LIB_ONLY=1 bash -c '. scripts/setup.sh && setup_sudo && detect_os && install_dnf_deps' && \
+		sudo dnf config-manager --set-enabled ol10_codeready_builder && \
+		sudo dnf install -y nbdkit-devel pkgconf-pkg-config make gcc openssh-clients git sudo; \
+	else \
+		sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
+			-o Dpkg::Options::="--force-confdef" \
+			-o Dpkg::Options::="--force-confold" \
+			nbdkit nbdkit-plugin-dev pkg-config qemu-system-x86 qemu-system-arm qemu-utils \
+			ovmf qemu-efi-aarch64 \
+			libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
+			iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
+			ovn-central ovn-host openvswitch-switch dhcpcd-base; \
+	fi
 
 install-go:
 	@echo -e "\n....Installing Go 1.27.3 for $(ARCH) ($(GO_ARCH))...."
@@ -438,14 +508,16 @@ distro-amd64:
 		-f build/Dockerfile.distro \
 		--output type=local,dest=dist/amd64/ \
 		../
-	@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ] && [ -f $(MICROVM_OUT_DIR)/initramfs.cpio.gz ]; then \
+	@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ] && [ -f $(MICROVM_OUT_DIR)/initramfs.cpio.gz ] && [ -f $(MICROVM_OUT_DIR)/qboot.rom ] && [ -f $(QBOOT_SOURCE) ]; then \
 		echo "[distro-amd64] staging microvm artifacts into tarball"; \
 		mkdir -p dist/amd64/microvm; \
-		cp $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz dist/amd64/microvm/; \
+		cp $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz $(MICROVM_OUT_DIR)/qboot.rom $(MICROVM_OUT_DIR)/qboot.LICENSE dist/amd64/microvm/; \
+		cp $(QBOOT_SOURCE) dist/; \
 	else \
 		echo "[distro-amd64] WARNING: microvm artifacts missing — tarball will not include them"; \
 		echo "[distro-amd64]          run 'make build-microvm-image' before 'make distro-amd64'"; \
 	fi
+	./scripts/lib/stage-sqs-store-microvm.sh $(SQS_WORKER_MICROVM_OUT_DIR) dist/amd64 sqs-worker
 	tar -czf dist/spinifex-$(VERSION)-linux-amd64.tar.gz -C dist/amd64 .
 	sha256sum dist/spinifex-$(VERSION)-linux-amd64.tar.gz > dist/spinifex-$(VERSION)-linux-amd64.tar.gz.sha256
 
@@ -464,7 +536,7 @@ distro-arm64:
 distro-clean:
 	rm -rf dist/
 
-.PHONY: test-package-check build build-ui build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-rds-postgres-image import-rds-postgres-image build-rds-mariadb-image import-rds-mariadb-image build-microvm-image install-microvm go_build preflight test test-cover test-race diff-coverage bench fuzz test-actions test-images test-build-scripts test-harness test-integration generate-aws-model-coverage aws-model-coverage test-segscan-oracle manifest-check manifest-lint manifest-lint-update \
+.PHONY: test-package-check build build-ui test-ui-e2e ui-e2e-container ui-baselines build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-rds-postgres-image import-rds-postgres-image build-rds-mariadb-image import-rds-mariadb-image build-microvm-image build-sqs-worker-microvm-image build-lambda-runtime-microvm-image install-microvm install-sqs-worker-microvm go_build preflight test test-cover test-race diff-coverage bench fuzz test-actions test-images test-build-scripts test-harness test-integration generate-aws-model-coverage aws-model-coverage test-segscan-oracle \
 	deploy reinstall clean \
 	install-system install-go install-aws quickinstall \
 	lint fix govulncheck nilaway \

README.md

@@ -54,7 +54,7 @@ For Terraform, start from the [example workbooks](https://docs.mulgadc.com/docs/
 
 ## Services
 
-The platform serves **392 AWS API operations** across the services below. See exactly which ones in the [coverage report](https://docs.mulgadc.com/coverage).
+The platform serves **405 AWS API operations** across the services below. See exactly which ones in the [coverage report](https://docs.mulgadc.com/coverage).
 
 | Service | What it is | Built on | Status |
 | --- | --- | --- | --- |
@@ -68,8 +68,10 @@ The platform serves **392 AWS API operations** across the services below. See ex
 | **ECR** | Container registry | Predastore, deduplicated per account | Available |
 | **ECS** | Container service | Spinifex instances | Available |
 | **RDS** | Databases | PostgreSQL and MariaDB, with automated backups | Available |
+| **Route 53** | DNS | [Northstar](https://github.com/mulgadc/northstar): authoritative DNS with hosted zones | Available |
 | **Bedrock** | AI deployment | vLLM for text generation and Hugging Face TEI for embeddings and reranking | Q4 2026 |
 | **Lambda** | Serverless functions | | Q1 2027 |
+| **SQS** | Message queues | | Q1 2027 |
 
 Roadmap services ship under the same AWS API surface, so code written for AWS today works the moment they land. [Track what's shipped in the release notes](https://github.com/mulgadc/spinifex/releases).
 

architecture/layering/allowlist_test.go

@@ -29,8 +29,17 @@ var allowlist = []allowance{
 
 	{From: "agents/rds/agent", To: "domains/rds/instance", Debt: "agents/rds/agent (dependency, narrowed)", Reason: "command protocol, wire and record types; needs a guest/controller wire contract"},
 
-	{From: "cmd/aws-model-coverage", To: "domains/acm/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared ECR/ACM inventories; named exception, must not construct a registration"},
-	{From: "cmd/aws-model-coverage", To: "domains/ecr/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared ECR/ACM inventories; named exception, must not construct a registration"},
+	{From: "cmd/aws-model-coverage", To: "domains/acm/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
+	{From: "cmd/aws-model-coverage", To: "domains/ecr/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
+	{From: "cmd/aws-model-coverage", To: "domains/route53/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
+	{From: "cmd/aws-model-coverage", To: "domains/elbv2/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
+	{From: "cmd/aws-model-coverage", To: "domains/rds/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
+	{From: "cmd/aws-model-coverage", To: "domains/ec2/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
+	{From: "cmd/aws-model-coverage", To: "domains/ecs/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
+	{From: "cmd/aws-model-coverage", To: "domains/eks/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
+	{From: "cmd/aws-model-coverage", To: "domains/iam/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
+	{From: "cmd/aws-model-coverage", To: "domains/sqs/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
+	{From: "cmd/aws-model-coverage", To: "domains/sts/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
 
 	{From: "runtime/daemon", To: "domains/ec2/awsapi/instance", Debt: "Instance adapter consumers outside EC2", Reason: "runtime/daemon/eks_cp_control.go and rds_deps.go call the instance adapter; needs an EC2 capability"},
 
@@ -52,9 +61,6 @@ var allowlist = []allowance{
 	{From: "domains/ec2/awsapi/instance", To: "runtime/compute/cache", Debt: "domains/ec2/awsapi/instance (dependency)", Reason: "node liveness and VM state; needs an EC2-owned projection"},
 	{From: "domains/ec2/awsapi/instance", To: "runtime/compute/vm", Debt: "domains/ec2/awsapi/instance (dependency)", Reason: "node liveness and VM state; needs an EC2-owned projection"},
 
-	{From: "domains/ec2/awsapi/spotinstance", To: "domains/admission/quota", Debt: "domains/ec2/awsapi/spotinstance (dependency)", Reason: "RequestSpotInstances.go uses IAMService and quota.Service"},
-	{From: "domains/ec2/awsapi/spotinstance", To: "domains/iam", Debt: "domains/ec2/awsapi/spotinstance (dependency)", Reason: "RequestSpotInstances.go uses IAMService and quota.Service"},
-
 	{From: "domains/ec2/eip", To: "domains/network/topology", Debt: "EC2 → network implementation", Reason: "EC2 reads network implementation; clears with the EC2-owned projection"},
 
 	{From: "domains/ec2/guestmetadata", To: "domains/dns", Debt: "Guest metadata: the cross-domain and runtime imports above", Reason: "guest-metadata relocation residual; needs capabilities, projections or contracts"},
@@ -91,42 +97,14 @@ var allowlist = []allowance{
 	{From: "domains/ochre", To: "domains/ec2/systeminstance", Debt: "Cross-domain", Reason: "consumes another domain's implementation rather than a capability"},
 	{From: "domains/ochre", To: "domains/iam", Debt: "IAM implementation consumers in STS, Ochre and ELBv2", Reason: "secret encrypt/decrypt helpers; needs a shared crypto boundary"},
 	{From: "domains/ochre", To: "domains/network/systemvpc", Debt: "Cross-domain", Reason: "consumes another domain's implementation rather than a capability"},
-	{From: "domains/ochre", To: "ingress/aws/bodyscope", Debt: "domains/ochre (action adapters)", Reason: "Bedrock action adapters still live in the Ochre package; they move to domains/ochre/awsapi"},
 	{From: "domains/ochre", To: "runtime/compute/gpu", Debt: "Domain → runtime", Reason: "GPU runtime state read directly; needs a domain-owned projection"},
 
 	{From: "domains/ochre/vector", To: "domains/iam", Debt: "Cross-domain", Reason: "consumes another domain's implementation rather than a capability"},
 
-	{From: "gateway", To: "domains/ec2/awsapi", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/ec2/awsapi/account", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/ec2/awsapi/capacityreservation", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/ec2/awsapi/eigw", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/ec2/awsapi/eip", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/ec2/awsapi/idem", Debt: "gateway/ec2.go (registration; ec2_idempotency.go)", Reason: "dispatch-time idempotency wrapper; moves with the EC2 dispatch map"},
-	{From: "gateway", To: "domains/ec2/awsapi/igw", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/ec2/awsapi/image", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/ec2/awsapi/instance", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/ec2/awsapi/key", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/ec2/awsapi/launchtemplate", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/ec2/awsapi/natgw", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/ec2/awsapi/placementgroup", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/ec2/awsapi/routetable", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/ec2/awsapi/snapshot", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/ec2/awsapi/spotinstance", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/ec2/awsapi/tags", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/ec2/awsapi/volume", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/ec2/awsapi/vpc", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/ec2/awsapi/zone", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
-
 	{From: "domains/rds/awsapi", To: "domains/ec2/awsapi/instance", Debt: "Instance adapter consumers outside EC2", Reason: "enginecatalog.go calls the instance adapter; needs an EC2 capability"},
 
 	{From: "domains/sts/awsapi", To: "domains/iam", Debt: "IAM implementation consumers in STS, Ochre and ELBv2", Reason: "GetCallerIdentity takes IAMService; needs an IAM capability"},
 
-	{From: "gateway", To: "domains/eks/awsapi", Debt: "gateway (service registration)", Reason: "IAM, STS, EKS and RDS dispatch stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/iam/awsapi", Debt: "gateway (service registration)", Reason: "IAM, STS, EKS and RDS dispatch stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/rds/awsapi", Debt: "gateway (service registration)", Reason: "IAM, STS, EKS and RDS dispatch stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/sts/awsapi", Debt: "gateway (service registration)", Reason: "IAM, STS, EKS and RDS dispatch stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/ecs/awsapi", Debt: "gateway (service registration)", Reason: "ECS and ELBv2 dispatch stays in gateway until the ingress registration seam"},
-	{From: "gateway", To: "domains/elbv2/awsapi", Debt: "gateway (service registration)", Reason: "ECS and ELBv2 dispatch stays in gateway until the ingress registration seam"},
 	{From: "domains/ecs", To: "domains/ec2/ebs/metadata", Debt: "close-out: handlers/ecs (ADR-0004 blocked-dependency list; inventory-ecs boundary imports)", Reason: "legacy ECS calls another domain's implementation as an internal API"},
 	{From: "domains/ecs", To: "domains/ec2/eip", Debt: "close-out: handlers/ecs (ADR-0004 blocked-dependency list; inventory-ecs boundary imports)", Reason: "legacy ECS calls another domain's implementation as an internal API"},
 	{From: "domains/ecs", To: "domains/ec2/instancetypes", Debt: "close-out: handlers/ecs (ADR-0004 blocked-dependency list; inventory-ecs boundary imports)", Reason: "legacy ECS calls another domain's implementation as an internal API"},
@@ -159,6 +137,4 @@ var allowlist = []allowance{
 	{From: "domains/rds/instance", To: "runtime/compute/vm", Debt: "close-out: handlers/rds (ADR-0004 blocked-dependency list; inventory-rds boundary imports)", Reason: "legacy RDS calls another domain's or runtime implementation directly"},
 
 	{From: "domains/sts", To: "domains/iam", Debt: "IAM implementation consumers in STS, Ochre and ELBv2", Reason: "STS reads IAM service, trust-policy and OIDC-provider state; needs an IAM capability"},
-
-	{From: "operator/cli", To: "gateway", Debt: "operator/cli (dependency)", Reason: "command tree imports implementations; needs public capabilities"},
 }

architecture/layering/layering_test.go

@@ -192,24 +192,47 @@ func TestADR0001_LayeringCheckerDetectsFaults(t *testing.T) {
 	}
 }
 
-// TestADR0001_InterimDaemonDirectionCoversRuntimeDaemon: the daemon left its
-// legacy root, so gateway importing runtime/daemon must still be rejected, and
-// the former handlers packages are held to it by R5 at their domain homes.
-func TestADR0001_InterimDaemonDirectionCoversRuntimeDaemon(t *testing.T) {
-	daemon, _ := classify("spinifex/runtime/daemon")
-	checks := []struct{ rule, from string }{
-		{"Interim_DaemonGatewayHandlersDirection", "spinifex/gateway"},
-		{"R5_DomainImportsNoRuntimeOrOperator", "spinifex/domains/ecs"},
+// TestADR0001_R2_IngressRejectsRuntimeOperatorAndDomains: the request pipeline
+// lives under ingress/aws, so it must not import the daemon, any runtime or
+// operator package, or a domain implementation.
+func TestADR0001_R2_IngressRejectsRuntimeOperatorAndDomains(t *testing.T) {
+	r := ruleByID("R2_IngressImportsNoImplementation")
+	from := "spinifex/ingress/aws/pipeline"
+	f, ok := classify(from)
+	if !ok || !r.subject(f, from) {
+		t.Fatalf("%s is not governed by R2", from)
 	}
-	for _, c := range checks {
-		r := ruleByID(c.rule)
-		f, _ := classify(c.from)
-		if !r.subject(f, c.from) || !r.forbids(f, daemon, c.from) {
-			t.Errorf("%s -> runtime/daemon is not rejected by %s", c.from, c.rule)
+	for _, to := range []string{
+		"spinifex/runtime/daemon", "spinifex/runtime/host/process", "spinifex/runtime/roles/awsgw",
+		"spinifex/operator/admin", "spinifex/domains/ecs", "spinifex/domains/iam/awsapi",
+	} {
+		target, _ := classify(to)
+		if !r.forbids(f, target, from) {
+			t.Errorf("%s -> %s is not rejected by R2", from, to)
 		}
 	}
-	other, _ := classify("spinifex/runtime/host/process")
-	if gw, _ := classify("spinifex/gateway"); ruleByID("Interim_DaemonGatewayHandlersDirection").forbids(gw, other, "spinifex/gateway") {
-		t.Error("gateway -> runtime/host/process is rejected, but it is not the daemon")
+}
+
+// TestADR0002_S7_RequestRecordRejectsOutsiders: the request record is readable
+// by packages under ingress/aws only.
+func TestADR0002_S7_RequestRecordRejectsOutsiders(t *testing.T) {
+	record, ok := classify("spinifex/ingress/aws/reqctx")
+	if !ok || !record.requestRecord {
+		t.Fatalf("ingress/aws/reqctx is not classified as the request record: %+v", record)
+	}
+	r := ruleByID("S7_RequestRecordPrivateToIngress")
+	for name, wantRejected := range map[string]bool{
+		"domains/ecs/awsapi":   true,
+		"runtime/roles/awsgw":  true,
+		"operator/admin":       true,
+		"ingress/aws/dispatch": false,
+	} {
+		from, ok := classify(fullPath(name))
+		if !ok {
+			t.Fatalf("%s is unclassified", name)
+		}
+		if got := r.forbids(from, record, name); got != wantRejected {
+			t.Errorf("%s -> ingress/aws/reqctx rejected=%v, want %v", name, got, wantRejected)
+		}
 	}
 }

architecture/layering/layers_test.go

@@ -29,12 +29,11 @@ const (
 type domainKind int
 
 const (
-	notDomain     domainKind = iota
-	domainImpl               // domains/<d>/... resource implementation
-	domainAWSAPI             // domains/<d>/awsapi/... AWS action adapters
-	domainProj               // domains/<d>/projection/... authorised view
-	legacyImpl               // handlers/<d>/... legacy domain implementation
-	legacyAdapter            // gateway/<d>/... legacy AWS action adapters
+	notDomain    domainKind = iota
+	domainImpl              // domains/<d>/... resource implementation
+	domainAWSAPI            // domains/<d>/awsapi/... AWS action adapters
+	domainProj              // domains/<d>/projection/... authorised view
+	legacyImpl              // handlers/<d>/... legacy domain implementation
 )
 
 type class struct {
@@ -43,6 +42,8 @@ type class struct {
 	domain string // owning domain for domain-shaped and agent packages
 	kind   domainKind
 	daemon bool // runtime/daemon, the interim daemon -> gateway -> handlers root
+	// requestRecord marks ingress/aws/reqctx, the request record ADR-0002:S7 keeps private to ingress.
+	requestRecord bool
 }
 
 func (c class) domainShaped() bool { return c.kind != notDomain }
@@ -58,17 +59,8 @@ func (c class) isLegacy(roots ...string) bool {
 	return slices.Contains(roots, c.legacy)
 }
 
-// legacyRoots are pre-ADR-0001 application roots still standing.
-var legacyRoots = map[string]bool{
-	"gateway": true,
-}
-
-// gatewayDomains names the domain each gateway/<service> adapter belongs to;
-// Bedrock is the AWS surface of the Ochre domain.
-var gatewayDomains = map[string]string{
-	"bedrock": "ochre", "ecs": "ecs", "eks": "eks", "elbv2": "elbv2", "iam": "iam",
-	"rds": "rds", "sts": "sts",
-}
+// legacyRoots are pre-ADR-0001 application roots still standing. None remain.
+var legacyRoots = map[string]bool{}
 
 // classify maps a module-relative package directory to its layer. ok is false
 // for a package that has no deliberate classification.
@@ -103,7 +95,7 @@ func classifyApp(app []string) (class, bool) {
 	case "foundation":
 		return class{layer: layerFoundation}, true
 	case "ingress":
-		return class{layer: layerIngress}, true
+		return class{layer: layerIngress, requestRecord: len(app) >= 3 && app[1] == "aws" && app[2] == "reqctx"}, true
 	case "runtime":
 		return class{layer: layerRuntime, daemon: len(app) >= 2 && app[1] == "daemon"}, true
 	case "operator":
@@ -140,8 +132,6 @@ func classifyApp(app []string) (class, bool) {
 	switch {
 	case app[0] == "handlers" && len(app) >= 2:
 		c.domain, c.kind = app[1], legacyImpl
-	case app[0] == "gateway" && len(app) >= 2 && gatewayDomains[app[1]] != "":
-		c.domain, c.kind = gatewayDomains[app[1]], legacyAdapter
 	}
 	return c, true
 }

architecture/layering/legacy_test.go

@@ -9,14 +9,11 @@ import (
 // frozenLegacyPackages is every production package still under a legacy root.
 // It only shrinks: a new entry is a new legacy package, a missing one is a
 // completed move whose line must be deleted.
-var frozenLegacyPackages = []string{
-	"gateway",
-	"gateway/spx",
-}
+var frozenLegacyPackages = []string{}
 
 // retiredLegacyRoots were legacy roots whose every package has moved to its
 // ADR-0001 target; recreating one would reopen a closed migration row.
-var retiredLegacyRoots = []string{"accountteardown", "admin", "daemon", "handlers", "lbagent", "services", "utils", "vpcd"}
+var retiredLegacyRoots = []string{"accountteardown", "admin", "daemon", "gateway", "handlers", "lbagent", "services", "utils", "vpcd"}
 
 // TestADR0001_LegacyRootsFrozen: the current-to-target map gives every legacy
 // root a destination, so none may gain a package and no retired root may return.
@@ -36,9 +33,6 @@ func TestADR0001_LegacyRootsFrozen(t *testing.T) {
 			present[name] = true
 		}
 	}
-	if len(present) == 0 {
-		t.Fatal("found no legacy packages; the freeze would pass vacuously")
-	}
 	frozen := map[string]bool{}
 	for _, p := range frozenLegacyPackages {
 		frozen[p] = true

architecture/layering/rules_test.go

@@ -1,6 +1,9 @@
 package layering
 
-import "slices"
+import (
+	"slices"
+	"strings"
+)
 
 // rule is one ADR-0001 dependency rule over a classified import edge.
 type rule struct {
@@ -123,10 +126,8 @@ var rules = []rule{
 			case t.layer == layerAgents:
 				return true
 			case t.layer == layerIngress:
-				return f.kind != domainAWSAPI && f.kind != legacyAdapter
-			case t.isLegacy("gateway") && !t.domainShaped():
-				return true
-			case t.isLegacy("handlers", "gateway"):
+				return f.kind != domainAWSAPI
+			case t.isLegacy("handlers"):
 				return f.layer == layerDomain
 			}
 			return false
@@ -152,18 +153,15 @@ var rules = []rule{
 		},
 	},
 	{
-		id:      "Interim_DaemonGatewayHandlersDirection",
-		clause:  "during migration the existing daemon -> gateway -> handlers rule remains the guardrail (Q-20, Q-29)",
-		hint:    "keep the dependency pointing daemon -> gateway -> handlers; invert it through a callback or capability",
-		subject: func(f class, _ string) bool { return f.isLegacy("handlers", "gateway") },
-		forbids: func(f, t class, _ string) bool {
-			switch {
-			case f.isLegacy("handlers"):
-				return t.isLegacy("gateway") || t.daemon
-			case f.isLegacy("gateway"):
-				return t.daemon
+		id:      "S7_RequestRecordPrivateToIngress",
+		clause:  "ADR-0002:S7: the request record is private to ingress, so no domain, runtime or operator package reads it; only packages under ingress/aws may import ingress/aws/reqctx",
+		hint:    "receive what you need through dispatch.Invocation instead of the request record",
+		subject: func(_ class, _ string) bool { return true },
+		forbids: func(_, t class, fromName string) bool {
+			if !t.requestRecord {
+				return false
 			}
-			return false
+			return !strings.HasPrefix(fromName, "ingress/aws/")
 		},
 	},
 }

build/Dockerfile.distro

@@ -1,6 +1,6 @@
 ARG VERSION=dev
 
-FROM golang:1.27.3-bookworm@sha256:972b4110ba0f47539a975c31e52c64765e1ff19f2d74469ae4a595c6676e80a5 AS builder
+FROM golang:1.27.3-bookworm@sha256:d9b6a1beb9474adc2a1ce842573639b44b2a0dd49b6669f51d3821baf9ec37c9 AS builder
 
 ARG VERSION
 
@@ -42,9 +42,16 @@ RUN COMMIT=$(cd spinifex && git rev-parse --short HEAD 2>/dev/null || echo "unkn
 # Build nbdkit viperblock plugin (.so)
 RUN cd viperblock && GOFIPS140=v1.0.0 go build -o /out/nbdkit-viperblock-plugin.so -buildmode=c-shared nbd/viperblock.go
 
+# One tarball ships to every OS. The ceiling is what both binaries need today,
+# so a rise is a decision rather than an accident; every supported OS has 2.39+.
+RUN spinifex/scripts/check-glibc-ceiling.sh 2.34 /out/spx /out/nbdkit-viperblock-plugin.so
+
 # Copy setup scripts and systemd units
 RUN cp spinifex/scripts/setup.sh /out/setup.sh 2>/dev/null || true
 RUN cp spinifex/scripts/setup-ovn.sh /out/setup-ovn.sh
+RUN cd spinifex/scripts && cp install-node.sh smoke-test.sh node-reset.sh uninstall-spx.sh /out/
+RUN mkdir -p /out/udev && cp spinifex/build/udev/* /out/udev/
+RUN mkdir -p /out/lib && cp spinifex/scripts/lib/distro.sh /out/lib/distro.sh
 RUN mkdir -p /out/systemd && cp spinifex/build/systemd/* /out/systemd/
 RUN mkdir -p /out/scripts && cp spinifex/build/scripts/* /out/scripts/
 RUN cp spinifex/build/logrotate/spinifex /out/logrotate-spinifex

build/microvm/lambda-runtime/init.sh

@@ -0,0 +1,68 @@
+#!/bin/sh
+mount -t proc  proc  /proc
+mount -t sysfs sys   /sys
+mount -t devtmpfs dev /dev
+
+# modprobe needs a usable modules.dep; insmod by explicit path does not, so
+# fall back to it. Handles .ko and .ko.gz.
+load_mod() {
+    modprobe "$1" 2>/dev/null && return 0
+    for ko in $(find /lib/modules -name "$1.ko" -o -name "$1.ko.gz" 2>/dev/null); do
+        insmod "$ko" 2>/dev/null && return 0
+        case "$ko" in
+            *.gz) gunzip -c "$ko" >/tmp/mod.ko 2>/dev/null \
+                  && insmod /tmp/mod.ko 2>/dev/null && return 0 ;;
+        esac
+    done
+    return 1
+}
+# lambda-agent reads its identity from fw_cfg and unloads qemu_fw_cfg before
+# anything else runs. virtio_net needs net_failover and failover, and the
+# insmod fallback resolves no dependencies, so load the chain bottom-up.
+load_mod qemu_fw_cfg || echo "[init] qemu_fw_cfg did not load" >/dev/console
+load_mod failover || true
+load_mod net_failover || true
+load_mod virtio_net || echo "[init] virtio_net did not load" >/dev/console
+
+ip link set lo up
+
+# The guest's addresses come from the same netcfg entry as other system VMs.
+NETCFG=/sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw
+i=0
+while [ "$i" -lt 25 ] && [ ! -f "$NETCFG" ]; do
+    sleep 0.04
+    i=$((i + 1))
+done
+if [ -f "$NETCFG" ]; then
+    # shellcheck source=/dev/null
+    . "$NETCFG"
+    for n in 0 1 2 3 4 5; do
+        eval mac=\$NIC${n}_MAC
+        [ -z "$mac" ] && continue
+        iface=""
+        for _ in 1 2 3 4 5; do
+            iface=$(ip -o link | awk -v m="$mac" 'tolower($0) ~ tolower(m) {print $2}' | tr -d :)
+            [ -n "$iface" ] && break
+            sleep 0.02
+        done
+        if [ -z "$iface" ]; then
+            echo "[init] no interface for MAC $mac (NIC$n)" >/dev/console
+            continue
+        fi
+        cidr="" gw="" default="" rdst="" rvia=""
+        eval cidr=\$NIC${n}_CIDR
+        eval gw=\$NIC${n}_GW
+        eval default=\$NIC${n}_DEFAULT
+        eval rdst=\$NIC${n}_ROUTE_DST
+        eval rvia=\$NIC${n}_ROUTE_VIA
+        ip link set "$iface" up
+        [ -n "$cidr" ] && ip addr add "$cidr" dev "$iface"
+        [ "$default" = "1" ] && [ -n "$gw" ] && ip route add default via "$gw" dev "$iface"
+        [ -n "$rdst" ] && [ -n "$rvia" ] && ip route add "$rdst" via "$rvia" dev "$iface"
+    done
+else
+    # lambda-agent then fails to reach the host and the guest powers off.
+    echo "[init] fw_cfg netcfg not available" >/dev/console
+fi
+
+exec /sbin/init

build/microvm/lambda-runtime/inittab

@@ -0,0 +1,6 @@
+# The power button loads here, not in /init: busybox init has its SIGUSR2
+# (poweroff) handler installed before sysinit runs, so a press can never reach
+# the pre-exec shell PID 1. A press before this line is lost; the host re-sends.
+::sysinit:/sbin/modprobe tiny_power_button power_signal=12
+::once:/usr/local/lib/lambda-runtime/run-agent
+::ctrlaltdel:/sbin/reboot

build/microvm/lambda-runtime/run-agent

@@ -0,0 +1,6 @@
+#!/bin/sh
+# Runs the Lambda guest agent, then powers the guest off whatever its status.
+/usr/local/bin/lambda-agent >/dev/console 2>&1
+status=$?
+echo "[run-agent] lambda-agent exited with status $status" >/dev/console
+poweroff -f

build/microvm/sqs-worker/init.sh

@@ -0,0 +1,100 @@
+#!/bin/sh
+mount -t proc  proc  /proc
+mount -t sysfs sys   /sys
+mount -t devtmpfs dev /dev
+
+# virtio_mmio is built-in (CONFIG_VIRTIO_MMIO=y) so virtio-net devices are
+# already enumerated by the time PID 1 runs — no modprobe needed.
+# qemu_fw_cfg and virtio_net are =m in stock Alpine linux-virt, so load them.
+#
+# modprobe needs a usable modules.dep; when the image is built on a host whose
+# depmod can't produce one, busybox modprobe silently no-ops and the module
+# never loads — fw_cfg then never appears and init panics. insmod by explicit
+# path needs no modules.dep, so fall back to it. Handles .ko and .ko.gz.
+load_mod() {
+    modprobe "$1" 2>/dev/null && return 0
+    for ko in $(find /lib/modules -name "$1.ko" -o -name "$1.ko.gz" 2>/dev/null); do
+        insmod "$ko" 2>/dev/null && return 0
+        case "$ko" in
+            *.gz) gunzip -c "$ko" >/tmp/mod.ko 2>/dev/null \
+                  && insmod /tmp/mod.ko 2>/dev/null && return 0 ;;
+        esac
+    done
+    return 1
+}
+load_mod qemu_fw_cfg || load_mod fw_cfg_sysfs || true
+# virtio_net depends on net_failover, which depends on failover. The load_mod
+# insmod fallback loads a single .ko with no dependency resolution, and
+# modules.dep is absent when the image is built on a kmod-less host (depmod
+# skipped), so modprobe can't resolve the chain either. Load it explicitly
+# bottom-up so the NIC comes up regardless of how the image was built —
+# without this the agent has no network and the LB hangs in provisioning.
+load_mod failover || true
+load_mod net_failover || true
+load_mod virtio_net || true
+
+# The guest's SQS store listens on 127.0.0.1 and the worker dials it there;
+# the kernel leaves lo down, so bring it up before init starts either.
+ip link set lo up
+
+NETCFG=/sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw
+i=0
+while [ "$i" -lt 25 ]; do
+    [ -f "$NETCFG" ] && break
+    sleep 0.04
+    i=$((i + 1))
+done
+
+if [ ! -f "$NETCFG" ]; then
+    # Dump host/guest fw_cfg state to the serial console before init exits so a
+    # "netcfg not available" panic is diagnosable from the captured console log
+    # (module-not-loaded vs qemu-not-exposing-by_name vs wrong machine type).
+    echo "[init] FATAL: fw_cfg netcfg not available" >&2
+    echo "[init] dmi: vendor=$(cat /sys/class/dmi/id/sys_vendor 2>/dev/null) product=$(cat /sys/class/dmi/id/product_name 2>/dev/null) bios=$(cat /sys/class/dmi/id/bios_version 2>/dev/null)" >&2
+    echo "[init] qemu_fw_cfg dir present: $([ -d /sys/firmware/qemu_fw_cfg ] && echo yes || echo NO)" >&2
+    echo "[init] modules: $(lsmod 2>/dev/null | awk 'NR>1{print $1}' | tr '\n' ',')" >&2
+    echo "[init] by_name listing:" >&2
+    ls -la /sys/firmware/qemu_fw_cfg/by_name/ 2>&1 | sed 's/^/[init]   /' >&2
+    ls -la /sys/firmware/qemu_fw_cfg/by_name/opt/ 2>&1 | sed 's/^/[init]   /' >&2
+    echo "[init] dmesg fw_cfg:" >&2
+    dmesg 2>/dev/null | grep -i fw_cfg | sed 's/^/[init]   /' >&2
+    exit 1
+fi
+
+. "$NETCFG"
+
+for n in 0 1 2 3 4 5; do
+    eval mac=\$NIC${n}_MAC
+    [ -z "$mac" ] && continue
+
+    iface=""
+    for attempt in 1 2 3 4 5; do
+        iface=$(ip -o link | awk -v m="$mac" 'tolower($0) ~ tolower(m) {print $2}' | tr -d :)
+        [ -n "$iface" ] && break
+        sleep 0.02
+    done
+
+    if [ -z "$iface" ]; then
+        echo "[init] WARNING: no interface found for MAC $mac (NIC$n), skipping" >&2
+        continue
+    fi
+
+    eval cidr=\$NIC${n}_CIDR
+    eval gw=\$NIC${n}_GW
+    eval default=\$NIC${n}_DEFAULT
+    eval rdst=\$NIC${n}_ROUTE_DST
+    eval rvia=\$NIC${n}_ROUTE_VIA
+
+    ip link set "$iface" up
+    [ -n "$cidr" ] && ip addr add "$cidr" dev "$iface"
+    [ "$default" = "1" ] && [ -n "$gw" ] && ip route add default via "$gw" dev "$iface"
+    [ -n "$rdst" ] && [ -n "$rvia" ] && ip route add "$rdst" via "$rvia" dev "$iface"
+done
+
+# The image takes no lb-agent environment. The worker reads the cluster CA
+# from fw_cfg itself; this copy only serves other tools in the guest.
+CA_CERT=/sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/ca-cert/raw
+mkdir -p /etc/ssl/certs
+[ -f "$CA_CERT" ] && cp "$CA_CERT" /etc/ssl/certs/ca-certificates.crt
+
+exec /sbin/init

build/microvm/sqs-worker/inittab

@@ -0,0 +1,9 @@
+# The power button loads here, not in /init: busybox init has its SIGUSR2
+# (poweroff) handler installed before sysinit runs, so a press can never reach
+# the pre-exec shell PID 1. A press before this line is lost; the host re-sends.
+::sysinit:/sbin/modprobe tiny_power_button power_signal=12
+::sysinit:/usr/local/lib/sqs-worker/sqs-worker-prepare
+::respawn:/usr/local/lib/sqs-worker/sqs-worker-store
+::respawn:/usr/local/lib/sqs-worker/sqs-worker-ready
+::ctrlaltdel:/sbin/reboot
+::shutdown:/usr/local/lib/sqs-worker/sqs-worker-stop

build/microvm/sqs-worker/sqs-worker-lib.sh

@@ -0,0 +1,203 @@
+# sqs-worker-lib.sh — shared by the SQS guest's store, readiness and stop
+# steps. Sourced, never executed. Every path and command is overridable so the
+# host test can drive the same code against test-owned processes and stub tools.
+#
+# The guest runs its own SQS nats-server on loopback and the worker beside it.
+# The server starts only after boot settled the receipt key; store-ready exists
+# only while the running watcher's last check of that server passed.
+
+SQSW_RUN=${SQSW_RUN:-/run/sqs-worker}
+SQSW_DATA=${SQSW_DATA:-/data}
+SQSW_CONSOLE=${SQSW_CONSOLE:-/dev/console}
+SQSW_MOUNTS=${SQSW_MOUNTS:-/proc/mounts}
+SQSW_BIN=${SQSW_BIN:-/usr/local/bin/sqs-worker}
+SQSW_LIB=${SQSW_LIB:-/usr/local/lib/sqs-worker/sqs-worker-lib.sh}
+SQSW_NATS_BIN=${SQSW_NATS_BIN:-/usr/local/bin/nats-server}
+SQSW_STORE_PORT=${SQSW_STORE_PORT:-4222}
+SQSW_STOP_BUDGET_TENTHS=${SQSW_STOP_BUDGET_TENTHS:-100}
+SQSW_RESPAWN_GAP=${SQSW_RESPAWN_GAP:-5}
+SQSW_PARK=${SQSW_PARK:-sleep 2147483647}
+SQSW_BOOT_RETRY=${SQSW_BOOT_RETRY:-30}
+UMOUNT=${UMOUNT:-umount}
+
+sqsw_log() { echo "[sqs-worker] $*" >> "$SQSW_CONSOLE"; }
+
+# sqsw_alive <pid>: running or stopped, not exited (a zombie counts as exited).
+sqsw_alive() { [ -r "/proc/$1/status" ] && ! grep -q '^State:[[:space:]]*Z' "/proc/$1/status" 2>/dev/null; }
+
+# sqsw_starttime <pid>: kernel start time (stat field 22). It survives exec, and
+# pairing it with the pid reduces PID-reuse risk; it is not an absolute guarantee.
+sqsw_starttime() { sed 's/.*) //' "/proc/$1/stat" 2>/dev/null | cut -d' ' -f20; }
+
+# sqsw_same <pid> <starttime>: the recorded process is still alive.
+sqsw_same() { [ -n "$2" ] && sqsw_alive "$1" && [ "$(sqsw_starttime "$1")" = "$2" ]; }
+
+# sqsw_write_config <file>: the guest's own server. It listens only on
+# loopback, with no monitor, route or cluster listener. Its max_payload admits
+# a 1 MiB SQS message with its headers (ADR-0011:S4); the 1MB default does not.
+sqsw_write_config() {
+    cat > "$1" <<CONF
+listen: "127.0.0.1:$SQSW_STORE_PORT"
+server_name: "sqs-store"
+max_payload: 2MiB
+jetstream {
+  store_dir: "$SQSW_DATA/jetstream"
+  sync_interval: always
+}
+CONF
+}
+
+# sqsw_prepare_store: write and test the server config once boot has settled
+# the receipt key. Returns non-zero, leaving the store unstarted, otherwise.
+sqsw_prepare_store() {
+    rm -f "$SQSW_RUN/store.conf"
+    [ -s "$SQSW_RUN/receipt-ready" ] || { sqsw_log "store: receipt key not settled, store not prepared"; return 1; }
+    umask 077
+    sqsw_write_config "$SQSW_RUN/store.conf.new" || return 1
+    if ! "$SQSW_NATS_BIN" -t -c "$SQSW_RUN/store.conf.new" > "$SQSW_RUN/config-test.out" 2>&1; then
+        sqsw_log "store: FATAL: server config test failed: $(tr '\n' ' ' < "$SQSW_RUN/config-test.out")"
+        return 1
+    fi
+    mv "$SQSW_RUN/store.conf.new" "$SQSW_RUN/store.conf"
+    sqsw_log "store: config written for 127.0.0.1:$SQSW_STORE_PORT; server not started yet"
+}
+
+# sqsw_park <why>: init respawns a step that exits, so a step that must not run
+# logs once and waits for shutdown instead of looping.
+sqsw_park() {
+    sqsw_log "$1"
+    exec $SQSW_PARK
+}
+
+# sqsw_record <name>: record this process's identity before the stopping
+# check, so a stop that misses the record has already set stopping.
+sqsw_record() { echo "$$ $(sqsw_starttime $$)" > "$SQSW_RUN/$1.pid"; }
+
+# sqsw_boot: settle the disk and receipt key once per guest boot. A transient
+# failure (exit 75) waits and exits so init runs it again; boot never remounts
+# a disk already mounted. Any other failure is final for this guest boot.
+sqsw_boot() {
+    [ ! -e "$SQSW_RUN/boot-refused" ] || sqsw_park "store: boot refused, not starting"
+    [ -s "$SQSW_RUN/receipt-ready" ] && return 0
+    # The boot child records itself before its stopping check, as the other
+    # steps do, so shutdown either sees its record or it never starts.
+    SQSW_RUN="$SQSW_RUN" SQSW_BIN="$SQSW_BIN" sh -c '. "$1"; sqsw_record boot; [ ! -e "$SQSW_RUN/stopping" ] || exit 75; exec "$SQSW_BIN" boot' \
+        sh "$SQSW_LIB" >> "$SQSW_CONSOLE" 2>&1 &
+    child=$!
+    rc=0
+    wait "$child" || rc=$?
+    rm -f "$SQSW_RUN/boot.pid"
+    [ ! -e "$SQSW_RUN/stopping" ] || sqsw_park "store: shutdown during boot, not starting"
+    case "$rc" in
+        0) return 0 ;;
+        75) sqsw_log "store: boot not settled, retrying in ${SQSW_BOOT_RETRY}s"
+            sleep "$SQSW_BOOT_RETRY"
+            exit 1 ;;
+        *) : > "$SQSW_RUN/boot-refused"
+           sqsw_park "store: boot refused (rc=$rc), not starting" ;;
+    esac
+}
+
+# sqsw_store: the respawned server body. It starts only on a prepared config.
+sqsw_store() {
+    mkdir -p "$SQSW_RUN"
+    sqsw_record store
+    [ ! -e "$SQSW_RUN/stopping" ] || sqsw_park "store: shutdown in progress, not starting"
+    sqsw_boot
+    if [ ! -s "$SQSW_RUN/store.conf" ] && ! sqsw_prepare_store; then
+        : > "$SQSW_RUN/boot-refused"
+        sqsw_park "store: not prepared, not starting"
+    fi
+    sqsw_log "store: starting pid=$$"
+    # ADR-0011:S11 bounds the store's Go heap; the guest must be larger.
+    GOMEMLIMIT=512MiB exec "$SQSW_NATS_BIN" -c "$SQSW_RUN/store.conf"
+}
+
+# sqsw_run: the respawned readiness body. A predecessor that was killed or
+# crashed could not remove its marker, so every start removes it first.
+sqsw_run() {
+    mkdir -p "$SQSW_RUN"
+    sqsw_record ready
+    if [ -e "$SQSW_RUN/store-ready" ]; then
+        rm -f "$SQSW_RUN/store-ready"
+        sqsw_log "ready: removed a store-ready marker left by an earlier watcher"
+    fi
+    [ ! -e "$SQSW_RUN/stopping" ] || sqsw_park "ready: shutdown in progress, not starting"
+    [ ! -e "$SQSW_RUN/boot-refused" ] || sqsw_park "ready: boot refused, readiness not started"
+    # Boot may still be retrying; check again after a gap rather than parking.
+    if [ ! -s "$SQSW_RUN/store.conf" ]; then
+        sleep "$SQSW_RESPAWN_GAP"
+        exit 0
+    fi
+    # Init respawns at once; a watcher that keeps exiting restarts at most
+    # once per gap instead of in a tight loop.
+    now=$(date +%s)
+    last=$(cat "$SQSW_RUN/ready.last" 2>/dev/null || echo 0)
+    wait=$((last + SQSW_RESPAWN_GAP - now))
+    if [ "$wait" -gt 0 ] && [ "$wait" -le "$SQSW_RESPAWN_GAP" ]; then sleep "$wait"; fi
+    date +%s > "$SQSW_RUN/ready.last"
+    sqsw_log "ready: starting watcher pid=$$"
+    exec "$SQSW_BIN" ready >> "$SQSW_CONSOLE" 2>&1
+}
+
+# sqsw_end <name>: SIGTERM the recorded process, SIGKILL only after the
+# budget. Returns non-zero while it is still alive.
+sqsw_end() {
+    rec=$(cat "$SQSW_RUN/$1.pid" 2>/dev/null || true)
+    pid=${rec%% *} st=${rec#* }
+    [ "$st" != "$rec" ] || st=""
+    if [ -z "$pid" ] || ! sqsw_same "$pid" "$st"; then
+        sqsw_log "stop: no running $1"
+        return 0
+    fi
+    kill -TERM "$pid" 2>/dev/null || true
+    sqsw_log "stop: SIGTERM sent to $1 pid=$pid"
+    i=0
+    while sqsw_same "$pid" "$st" && [ "$i" -lt "$SQSW_STOP_BUDGET_TENTHS" ]; do sleep 0.1; i=$((i + 1)); done
+    if sqsw_same "$pid" "$st"; then
+        sqsw_log "stop: $1 pid=$pid still running after $((SQSW_STOP_BUDGET_TENTHS / 10))s, SIGKILL"
+        kill -KILL "$pid" 2>/dev/null || true
+        j=0
+        while sqsw_same "$pid" "$st" && [ "$j" -lt 30 ]; do sleep 0.1; j=$((j + 1)); done
+    fi
+    if sqsw_same "$pid" "$st"; then
+        sqsw_log "stop: $1 pid=$pid STILL ALIVE"
+        return 1
+    fi
+    sqsw_log "stop: $1 pid=$pid exited"
+    rm -f "$SQSW_RUN/$1.pid"
+}
+
+# sqsw_stop: block any respawn, drop the marker, stop the watcher, the server
+# and any boot in progress, and unmount the disk only once all are confirmed
+# gone. A survivor keeps the disk mounted.
+sqsw_stop() {
+    mkdir -p "$SQSW_RUN"
+    : > "$SQSW_RUN/stopping"
+    rm -f "$SQSW_RUN/store-ready"
+    if ! sqsw_end ready || ! sqsw_end store || ! sqsw_end boot; then
+        rm -f "$SQSW_RUN/store-ready"
+        sqsw_log "stop: leaving $SQSW_DATA mounted"
+        return 1
+    fi
+    rm -f "$SQSW_RUN/store-ready"
+    sync
+    if grep -q " $SQSW_DATA " "$SQSW_MOUNTS" 2>/dev/null; then
+        rc=0
+        "$UMOUNT" "$SQSW_DATA" || rc=$?
+        sqsw_log "stop: umount $SQSW_DATA rc=$rc"
+    fi
+}
+
+# sqsw_shutdown: the only shutdown action. The generic read-only remount runs
+# only after both processes stopped; a survivor keeps every mount as it is.
+sqsw_shutdown() {
+    if ! sqsw_stop; then
+        sync
+        sqsw_log "shutdown: a guest process survived, skipping umount -a -r"
+        return 1
+    fi
+    rc=0
+    "$UMOUNT" -a -r || rc=$?
+    echo "[init] shutdown: umount -a -r rc=$rc" >> "$SQSW_CONSOLE"
+}

build/microvm/sqs-worker/sqs-worker-prepare

@@ -0,0 +1,9 @@
+#!/bin/sh
+# sysinit: load the disk drivers and clear this boot's markers. The respawned
+# store step mounts the disk, settles the receipt key and starts the store.
+. /usr/local/lib/sqs-worker/sqs-worker-lib.sh
+modprobe virtio_blk 2>/dev/null || true
+modprobe ext4 2>/dev/null || true
+mkdir -p "$SQSW_RUN"
+rm -f "$SQSW_RUN/receipt-ready" "$SQSW_RUN/store.conf" "$SQSW_RUN/boot-refused"
+exit 0

build/microvm/sqs-worker/sqs-worker-ready

@@ -0,0 +1,5 @@
+#!/bin/sh
+# Respawned by init: the worker, reporting ready only while the local store
+# answers. A watcher that must not run parks instead of exiting.
+. /usr/local/lib/sqs-worker/sqs-worker-lib.sh
+sqsw_run

build/microvm/sqs-worker/sqs-worker-stop

@@ -0,0 +1,5 @@
+#!/bin/sh
+# The only shutdown action: stop the worker, then the store, and unmount the
+# data disk, then run the generic read-only remount only if both are gone.
+. /usr/local/lib/sqs-worker/sqs-worker-lib.sh
+sqsw_shutdown

build/microvm/sqs-worker/sqs-worker-store

@@ -0,0 +1,5 @@
+#!/bin/sh
+# Respawned by init: the guest's own SQS nats-server, started only on a
+# prepared config. A store that must not run parks instead of exiting.
+. /usr/local/lib/sqs-worker/sqs-worker-lib.sh
+sqsw_store

build/systemd/spinifex-daemon.service

@@ -1,10 +1,10 @@
-# spinifex-unit-version: 3
+# spinifex-unit-version: 6
 [Unit]
 Description=Spinifex Daemon (VM Orchestration)
 PartOf=spinifex.target
-After=spinifex-nats.service spinifex-viperblock.service openvswitch-switch.service ovn-controller.service openvswitch-ipsec.service
-# Wants (not Requires): stay bootable when openvswitch-ipsec is absent
-Wants=openvswitch-ipsec.service
+After=spinifex-nats.service spinifex-viperblock.service ovsdb-server.service ovs-vswitchd.service ovn-controller.service openvswitch-ipsec.service spinifex-hostnet.service
+# Wants (not Requires): boot without openvswitch-ipsec, survive OVS restarts via hostnet
+Wants=openvswitch-ipsec.service spinifex-hostnet.service
 
 [Service]
 EnvironmentFile=-/etc/spinifex/telemetry.env
@@ -14,6 +14,8 @@ Type=simple
 Slice=spinifex-system.slice
 User=spinifex-daemon
 Group=spinifex
+ExecStartPre=/usr/local/bin/spx admin hostnet check
+ExecStartPre=-+/usr/local/bin/spx admin node-facts collect
 ExecStart=/usr/local/bin/spx service spinifex start
 Restart=on-failure
 RestartSec=5

build/systemd/spinifex-firewall.service

@@ -1,11 +1,10 @@
-# spinifex-unit-version: 1
+# spinifex-unit-version: 2
 [Unit]
 Description=Spinifex host firewall (nftables input policy)
 Documentation=file:/etc/spinifex/firewall/spinifex.nft
 Wants=network-pre.target
 Before=network-pre.target shutdown.target
 DefaultDependencies=no
-ConditionPathExists=/etc/spinifex/firewall/peers.nft
 
 [Service]
 Type=oneshot
@@ -13,8 +12,11 @@ RemainAfterExit=yes
 StandardInput=null
 ProtectSystem=full
 ProtectHome=true
-ExecStart=/usr/local/lib/spinifex/spinifex-firewall-apply
-ExecReload=/usr/local/lib/spinifex/spinifex-firewall-apply
+# Unconditional: connection tracking loads on every node, so a flow that opens
+# while the policy is not armed survives the arm. The policy itself still loads
+# only where the daemon has written a peer file.
+ExecStart=/usr/local/lib/spinifex/spinifex-firewall-apply boot
+ExecReload=/usr/local/lib/spinifex/spinifex-firewall-apply boot
 TimeoutStopSec=10s
 
 # No ExecStop: flushing on stop would take vpcd's MASQUERADE and per-EIP FORWARD

build/systemd/spinifex-hostnet.service

@@ -0,0 +1,25 @@
+# spinifex-unit-version: 2
+[Unit]
+Description=Spinifex host network wiring (veth pairs, mgmt address, encap route)
+# PartOf/WantedBy ovs-vswitchd: re-applied whenever OVS starts; daemon and vpcd only Want it
+Requires=ovs-vswitchd.service
+After=ovs-vswitchd.service NetworkManager-wait-online.service
+PartOf=ovs-vswitchd.service
+# Stopping OVS stops ovn-controller through its Requires= chain and starting OVS
+# does not bring it back, so the start that re-applies hostnet restarts it too
+Wants=ovn-controller.service
+Before=spinifex.target
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/usr/local/bin/spx admin hostnet apply
+# Covers the 120s wait behind setup-ovn.sh plus the 60s wait for links
+TimeoutStartSec=240
+TimeoutStopSec=10
+ProtectSystem=full
+ProtectHome=yes
+PrivateTmp=yes
+
+[Install]
+WantedBy=spinifex.target ovs-vswitchd.service

build/systemd/spinifex-vpcd.service

@@ -1,9 +1,9 @@
-# spinifex-unit-version: 1
+# spinifex-unit-version: 3
 [Unit]
 Description=Spinifex VPC Daemon (OVN Networking)
 PartOf=spinifex.target
-After=spinifex-nats.service spinifex-daemon.service systemd-networkd.service openvswitch-switch.service ovn-controller.service
-Wants=spinifex-nats.service ovn-controller.service
+After=spinifex-nats.service spinifex-daemon.service systemd-networkd.service ovsdb-server.service ovs-vswitchd.service ovn-controller.service spinifex-hostnet.service
+Wants=spinifex-nats.service ovn-controller.service spinifex-hostnet.service
 
 [Service]
 EnvironmentFile=-/etc/spinifex/telemetry.env
@@ -11,6 +11,7 @@ Type=simple
 Slice=spinifex-system.slice
 User=spinifex-vpcd
 Group=spinifex
+ExecStartPre=/usr/local/bin/spx admin hostnet check
 ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
 ExecStart=/usr/local/bin/spx service vpcd start
 Restart=on-failure

build/systemd/spinifex.target

@@ -1,9 +1,9 @@
-# spinifex-unit-version: 1
+# spinifex-unit-version: 4
 [Unit]
 Description=Spinifex Infrastructure Platform
 After=network-online.target
 Wants=network-online.target
-Wants=spinifex-nats.service spinifex-predastore.service spinifex-northstar.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service spinifex-qmp-collector.service spinifex-shutdown.service spinifex-nats-watchdog.timer openvswitch-switch.service ovn-controller.service
+Wants=spinifex-nats.service spinifex-predastore.service spinifex-northstar.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service spinifex-qmp-collector.service spinifex-shutdown.service spinifex-nats-watchdog.timer ovsdb-server.service ovs-vswitchd.service ovn-controller.service
 
 [Install]
 WantedBy=multi-user.target

cmd/aws-model-coverage/main.go

@@ -4,12 +4,21 @@ import (
 	"flag"
 	"fmt"
 	"os"
+	"slices"
 
 	"github.com/mulgadc/predastore/s3api"
 	"github.com/mulgadc/spinifex/internal/awsmodel"
 	acmawsapi "github.com/mulgadc/spinifex/spinifex/domains/acm/awsapi"
+	ec2awsapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi"
 	awsapi "github.com/mulgadc/spinifex/spinifex/domains/ecr/awsapi"
-	"github.com/mulgadc/spinifex/spinifex/gateway"
+	ecsawsapi "github.com/mulgadc/spinifex/spinifex/domains/ecs/awsapi"
+	eksawsapi "github.com/mulgadc/spinifex/spinifex/domains/eks/awsapi"
+	elbv2awsapi "github.com/mulgadc/spinifex/spinifex/domains/elbv2/awsapi"
+	iamawsapi "github.com/mulgadc/spinifex/spinifex/domains/iam/awsapi"
+	rdsawsapi "github.com/mulgadc/spinifex/spinifex/domains/rds/awsapi"
+	route53awsapi "github.com/mulgadc/spinifex/spinifex/domains/route53/awsapi"
+	sqsawsapi "github.com/mulgadc/spinifex/spinifex/domains/sqs/awsapi"
+	stsawsapi "github.com/mulgadc/spinifex/spinifex/domains/sts/awsapi"
 	"github.com/mulgadc/spinifex/spinifex/ingress/aws/dispatch"
 
 	_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
@@ -48,10 +57,26 @@ func fail(err error) {
 func compareAll() ([]awsmodel.OperationCoverage, error) {
 	// Registered services contribute their declared inventory, which needs no
 	// live dependencies; nothing here serves a request.
-	inventories := gateway.AWSOperationInventory(map[string]dispatch.Inventory{
-		awsapi.ServiceName:    awsapi.OperationInventory(),
-		acmawsapi.ServiceName: acmawsapi.OperationInventory(),
+	inventories := dispatch.AWSOperationInventory(map[string]dispatch.Inventory{
+		awsapi.ServiceName:        awsapi.OperationInventory(),
+		acmawsapi.ServiceName:     acmawsapi.OperationInventory(),
+		route53awsapi.ServiceName: route53awsapi.OperationInventory(),
+		elbv2awsapi.InventoryName: elbv2awsapi.OperationInventory(),
+		rdsawsapi.ServiceName:     rdsawsapi.OperationInventory(),
+		ec2awsapi.ServiceName:     ec2awsapi.OperationInventory(),
+		ecsawsapi.ServiceName:     ecsawsapi.OperationInventory(),
+		eksawsapi.ServiceName:     eksawsapi.OperationInventory(),
+		iamawsapi.ServiceName:     iamawsapi.OperationInventory(),
+		sqsawsapi.ServiceName:     sqsawsapi.OperationInventory(),
+		stsawsapi.ServiceName:     stsawsapi.OperationInventory(),
 	})
+	// The loop below walks the embedded models, so a served service with no
+	// model would otherwise drop out of the report without a word.
+	for name := range inventories {
+		if !slices.Contains(awsmodel.Services(), awsmodel.Service(name)) {
+			return nil, fmt.Errorf("no embedded AWS model for served service %s", name)
+		}
+	}
 	coverages := make([]awsmodel.OperationCoverage, 0, len(awsmodel.Services()))
 	for _, service := range awsmodel.Services() {
 		inventory, ok := inventories[string(service)]

cmd/eks-etcd-member/main.go

@@ -0,0 +1,50 @@
+// eks-etcd-member removes a dead peer from the embedded etcd cluster of an EKS
+// K3s control-plane VM. A member that is already gone exits 0; any other
+// failure exits non-zero.
+package main
+
+import (
+	"context"
+	"flag"
+	"fmt"
+	"io"
+	"os"
+	"time"
+
+	_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
+
+	"github.com/mulgadc/spinifex/spinifex/agents/eks/etcdmember"
+)
+
+func main() {
+	os.Exit(run(os.Args[1:], os.Stdout, os.Stderr))
+}
+
+func run(args []string, stdout, stderr io.Writer) int {
+	if len(args) < 1 || args[0] != "remove" {
+		fmt.Fprintln(stderr, "usage: eks-etcd-member remove --peer-ip 10.0.0.12 [--self-ip 10.0.0.10]")
+		return 2
+	}
+	fs := flag.NewFlagSet("remove", flag.ContinueOnError)
+	fs.SetOutput(stderr)
+	var cfg etcdmember.Config
+	endpoint := fs.String("endpoint", etcdmember.DefaultEndpoint, "etcd client endpoint")
+	certDir := fs.String("cert-dir", etcdmember.DefaultCertDir, "directory holding the K3s etcd client.crt, client.key and server-ca.crt")
+	timeout := fs.Duration("timeout", 30*time.Second, "overall deadline")
+	fs.StringVar(&cfg.PeerIP, "peer-ip", "", "peer URL host of the member to remove")
+	fs.StringVar(&cfg.SelfIP, "self-ip", "", "this host's own address; removal of it is refused")
+	if err := fs.Parse(args[1:]); err != nil {
+		return 2
+	}
+
+	ctx, cancel := context.WithTimeout(context.Background(), *timeout)
+	defer cancel()
+
+	cli, err := etcdmember.Dial(*endpoint, *certDir, *timeout)
+	if err != nil {
+		fmt.Fprintln(stderr, "etcd member remove failed:", err)
+		return 1
+	}
+	defer func() { _ = cli.Close() }()
+	return etcdmember.Run(ctx, cli, cfg, stdout)
+}

cmd/eks-etcd-member/main_test.go

@@ -0,0 +1,63 @@
+package main
+
+import (
+	"bytes"
+	"testing"
+
+	"github.com/mulgadc/spinifex/internal/testkit/fakeetcd"
+	"github.com/stretchr/testify/assert"
+)
+
+var fakeMembers = []fakeetcd.Member{
+	{ID: 0xa1, Name: "cp-a", PeerURLs: []string{"https://10.0.0.10:2380"}},
+	{ID: 0xb2, Name: "cp-b", PeerURLs: []string{"https://10.0.0.12:2380"}},
+}
+
+func TestRunRequiresTheRemoveSubcommand(t *testing.T) {
+	t.Parallel()
+	for _, args := range [][]string{nil, {"list"}} {
+		var out, errOut bytes.Buffer
+		assert.Equal(t, 2, run(args, &out, &errOut))
+		assert.Contains(t, errOut.String(), "usage: eks-etcd-member remove")
+	}
+}
+
+func TestRunRejectsUnknownFlags(t *testing.T) {
+	t.Parallel()
+	var out, errOut bytes.Buffer
+	assert.Equal(t, 2, run([]string{"remove", "--bogus"}, &out, &errOut))
+}
+
+func TestRunFailsWhenEtcdCertificatesAreMissing(t *testing.T) {
+	t.Parallel()
+	var out, errOut bytes.Buffer
+	code := run([]string{"remove", "--peer-ip", "10.0.0.12", "--cert-dir", t.TempDir()}, &out, &errOut)
+	assert.Equal(t, 1, code)
+	assert.Contains(t, errOut.String(), "etcd member remove failed")
+}
+
+func TestRunRemovesTheNamedPeer(t *testing.T) {
+	t.Parallel()
+	srv := fakeetcd.Start(t, 0xa1, fakeMembers)
+	var out, errOut bytes.Buffer
+
+	code := run([]string{"remove", "--peer-ip", "10.0.0.12", "--self-ip", "10.0.0.10",
+		"--endpoint", srv.Endpoint, "--cert-dir", srv.CertDir}, &out, &errOut)
+
+	assert.Zero(t, code, errOut.String()+out.String())
+	assert.Contains(t, out.String(), "removed etcd member 10.0.0.12")
+	assert.Equal(t, []uint64{0xb2}, srv.Removed())
+}
+
+func TestRunTreatsAnAbsentPeerAsSuccess(t *testing.T) {
+	t.Parallel()
+	srv := fakeetcd.Start(t, 0xa1, fakeMembers)
+	var out, errOut bytes.Buffer
+
+	code := run([]string{"remove", "--peer-ip", "10.0.0.99",
+		"--endpoint", srv.Endpoint, "--cert-dir", srv.CertDir}, &out, &errOut)
+
+	assert.Zero(t, code)
+	assert.Contains(t, out.String(), "member already absent")
+	assert.Empty(t, srv.Removed())
+}

cmd/eks-gateway-fetch/main.go

@@ -6,6 +6,7 @@
 //
 //	eks-gateway-fetch -resource addons   # GET /clusters/{cluster}/internal-addons/{accountId}
 //	eks-gateway-fetch -resource recovery -instance-id i-… # GET /clusters/{cluster}/internal-recovery/{accountId}/{instanceId}
+//	eks-gateway-fetch -resource workers  # GET /clusters/{cluster}/internal-workers/{accountId}
 //
 // Flags default to environment variables seeded by cloud-init:
 // EKS_GATEWAY_URL, EKS_GATEWAY_CA, EKS_ACCESS_KEY, EKS_SECRET_KEY, EKS_REGION,
@@ -33,7 +34,7 @@ func main() {
 	flag.StringVar(&cfg.AccountID, "account-id", os.Getenv("EKS_ACCOUNT_ID"), "Cluster account ID")
 	flag.StringVar(&cfg.ClusterName, "cluster", os.Getenv("EKS_CLUSTER_NAME"), "Cluster name")
 	flag.StringVar(&cfg.InstanceID, "instance-id", os.Getenv("EKS_INSTANCE_ID"), "This member's EC2 instance ID (recovery resource)")
-	flag.StringVar(&cfg.Resource, "resource", "addons", "Resource to fetch: addons | recovery")
+	flag.StringVar(&cfg.Resource, "resource", "addons", "Resource to fetch: addons | recovery | workers")
 	flag.Parse()
 
 	if err := gatewayfetch.Run(context.Background(), cfg); err != nil {

cmd/eks-pod-identity-webhook/main.go

@@ -0,0 +1,32 @@
+// Command eks-pod-identity-webhook is the IRSA mutating admission webhook run
+// on every EKS control-plane server; the implementation is in
+// agents/eks/podidentity. It listens on loopback only and is called by the
+// local kube-apiserver.
+package main
+
+import (
+	"context"
+	"log/slog"
+	"os"
+	"os/signal"
+	"syscall"
+
+	_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
+
+	"github.com/mulgadc/spinifex/spinifex/agents/eks/podidentity"
+)
+
+func main() {
+	log := slog.New(slog.NewTextHandler(os.Stderr, nil))
+	cfg, err := podidentity.ParseServiceConfig(os.Args[1:], os.Getenv, os.Stderr)
+	if err != nil {
+		log.Error("invalid configuration", "error", err)
+		os.Exit(2)
+	}
+	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
+	defer stop()
+	if err := podidentity.Run(ctx, cfg, log); err != nil {
+		log.Error("pod identity webhook stopped", "error", err)
+		os.Exit(1)
+	}
+}

cmd/eks-token-webhook/main.go

@@ -29,6 +29,7 @@ func loadConfig() (tokenwebhook.Config, error) {
 		CertPath:    envOr("EKS_WEBHOOK_CERT", "/etc/spinifex-eks/token-webhook.crt"),
 		KeyPath:     envOr("EKS_WEBHOOK_KEY", "/etc/spinifex-eks/token-webhook.key"),
 		Kubeconfig:  envOr("EKS_WEBHOOK_KUBECONFIG", "/etc/spinifex-eks/token-webhook.kubeconfig"),
+		ReadyAddr:   os.Getenv("EKS_WEBHOOK_READY_ADDR"),
 	}
 	// Static SigV4 creds are optional: when absent, eksgw.New signs with the
 	// AWS SDK chain (IMDS instance-role creds), the same path the CP VM's

cmd/lambda-agent/main.go

@@ -0,0 +1,24 @@
+// lambda-agent runs inside a Lambda guest.
+//
+// A normal build reads its per-generation identity from fw_cfg, unloads
+// fw_cfg so nothing started later can read the guest key, and opens the
+// authenticated control connection to the host. It accepts no invocations and
+// never starts the function's bootstrap: the message protocol and grant
+// fencing do not exist yet. The agent never listens for the host.
+//
+// A lambdatestlink build instead relays invocations over an unauthenticated
+// test carrier for local proofs; it is not the guest protocol.
+package main
+
+import (
+	"context"
+	"flag"
+	"os"
+)
+
+func main() { os.Exit(run()) }
+
+func run() int {
+	flag.Parse()
+	return start(context.Background())
+}

cmd/lambda-agent/start_tcp.go

@@ -0,0 +1,84 @@
+//go:build !lambdatestlink
+
+package main
+
+import (
+	"context"
+	"crypto/tls"
+	"errors"
+	"fmt"
+	"io"
+	"log/slog"
+
+	"github.com/mulgadc/spinifex/spinifex/agents/lambda/hostlink"
+)
+
+// Exit statuses: the guest refused to start, the control connection failed,
+// or the host closed it.
+const (
+	exitRefused = 2
+	exitLink    = 1
+	exitClosed  = 0
+)
+
+// errNoProtocol is returned when the host sends anything: this build defines no
+// message protocol, so it accepts no work.
+var errNoProtocol = errors.New("host sent data, but this build accepts no invocations")
+
+type startup struct {
+	fwCfgDir  string
+	fwCfgRoot string
+	unload    func() error
+}
+
+func start(ctx context.Context) int {
+	s := startup{fwCfgDir: hostlink.FwCfgDir, fwCfgRoot: hostlink.FwCfgRoot, unload: hostlink.UnloadFwCfg}
+	code, err := s.run(ctx)
+	if err != nil {
+		slog.Error("lambda-agent stopping", "exit", code, "err", err)
+	}
+	return code
+}
+
+// run loads the identity, seals fw_cfg whether or not the load succeeded,
+// then authenticates to the host and holds the connection until it closes.
+func (s startup) run(ctx context.Context) (int, error) {
+	id, loadErr := hostlink.LoadIdentity(s.fwCfgDir)
+	if err := hostlink.SealFwCfg(s.fwCfgRoot, s.unload); err != nil {
+		return exitRefused, errors.Join(loadErr, err)
+	}
+	if loadErr != nil {
+		return exitRefused, loadErr
+	}
+	conn, err := hostlink.Dial(ctx, id)
+	if err != nil {
+		if ctx.Err() != nil {
+			err = fmt.Errorf("%w: %w", err, context.Cause(ctx))
+		}
+		return exitLink, err
+	}
+	defer conn.Close()
+	slog.Info("control connection open; invocations are not enabled in this build", "host", id.HostAddr)
+	if err := awaitHostClose(ctx, conn); err != nil {
+		return exitLink, err
+	}
+	return exitClosed, nil
+}
+
+// awaitHostClose reads until the host closes the connection. Under TLS 1.3 a
+// host that rejects the guest certificate is only seen here, on first read.
+func awaitHostClose(ctx context.Context, conn *tls.Conn) error {
+	stop := context.AfterFunc(ctx, func() { _ = conn.Close() })
+	defer stop()
+	n, err := conn.Read(make([]byte, 1))
+	switch {
+	case n > 0:
+		return errNoProtocol
+	case errors.Is(err, io.EOF):
+		return nil
+	case ctx.Err() != nil:
+		return fmt.Errorf("control connection: %w", context.Cause(ctx))
+	default:
+		return fmt.Errorf("control connection: %w", err)
+	}
+}

cmd/lambda-agent/start_tcp_test.go

@@ -0,0 +1,256 @@
+//go:build !lambdatestlink
+
+package main
+
+import (
+	"context"
+	"crypto/tls"
+	"errors"
+	"net"
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+
+	lambdav1 "github.com/mulgadc/spinifex/contracts/lambda/v1"
+	"github.com/mulgadc/spinifex/spinifex/runtime/compute/lambda/guestconn"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+const scope lambdav1.Scope = "ap-southeast-2/000000000001/fn-1/env-1/gen-1"
+
+// guest is a fake fw_cfg tree; unload removes it as unloading the module would.
+type guest struct {
+	root     string
+	unloads  int
+	sealedAt chan struct{}
+}
+
+func newGuest(t *testing.T, entries map[string][]byte) *guest {
+	t.Helper()
+	g := &guest{root: filepath.Join(t.TempDir(), "qemu_fw_cfg"), sealedAt: make(chan struct{}, 1)}
+	for name, b := range entries {
+		dir := filepath.Join(g.root, "by_name", lambdav1.FwCfgPrefix, name)
+		require.NoError(t, os.MkdirAll(dir, 0o700))
+		require.NoError(t, os.WriteFile(filepath.Join(dir, "raw"), b, 0o600))
+	}
+	return g
+}
+
+func (g *guest) startup() startup {
+	return startup{
+		fwCfgDir:  filepath.Join(g.root, "by_name", lambdav1.FwCfgPrefix),
+		fwCfgRoot: g.root,
+		unload: func() error {
+			g.unloads++
+			g.sealedAt <- struct{}{}
+			return os.RemoveAll(g.root)
+		},
+	}
+}
+
+type host struct {
+	ln   *net.TCPListener
+	gen  *guestconn.Generation
+	conn chan accepted
+}
+
+func newHost(t *testing.T) *host {
+	t.Helper()
+	ln, err := net.ListenTCP("tcp", &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1)})
+	require.NoError(t, err)
+	t.Cleanup(func() { ln.Close() })
+	gen, err := guestconn.NewGeneration(scope, time.Hour)
+	require.NoError(t, err)
+	return &host{ln: ln, gen: gen, conn: make(chan accepted, 1)}
+}
+
+func (h *host) entries() map[string][]byte { return h.gen.GuestEntries(h.ln.Addr().String()) }
+
+type accepted struct {
+	conn *tls.Conn
+	err  error
+}
+
+// accept runs the host end until ctx ends and reports what it accepted.
+func (h *host) accept(ctx context.Context) {
+	go func() {
+		c, err := h.gen.AcceptGuest(ctx, h.ln, 2*time.Second)
+		h.conn <- accepted{c, err}
+	}()
+}
+
+// guest waits for the host to authenticate the guest.
+func (h *host) guest(t *testing.T) *tls.Conn {
+	t.Helper()
+	a := wait(t, h.conn, "host to authenticate the guest")
+	require.NoError(t, a.err)
+	return a.conn
+}
+
+// wait bounds every test wait so a stalled host or startup fails promptly.
+func wait[T any](t *testing.T, ch <-chan T, what string) T {
+	t.Helper()
+	select {
+	case v := <-ch:
+		return v
+	case <-time.After(10 * time.Second):
+		t.Fatalf("timed out waiting for %s", what)
+		panic("unreachable")
+	}
+}
+
+type result struct {
+	code int
+	err  error
+}
+
+func runAsync(ctx context.Context, s startup) <-chan result {
+	out := make(chan result, 1)
+	go func() {
+		code, err := s.run(ctx)
+		out <- result{code, err}
+	}()
+	return out
+}
+
+func TestStartupAuthenticatesAfterSealingFwCfg(t *testing.T) {
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	h := newHost(t)
+	h.accept(ctx)
+	g := newGuest(t, h.entries())
+	done := runAsync(ctx, g.startup())
+
+	conn := h.guest(t)
+	select {
+	case <-g.sealedAt:
+	default:
+		t.Fatal("guest dialled before fw_cfg was sealed")
+	}
+	_, err := os.Stat(g.root)
+	assert.ErrorIs(t, err, os.ErrNotExist, "fw_cfg still readable while connected")
+
+	require.NoError(t, conn.Close())
+	res := wait(t, done, "startup to return")
+	assert.Equal(t, exitClosed, res.code)
+	assert.NoError(t, res.err)
+	assert.Equal(t, 1, g.unloads)
+}
+
+func TestStartupRefusesHostData(t *testing.T) {
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	h := newHost(t)
+	h.accept(ctx)
+	done := runAsync(ctx, newGuest(t, h.entries()).startup())
+
+	conn := h.guest(t)
+	defer conn.Close()
+	_, err := conn.Write([]byte{0})
+	require.NoError(t, err)
+	res := wait(t, done, "startup to return")
+	assert.Equal(t, exitLink, res.code)
+	assert.ErrorIs(t, res.err, errNoProtocol)
+}
+
+// The host pins another generation's guest; the guest learns of the refusal
+// on its first read and fails with a link error, never a clean close.
+func TestStartupFailsWhenHostRefusesGuest(t *testing.T) {
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	h := newHost(t)
+	h.accept(ctx)
+	other, err := guestconn.NewGeneration(scope, time.Hour)
+	require.NoError(t, err)
+	entries := h.entries()
+	wrong := other.GuestEntries(h.ln.Addr().String())
+	entries[lambdav1.EntryGuestKey], entries[lambdav1.EntryGuestCert] = wrong[lambdav1.EntryGuestKey], wrong[lambdav1.EntryGuestCert]
+	g := newGuest(t, entries)
+
+	res := wait(t, runAsync(ctx, g.startup()), "startup to return")
+	assert.Equal(t, exitLink, res.code)
+	require.Error(t, res.err)
+	assert.Contains(t, res.err.Error(), "bad certificate")
+	assert.Equal(t, 1, g.unloads)
+	select {
+	case a := <-h.conn:
+		require.Error(t, a.err, "host accepted a guest it does not pin")
+	default:
+	}
+}
+
+func TestStartupRefusesWithoutDialling(t *testing.T) {
+	h := newHost(t)
+	cases := map[string]struct {
+		entries map[string][]byte
+		unload  func(g *guest) func() error
+		want    string
+	}{
+		"missing entry": {
+			entries: func() map[string][]byte { e := h.entries(); delete(e, lambdav1.EntryHostCert); return e }(),
+			want:    "read host-cert",
+		},
+		"fw_cfg stays readable": {
+			entries: h.entries(),
+			unload:  func(*guest) func() error { return func() error { return errors.New("module is built in") } },
+			want:    "fw_cfg still readable",
+		},
+		"unload claims success but tree remains": {
+			entries: h.entries(),
+			unload:  func(*guest) func() error { return func() error { return nil } },
+			want:    "unload reported success",
+		},
+	}
+	for name, tc := range cases {
+		t.Run(name, func(t *testing.T) {
+			g := newGuest(t, tc.entries)
+			s := g.startup()
+			if tc.unload != nil {
+				s.unload = tc.unload(g)
+			}
+			code, err := s.run(context.Background())
+			assert.Equal(t, exitRefused, code)
+			assert.ErrorContains(t, err, tc.want)
+		})
+	}
+
+	// Nothing dialled the host in any case above.
+	require.NoError(t, h.ln.SetDeadline(time.Now().Add(50*time.Millisecond)))
+	_, err := h.ln.Accept()
+	var ne net.Error
+	require.ErrorAs(t, err, &ne)
+	assert.True(t, ne.Timeout())
+}
+
+// A missing entry still seals fw_cfg: the key may be present even when
+// another entry is not.
+func TestStartupSealsEvenWhenIdentityIsIncomplete(t *testing.T) {
+	h := newHost(t)
+	e := h.entries()
+	delete(e, lambdav1.EntryHostAddr)
+	g := newGuest(t, e)
+	code, err := g.startup().run(context.Background())
+	assert.Equal(t, exitRefused, code)
+	assert.ErrorContains(t, err, "read host-addr")
+	assert.Equal(t, 1, g.unloads)
+	_, statErr := os.Stat(g.root)
+	assert.ErrorIs(t, statErr, os.ErrNotExist)
+}
+
+func TestStartupStopsWithContext(t *testing.T) {
+	ctx, cancel := context.WithCancelCause(context.Background())
+	defer cancel(nil)
+	h := newHost(t)
+	h.accept(ctx)
+	done := runAsync(ctx, newGuest(t, h.entries()).startup())
+	conn := h.guest(t)
+	defer conn.Close()
+
+	reason := errors.New("guest shutting down")
+	cancel(reason)
+	res := wait(t, done, "startup to return")
+	assert.Equal(t, exitLink, res.code)
+	assert.ErrorIs(t, res.err, reason)
+}

cmd/lambda-agent/start_testlink.go

@@ -0,0 +1,87 @@
+//go:build lambdatestlink
+
+package main
+
+import (
+	"context"
+	"errors"
+	"flag"
+	"fmt"
+	"log/slog"
+	"net"
+	"net/http"
+	"os"
+	"time"
+
+	"github.com/mulgadc/spinifex/spinifex/agents/lambda/agent"
+	"github.com/mulgadc/spinifex/spinifex/agents/lambda/hostlink"
+	"github.com/mulgadc/spinifex/spinifex/agents/lambda/runtimeapi"
+)
+
+const (
+	runtimeAddr = "127.0.0.1:9001"
+	taskRoot    = "/var/task"
+)
+
+var (
+	linkPath = flag.String("link", os.Getenv("LAMBDA_AGENT_LINK"), "test carrier device path")
+	handler  = flag.String("handler", os.Getenv("_HANDLER"), "function handler passed to the bootstrap")
+)
+
+func start(parent context.Context) int {
+	if *linkPath == "" {
+		slog.Error("no test carrier configured; refusing to start")
+		return 2
+	}
+	// The bootstrap is tenant code; fw_cfg must be unreadable before it runs.
+	if err := hostlink.SealFwCfg(hostlink.FwCfgRoot, hostlink.UnloadFwCfg); err != nil {
+		slog.Error("seal fw_cfg; refusing to start", "err", err)
+		return 2
+	}
+	dev, err := os.OpenFile(*linkPath, os.O_RDWR, 0)
+	if err != nil {
+		slog.Error("open test carrier; refusing to start", "err", err)
+		return 2
+	}
+	defer dev.Close()
+	link := hostlink.NewTestStream(dev)
+
+	srv := runtimeapi.New()
+	ln, err := net.Listen("tcp", runtimeAddr)
+	if err != nil {
+		slog.Error("listen runtime api", "err", err)
+		return 1
+	}
+	hs := &http.Server{Handler: srv.Handler(), ReadHeaderTimeout: 10 * time.Second}
+	go func() { _ = hs.Serve(ln) }()
+	defer hs.Close()
+
+	ctx, cancel := context.WithCancelCause(parent)
+	defer cancel(nil)
+
+	cmd, err := agent.StartBootstrap(ctx, taskRoot+"/bootstrap", []string{
+		"AWS_LAMBDA_RUNTIME_API=" + runtimeAddr,
+		"_HANDLER=" + *handler,
+		"LAMBDA_TASK_ROOT=" + taskRoot,
+		"PATH=/bin:/sbin:/usr/bin",
+	}, os.Stdout, os.Stderr)
+	if err != nil {
+		slog.Error("start bootstrap", "err", err)
+		return 1
+	}
+	exited := make(chan struct{})
+	go func() {
+		defer close(exited)
+		err := cmd.Wait()
+		cancel(fmt.Errorf("bootstrap exited: %w", err))
+	}()
+
+	err = agent.Run(ctx, link, srv)
+	cancel(errors.New("agent stopping"))
+	<-exited
+	if err != nil {
+		slog.Error("agent stopped", "err", err)
+		return 1
+	}
+	return 0
+}

cmd/sqs-worker/main.go

@@ -0,0 +1,140 @@
+// sqs-worker runs inside the SQS microVM, next to its own SQS NATS server.
+//
+// Usage:
+//
+//	sqs-worker boot
+//	sqs-worker ready
+//
+// boot selects and mounts the guest's data disk named by fw_cfg, asks the
+// gateway for its receipt key decision with its instance-role credentials,
+// then loads the recorded key, or creates one on a new volume and records its
+// fingerprint, before the guest's SQS store first starts on that disk. ready
+// then keeps checking the guest's own loopback JetStream, keeping a readiness
+// marker only while it answers, and serves SQS owner calls against it, pulling
+// them from the gateway with the same credentials.
+//
+// Serving is refused unless the gateway reports a single-node host cluster.
+// A disk whose receipt key is missing or does not match the recorded one
+// refuses to serve; recovering it needs a procedure not yet designed, and
+// nothing deletes or replaces the volume.
+package main
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"os"
+	"os/signal"
+	"syscall"
+	"time"
+
+	"github.com/nats-io/nats.go/jetstream"
+
+	"github.com/mulgadc/spinifex/internal/gwsign"
+	"github.com/mulgadc/spinifex/internal/sqsworkergw"
+	"github.com/mulgadc/spinifex/spinifex/agents/sqs/workerboot"
+	"github.com/mulgadc/spinifex/spinifex/agents/sqs/workerkey"
+	"github.com/mulgadc/spinifex/spinifex/domains/sqs/workerserve"
+	"github.com/mulgadc/spinifex/spinifex/foundation/state/clustersize"
+)
+
+// The cluster CA the daemon supplies; the only root the client trusts.
+const caPath = "/sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/ca-cert/raw"
+
+// newClaimer signs with the IMDS instance-role credentials only; there is
+// no static-key fallback.
+func newClaimer() workerboot.NewClaimer {
+	return func(region, gatewayURL string) (workerkey.Claimer, error) {
+		c, err := newClient(region, gatewayURL)
+		if err != nil {
+			return nil, err
+		}
+		return workerboot.GatewayClaimer{Client: c}, nil
+	}
+}
+
+func newClient(region, gatewayURL string) (*sqsworkergw.Client, error) {
+	ca, err := os.ReadFile(caPath)
+	if err != nil {
+		return nil, fmt.Errorf("read cluster CA: %w", err)
+	}
+	return sqsworkergw.New(gatewayURL, region, ca, gwsign.NewInstanceRole())
+}
+
+func main() {
+	if len(os.Args) != 2 || (os.Args[1] != "boot" && os.Args[1] != "ready") {
+		fmt.Fprintln(os.Stderr, "usage: sqs-worker boot|ready")
+		os.Exit(2)
+	}
+	logf := func(msg string) { fmt.Fprintf(os.Stderr, "[sqs-worker] %s\n", msg) }
+	run := boot
+	if os.Args[1] == "ready" {
+		run = ready
+	}
+	if err := run(logf); err != nil {
+		if os.Args[1] == "boot" && workerboot.Transient(err) {
+			logf("boot not settled, will retry: " + err.Error())
+			os.Exit(exitTransient)
+		}
+		logf("FATAL: " + err.Error())
+		os.Exit(1)
+	}
+}
+
+// exitTransient tells the guest's supervisor that boot may be run again.
+const exitTransient = 75
+
+func config(logf func(string)) workerboot.Config {
+	return workerboot.Config{
+		FwCfgDir:      "/sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/sqs-worker",
+		SysBlock:      "/sys/block",
+		DevDir:        "/dev",
+		MountInfo:     "/proc/self/mountinfo",
+		DataDir:       "/data",
+		RunDir:        "/run/sqs-worker",
+		Disk:          workerboot.SystemDisk{E2fsck: "/sbin/e2fsck"},
+		IsBlockDevice: workerboot.IsBlockDevice,
+		NewClaimer:    newClaimer(),
+		Attempts:      60,
+		Wait:          5 * time.Second,
+		Log:           logf,
+	}
+}
+
+func boot(logf func(string)) error {
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
+	defer cancel()
+	_, err := workerboot.Boot(ctx, config(logf))
+	return err
+}
+
+// ready runs until init stops it; the marker is removed on the way out.
+func ready(logf func(string)) error {
+	ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
+	defer cancel()
+	// The store is one guest, so its streams and buckets are R1.
+	if err := clustersize.Declare(1); err != nil {
+		return err
+	}
+	err := workerboot.Watch(ctx, workerboot.ReadyConfig{
+		Config:      config(logf),
+		Checker:     workerboot.NATSChecker{},
+		CallTimeout: 30 * time.Second,
+		Refresh:     30 * time.Second,
+		Refused:     5 * time.Minute,
+		Serve: &workerboot.ServeConfig{
+			Connector: workerboot.NATSConnector{},
+			Run: func(ctx context.Context, region, gatewayURL string, js jetstream.JetStream, receiptKey []byte) error {
+				c, err := newClient(region, gatewayURL)
+				if err != nil {
+					return err
+				}
+				return workerserve.Run(ctx, c, js, receiptKey, logf)
+			},
+		},
+	})
+	if errors.Is(err, context.Canceled) {
+		return nil
+	}
+	return err
+}

contracts/admission/v1/README.md

@@ -10,7 +10,7 @@ quota state.
 a request/reply route on which a gateway, in the `awsgw-quota-seal` queue
 group, irreversibly seals one account's quota override record.
 `domains/admission/quota` owns it: `awsgw` serves it.
-`accountteardown` is the consumer: it calls this subject to fence every
+`operator/accountlifecycle` is the consumer: it calls this subject to fence every
 override writer at the quota owner before the account record is removed.
 
 The request names the account in its body (`account_id`) and may carry an
@@ -31,10 +31,20 @@ Any current gateway may answer; the leader-accepted revision is the evidence
 whichever one does, and the override bucket is opened at seal time even on a
 gateway whose quotas were disabled at startup.
 
+## Seal-quota-override capability
+
+`AccountQuotaOverrideSealCapabilitySubject(node)` (`admission.v1.AccountQuotaOverrideSealCapability.<node>`) is a per-node, read-only request/reply route on which `operator/accountlifecycle` asks one configured gateway whether it supports the quota override seal, immediately before every seal attempt.
+The quota owner (`domains/admission/quota`) owns it: `awsgw` serves it with no queue group, so a request addressed to one node reaches only subscribers configured as that node, and two gateways sharing a node ID both answer.
+The consumer's quota seal reaper probes every configured node that runs `awsgw` and sends no seal request unless every one answers compatible.
+
+`AccountQuotaOverrideSealCapabilityVersion` is a claim about the whole binary's writers, not about the responder: version 1 means every quota override writer in the binary is revision-guarded and refuses a sealed record, and the binary serves the seal subject.
+A gateway binary without this responder has no subscriber, so the probe for that node reads as incomplete (`no-reply`), the same as a stopped or partitioned node.
+The reply proves nothing beyond the moment it was sent: a gateway downgraded or replaced after a successful probe is outside the guarantee.
+
 ### Compatibility boundary
 
-The subject name carries the contract major; `account_quota_seal_test.go`
-preserves the version-1 JSON shape.
+The subject name carries the contract major; `account_quota_seal_test.go` and `account_quota_seal_capability_test.go`
+preserve the version-1 JSON shapes and the subject builder.
 Producer and consumer are in the coordinated Spinifex release, so the window
 is N and N-1.
 A gateway that predates the subject has no subscription, so the request gets

contracts/admission/v1/account_quota_seal_capability.go

@@ -0,0 +1,39 @@
+package admissionv1
+
+import "strings"
+
+// AccountQuotaOverrideSealCapabilityVersion is the version this binary reports.
+// Version 1: every quota override writer in the binary is revision-guarded and
+// refuses a sealed record, and the seal subject is served.
+const AccountQuotaOverrideSealCapabilityVersion = 1
+
+// AccountQuotaOverrideSealCapabilitySubject is one gateway's read-only
+// seal-capability route, addressed per node so a probe reaches exactly the
+// node it asks about.
+func AccountQuotaOverrideSealCapabilitySubject(node string) string {
+	return "admission.v1.AccountQuotaOverrideSealCapability." + node
+}
+
+// ValidSealCapabilityNode reports whether node is exactly one NATS subject
+// token. Responder and probe both check it before building a subject, so
+// neither can address a wildcard or another node's route.
+func ValidSealCapabilityNode(node string) bool {
+	return node != "" && !strings.ContainsAny(node, ".*>\t\n\r ")
+}
+
+// AccountQuotaOverrideSealCapabilityRequest asks node whether it supports the
+// quota override seal. CorrelationRef is an opaque caller reference echoed for
+// logs; it grants nothing and is never a fence or authority to write.
+type AccountQuotaOverrideSealCapabilityRequest struct {
+	Node           string `json:"node"`
+	CorrelationRef string `json:"correlation_ref,omitempty"`
+}
+
+// AccountQuotaOverrideSealCapabilityReply reports the responding node's
+// capability version. It carries no account data and proves nothing beyond
+// the moment it was sent.
+type AccountQuotaOverrideSealCapabilityReply struct {
+	Node              string `json:"node"`
+	CapabilityVersion int    `json:"capability_version"`
+	CorrelationRef    string `json:"correlation_ref,omitempty"`
+}

contracts/admission/v1/account_quota_seal_capability_test.go

@@ -0,0 +1,43 @@
+package admissionv1
+
+import (
+	"encoding/json"
+	"testing"
+
+	"github.com/stretchr/testify/require"
+)
+
+func TestAccountQuotaOverrideSealCapabilitySubject(t *testing.T) {
+	require.Equal(t, "admission.v1.AccountQuotaOverrideSealCapability.node-1", AccountQuotaOverrideSealCapabilitySubject("node-1"))
+	require.Equal(t, "admission.v1.AccountQuotaOverrideSealCapability.node-2", AccountQuotaOverrideSealCapabilitySubject("node-2"))
+}
+
+func TestAccountQuotaOverrideSealCapabilityRequestJSONContract(t *testing.T) {
+	data, err := json.Marshal(AccountQuotaOverrideSealCapabilityRequest{Node: "node-1", CorrelationRef: "req-1"})
+	require.NoError(t, err)
+	require.JSONEq(t, `{"node":"node-1","correlation_ref":"req-1"}`, string(data))
+
+	data, err = json.Marshal(AccountQuotaOverrideSealCapabilityRequest{Node: "node-1"})
+	require.NoError(t, err)
+	require.JSONEq(t, `{"node":"node-1"}`, string(data))
+}
+
+func TestAccountQuotaOverrideSealCapabilityReplyJSONContract(t *testing.T) {
+	reply := AccountQuotaOverrideSealCapabilityReply{Node: "node-1", CapabilityVersion: 1, CorrelationRef: "req-1"}
+	data, err := json.Marshal(reply)
+	require.NoError(t, err)
+	require.JSONEq(t, `{"node":"node-1","capability_version":1,"correlation_ref":"req-1"}`, string(data))
+
+	data, err = json.Marshal(AccountQuotaOverrideSealCapabilityReply{Node: "node-1", CapabilityVersion: 1})
+	require.NoError(t, err)
+	require.JSONEq(t, `{"node":"node-1","capability_version":1}`, string(data))
+}
+
+func TestValidSealCapabilityNode(t *testing.T) {
+	for _, node := range []string{"node-1", "node_2", "n3"} {
+		require.True(t, ValidSealCapabilityNode(node), node)
+	}
+	for _, node := range []string{"", "a.b", "*", ">", "node-*", "node >", "node\t1", "node\n1"} {
+		require.False(t, ValidSealCapabilityNode(node), "%q", node)
+	}
+}

contracts/cluster/v1/facts.go

@@ -0,0 +1,211 @@
+package clusterv1
+
+import "regexp"
+
+const (
+	// NodeFactsSubject fans out a request for each daemon's hardware and
+	// cabling description. Its reply payload is NodeFacts.
+	NodeFactsSubject = "spinifex.node.facts"
+
+	nodeNetTestPrefix = "spinifex.node.nettest."
+)
+
+var nodeIDPattern = regexp.MustCompile(`^[A-Za-z0-9_-]{1,64}$`)
+
+// ValidNodeID reports whether a node name is safe to place in a subject: no
+// NATS token separator or wildcard can get through.
+func ValidNodeID(node string) bool {
+	return nodeIDPattern.MatchString(node)
+}
+
+// NetTestServeSubject asks one node to start a one-shot throughput server.
+func NetTestServeSubject(node string) string {
+	return nodeNetTestPrefix + node + ".serve"
+}
+
+// NetTestRunSubject asks one node to run the client side of a throughput test.
+func NetTestRunSubject(node string) string {
+	return nodeNetTestPrefix + node + ".run"
+}
+
+// Planes a node's NICs can carry, named as the installer names them.
+const (
+	PlaneWAN = "wan"
+	PlaneLAN = "lan"
+	PlaneVPC = "vpc"
+)
+
+// NetTestPort is the TCP port reserved for node-to-node throughput tests.
+const NetTestPort = 5001
+
+// NodeFacts describes one node's hardware, OS and cabling. Fields a host
+// cannot report are left empty, and Sources says why.
+type NodeFacts struct {
+	Node        string            `json:"node"`
+	CollectedAt string            `json:"collected_at,omitempty"`
+	FormFactor  string            `json:"form_factor"`
+	Virt        string            `json:"virt,omitempty"`
+	System      SystemFacts       `json:"system"`
+	OS          OSFacts           `json:"os"`
+	CPU         CPUFacts          `json:"cpu"`
+	Memory      MemoryFacts       `json:"memory"`
+	NICs        []NICFacts        `json:"nics"`
+	Planes      []PlaneFacts      `json:"planes"`
+	Sources     map[string]string `json:"sources"`
+}
+
+// SystemFacts is the machine's identity from DMI.
+type SystemFacts struct {
+	Vendor       string `json:"vendor,omitempty"`
+	Product      string `json:"product,omitempty"`
+	Family       string `json:"family,omitempty"`
+	Version      string `json:"version,omitempty"`
+	Serial       string `json:"serial,omitempty"`
+	UUID         string `json:"uuid,omitempty"`
+	ChassisType  string `json:"chassis_type,omitempty"`
+	ChassisAsset string `json:"chassis_asset,omitempty"`
+	BoardVendor  string `json:"board_vendor,omitempty"`
+	BoardProduct string `json:"board_product,omitempty"`
+	BoardSerial  string `json:"board_serial,omitempty"`
+	BIOSVendor   string `json:"bios_vendor,omitempty"`
+	BIOSVersion  string `json:"bios_version,omitempty"`
+	BIOSDate     string `json:"bios_date,omitempty"`
+	DeviceModel  string `json:"device_model,omitempty"`
+}
+
+// OSFacts names the running OS and Spinifex build.
+type OSFacts struct {
+	Name          string `json:"name,omitempty"`
+	Kernel        string `json:"kernel,omitempty"`
+	Arch          string `json:"arch,omitempty"`
+	Version       string `json:"spx_version,omitempty"`
+	Commit        string `json:"spx_commit,omitempty"`
+	UptimeSeconds int64  `json:"uptime_seconds,omitempty"`
+	Hostname      string `json:"hostname,omitempty"`
+}
+
+// CPUFacts summarises the sockets. Threads and Cores are totals. Features are
+// the flags worth naming; Flags is every flag the CPU reports.
+type CPUFacts struct {
+	Model          string   `json:"model,omitempty"`
+	Vendor         string   `json:"vendor,omitempty"`
+	Sockets        int      `json:"sockets"`
+	CoresPerSocket int      `json:"cores_per_socket,omitempty"`
+	Cores          int      `json:"cores"`
+	Threads        int      `json:"threads"`
+	MaxMHz         int      `json:"max_mhz,omitempty"`
+	CurrentMHz     int      `json:"current_mhz,omitempty"`
+	Virtualization string   `json:"virtualization,omitempty"`
+	Features       []string `json:"features,omitempty"`
+	Flags          []string `json:"flags,omitempty"`
+}
+
+// MemoryFacts groups identical DIMMs so a 32-slot board reads as one line.
+type MemoryFacts struct {
+	TotalBytes    uint64        `json:"total_bytes"`
+	SlotsUsed     int           `json:"slots_used"`
+	SlotsTotal    int           `json:"slots_total"`
+	MaxCapacity   string        `json:"max_capacity,omitempty"`
+	ECC           string        `json:"ecc,omitempty"`
+	Groups        []MemoryGroup `json:"groups"`
+	OSTotalBytes  uint64        `json:"os_total_bytes,omitempty"`
+	OSAvailBytes  uint64        `json:"os_available_bytes,omitempty"`
+	HugePageBytes uint64        `json:"huge_page_bytes,omitempty"`
+}
+
+// MemoryGroup is a set of DIMMs alike in every field shown. ConfiguredMTs
+// below SpeedMTs means the board runs them slower than rated.
+type MemoryGroup struct {
+	Count         int      `json:"count"`
+	SizeBytes     uint64   `json:"size_bytes"`
+	Type          string   `json:"type,omitempty"`
+	TypeDetail    string   `json:"type_detail,omitempty"`
+	FormFactor    string   `json:"form_factor,omitempty"`
+	SpeedMTs      int      `json:"speed_mts,omitempty"`
+	ConfiguredMTs int      `json:"configured_mts,omitempty"`
+	Manufacturer  string   `json:"manufacturer,omitempty"`
+	PartNumber    string   `json:"part_number,omitempty"`
+	Rank          string   `json:"rank,omitempty"`
+	Serials       []string `json:"serials,omitempty"`
+}
+
+// NICFacts is one physical network interface and, where LLDP reports it, the
+// switch port it is cabled to.
+type NICFacts struct {
+	Name      string        `json:"name"`
+	MAC       string        `json:"mac,omitempty"`
+	Driver    string        `json:"driver,omitempty"`
+	Firmware  string        `json:"firmware,omitempty"`
+	BusInfo   string        `json:"bus_info,omitempty"`
+	Model     string        `json:"model,omitempty"`
+	SpeedMbps int           `json:"speed_mbps,omitempty"`
+	MaxMbps   int           `json:"max_mbps,omitempty"`
+	Duplex    string        `json:"duplex,omitempty"`
+	Port      string        `json:"port,omitempty"`
+	MTU       int           `json:"mtu,omitempty"`
+	State     string        `json:"state"`
+	Master    string        `json:"master,omitempty"`
+	Planes    []string      `json:"planes,omitempty"`
+	Neighbor  *LLDPNeighbor `json:"neighbor,omitempty"`
+}
+
+// LLDPNeighbor is what the switch on the other end of a NIC advertises.
+type LLDPNeighbor struct {
+	ChassisID         string   `json:"chassis_id"`
+	SystemName        string   `json:"system_name,omitempty"`
+	SystemDescription string   `json:"system_description,omitempty"`
+	PortID            string   `json:"port_id,omitempty"`
+	PortDescription   string   `json:"port_description,omitempty"`
+	Capabilities      []string `json:"capabilities,omitempty"`
+}
+
+// PlaneFacts is the address a plane uses on this node and the NICs under it.
+type PlaneFacts struct {
+	Plane   string   `json:"plane"`
+	Bridge  string   `json:"bridge,omitempty"`
+	Address string   `json:"address,omitempty"`
+	Prefix  int      `json:"prefix,omitempty"`
+	NICs    []string `json:"nics,omitempty"`
+	MTU     int      `json:"mtu,omitempty"`
+}
+
+// NetTestServeRequest starts a server bound to the node's address on Plane.
+type NetTestServeRequest struct {
+	Plane           string `json:"plane"`
+	DurationSeconds int    `json:"duration_seconds"`
+}
+
+// NetTestServeResponse names the address the server listens on.
+type NetTestServeResponse struct {
+	Address string `json:"address,omitempty"`
+	Port    int    `json:"port,omitempty"`
+	Error   string `json:"error,omitempty"`
+}
+
+// NetTestRunRequest runs the client towards Address from the node's own
+// address on Plane.
+type NetTestRunRequest struct {
+	Plane           string `json:"plane"`
+	Address         string `json:"address"`
+	Port            int    `json:"port"`
+	DurationSeconds int    `json:"duration_seconds"`
+	Streams         int    `json:"streams"`
+}
+
+// NetTestRunResponse is one direction of a pair's measurement.
+type NetTestRunResponse struct {
+	SourceAddress string    `json:"source_address,omitempty"`
+	BitsPerSecond float64   `json:"bits_per_second"`
+	Bytes         int64     `json:"bytes"`
+	Seconds       float64   `json:"seconds"`
+	Streams       int       `json:"streams"`
+	StreamBPS     []float64 `json:"stream_bps,omitempty"`
+	RTTMinMillis  float64   `json:"rtt_min_ms,omitempty"`
+	RTTAvgMillis  float64   `json:"rtt_avg_ms,omitempty"`
+	RTTMaxMillis  float64   `json:"rtt_max_ms,omitempty"`
+	PacketLoss    float64   `json:"packet_loss_percent,omitempty"`
+	LinkMbps      int       `json:"link_mbps,omitempty"`
+	MTU           int       `json:"mtu,omitempty"`
+	Tool          string    `json:"tool,omitempty"`
+	Error         string    `json:"error,omitempty"`
+}

contracts/cluster/v1/heartbeat.go

@@ -0,0 +1,29 @@
+package clusterv1
+
+import "time"
+
+// ClusterStateBucket is the KV bucket holding each daemon's heartbeat alongside
+// other cluster-wide state.
+const ClusterStateBucket = "spinifex-cluster-state"
+
+// HeartbeatKey is the ClusterStateBucket key a node writes its heartbeat under.
+func HeartbeatKey(node string) string {
+	return "heartbeat." + node
+}
+
+// HeartbeatTimestamp is the read side of a heartbeat for node liveness. It
+// decodes only the timestamp, so a reader does not depend on the capacity
+// figures the daemon writes beside it.
+type HeartbeatTimestamp struct {
+	Timestamp string `json:"timestamp"`
+}
+
+// Time returns when the heartbeat was written. A timestamp that is missing or
+// unparseable vouches for nothing, so ok is false.
+func (h HeartbeatTimestamp) Time() (written time.Time, ok bool) {
+	parsed, err := time.Parse(time.RFC3339, h.Timestamp)
+	if err != nil {
+		return time.Time{}, false
+	}
+	return parsed.UTC(), true
+}

contracts/cluster/v1/heartbeat_test.go

@@ -0,0 +1,35 @@
+package clusterv1_test
+
+import (
+	"encoding/json"
+	"testing"
+	"time"
+
+	clusterv1 "github.com/mulgadc/spinifex/contracts/cluster/v1"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// The bucket and key are the deployed layout: a rolling upgrade has old daemons
+// writing and new readers reading, so these strings cannot change.
+func TestHeartbeatKey_IsTheDeployedLayout(t *testing.T) {
+	assert.Equal(t, "spinifex-cluster-state", clusterv1.ClusterStateBucket)
+	assert.Equal(t, "heartbeat.node-a", clusterv1.HeartbeatKey("node-a"))
+}
+
+func TestHeartbeatTimestamp_ReadsAFullDaemonHeartbeat(t *testing.T) {
+	full := `{"node":"node-a","epoch":3,"timestamp":"2026-10-30T01:02:03Z","vm_count":4,"allocated_vcpu":8}`
+	var h clusterv1.HeartbeatTimestamp
+	require.NoError(t, json.Unmarshal([]byte(full), &h))
+
+	written, ok := h.Time()
+	require.True(t, ok)
+	assert.True(t, written.Equal(time.Date(2026, 10, 30, 1, 2, 3, 0, time.UTC)))
+}
+
+func TestHeartbeatTimestamp_UndatedBeatVouchesForNothing(t *testing.T) {
+	for _, ts := range []string{"", "yesterday"} {
+		_, ok := clusterv1.HeartbeatTimestamp{Timestamp: ts}.Time()
+		assert.False(t, ok, "timestamp %q", ts)
+	}
+}

contracts/ec2/v1/volume_held.go

@@ -0,0 +1,7 @@
+package ec2v1
+
+// VolumeHeldElsewhereText is the text of a start refused because another node
+// holds one of the instance's volumes. It crosses NATS as message text, so
+// readers match on it; the provider that refuses and every reader share this
+// string so they cannot drift.
+const VolumeHeldElsewhereText = "volume is leased by another owner"

contracts/ecs/v1/messages.go

@@ -126,8 +126,8 @@ type Assign struct {
 	ENIPrivateIP  string `json:"eniPrivateIp,omitempty"`
 	ENISubnetID   string `json:"eniSubnetId,omitempty"`
 	// TaskRoleARN is the task's IAM role (from the taskdef); when set, the agent
-	// serves its credentials at 169.254.170.2 under CredID, defaulting CredID to
-	// the taskID when the scheduler leaves it empty.
+	// serves its credentials at 169.254.170.2 under CredID, a scheduler-minted
+	// secret. An assign without a CredID gets no credentials.
 	CredID      string `json:"credId,omitempty"`
 	TaskRoleARN string `json:"taskRoleArn,omitempty"`
 	// ExecutionRoleARN is the task's execution role (from the taskdef); when set,

contracts/lambda/v1/README.md

@@ -0,0 +1,66 @@
+# Lambda v1 contracts
+
+This package will own the Lambda process contracts declared by ADR-0009:S13.
+It holds the L-GUEST declaration and, in `guest.go`, the connection vocabulary both ends share: the scope, the URI SAN roles, the ALPN identifier and the fw_cfg entry names. The transport and authentication are decided (ADR-0009:S17); the invocation message types and framing are not chosen, so no message types are defined here.
+It is not an AWS tenant-facing API.
+
+## L-GUEST: node executor to guest agent
+
+**Owner:** the Lambda domain.
+
+**Producers and consumers:** the node-local executor (`runtime/compute/lambda`) and the in-guest Lambda agent (`agents/lambda`), in both directions.
+
+**Operations:** bootstrap with scoped material, readiness, invoke, completion, bounded logs, shutdown.
+
+**Transport:** TCP over the microVM's network (ADR-0009:S17).
+The agent dials one connection to a listener the executor owns for the environment, and the guest exposes no listener for host control.
+At most one authenticated control connection is active per environment, and the agent may reconnect; how a prior connection and its in-flight work are handled is open (ADR-0009 M1).
+The network is shared and untrusted: no source address, port or bridge membership is identity or authority.
+Tenant code may be able to reach the listener, but it cannot open the agent's connection or key material, so any connection it makes fails authentication.
+
+**Authentication:** mutual TLS 1.3, bound to the environment ID and realization generation, using material the user runtime cannot read (ADR-0009:S17).
+For each generation the executor creates ephemeral self-signed host and guest certificates whose single URI SAN is `spinifex-lambda://<Region>/<account>/<function ID>/<environment ID>/<generation>/<role>`, with role `host` or `guest`.
+Each end validates the peer certificate against the one pinned peer certificate and separately requires that URI SAN with the opposite role; hostname verification is not relied on.
+ALPN carries the L-GUEST major version, `spinifex-lambda-guest/1`, and an unsupported version fails the handshake.
+Under TLS 1.3 the guest's handshake can complete before the host refuses its certificate, so the guest treats the connection as accepted only after the host's first message.
+
+**Bootstrap material:** the executor passes these fw_cfg entries under `opt/spinifex/lambda/`, and the agent reads them as root and unloads `qemu_fw_cfg` before user code starts.
+
+| Entry | Content |
+|---|---|
+| `guest-key` | Guest private key, PEM |
+| `guest-cert` | Guest certificate, PEM |
+| `host-cert` | The one pinned host certificate, PEM |
+| `expect` | `<Region>/<account>/<function ID>/<environment ID>/<generation>` |
+| `host-addr` | Host listener as `host:port` |
+
+The guest's own interface settings come from `opt/spinifex/netcfg`, the entry other system VMs use.
+The agent unloads `qemu_fw_cfg` even when an entry is missing, and refuses to start if the fw_cfg tree is still readable afterwards.
+Until the message format exists, the agent authenticates and holds the connection but accepts no host message and starts no user code.
+
+**Authority:** one environment's attempt only.
+The user runtime never holds these keys, and sharing the guest with the agent gives user code no platform authority.
+
+**Idempotency and fencing:** an invoke carries the attempt-grant identity and the attempt ID.
+The agent accepts a run command only under a current grant, and a completion under a superseded grant is discarded and reported.
+How a grant is enforced and proven ended is open (ADR-0009 M1).
+
+**Errors and versions:** the agent and executor fail closed on an unknown protocol version or an identity mismatch.
+
+**Readiness:** environment readiness needs authenticated, current-generation agent evidence; a running hypervisor process does not count.
+
+**Payloads:** the contract must carry a synchronous payload and response of up to 6 MB and an asynchronous payload of up to 1 MB, plus framing, over a path whose size bound is declared.
+How payloads move is open (ADR-0009 M3).
+
+### Compatibility boundary
+
+The agent ships inside the runtime image pinned by each published version, so it is outside the coordinated release train.
+The executor supports every L-GUEST major carried by a still-supported runtime image.
+Retiring an image follows a published runtime-deprecation lifecycle that blocks create and update before it blocks invoke.
+
+Evidence required before the first agent image ships: a function on an old image invoked by a new executor, and the reverse during rollback.
+
+## L-RUNTIME
+
+The guest agent serves the AWS Lambda Runtime API, version `2018-06-01`, to the user runtime at `AWS_LAMBDA_RUNTIME_API`.
+Spinifex does not version this AWS contract, and conformance is measured against AWS.

contracts/lambda/v1/guest.go

@@ -0,0 +1,54 @@
+// Package lambdav1 holds the Lambda process-contract vocabulary shared by the
+// node executor and the guest agent. See README.md for the declarations.
+package lambdav1
+
+import (
+	"fmt"
+	"strings"
+)
+
+// ALPNv1 names L-GUEST major version 1; a peer offering any other fails the handshake.
+const ALPNv1 = "spinifex-lambda-guest/1"
+
+// FwCfgPrefix prefixes every fw_cfg entry the executor passes the guest.
+const FwCfgPrefix = "opt/spinifex/lambda/"
+
+// fw_cfg entry names under FwCfgPrefix.
+const (
+	EntryGuestKey  = "guest-key"
+	EntryGuestCert = "guest-cert"
+	EntryHostCert  = "host-cert"
+	EntryScope     = "expect"
+	EntryHostAddr  = "host-addr"
+)
+
+// Roles named as the last URI SAN path segment.
+const (
+	RoleHost  = "host"
+	RoleGuest = "guest"
+)
+
+const uriScheme = "spinifex-lambda"
+
+// Scope names one environment generation:
+// Region/account/function ID/environment ID/generation.
+type Scope string
+
+// ParseScope accepts exactly five non-empty, slash-separated segments.
+func ParseScope(s string) (Scope, error) {
+	parts := strings.Split(s, "/")
+	if len(parts) != 5 {
+		return "", fmt.Errorf("lambdav1: scope %q: want 5 segments, got %d", s, len(parts))
+	}
+	for _, p := range parts {
+		if p == "" || strings.ContainsAny(p, " \t\r\n?#%") {
+			return "", fmt.Errorf("lambdav1: scope %q: empty or invalid segment", s)
+		}
+	}
+	return Scope(s), nil
+}
+
+// URI is the single URI SAN a certificate for role in this scope carries.
+func (s Scope) URI(role string) string {
+	return uriScheme + "://" + string(s) + "/" + role
+}

contracts/lambda/v1/guest_test.go

@@ -0,0 +1,20 @@
+package lambdav1
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestParseScope(t *testing.T) {
+	s, err := ParseScope("ap-southeast-2/000000000001/fn-1/env-1/gen-1")
+	require.NoError(t, err)
+	assert.Equal(t, "spinifex-lambda://ap-southeast-2/000000000001/fn-1/env-1/gen-1/host", s.URI(RoleHost))
+	for _, bad := range []string{
+		"", "a/b/c/d", "a/b/c/d/e/f", "a//c/d/e", "a/b/c/d/e/", "a/b/c/d/e?x", "a/b/c/d/e#x", "a b/b/c/d/e",
+	} {
+		_, err := ParseScope(bad)
+		assert.Error(t, err, bad)
+	}
+}

contracts/operator/v1/README.md

@@ -8,3 +8,8 @@ administration. It is separate from AWS tenant-facing service contracts.
 roles and Reed-Solomon configuration. It is an operational status contract,
 not a generic provider abstraction; do not change its subject or JSON shape
 incompatibly. Add a new version when that is required.
+
+`admin.go` holds the `/admin/<Method>` request and response bodies the private admin surface shares with the CLI.
+`AdminMethodNames` is the list of callable methods, and the gateway router builds its accept set from it.
+`AccountQuotaRequest`, `AccountQuotaResponse` and `DeleteAccountResponse` stay in the gateway until the `/admin` handlers move, because they embed types the contracts layer may not import.
+Do not change a field name, JSON tag or omitempty incompatibly; add a new version when that is required.

contracts/operator/v1/admin.go

@@ -0,0 +1,138 @@
+package operatorv1
+
+import "slices"
+
+// adminMethodNames is the callable /admin/<Method> list. The gateway router
+// builds its accept set from it, so a name cannot be served without being listed.
+var adminMethodNames = []string{
+	"CreateAccount",
+	"DeleteAccount",
+	"DescribeAccountDeletion",
+	"GetAccountQuota",
+	"ListAccounts",
+	"PutAccountQuota",
+}
+
+// AdminMethodNames returns the callable /admin/<Method> names in sorted order.
+// Tooling that mints a credential grants these by name, so the list has to be
+// the one the router serves rather than a copy that can drift out of step.
+func AdminMethodNames() []string {
+	return slices.Sorted(slices.Values(adminMethodNames))
+}
+
+// Deletion job states.
+const (
+	DeletionStateRunning   = "RUNNING"
+	DeletionStateCompleted = "COMPLETED"
+	DeletionStateFailed    = "FAILED"
+)
+
+// CreateAccountRequest is the /admin/CreateAccount request body.
+type CreateAccountRequest struct {
+	Name        string `json:"name"`
+	ClientToken string `json:"clientToken"`
+	Source      string `json:"source,omitempty"`
+}
+
+// CreateAccountResponse is the /admin/CreateAccount success body. It is stored
+// verbatim (with the secret encrypted) so a replay returns an identical answer.
+type CreateAccountResponse struct {
+	AccountID       string `json:"accountId"`
+	AccountName     string `json:"accountName"`
+	AdminUser       string `json:"adminUser"`
+	AccessKeyID     string `json:"accessKeyId"`
+	SecretAccessKey string `json:"secretAccessKey"`
+	DefaultVpcID    string `json:"defaultVpcId"`
+}
+
+// DeleteAccountRequest is the /admin/DeleteAccount request body.
+type DeleteAccountRequest struct {
+	AccountID   string `json:"accountId"`
+	AccountName string `json:"accountName"`
+	ClientToken string `json:"clientToken"`
+	Force       bool   `json:"force,omitempty"`
+	DryRun      bool   `json:"dryRun,omitempty"`
+}
+
+// DescribeAccountDeletionRequest is the /admin/DescribeAccountDeletion body.
+type DescribeAccountDeletionRequest struct {
+	AccountID string `json:"accountId"`
+}
+
+// AccountSummary is one row of /admin/ListAccounts. It carries no credential
+// material: the caller can already create and delete accounts, but that is no
+// reason for a listing to hand out keys.
+type AccountSummary struct {
+	AccountID   string `json:"accountId"`
+	AccountName string `json:"accountName"`
+	Status      string `json:"status"`
+	CreatedAt   string `json:"createdAt,omitempty"`
+}
+
+// ListAccountsResponse is the /admin/ListAccounts success body.
+type ListAccountsResponse struct {
+	Accounts []AccountSummary `json:"accounts"`
+	Count    int              `json:"count"`
+}
+
+// IAMQuiescence is the coordinator's recorded view of IAM quiescence for one
+// fence. Recording only: it admits nothing, and QuiescedScope stays empty until
+// a final inventory re-read exists to catch a late write.
+type IAMQuiescence struct {
+	OperationID   string            `json:"operationId"`
+	Fence         uint64            `json:"fence"`
+	Generation    uint64            `json:"generation"`
+	QuiescedScope []string          `json:"quiescedScope"`
+	Unresolved    []UnresolvedScope `json:"unresolved"`
+	Consumers     []ConsumerBasis   `json:"consumers"`
+	Writers       []AccountedWriter `json:"writers"`
+	Inventory     []InventoryStatus `json:"inventory"`
+	EvaluatedAt   string            `json:"evaluatedAt"`
+}
+
+// UnresolvedScope names a scope or writer that blocks quiescence, with a fixed
+// reason. It is never a timeout.
+type UnresolvedScope struct {
+	Scope  string `json:"scope"`
+	Class  string `json:"class"`
+	Reason string `json:"reason"`
+}
+
+// ConsumerBasis cites an enforcement consumer and the accepted contract that
+// lets it count as having applied the generation.
+type ConsumerBasis struct {
+	Consumer string `json:"consumer"`
+	Basis    string `json:"basis"`
+}
+
+// AccountedWriter is a writer that cannot be fenced and is covered by the
+// owner's inventory instead. Ref names the tracked work that closes the gap.
+type AccountedWriter struct {
+	Writer string `json:"writer"`
+	Basis  string `json:"basis"`
+	Ref    string `json:"ref"`
+}
+
+// InventoryStatus is the owner inventory's status for one kind at evaluation.
+type InventoryStatus struct {
+	Kind   string `json:"kind"`
+	Status string `json:"status"`
+}
+
+// Classes of an UnresolvedScope.
+const (
+	UnresolvedClassFence    = "fence"
+	UnresolvedClassConsumer = "consumer"
+	UnresolvedClassWriter   = "writer"
+)
+
+// Fixed reasons of an UnresolvedScope.
+const (
+	ReasonFenceMismatch                = "fence-mismatch"
+	ReasonIAMReadbackFailed            = "iam-readback-failed"
+	ReasonNoAcceptedEnforcement        = "no-accepted-enforcement-contract"
+	ReasonNoCompleteOwnerInventory     = "no-complete-owner-inventory"
+	ReasonParticipantNotDeclared       = "participant-not-declared"
+	ReasonOwnerInventoryIncomplete     = "owner-inventory-incomplete"
+	ReasonFinalInventoryRereadNotWired = "final-inventory-reread-not-wired"
+)

contracts/operator/v1/admin_test.go

@@ -0,0 +1,65 @@
+package operatorv1
+
+import (
+	"encoding/json"
+	"testing"
+
+	"github.com/stretchr/testify/require"
+)
+
+func goldenJSON(t *testing.T, v any, want string) {
+	t.Helper()
+	data, err := json.Marshal(v)
+	require.NoError(t, err)
+	require.JSONEq(t, want, string(data))
+}
+
+func TestAdminWire_CreateAccountRequest(t *testing.T) {
+	goldenJSON(t, CreateAccountRequest{Name: "acme", ClientToken: "tok-1", Source: "cli"},
+		`{"name":"acme","clientToken":"tok-1","source":"cli"}`)
+	goldenJSON(t, CreateAccountRequest{Name: "acme", ClientToken: "tok-1"},
+		`{"name":"acme","clientToken":"tok-1"}`)
+}
+
+func TestAdminWire_CreateAccountResponse(t *testing.T) {
+	goldenJSON(t, CreateAccountResponse{
+		AccountID: "000000000042", AccountName: "acme", AdminUser: "admin",
+		AccessKeyID: "AKIAEXAMPLE", SecretAccessKey: "secret", DefaultVpcID: "vpc-1",
+	}, `{"accountId":"000000000042","accountName":"acme","adminUser":"admin","accessKeyId":"AKIAEXAMPLE","secretAccessKey":"secret","defaultVpcId":"vpc-1"}`)
+}
+
+func TestAdminWire_DeleteAccountRequest(t *testing.T) {
+	goldenJSON(t, DeleteAccountRequest{AccountID: "000000000042", AccountName: "acme", ClientToken: "tok-2", Force: true, DryRun: true},
+		`{"accountId":"000000000042","accountName":"acme","clientToken":"tok-2","force":true,"dryRun":true}`)
+	goldenJSON(t, DeleteAccountRequest{AccountID: "000000000042", AccountName: "acme", ClientToken: "tok-2"},
+		`{"accountId":"000000000042","accountName":"acme","clientToken":"tok-2"}`)
+}
+
+func TestAdminWire_DescribeAccountDeletionRequest(t *testing.T) {
+	goldenJSON(t, DescribeAccountDeletionRequest{AccountID: "000000000042"}, `{"accountId":"000000000042"}`)
+}
+
+func TestAdminWire_DeletionStates(t *testing.T) {
+	require.Equal(t, "RUNNING", DeletionStateRunning)
+	require.Equal(t, "COMPLETED", DeletionStateCompleted)
+	require.Equal(t, "FAILED", DeletionStateFailed)
+}
+
+func TestAdminWire_ListAccountsResponse(t *testing.T) {
+	goldenJSON(t, ListAccountsResponse{
+		Accounts: []AccountSummary{
+			{AccountID: "000000000042", AccountName: "acme", Status: "ACTIVE", CreatedAt: "2026-01-02T03:04:05Z"},
+			{AccountID: "000000000043", AccountName: "bolt", Status: "TERMINATING"},
+		},
+		Count: 2,
+	}, `{"accounts":[
+		{"accountId":"000000000042","accountName":"acme","status":"ACTIVE","createdAt":"2026-01-02T03:04:05Z"},
+		{"accountId":"000000000043","accountName":"bolt","status":"TERMINATING"}],"count":2}`)
+}
+
+func TestAdminWire_AdminMethodNames(t *testing.T) {
+	require.Equal(t, []string{
+		"CreateAccount", "DeleteAccount", "DescribeAccountDeletion",
+		"GetAccountQuota", "ListAccounts", "PutAccountQuota",
+	}, AdminMethodNames())
+}

contracts/telemetry/v1/live.go

@@ -0,0 +1,71 @@
+package telemetryv1
+
+import (
+	"regexp"
+)
+
+// LiveSamplePeriodMillis is how often a watched instance is sampled for the
+// console's live view.
+const LiveSamplePeriodMillis = 500
+
+// LiveWatchLeaseMillis is how long one watch keeps the live sampler running.
+// Watchers renew well inside it, so a viewer that goes away costs at most this.
+const LiveWatchLeaseMillis = 30_000
+
+// LiveRingSamples bounds the in-memory history a live sampler keeps for a
+// reconnecting viewer: five minutes at LiveSamplePeriodMillis.
+const LiveRingSamples = 5 * 60 * 1000 / LiveSamplePeriodMillis
+
+// Live subjects sit outside metrics.> so nothing that ingests the periodic
+// CloudWatch batches ever reads a sub-second rate as a period. The account is part
+// of the subject, so only the poller for that account's instance answers.
+const (
+	liveWatchSubjectPrefix  = "telemetry.watch.ec2."
+	liveSampleSubjectPrefix = "telemetry.live.ec2."
+)
+
+var (
+	liveAccountPattern  = regexp.MustCompile(`^[0-9]{12}$`)
+	liveInstancePattern = regexp.MustCompile(`^i-[0-9a-f]{8,17}$`)
+)
+
+// ValidLiveTarget reports whether accountID and instanceID are safe to place
+// in a live subject: no NATS token separators or wildcards can get through.
+func ValidLiveTarget(accountID, instanceID string) bool {
+	return liveAccountPattern.MatchString(accountID) && liveInstancePattern.MatchString(instanceID)
+}
+
+// LiveWatchSubject is where a viewer asks the owning node to keep sampling. A
+// request on it is answered with the ring since LiveWatch.SinceMillis.
+func LiveWatchSubject(accountID, instanceID string) string {
+	return liveWatchSubjectPrefix + accountID + "." + instanceID
+}
+
+// LiveSampleSubject carries one LiveSample per sampling period while watched.
+func LiveSampleSubject(accountID, instanceID string) string {
+	return liveSampleSubjectPrefix + accountID + "." + instanceID
+}
+
+// LiveWatch is the body of a watch. SinceMillis is the newest sample the
+// viewer already has; zero asks for the whole ring.
+type LiveWatch struct {
+	SinceMillis int64 `json:"since_ms,omitempty"`
+}
+
+// LiveRing answers a watch request with the samples newer than SinceMillis.
+type LiveRing struct {
+	Samples []LiveSample `json:"samples"`
+}
+
+// LiveSample is one live measurement. Every value is a rate over the interval
+// ending at TimeMillis, measured against the previous sample's clock.
+type LiveSample struct {
+	TimeMillis int64   `json:"t"`
+	CPUPercent float64 `json:"cpu"`
+	// NetworkIn and NetworkOut are bytes per second from the guest's side.
+	NetworkIn  float64 `json:"netIn"`
+	NetworkOut float64 `json:"netOut"`
+	// PacketsIn and PacketsOut are packets per second from the guest's side.
+	PacketsIn  float64 `json:"pktIn"`
+	PacketsOut float64 `json:"pktOut"`
+}

contracts/telemetry/v1/live_test.go

@@ -0,0 +1,42 @@
+package telemetryv1
+
+import (
+	"encoding/json"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// The console parses these names, so a rename here breaks its live charts.
+func TestLiveSampleJSONContract(t *testing.T) {
+	data, err := json.Marshal(LiveSample{
+		TimeMillis: 1793450123250, CPUPercent: 12.5,
+		NetworkIn: 48213, NetworkOut: 1290, PacketsIn: 41, PacketsOut: 12,
+	})
+	require.NoError(t, err)
+	require.JSONEq(t, `{"t":1793450123250,"cpu":12.5,"netIn":48213,"netOut":1290,"pktIn":41,"pktOut":12}`, string(data))
+}
+
+func TestValidLiveTarget(t *testing.T) {
+	assert.True(t, ValidLiveTarget("123456789012", "i-0123456789abcdef0"))
+	assert.True(t, ValidLiveTarget("123456789012", "i-0abc1234"))
+
+	for _, tc := range []struct{ account, instance string }{
+		{"", "i-0123456789abcdef0"},
+		{"12345678901", "i-0123456789abcdef0"},
+		{"123456789012", ""},
+		{"123456789012", ">"},
+		{"123456789012", "*"},
+		{"123456789012", "i-0123.abcdef"},
+		{"123456789012", "i-0123456789ABCDEF0"},
+		{"123456789012", "vol-0123456789abcdef0"},
+	} {
+		assert.False(t, ValidLiveTarget(tc.account, tc.instance), "%q %q", tc.account, tc.instance)
+	}
+}
+
+func TestLiveSubjectsAreOutsideMetrics(t *testing.T) {
+	assert.Equal(t, "telemetry.watch.ec2.123456789012.i-0abc1234", LiveWatchSubject("123456789012", "i-0abc1234"))
+	assert.Equal(t, "telemetry.live.ec2.123456789012.i-0abc1234", LiveSampleSubject("123456789012", "i-0abc1234"))
+}

contracts/telemetry/v1/node_live.go

@@ -0,0 +1,53 @@
+package telemetryv1
+
+import "regexp"
+
+// Node live subjects mirror the instance ones, keyed by node name. Only the
+// super-admin account's gateway handler subscribes to them.
+const (
+	nodeLiveWatchSubjectPrefix  = "telemetry.watch.node."
+	nodeLiveSampleSubjectPrefix = "telemetry.live.node."
+)
+
+var liveNodePattern = regexp.MustCompile(`^[A-Za-z0-9_-]{1,64}$`)
+
+// ValidLiveNode reports whether a node name is safe to place in a subject.
+func ValidLiveNode(node string) bool {
+	return liveNodePattern.MatchString(node)
+}
+
+// NodeLiveWatchSubject is where a viewer asks a node to keep sampling itself.
+// A request on it is answered with a NodeLiveRing.
+func NodeLiveWatchSubject(node string) string {
+	return nodeLiveWatchSubjectPrefix + node
+}
+
+// NodeLiveSampleSubject carries one NodeLiveSample per period while watched.
+func NodeLiveSampleSubject(node string) string {
+	return nodeLiveSampleSubjectPrefix + node
+}
+
+// NodeLiveRing answers a watch with the samples newer than its SinceMillis.
+type NodeLiveRing struct {
+	Samples []NodeLiveSample `json:"samples"`
+}
+
+// NodeLiveSample is one host measurement. Rates are per second over the
+// interval ending at TimeMillis.
+type NodeLiveSample struct {
+	TimeMillis int64   `json:"t"`
+	CPUPercent float64 `json:"cpu"`
+	// MemoryUsedPercent is used memory less page cache, as free(1) reports it.
+	MemoryUsedPercent float64 `json:"mem"`
+	Load1             float64 `json:"load1"`
+	// Planes holds the traffic of each plane's physical NICs, keyed wan, lan, vpc.
+	Planes map[string]NodeLivePlane `json:"planes"`
+}
+
+// NodeLivePlane is one plane's traffic from the host's side.
+type NodeLivePlane struct {
+	NetworkIn  float64 `json:"netIn"`
+	NetworkOut float64 `json:"netOut"`
+	PacketsIn  float64 `json:"pktIn"`
+	PacketsOut float64 `json:"pktOut"`
+}

contracts/viperblockd/legacy/v1/README.md

@@ -60,6 +60,10 @@ moved to another node; the local daemon subscribes to stop the guest that
 was using it. Node-addressed, like mount and unmount: the guest is local, and
 no other node's daemon has anything to stop.
 
+The event is sent as a request and repeated until the daemon replies with `VolumeFencedAck`, because the fence often runs while the node's NATS server is restarting and a single publish can reach no subscriber.
+The daemon replies only when the event carries a reply subject, so a plain publish is still accepted.
+Repeats are harmless, since fencing a guest that is already fenced changes nothing.
+
 ## Abandon route
 
 `VolumeAbandonSubject(node)` is `ebs.<node>.abandon`. The daemon uses

contracts/viperblockd/legacy/v1/fence.go

@@ -19,3 +19,11 @@ type VolumeFencedEvent struct {
 	Winner string `json:"winner"`
 	Reason string `json:"reason"`
 }
+
+// VolumeFencedAck is the daemon's answer to a VolumeFencedEvent. Any answer
+// ends the sender's retries; Instance names the guest stopped, if one was, and
+// Error says why the daemon could not act on the event.
+type VolumeFencedAck struct {
+	Instance string `json:"instance,omitempty"`
+	Error    string `json:"error,omitempty"`
+}

contracts/viperblockd/legacy/v1/fence_test.go

@@ -37,3 +37,18 @@ func TestVolumeFencedEvent_JSON(t *testing.T) {
 	require.NoError(t, json.Unmarshal([]byte(golden), &decoded))
 	assert.Equal(t, event, decoded)
 }
+
+// TestVolumeFencedAck_JSON pins the daemon's answer, which is what ends
+// viperblockd's retries of a fence notice.
+func TestVolumeFencedAck_JSON(t *testing.T) {
+	golden := `{"instance":"i-1","error":"bad event"}`
+	ack := viperblocklegacyv1.VolumeFencedAck{Instance: "i-1", Error: "bad event"}
+
+	marshaled, err := json.Marshal(ack)
+	require.NoError(t, err)
+	assert.JSONEq(t, golden, string(marshaled))
+
+	empty, err := json.Marshal(viperblocklegacyv1.VolumeFencedAck{})
+	require.NoError(t, err)
+	assert.JSONEq(t, `{}`, string(empty), "an ack for a volume with no local guest carries nothing")
+}

docs/COMMANDS.md

@@ -26,7 +26,7 @@ Every service is `PartOf=spinifex.target` and ordered by its own `After=`/`Wants
 | `spinifex-qmp-collector.service` | `spx service qmp-collector start` | Reads the `contracts/telemetry/v1` per-VM sidecar, polls telemetry QMP sockets and tap counters, publishes to NATS `metrics.ec2.*` |
 | `spinifex-shutdown.service` | `/bin/true`; the work is its `ExecStop` | Runs `spx admin node drain --local --unless-restarting`. `After=` every service, so it stops first and drains guests while the stack is still up |
 | `spinifex-nats-watchdog.service` / `.timer` | `nats-js-watchdog.sh` | JetStream liveness probe every 30s. `try-restart`s NATS when JetStream stops accepting writes while the process stays alive — what `Restart=on-failure` cannot catch |
-| `spinifex-firewall.service` | `spinifex-firewall-apply` | Oneshot nftables input policy, `WantedBy=sysinit.target` rather than the target. No `ExecStop`: remove it with `nft delete table inet spinifex_filter` |
+| `spinifex-firewall.service` | `spinifex-firewall-apply boot` | Oneshot nftables input policy, `WantedBy=sysinit.target` rather than the target. Loads the `inet spinifex_conntrack` tracking table on every node, and the policy only once the daemon has written a peer file. No `ExecStop`: remove the policy with `spinifex-firewall-apply disable` |
 | `regenerate-ssh-host-keys.service` | `ssh-keygen -A` | Oneshot for AMI-built nodes, conditional on the host key being absent |
 
 #### Operations
@@ -95,7 +95,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 
 | Command | Flags | Description |
 |---------|-------|-------------|
-| `spx admin init` | `--nodes`, `--node`, `--bind`, `--advertise` (external IP off-host clients dial; auto-detected from the default route), `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--predastore-compaction-interval` (seconds; test clusters set a short one), `--services`, `--formation-timeout`, `--token-ttl`, `--force`, `--discard-jetstream` (with `--nodes` >= 2, remove this node's existing JetStream streams once every node has joined; without it a multi-node init over a store that holds streams is refused, and it never proceeds while NATS is running), `--skip-host-dns`, `--email` (operator address for update and security notices), `--no-telemetry`, `--gpu-passthrough`, `--ipsec` (default true; encrypt intra-AZ Geneve via OVN native IPsec cluster-wide — disable only for a trusted single-rack lab), `--host` (leader to join; falls back to multicast discovery), `--external-mode` (`pool` \| `nat` routed NAT for non-bridgeable uplinks, single-node; pair with `setup-ovn.sh --nat-uplink`; note: system instances (ECS/EKS/load-balancer agents) are unsupported in nat v1 — they require `pool` mode), `--external-iface` (WAN NIC for br-external, auto-detected), `--external-source` (`dhcp` \| `static`), `--external-pool` (`start-end`), `--external-prefix-len` (default 24), `--external-gateway` (auto-detected), `--external-dns` (comma-separated IPv4 addresses, at most 3; the DNS servers written to the external pool's `dns_servers` and handed to guests, verbatim; when omitted they are detected from the host resolver at init time, so set it to give every node of a cluster the same value), `--external-bind-bridge` (default `br-wan` when `--external-source=dhcp`), `--gateway-ip` (OVN gateway router's external IP for SNAT; defaults to the pool's range start, required for `nat` mode without DHCP) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` → points the host resolver at this node's local northstar so the Spinifex zones (ELB/EKS names) resolve from the node itself (opt out with `--skip-host-dns`). `--force` on an already-initialized node is idempotent for crypto: it preserves the existing keys (master.key, predastore/viperblock encryption keys), system + admin credentials, and CA, and refreshes only the config files and the CA-signed server certificate (so a changed bind IP / SANs is picked up without breaking already-joined nodes or CA-baked AMIs). A genuine clean slate requires removing the data dirs first. |
+| `spx admin init` | `--nodes`, `--node`, `--bind`, `--advertise` (external IP off-host clients dial; auto-detected from the default route), `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--predastore-compaction-interval` (seconds; test clusters set a short one), `--services`, `--formation-timeout`, `--token-ttl`, `--force`, `--discard-jetstream` (with `--nodes` >= 2, remove this node's existing JetStream streams once every node has joined; without it a multi-node init over a store that holds streams is refused, and it never proceeds while NATS is running), `--skip-host-dns`, `--email` (operator address for update and security notices), `--no-telemetry`, `--gpu-passthrough`, `--ipsec` (default true; encrypt intra-AZ Geneve via OVN native IPsec cluster-wide — disable only for a trusted single-rack lab; on a host whose IPsec stack is libreswan it defaults to false and `--ipsec=true` is refused), `--host` (leader to join; falls back to multicast discovery), `--external-mode` (`pool` \| `nat` routed NAT for non-bridgeable uplinks, single-node; pair with `setup-ovn.sh --nat-uplink`; note: system instances (ECS/EKS/load-balancer agents) are unsupported in nat v1 — they require `pool` mode), `--external-iface` (WAN NIC for br-external, auto-detected), `--external-source` (`dhcp` \| `static`), `--external-pool` (`start-end`), `--external-prefix-len` (default 24), `--external-gateway` (auto-detected), `--external-dns` (comma-separated IPv4 addresses, at most 3; the DNS servers written to the external pool's `dns_servers` and handed to guests, verbatim; when omitted they are detected from the host resolver at init time, so set it to give every node of a cluster the same value), `--external-bind-bridge` (default `br-wan` when `--external-source=dhcp`), `--gateway-ip` (OVN gateway router's external IP for SNAT; defaults to the pool's range start, required for `nat` mode without DHCP) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` → points the host resolver at this node's local northstar so the Spinifex zones (ELB/EKS names) resolve from the node itself (opt out with `--skip-host-dns`). `--force` on an already-initialized node is idempotent for crypto: it preserves the existing keys (master.key, predastore/viperblock encryption keys), system + admin credentials, and CA, and refreshes only the config files and the CA-signed server certificate (so a changed bind IP / SANs is picked up without breaking already-joined nodes or CA-baked AMIs). A genuine clean slate requires removing the data dirs first. |
 | `spx admin join` | `--host` (required), `--node` (required), `--token` (required), `--bind`, `--advertise` (defaults to `--bind`, or the auto-detected WAN IP when `--bind` is 0.0.0.0), `--port`, `--region`, `--az`, `--cluster-bind`, `--cluster-routes`, `--data-dir`, `--services`, `--skip-host-dns`, `--email`, `--no-telemetry`, `--predastore-compaction-interval`, `--join-timeout` (how long to keep retrying while the formation server is unreachable, default 20m), `--force` (join even though this node is already initialized or holds JetStream streams, **discarding its own CA, master key and JetStream store**; the store is removed only after formation succeeds, and the join is refused while NATS is running even with `--force`) | Connects to leader node with join token (Authorization: Bearer header) → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations → points the host resolver at this node's local northstar so the Spinifex zones resolve from the node itself (opt out with `--skip-host-dns`) |
 
 ### Version
@@ -127,7 +127,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 
 | Command | Flags | Description |
 |---------|-------|-------------|
-| `spx admin preflight` | `--quiet` (print only non-OK rows) | Reports whether this host has the privileged helper scripts and sudoers grants the installed build depends on. `spx version` knows the build but not the host: `setup.sh` and `setup-ovn.sh` install those assets, and a binary-swap deploy (`make deploy`, or a manual copy) skips them. It also reports any running service still executing a replaced binary — overwriting `/usr/local/bin/spx` while `spinifex.target` is up does not move services onto it, so each keeps executing the replaced inode until it restarts and a node can serve a mixture of two builds indefinitely. Checks compare content, not existence, so a present-but-stale asset is caught. Exits non-zero if any asset is Missing, Stale or Ungranted. |
+| `spx admin preflight` | `--quiet` (print only non-OK rows) | Reports whether this host has the privileged helper scripts and sudoers grants the installed build depends on. `spx version` knows the build but not the host: `setup.sh` and `setup-ovn.sh` install those assets, and a binary-swap deploy (`make deploy`, or a manual copy) skips them. It also reports any running service still executing a replaced binary — overwriting `/usr/local/bin/spx` while `spinifex.target` is up does not move services onto it, so each keeps executing the replaced inode until it restarts and a node can serve a mixture of two builds indefinitely. Checks compare content, not existence, so a present-but-stale asset is caught. On x86_64 it also checks that the QEMU binary the daemon will run lists the `microvm` machine type and that the microvm firmware `/usr/share/spinifex/microvm/qboot.rom` is installed, since load balancers boot on both. It runs the host network check `spinifex-daemon` and `spinifex-vpcd` run before they start, and fails while firewalld or ufw is active, since the Spinifex nftables policy is the only supported node firewall. It reports a Warning, without failing, when systemd-resolved is active but `/etc/resolv.conf` is not its stub, since lookups then bypass the route that sends Spinifex names to northstar. Exits non-zero if any asset is Missing, Stale, Ungranted, Unsupported or Failed. |
 | `spx admin node js-probe` | `--timeout` (default: 10s) | Round-trips a canary key through the cluster-state KV bucket and exits non-zero if the write or the read-back fails. JetStream can latch a write error per message block and keep serving reads — nats-server's filestore records the failure on the block and returns it on every later write, and nothing clears it short of a restart. The HTTP health endpoint reports the server-wide JetStream state, which that latch never touches, so only an actual write observes it. |
 | `spx admin banner` | `--boot-check` | Writes the node information banner to `/etc/issue` (shown before login on the physical/serial console) and `/etc/motd` (shown after SSH login). With `--boot-check` it also detects a management IP that has changed since last boot and updates `/etc/spinifex/node.conf` accordingly. |
 
@@ -136,7 +136,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 | Command | Flags | Description |
 |---------|-------|-------------|
 | `spx admin volumes orphans` | — | Lists volumes the storage provider holds that the control plane has no record of. Their blocks consume space but they have no API handle, so they can be neither described nor deleted through the EC2 API. **Reports only, never deletes**: the evidence that an orphan is still wanted is the record that went missing, so removal is an operator decision taken per volume. |
-| `spx admin volumes unsealed` | — | Lists volumes a node has open, or had open, without a confirmed seal to the backend — the node named holds writes predastore may not have. Starting such a volume elsewhere is allowed, and is preferred over leaving the instance unable to run, but it opens from the last checkpoint that did reach the backend. This is the list of volumes where that would cost something. |
+| `spx admin volumes unsealed` | — | Lists volumes a node has open, or had open, without a confirmed seal to the backend — the node named holds writes predastore may not have. Starting such a volume elsewhere is refused while that node is alive (its daemon heartbeat is under 60 seconds old, or cannot be read) and is otherwise allowed, in which case it opens from the last checkpoint that did reach the backend. This is the list of volumes where that would cost something. |
 | `spx admin storage status` | `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA) | Sends the signed `GetStorageStatus` action to the AWS gateway with credentials from the standard AWS chain (e.g. `AWS_PROFILE=spinifex`) and prints what the console's Service Metrics page shows: predastore encoding, meta and blob node counts, cluster health, a meta node table (health, raft state, leader, term, commit and applied index) and the blob nodes. Cluster health is `Degraded` when any meta node is unreachable or the nodes reporting raft state do not agree on one leader. A meta node read only through its host's `/readyz` shows healthy with blank raft fields. |
 
 ### Upgrade Management
@@ -152,12 +152,12 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 | `spx admin account create` | `--name` (required) | Connects to NATS → refuses if an account already holds the name (case- and whitespace-insensitive) → reserves the name in `spinifex-account-names` KV → CAS loop on `spinifex-account-counter:next_id` for sequential 12-digit ID → creates Account record in `spinifex-accounts` KV → creates default VPC → creates `admin` user, AdministratorAccess policy (Action:*, Resource:*) and one access key → prints credentials and writes a `spinifex-<name>` AWS CLI profile |
 | `spx admin account create --remote` | `--name` (required), `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA), `--client-token` (default: generated), `--source` (default: `spx-cli`) | Same account, created through `POST /admin/CreateAccount` over SigV4 instead of NATS — the path the public signup form uses, so a shell can exercise it. Credentials come from the standard AWS chain (env vars or `AWS_PROFILE`), never from the cluster master key; the caller must be an IAM user in the super-admin account holding `spinifex:CreateAccount`. Reuse the printed `--client-token` to retry: a fresh token would create a second account. |
 | `spx admin account list` | — | Connects to NATS → IAMService.ListAccounts() → prints table with Account ID, Name, Status, Created |
-| `spx admin account list --remote` | `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA) | Same listing read through `POST /admin/ListAccounts` over SigV4 instead of NATS, so an off-cluster operator or the load-test harness sees its tenants without SSH access to a node. Credentials come from the standard AWS chain; the caller must be an IAM user in the super-admin account holding `spinifex:ListAccounts`. |
+| `spx admin account list --remote` | `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA) | Same listing read through `POST /admin/ListAccounts` over SigV4 instead of NATS, so an off-cluster operator or the load-test harness sees its tenants without SSH access to a node. Credentials come from the standard AWS credential chain: on a cluster node that means `AWS_PROFILE=spinifex` (there is no default profile, so without it the command fails to find credentials), or equivalent credentials; the caller must be an IAM user in the super-admin account holding `spinifex:ListAccounts`. |
 | `spx admin account quota get <account-id>` | `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA) | Reads the limits in force for one account through `POST /admin/GetAccountQuota` over SigV4, and reports for each dimension whether it came from the `[quota]` block in `awsgw.toml` (`config`) or from an override stored against the account (`override`). Credentials come from the standard AWS chain; the caller must be an IAM user in the super-admin account holding `spinifex:GetAccountQuota`. |
 | `spx admin account quota set <account-id>` | `--vcpus`, `--vpcs`, `--subnets`, `--eips`, `--volumes`, `--volumes-gib`, `--rds-instances`, `--load-balancers`, `--clear`, plus `--endpoint`, `--region`, `--ca-bundle` | Overrides individual limits for one account through `POST /admin/PutAccountQuota`, leaving every dimension not named inheriting the configured baseline. `-1` uncaps a dimension; `0` is a real limit that denies every request on it. `--clear` removes the whole override record and cannot be combined with a limit flag. The account must already exist. Prints the resulting limits, so a caller never has to guess what it applied. The caller must hold `spinifex:PutAccountQuota`. |
-| `spx admin account delete <account-id>` | `--dry-run`, `--force`, `--yes` (requires `--name`), `--name` (account name, must match the stored record) | Prints the inventory it is about to remove → confirms the account name (prompted unless `--name` is given) → marks the account `TERMINATING`, which blocks its credentials cluster-wide → deletes its resources in dependency order (compute, attachments, storage, network, platform, identity, account), waiting for each stage to actually drain before starting the next → releases the name reservation and quota counter → deletes the account record. Anything that will not delete is reported as **stuck** with the reason, and the account stays `TERMINATING` so the residue keeps an owner. `000000000000` and `000000000001` are refused. `--dry-run` lists and changes nothing. The compute stage reaps ECS clusters, EKS nodegroups and clusters, and RDS instances before it reaps plain instances, because their backing VMs belong to the system account and a tenant-scoped instance listing cannot see them. The storage stage reaps the account's S3 buckets: uploads in flight are aborted, then the objects are removed a page at a time, then the bucket. Only buckets the account owns are ever listed, so the shared platform buckets are out of reach rather than filtered out. Platform-managed serving endpoints are reaped ahead of instances, because only their own delete releases the ENI and attached data volume alongside the VM, and spot instance requests are cancelled first so a live request cannot launch a replacement for something already terminated. The network stage reaps load balancers, then target groups, then NAT gateways and egress-only internet gateways before the subnets, security groups and addresses they hold. The platform stage reaps ACM certificates. |
+| `spx admin account delete <account-id>` | `--dry-run`, `--force`, `--yes` (requires `--name`), `--name` (account name, must match the stored record), `--endpoint`, `--region`, `--ca-bundle`, `--client-token` | Always runs through `POST /admin/DeleteAccount` over SigV4 and follows `DescribeAccountDeletion` to completion; the CLI never tears an account down in its own process, so a CLI delete and an API delete cannot race. Credentials come from the standard AWS chain; the caller must be an IAM user in the super-admin account holding `spinifex:DeleteAccount` and `spinifex:DescribeAccountDeletion`. Prints the inventory it is about to remove → confirms the account name (prompted unless `--name` is given) → marks the account `TERMINATING`, which blocks its credentials cluster-wide → deletes its resources in dependency order (compute, attachments, storage, network, platform, identity, account), waiting for each stage to actually drain before starting the next → seals the account's quota override record so no later write can recreate it, which needs every gateway node the cluster configures to answer a read-only probe first (otherwise nothing is sealed and the account stays `TERMINATING`) → releases the name reservation and quota counter → deletes the account record. Anything that will not delete is reported as **stuck** with the reason, and the account stays `TERMINATING` so the residue keeps an owner. `000000000000` and `000000000001` are refused. `--dry-run` lists and changes nothing. The compute stage reaps ECS clusters, EKS nodegroups and clusters, and RDS instances before it reaps plain instances, because their backing VMs belong to the system account and a tenant-scoped instance listing cannot see them. The storage stage reaps the account's S3 buckets: uploads in flight are aborted, then the objects are removed a page at a time, then the bucket. Only buckets the account owns are ever listed, so the shared platform buckets are out of reach rather than filtered out. Platform-managed serving endpoints are reaped ahead of instances, because only their own delete releases the ENI and attached data volume alongside the VM, and spot instance requests are cancelled first so a live request cannot launch a replacement for something already terminated. The network stage reaps load balancers, then target groups, then NAT gateways and egress-only internet gateways before the subnets, security groups and addresses they hold. The platform stage reaps ACM certificates. |
 | `spx admin account delete --force` | as above | Escalates only for resources the ordinary API refuses: it clears a volume attachment in the control plane without the guest's cooperation (`ec2.ForceDetachVolume`), hard-destroys after a graceful attempt times out, and treats an already-missing resource as deleted. It never reorders the stages and never widens scope beyond the account. Use it once the guest is being destroyed — this is the escape from a volume stuck attached to an instance that will not terminate, where neither can otherwise be deleted. |
-| `spx admin account delete --remote` | as above, plus `--endpoint`, `--region`, `--ca-bundle`, `--client-token` | Same teardown driven through `POST /admin/DeleteAccount` over SigV4 instead of NATS, then follows `DescribeAccountDeletion` to completion, printing each stage as it lands. Credentials come from the standard AWS chain; the caller must be an IAM user in the super-admin account holding `spinifex:DeleteAccount` and `spinifex:DescribeAccountDeletion`. |
+| `spx admin account delete --remote` | as above | Accepted for existing scripts and ignored: every delete already goes through the admin API. |
 | `spx admin principal create <name>` | `--grant` (repeatable or comma-separated; default: every admin method — `CreateAccount`, `DeleteAccount`, `DescribeAccountDeletion`, `GetAccountQuota`, `ListAccounts`, `PutAccountQuota`) | Creates an IAM user in the super-admin account with an inline `spx-admin-methods` policy allowing the requested methods, revokes any access key left from a previous run, and mints one new key. Each method is named rather than granted as `spinifex:*`, so a later addition to the admin surface is not authorised by an existing key. The principal's grants are exactly that one policy: re-running with fewer `--grant` values narrows it, and any other inline policy on the user is removed, so a rename cannot leave a wider grant behind. The secret is printed once; store it in a secret manager or an AWS profile, never in a config file. `admin` is refused as a name — it is the account's own administrator and rotating its key would revoke the bootstrap credential. |
 | `spx admin principal list` | — | Lists every principal in the super-admin account with its access-key count and what it may call. An attached policy is reported by name (`AdministratorAccess (attached)`) rather than expanded, which is how an unscoped principal is spotted. |
 | `spx admin principal revoke <name>` | — | Deletes every access key the named principal holds, leaving the user and its policy in place. Immediate and cluster-wide; re-issue with `principal create`. |
@@ -181,7 +181,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 | `spx admin gpu status` | `--node` (default: local node) | Queries `spinifex.node.status` fan-out → finds the target node response → prints Node, GPU hardware (model list or "none detected"), IOMMU state, vfio-pci state, passthrough enabled/disabled, and GPU pool allocation (`allocated/total`). Also lists GPU-capable instance types when passthrough is active. |
 | `spx admin gpu enable` | — | Checks current passthrough state via NATS → errors if already enabled or prerequisites not met (directs to `setup`) → writes `gpu_passthrough = true` to `spinifex.toml` via `admin.SetGPUPassthrough` → sends SIGHUP to `spinifex-daemon` via `systemctl kill -s HUP` → polls node status for up to 30 s until daemon confirms new state → prints final `gpu status` output. Must be run directly on the target host. |
 | `spx admin gpu disable` | — | Checks current passthrough state via NATS → errors if already disabled or if `AllocGPUs > 0` (must terminate GPU instances first) → writes `gpu_passthrough = false` → sends SIGHUP to `spinifex-daemon` → polls for up to 30 s → prints final `gpu status` output. Must be run directly on the target host. |
-| `spx admin gpu setup` | — | Idempotent host configuration for GPU passthrough. Steps (skipped if already applied): detect GPUs via `gpu.Discover` → collect PCI IDs for all IOMMU-group siblings → check/enable IOMMU in GRUB (`intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt`) → write vfio udev rule (`/etc/udev/rules.d/99-spinifex-vfio.rules`) → blacklist nouveau (`/etc/modprobe.d/blacklist-nouveau.conf`) → blacklist amdgpu if AMD GPU present (`/etc/modprobe.d/blacklist-amdgpu.conf`) → write vfio-pci early binding config (`/etc/modprobe.d/vfio-pci.conf`) → add vfio modules to initramfs. If any change requires a reboot: runs `update-initramfs -u` and exits with reboot instructions. After reboot: verifies `vfio_pci` module is loaded → verifies each GPU is bound to `vfio-pci` (binds explicitly via `driver_override` if unbound) → calls `gpu enable` to activate passthrough. |
+| `spx admin gpu setup` | — | Idempotent host configuration for GPU passthrough. Resolves its boot tools first (`update-grub` or `grubby`, `update-initramfs` or `dracut`) and stops before any change if either pair has neither or both. Steps (skipped if already applied): detect GPUs via `gpu.Discover` → collect PCI IDs for all IOMMU-group siblings → if IOMMU is inactive, add `intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt` to `GRUB_CMDLINE_LINUX_DEFAULT` and run `update-grub`, or run `grubby --update-kernel=ALL --args=...` and check the default entry and `/etc/kernel/cmdline` carry them → write vfio udev rule (`/etc/udev/rules.d/99-spinifex-vfio.rules`) → blacklist nouveau (`/etc/modprobe.d/blacklist-nouveau.conf`) → blacklist amdgpu if AMD GPU present (`/etc/modprobe.d/blacklist-amdgpu.conf`) → write vfio-pci early binding config (`/etc/modprobe.d/vfio-pci.conf`) → add vfio modules to `/etc/initramfs-tools/modules`, or write `/etc/dracut.conf.d/spinifex-vfio.conf` (`force_drivers` for the vfio modules, `install_items` for the `modprobe.d` files). If any change requires a reboot: runs `update-initramfs -u`, or `dracut -f --regenerate-all` and checks every kernel's image with `lsinitrd`, then exits with reboot instructions. After reboot: verifies `vfio_pci` module is loaded → verifies every PCI function in each GPU's slot is bound to `vfio-pci` (binds explicitly via `driver_override` if unbound, fails naming any function on another driver) → calls `gpu enable` to activate passthrough. |
 | `spx admin gpu mig status` | — (must be run on target host) | Runs `gpu.Discover()` locally → for each GPU prints: PCI address, model, MIG capability, MIG mode (enabled/disabled/N/A); for GPUs with MIG enabled lists active MIG slices with GI ID, profile name, and mdev path. |
 | `spx admin gpu mig enable` | `--profile <name>` (required, e.g. `1g.10gb`), `--gpu <pci-addr>` (optional, default: all MIG-capable GPUs) | Checks no GPU instances running (NATS) → discovers MIG-capable GPUs via `gpu.Discover()` (filtered by `--gpu` if set) → enables MIG mode on each target (`gpu.EnableMIGMode`) → lists available profiles (`gpu.ListProfiles`) and validates requested profile name → destroys any existing instances (`gpu.DestroyAllInstances`) → creates new instances filling GPU capacity (`gpu.CreateInstances`) → writes `mig_profile` to `spinifex.toml` via `admin.SetMIGProfile` → sends SIGHUP to `spinifex-daemon`. Must be run directly on the target host. |
 | `spx admin gpu mig disable` | `--gpu <pci-addr>` (optional, default: all MIG-capable GPUs) | Checks no GPU instances running (NATS) → discovers MIG-capable GPUs via `gpu.Discover()` (filtered by `--gpu` if set) → destroys all GPU instances (`gpu.DestroyAllInstances`) → disables MIG mode (`gpu.DisableMIGMode`) → clears `mig_profile` in `spinifex.toml` via `admin.SetMIGProfile` → sends SIGHUP to `spinifex-daemon`. Must be run directly on the target host. |
@@ -192,6 +192,35 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 |---------|-------|-------------|
 | `spx admin eks restore-snapshot` | `--cluster` (required), `--snapshot` (optional, defaults to the latest snapshot in predastore), `--account` (optional, defaults to the bootstrap account) | Single-CP total-loss DR path (fail-safe): validates the snapshot exists in predastore BEFORE any mutation (a typo'd/missing key hard-fails, never resets into an empty datastore) → launches a fresh control-plane VM as a cluster-init seed (replaying the persisted create-time launch template) → sets a required-snapshot `RecoveryDirective` (`cluster-reset`) so the boot-time recovery agent aborts rather than resets-into-empty if it cannot fetch the snapshot → persists the replacement in cluster meta BEFORE re-pointing the NLB (so an NLB failure is convergeable by the reconciler, returned as a provisional status, not a hard error) → re-points the cluster NLB's apiserver and konnectivity target groups from the old CP's ENI to the new one → fences the old CP with retries, failing loudly if it cannot be confirmed terminated (split-brain guard). Any failure before the meta commit unwinds the fresh CP (terminate + clear directive) so a re-run does not stack a second resetting control plane. The returned status is provisional — success means the sequence completed, not that etcd is restored and serving; verify cluster health. HA clusters (a spread with a potentially surviving quorum) are rejected — recover those via quorum reformation instead. |
 
+### SQS System Instances
+
+These commands enable SQS on a single-node cluster and start its one system VM.
+A bound VM is not a ready service: `spx admin sqs status` reports readiness.
+
+The SQS VM runs two processes on one encrypted data volume: the Region's dedicated SQS NATS server, which listens only on the guest's loopback address and is separate from platform NATS, and the worker, which serves SQS owner calls from that local store. The worker starts serving only after the local store answers and the receipt-signing key on the volume matches the fingerprint the platform recorded.
+
+SQS supports only a fresh single-node install. A worker record without a supported launch contract, or any record of the separate SQS store VM from an earlier dev build, cannot be upgraded: `enable` refuses it, and any existing volume is kept, not inspected, emptied or replaced. An existing separate store VM is refused on relaunch with its volume kept, and an existing worker VM is not restarted into a supported mode; upgrading such an install is unsupported.
+
+**Do not put tenant production data in SQS.** SQS-managed per-message encryption is not implemented: `GetQueueAttributes` reports `SqsManagedSseEnabled=false`, and a request to enable SSE-SQS or SSE-KMS is rejected.
+The SQS data volume must be encrypted, which is a separate layer under every queue and not SSE.
+There is no `DeleteQueue` or `PurgeQueue`, so a queue and its unexpired messages cannot be removed once created.
+Retention is checked when a message is received, against the queue's retention at that moment, and nothing deletes an expired message: the store drops a body only at its age backstop of 14 days and 1 hour.
+So a message past a short retention can be delivered again if `MessageRetentionPeriod` is later raised while its body is still stored.
+Account termination does not yet remove SQS queues or messages, and there is no disable: the VM's volume, receipt key and data are kept.
+
+Each sends a signed `spinifex` admin action (`EnableSQS`, `DescribeSQS`, `EnsureSQSWorker`) to the AWS gateway using the standard AWS credential chain, for example `AWS_PROFILE=spinifex`. The caller must be in the super-admin account (`000000000001`) and hold `spinifex:<Action>`. The VM, volume, private VPC and ENI belong to the system account (`000000000000`). `--endpoint`, `--region` and `--ca-bundle` default to this node's gateway, region and `config/ca.pem`; off a node, `--endpoint` and `--region` are required. `ensure-worker` and `enable` wait up to 11 minutes, and `status` up to 1 minute.
+
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx admin sqs enable` | `--node` (required), `--instance-type` (required, more than 512 MiB memory), `--volume-size-gib` (required, 1–16384), `--endpoint`, `--region`, `--ca-bundle` | Records SQS as enabled for the Region, pinned to `--node`, then launches or reports the SQS VM there. The desired state is recorded first, so rerunning the command resumes an interrupted enable. More than one node in the cluster is refused before anything is recorded. A worker record with a missing or unsupported launch contract, or a record of the separate store VM, is also refused before anything is recorded or launched; any existing volume is kept, and upgrading it is not supported. |
+| `spx admin sqs status` | `--endpoint`, `--region`, `--ca-bundle` | Shows the requested state, node, generation, VM state, receipt key state, whether a worker is polling this gateway, whether this gateway serves the SQS API, and `Ready`, which is true only when all of them hold. `Unsupported` names a record whose missing or unsupported launch contract blocks enable. |
+| `spx admin sqs ensure-worker` | `--node` (required), `--instance-type` (required, more than 512 MiB memory), `--volume-size-gib` (required, 1–16384), `--endpoint`, `--region`, `--ca-bundle` | Launches the Region's one SQS VM on `--node`, or reports the VM already bound there: node, region, instance, instance type and generation. The VM is pinned to its first node; another node is refused with `InvalidParameterValue`, and there is no takeover. It gets a private system VPC with one /24 subnet and no internet gateway, NAT or Elastic IP, and an encrypted system-account data volume of `--volume-size-gib` that holds the queue data and the receipt-signing key the guest creates on first boot. The platform records only the key's fingerprint. A different `--instance-type` is accepted only while no volume, network interface or launch step has been attempted; once any has, it is `IncorrectState`, whether or not the step completed. A different `--volume-size-gib` than the recorded one is `IncorrectState`. |
+
+`ServiceUnavailable` never proves that no SQS resources exist, because an earlier attempt may already have created some. Its message says what it does establish. When no node serves the subject, this request was not delivered. When a volume create was not sent, that particular create did not happen. When the node cannot boot the guest yet or its launcher lacks a capability, this request stopped before allocating or recording anything. When the gateway times out waiting for the node's reply, nothing is known: the launch may still be running, so never assume it created nothing. In every case, call the command again with the same arguments. A retry first checks the durable record of the attempt and continues only when that is safe; it can proceed to the first launch after a create that was proven unsent, and otherwise reports the recorded outcome as bound or `IncorrectState`, or returns `ServiceUnavailable` again. `IncorrectState` from a volume, network interface or launch step means that step was recorded as sent but its outcome was never observed. It is never resent, the message names the volume when its ID is known, and no tool reconciles it yet: recovery is manual.
+
+`bound` says only that the VM was launched for the current record. The worker is ready only when the guest's local store answers, and it serves only while its receipt key matches the recorded fingerprint; a missing or different key once one is recorded refuses serving and is never replaced.
+Real VPC routing, IMDS and gateway TLS from the guest have not been tested on a live host.
+
 ## Private Admin API
 
 Not an AWS API: a Spinifex-internal surface on the AWS gateway, JSON in and JSON out, one method per path segment at `POST /admin/<Method>`. It exists so a self-service signup form, a provisioning system or a load-test harness can create and remove tenant accounts without an operator shell on a node.

docs/DESIGN.md

@@ -33,67 +33,32 @@ The gateway (`spinifex/runtime/roles/awsgw/awsgw.go`) is the entry point:
 // Connect to NATS (retries while the local broker comes up)
 natsConn, err := natsmsg.ConnectNATSWithRetry(...)
 
-// Load IAM (master key + JetStream KV) and create the gateway
-gw := gateway.GatewayConfig{
-    NATSConn:   natsConn,
-    Config:     nodeConfig.AWSGW.Config,
-    IAMService: iamService,
+// Load IAM (master key + JetStream KV), then compose the gateway: composeGateway
+// sets every production hook and wires the service registry from each domain's
+// awsapi registration.
+gw, err := composeGateway(gatewayInputs{
+    NATS:        natsConn,
+    IAM:         iamService,
+    Credentials: credentials,
+    Admission:   admission,
     // ...
-}
+}, ecrControlPlane, acmControlPlane, ec2ControlPlane /* ... */)
 
 // Serve over TLS
 server.ListenAndServeTLS("", "")
 ```
 
-Request routing (`spinifex/gateway/gateway.go`):
+Request routing (`spinifex/ingress/aws/pipeline/gateway.go`):
 
-1. **Authentication**: SigV4 middleware validates AWS credentials and resolves the account ID
-2. **Throttling**: Per-account+action token bucket rejects bursts post-auth
-3. **Service Detection**: Extracts service name from the Authorization header
-4. **Action Dispatch**: Routes to service-specific handler. A service registered through the ingress dispatch registry (`ingress/aws/dispatch`, wired by `runtime/roles/awsgw`) is dispatched by its registration; currently those are the ECR and ACM control planes, registered from `domains/ecr/awsapi` and `domains/acm/awsapi`. Every other service uses the legacy switch:
-
-```go
-switch svc {
-case "ec2":
-    err = gw.EC2_Request(w, r)
-case "iam":
-    err = gw.IAM_Request(w, r)
-case "sts":
-    err = gw.STS_Request(w, r)
-case "elasticloadbalancing":
-    err = gw.ELBv2_Request(w, r)
-case "eks":
-    err = gw.EKS_Request(w, r)
-case "bedrock":
-    err = gw.Bedrock_Request(w, r)
-// ... bedrock-runtime, bedrock-agent, bedrock-agent-runtime
-case "ecs":
-    err = gw.ECS_Request(w, r)
-case "rds":
-    err = gw.RDS_Request(w, r)
-case "tagging":
-    err = gw.Tagging_Request(w, r)
-case "spinifex":
-    err = gw.Spinifex_Request(w, r)
-}
-```
+1. **Public routes**: `/oidc/eks/...` and `/v2` (the ECR registry) are mounted ahead of SigV4 by `runtime/roles/awsgw`
+2. **Authentication**: SigV4 middleware validates AWS credentials and resolves the account ID
+3. **Throttling**: Per-account+action token bucket rejects bursts post-auth
+4. **Service Detection**: Extracts service name from the Authorization header
+5. **Action Dispatch**: Every service is a registration (`ingress/aws/dispatch`) built by its domain's `awsapi.NewRegistration` and wired by `runtime/roles/awsgw`. The pipeline dispatches by registered service name and holds no per-service code. The private `/admin/<Method>` surface dispatches to the registration `awsgw` names as `Admin.Service`.
 
 ### 3. EC2 Handler
 
-The EC2 handler (`spinifex/gateway/ec2.go`) parses the `Action` parameter and delegates to specific handlers via the generic `ec2Handler` and `ec2HandlerWithReq` wrappers. The account ID resolved by SigV4 auth is threaded into every handler so requests are scoped per IAM principal:
-
-```go
-"RunInstances": ec2HandlerWithReq(func(ctx context.Context, input *ec2.RunInstancesInput, gw *GatewayConfig, accountID string, r *http.Request) (any, error) {
-    // ... PassRole and launch-quota checks, then the vCPU quota charge
-    return ec2instanceapi.RunInstances(ctx, input, gw.NATSConn, gw.IAMService, accountID, passRoleCheck, launchQuotaCheck, gw.ExpectedNodes)
-}),
-"DescribeInstances": ec2Handler(func(ctx context.Context, input *ec2.DescribeInstancesInput, gw *GatewayConfig, accountID string) (any, error) {
-    return ec2instanceapi.DescribeInstancesChecked(ctx, input, gw.NATSConn, 0, gw.NodeIDs, accountID)
-}),
-// ... + volumes, snapshots, VPCs, subnets, route tables, IGWs, NAT gateways,
-// security groups, network interfaces, elastic IPs, placement groups, key pairs,
-// images, tags, capacity reservations, spot instances
-```
+The EC2 registration (`spinifex/domains/ec2/awsapi`, `NewRegistration`) parses the `Action` parameter and looks the handler up in its action table. The account ID resolved by SigV4 auth is threaded into every handler so requests are scoped per IAM principal. Each action's handler is built from the owning EC2 resource package (`domains/ec2/instance`, `volume`, `vpc` and so on), with PassRole, quota and client-token idempotency wrapped on at the action table rather than inside the resource.
 
 ### 4. NATS Messaging
 
@@ -181,12 +146,17 @@ type Daemon struct {
 
 Tracks available and allocated CPU/memory to prevent overcommit, and drives the dynamic `ec2.RunInstances.{type}` subscriptions:
 
+One instance vCPU is one CPU the kernel has online: a thread with SMT on, a core with SMT off or on ARM, as AWS counts a vCPU.
+The OS and Spinifex's own services keep 1 CPU on a host with up to 8, 2 up to 16, 4 up to 63 and 8 from 64, and every other CPU is schedulable one-to-one with no overcommit.
+[ADR-0015](https://github.com/mulgadc/mulga/blob/main/docs/adr/0015-host-cpu-capacity-and-reservation.md) fixes this rule.
+
+
 ```go
 type ResourceManager struct {
     mu            sync.RWMutex
-    hostVCPU      int                                // raw runtime.NumCPU
+    hostVCPU      int                                // online CPUs (ADR-0015:S1)
     hostMemGB     float64                            // raw /proc/meminfo MemTotal
-    reservedVCPU  int                                // held back for spinifex services
+    reservedVCPU  int                                // ADR-0015:S2 tier: 1, 2, 4 or 8
     reservedMem   float64                            // held back for spinifex services
     allocatedVCPU int
     allocatedMem  float64
@@ -229,6 +199,55 @@ type EBSRequest struct {
 
 Viperblock responds with an NBD URI that QEMU uses to access the block device.
 
+#### Direct-boot microVM data volumes
+
+A direct-boot microVM keeps the launcher's saved `vm.Config` as a clean base, and `startQEMU` adds data volumes only to a launch-only copy of it (`runtime/compute/vm/lifecycle.go`).
+At each start the daemon takes one snapshot of the instance's already-mounted data requests, probes each NBD or host block export from that snapshot, and builds the command from the same snapshot, so the launched endpoint is the one that was checked and is never written back into the saved base.
+The microVM boots with `pcie=on`, so its NICs are `virtio-net-pci` devices with MSI-X interrupts instead of `virtio-net-device` MMIO devices.
+Each data volume becomes a `virtio-blk-pci` device on the default PCIe bus with the same id, node name, I/O thread, serial and `werror=stop,rerror=stop` policy as the q35 device, and takes no PCIe hot-plug port.
+A microVM started without `pcie=on` keeps the `virtio-*-device` MMIO transports.
+
+Boot and EFI requests are rejected on this path, and a direct-boot instance with no data requests launches its base unchanged.
+Disk-boot (q35) instances keep the existing PCI hot-plug path.
+
+Hot-attaching a volume to a running direct-boot microVM is not supported, because the microVM has no PCIe hot-plug root port to plug into.
+`AttachVolume` refuses it before mounting the volume or changing any instance or volume state, and the EC2 API returns `UnsupportedOperation`.
+
+This covers only requests that are already mounted when the instance starts; nothing yet supplies data volumes to a system instance, and the path is verified with fake-QEMU unit tests, not with a real EBS volume in a guest.
+
+#### Volume leases
+
+Before opening an engine on a volume, the Viperblock worker takes a lease: one entry per volume in the `VIPERBLOCK_VOLUME_LEASES` JetStream KV bucket, which expires 45 seconds after its last write (`providers/ebs/viperblock/volume_lease.go`).
+A renewal becomes eligible once the last confirmed write is 10 seconds old, and the holder checks for that periodically, so the rewrite lands some time after that point.
+Each rewrite is conditioned on the revision the holder last wrote, and a refused write that finds the entry gone or naming another owner or generation fences the export as a takeover.
+It may keep writing for 40 seconds after the last write the store acknowledged, measured from when that write was sent, because the server starts the entry's expiry when it applies the write and never earlier.
+A slow acknowledgement therefore shortens the remaining validity rather than extending it, and an acquisition acknowledged after its validity has already run out is refused as retryable.
+
+For a holder that cannot renew, a timer is armed for the validity deadline, leaving a nominal 5 seconds before the entry can expire; when it fires, the export's nbdkit is fenced with SIGKILL (`volume_fence.go`).
+The fence runs on its own goroutine, so Go scheduling, that goroutine and process termination all delay the kill, and enough delay can use up the whole margin; a kill held in uninterruptible sleep, which the fence waits up to 30 seconds for, is the extreme case.
+Once a lease is lost or has lapsed it stays lost: a renewal that succeeds afterwards is ignored, and another open on the same node cannot share it while the fence runs.
+
+These rules bound only the holder's own clock.
+The storage backend does not reject a stale writer, so a paused or partitioned host that resumes after its validity can still write; generation enforcement at the write boundary is required by Q-24 in the umbrella repository's `docs/specs/viperblock/rfc3-1_overview.md` and is not implemented.
+An export that survives a daemon restart writes without a renewed lease until recovery adopts it, and a survivor that recovery cannot adopt while a guest still uses it is left running without a lease or a fence.
+
+#### Unsealed writes and start elsewhere (interim)
+
+A volume is unsealed while its node holds writes the backend has not confirmed; the node records this in a dirty marker naming itself as owner (`providers/ebs/viperblock/volume_dirty.go`).
+Opening such a volume on another node loses those writes, so two guards stop it while the previous owner may still hold them.
+This is an interim guard until the instance lifecycle work defines ownership, and it is not a fence.
+
+When `ec2.start` is forwarded to the node that last ran a stopped instance and that node refuses for capacity, the forwarding node normally starts the instance locally.
+If the refusing node still holds unsealed writes for any of the instance's volumes, it sets the `X-Unsealed-Writes` NATS reply header and the forwarder relays `InsufficientInstanceCapacity` instead of starting elsewhere.
+The check asks the owner's viperblockd over the per-node `unsealed` verb with a 5 second bound, and a check that fails or times out counts as holding, so the refusal is kept.
+
+Independently, a node that finds another node's dirty marker on a volume refuses to open it when that node's daemon heartbeat (`heartbeat.<node>` in the cluster-state bucket) is under 60 seconds old.
+It also refuses when the heartbeat cannot be read or no heartbeat source is wired, and the refusal maps to the held-elsewhere error that recovery retries.
+A previous owner that has stopped heartbeating is opened over with a warning, as before.
+The heartbeat age is compared across node clocks, so the 60 second window is also the clock-skew allowance.
+
+EKS control-plane recovery treats both refusals, capacity and held-elsewhere, as retryable: it keeps retrying the stopped member and does not count them against its restart attempts.
+
 ### Predastore (S3)
 
 Object storage used for:
@@ -256,6 +275,60 @@ file. It does not connect to Predastore to do so, and no Predastore node
 exposes a status endpoint. This is an operator status interface, not a generic
 storage-provider contract.
 
+### SQS message store (internal)
+
+`domains/sqs/message` defines the store-neutral interface the SQS domain reaches its message store through (ADR-0011:S6), with a JetStream adapter in `domains/sqs/message/jetstream` and a shared conformance suite in `domains/sqs/message/conformance`.
+It is used only by the SQS worker inside the SQS system instance and does not itself deliver an SQS API.
+Spinifex ships no host service, unit, user or firewall rule for the dedicated JetStream server; the system instance that runs it is described under *SQS system instance (operator-launched)*.
+
+The interface covers durable and idempotent append, body and metadata reads, conditional state writes and deletes, per-namespace counts, namespace enumeration, an ordered change feed, a bounded paginated scan, namespace deletion and a sweep of tombstoned bodies.
+A namespace carries an immutable message-shard and state-shard allocation chosen when the queue is created, so a later change in shard counts never moves an existing queue.
+
+The adapter provisions a fixed set of shared message streams (`SQS_M_<shard>`, one subject per queue) and shared state buckets (`sqs_state_<shard>`), keeps historical ones, and applies a 14-day-plus-one-hour retention backstop to both.
+It supports one node (R1) and three nodes (R3) only and refuses any other declared cluster size before provisioning.
+
+Reads that must observe a completed write or delete go to the stream leader: `GetState`, the state-key pass behind `Count` and `Namespaces`, and the body pass behind `Namespaces` and `Scan`.
+This avoids the follower-served direct gets and follower-placed consumers that returned purged data in local three-node fault-injection tests; it is not a general linearizability guarantee.
+Each leader read is one request per record, and the body pass transfers each full body internally even though `Scan` returns metadata only, which is a material cost for a large backlog.
+A state-key pass is bounded but is not a snapshot when a key is overwritten during it.
+
+The feed reads through an in-memory, headers-only ordered consumer.
+If that consumer delivers nothing for about two seconds while the stream holds a change after the feed's cursor, the feed replaces it from the cursor, so a stored change does not wait on a stalled consumer; the cause of those stalls is not established.
+`Close` is terminal, waits for consumer opens and deletes in progress and retries failed deletes, but cleanup is best effort for a create whose outcome is unknown: a consumer that lands after its last delete is removed by the server's one-minute inactive threshold.
+
+Coverage is unit and conformance tests on in-process R1 and R3 servers; there is no end-to-end or multi-host validation of this code, and earlier multi-host evidence predates the final feed.
+
+### SQS system instance (operator-launched)
+
+One system-owned microVM per Region runs both the SQS store and the SQS worker as separate processes, on one encrypted data volume pinned to one node (ADR-0011 Amendment 5, ADR-0012 Amendment 6).
+Only an operator activates it with the super-admin `EnsureSQSWorker` or `EnableSQS` actions; nothing starts it at daemon startup, and platform NATS is neither used nor changed for SQS data.
+This first profile is fresh single-node only: a host cluster of more than one node is refused before anything is recorded or allocated, and there is no disable.
+
+`make build-sqs-worker-microvm-image` builds the one `sqs-worker` image: the static `sqs-worker` binary, the pinned `nats-server`, the ext4 tools and a network `/init`; releases ship it as `microvm-sqs-worker`, which `setup.sh` installs to `/usr/share/spinifex/microvm/sqs-worker`.
+At boot the guest reads three firmware config values under `opt/spinifex/sqs-worker/` (the data disk's serial, its Region and the gateway URL on this node's mgmt bridge) and refuses before touching any disk if one is missing or malformed.
+It selects exactly one disk with that serial, runs `e2fsck -p`, mounts it at `/data` and confirms from `/proc/self/mountinfo` that the whole disk is what is mounted; the guest never formats.
+
+The store is a dedicated `nats-server` listening only on `127.0.0.1`, with its JetStream directory at `/data/jetstream` and `GOMEMLIMIT=512MiB` (ADR-0011:S11), so nothing outside the VM can reach it and it needs no network identity of its own.
+The store starts only after the receipt key is settled, and the worker connects only after the store answers JetStream account info, so an unavailable store never appears ready.
+
+The receipt-signing key lives on the data disk under `/data/sqs-worker-key`, written to a temporary file, synced, renamed and its directory synced before its fingerprint is recorded.
+`domains/sqs/workerident` holds the platform's receipt-key state on the worker's identity record, beside its generation, claim, pinned node and volume: `pending` is set only when the record first creates its volume, and `recorded` holds the fingerprint.
+Under `pending` the guest may create a key, or reuse a valid key it already wrote before a crash, only while no `/data/jetstream` directory exists; any store data under `pending` refuses, because it proves the volume is not new.
+Under `recorded` a missing, unreadable or different key refuses the boot and is never regenerated; a record with no receipt-key state is refused as not proven new.
+`RecordReceiptKey` writes the fingerprint by compare-and-swap that re-checks the current instance, generation, claim and volume, and a lost reply is answered by reading the record again, never by a new key.
+A replacement worker reuses the pinned volume and keeps its predecessor's receipt-key state.
+
+The worker reaches the gateway over TLS 1.3 trusting only the cluster CA, signing with instance-role credentials from the metadata service only.
+The internal `spinifex-sqs` actions are `ClaimReceiptKey`, `RecordReceiptKey`, `PollWork` and `CompleteWork`; the gateway admits only an assumed-role session of `sqsWorkerRole` in the system account and resolves the instance through `workerident.Authorize`, a point-in-time check against the current bound record.
+The worker pulls tenant work from its own node's gateway relay and posts results back, so no SQS payload crosses platform NATS; the relay holds a bounded, in-memory set of waiting requests and answers `ServiceUnavailable` when it is full or no worker is polling.
+
+`EnsureSQSWorker` takes a `NodeId`, an `InstanceType` and a `VolumeSizeGiB`; the instance type's memory must exceed the store's 512 MiB limit, and a later request for a different type or size is refused as `IncorrectState`.
+`EnableSQS` records the Region's desired state, pinned to one node, and then ensures the worker; `status` reports it ready only when the worker is bound, polling and the tenant API is registered.
+A store launch record or VM from the earlier separate-store design, or a worker record without a supported launch contract, is refused with its volume kept: enable stops before recording anything, and a relaunch of an old store VM is marked failed rather than started.
+
+Evidence is local: unit tests, the guest shell test and an AWS SDK test that creates a queue and sends, receives and deletes a message through the real worker loop against a real local JetStream store and a restart.
+A real guest boot, its metadata route on a real ENI and the signed path on a live node are unproven, Viperblock's at-rest key is held by the platform, and SQS-managed message encryption is not implemented, so the profile must not hold tenant production data.
+
 ## Configuration
 
 Cluster configuration (`spinifex/bootstrap/config/config.go`):

docs/PACKAGE_BOUNDARY_MIGRATION.md

@@ -60,7 +60,7 @@ intact; ADR-0001 explicitly defers that design.
 
 | Current path | Standing | Target ownership | Rule |
 |---|---|---|---|
-| `cmd/aws-model-coverage` | Qualification generator | Thin main; reusable code in `internal/tooling` | Produces the dispatch-versus-model inventory; never imported by product packages. |
+| `cmd/aws-model-coverage` | Qualification generator | Thin main; reusable code in `internal/tooling` | Produces the dispatch-versus-model inventory; never imported by product packages. Debt: it reads each domain's declared `OperationInventory` from `domains/<service>/awsapi` (ACM, EC2, ECR, ECS, EKS, ELBv2, IAM, RDS, Route 53, SQS, STS) and never constructs a registration. |
 | `cmd/dhcptest` | Development diagnostic | Thin main; reusable code in `internal/tooling` if extraction is useful | It is a DHCP probe, not a supported `spx admin` command. |
 | `cmd/spx-loadgen` | Qualification/load tool | Thin main; reusable code in `internal/testkit` | It drives test load and is not shipped to nodes. |
 | `tests/e2e/*/cmd/*`, `tests/bench/*/cmd/*` | Test and benchmark-private tools | Remain test-private | They are invoked by a test or benchmark harness, not a product process. |
@@ -351,8 +351,27 @@ while the branch is in flight; they do not describe the umbrella repository's
 | `spinifex#1269` | `spinifex/handlers/elbv2` | `spinifex/domains/elbv2` (package `elbv2`) | A mechanical whole-package relocation under the 2026-10-26 ADR-0001 amendment: the ELBv2 control plane keeps its files, tests and current dependencies, and importers alias it `elbv2domain`. No identifier, API, NATS subject, KV bucket, metric, error string or behaviour changes. Its allowlist entries are now sourced from `domains/elbv2`. Q-30's data-plane boundary stays deferred, recorded as `agents/elbv2` (Q-30 data-plane boundary) debt. |
 | `spinifex#1271` | `spinifex/handlers/ecs` | `spinifex/domains/ecs` (package `ecs`) | A mechanical whole-package relocation: ADR-0001's `handlers/ecs` row targets `domains/ecs`, the files are already organised by resource, and resource packages are introduced only where callers need a compiler boundary. The ECS control plane keeps its files, tests and current dependencies alongside the already-extracted `domains/ecs/taskdefinition`, and importers alias it `ecsdomain`. No identifier, API, NATS subject, KV bucket, metric, error string or behaviour changes. Its allowlist entries are now sourced from `domains/ecs`. `handlers` has no package left and is a retired legacy root; R5 holds `domains/ecs` to the interim rule's ban on importing the daemon. |
 | `spinifex#1273` | `spinifex/gateway/{ecs,elbv2}` | `spinifex/domains/{ecs,elbv2}/awsapi` (`awsapi`, imported as `ecsawsapi`, `elbv2awsapi`) | A mechanical relocation of the ECS and ELBv2 AWS action adapters to their ADR-0001 domain homes, as `spinifex#1265` did for IAM, STS, EKS and RDS; APIs, NATS subjects, KV buckets and behaviour are unchanged. The root `gateway` dispatch still imports each package, recorded as `gateway` (service registration) debt, and `domains/admission/quota` keeps its live ELBv2 counter import, now of `domains/elbv2/awsapi`. |
+| `spinifex#1336` | `spinifex/gateway/tagging.go` (`Tagging_Request`, `taggingActions`, `taggingActionFromTarget`) and the `GatewayConfig.TaggingResources` field | `spinifex/domains/tagging/awsapi` (`NewRegistration`, `Deps`, `Actions`, `OperationInventory`), composed by `spinifex/runtime/roles/awsgw` | Resource Groups Tagging registers through the ADR-0002 seam like ACM and Route 53. The check order is kept: action, scope, authorization, account, body, aggregator, so a denied caller still gets `AccessDenied` before any body error and an unwired aggregator answers `ServerInternal`. The `checkPolicyResources` stand-in of `*` for every action is unchanged. The legacy switch case, the `supportedServices` entry and the `jsonErrorService` case are removed, and the registration declares the JSON 1.1 envelope. No allowlist row existed for this path. The nil-NATS check moves to the gateway-wide gate, which answers `ServiceUnavailableException` before dispatch as it did for every service. `cmd/aws-model-coverage` output is byte-identical because there is no embedded Tagging model. The characterization tests that pin the pipeline behaviour were committed first and pass unchanged against the registration; only the `withTagging` test helper changed. |
+| `spinifex#1336` | `spinifex/gateway/elbv2.go` (`ELBv2_Request`, `elbv2Actions`, `elbv2Handler`, `accountLoadBalancerLimit`) | `spinifex/domains/elbv2/awsapi` (`NewRegistration`, `Deps`, `Quota`, `ActionNames`, `OperationInventory`), `spinifex/domains/admission/quota` (`Service.LoadBalancerLimit`), `spinifex/ingress/aws/query` (`ParseArgs`), composed by `spinifex/runtime/roles/awsgw` | ELBv2 registers through the ADR-0002 seam as the first AWS Query service. The check order is kept: query parse, action, account, input parse, scope, authorization, NATS, dispatch, so a denied caller still gets `AccessDenied` before the handler runs. The gateway members ELBv2 read are now supplied as `Deps`: `NATSConn` as `Deps.NATS`, `Quota` as the `Deps.Quota` interface declared in `awsapi` because `domains/admission/quota` imports `awsapi`, `Region` as `Invocation.Region`, and the policy gate as `Invocation.Authorize`. The `DescribeAccountLimits` cap resolution moves to `Service.LoadBalancerLimit` unchanged, and a nil `Deps.Quota` enforces nothing and reports the AWS default. AWSGW now wires the registry after it builds the quota service, because the registration captures it. The registration declares the XML envelope and the inventory name `elasticloadbalancingv2`, so `cmd/aws-model-coverage` keys it explicitly and its output is byte-identical. `gateway.ParseAWSQueryArgs` delegates to `query.ParseArgs`. The legacy switch case, the `supportedServices` entry, the inventory entry and the `gateway -> domains/elbv2/awsapi` allowlist row are removed. `cmd/aws-model-coverage` imports `domains/elbv2/awsapi` for the inventory, under a fourth named-exception allowlist row whose reason is now generic for all four. Finding, not fixed here: the XML for `DescribeAccountLimits` and `DescribeLoadBalancers` lists member fields in a different order from run to run because the marshal is map-ordered, so the characterization tests pin decoded content rather than bytes. The characterization tests that pin the pipeline behaviour were committed first and pass unchanged against the registration; only the `withELBv2` and `serveELBv2` test helpers and the setup of two older gateway tests changed. |
+| `spinifex#1337` | `spinifex/gateway/spinifex.go` (`Spinifex_Request`, `spinifexAdminActions`) | `spinifex/operator/admin` (`NewRegistration`, `Deps`, `ModelAccess`, `ActionNames`), composed by `spinifex/runtime/roles/awsgw` | The `spinifex` Query service registers through the ADR-0002 seam with the XML error envelope it had. The check order is kept: query parse, action, authorization with the wildcard resource, account, the super-admin account gate, then the action, so a denied caller learns nothing about which actions exist. The gateway members the actions read are now `Deps`: `Version` and `Commit` strings, `NATSConn` as `Deps.NATS`, `RootCAs`, `DiscoverActiveNodes` as `Deps.ActiveNodes`, `BedrockAccessAdmin` as the `ModelAccess` interface, and the credential and access resolvers as `Deps.Credentials` and `Deps.Access`. The query actions need only the caller account, so the actor is not widened. The cluster-unavailable answer stays in the shared request path ahead of authorization. `/admin/{method}` moved onto the same registration in `mulga-3nbla.7`, recorded below. |
+| `spinifex#1344` | `spinifex/gateway/rds.go` (`RDS_Request`, `rdsCaller`) | `spinifex/domains/rds/awsapi` (`NewRegistration`, `Deps`, `Quota`, `CallerFromActor`, `OperationInventory`), composed by `spinifex/runtime/roles/awsgw` | RDS registers through the ADR-0002 seam with the XML envelope it had. The check order is kept: query parse, action, account, the principal-class gate for the internal agent actions, scope, authorization, NATS, quota and dispatch, so a denied caller still gets `AccessDenied` before the handler runs and no customer grant is evaluated against an agent action. The gateway members RDS read are now supplied as `Deps`: `NATSConn` as `Deps.NATS`, `ExpectedNodes` as `Deps.ExpectedNodes`, `Quota.EnforceRDSInstances` as the `Deps.Quota` interface, `Region` as `Invocation.Region` and the policy gate as `Invocation.Authorize`. A nil `Deps.Quota` skips the DB-instance cap. The caller comes from `dispatch.Actor`: the session name for an assumed-role session and the user name otherwise, and the role name from `Actor.RoleName()`, which parses the underlying role ARN and never the session name. The account comes from `Invocation.AccountID`, so a caller with no composable ARN still reaches the policy check and is refused there as before. The legacy switch case, the `supportedServices` entry, the inventory entry and the `gateway -> domains/rds/awsapi` allowlist row are removed, and `cmd/aws-model-coverage` reads the inventory under a fifth named-exception allowlist row. The characterization tests that pin the pipeline behaviour were committed first with spinifex#1336 and pass unchanged against the registration; only the `withRDS` and `serveRDS` test helpers and the setup of the direct-call RDS tests changed. |
 | `#1277` | `spinifex/accountteardown` | `spinifex/operator/accountlifecycle` (package `accountlifecycle`, imported as `accountteardown`) | A mechanical whole-package relocation under the 2026-10-27 ADR-0001 amendment: the account deletion coordinator keeps its APIs, NATS subjects, KV access and behaviour; importers keep the `accountteardown` name as an alias, so only their import paths change. Replacing its direct domain and adapter cleanup calls with owner capabilities stays deferred debt. |
 | `#1285` | `spinifex/gateway/requestaudit.go` and `slogRequestLogger` from `spinifex/gateway/gateway.go` | `spinifex/ingress/aws/audit` (package `audit`) | The first generic-ingress extraction from the root gateway: the request-audit middleware and access-log middleware move unchanged in behaviour. The gateway's authentication flow and dispatch still populate the record, now through the exported `audit.From` and `Record` setters at the same call sites; the package imports only `foundation/telemetry` and `ingress/http`. Its unit tests moved with it; the tests that run it behind SigV4 stay in `gateway`. |
+| `spinifex#1334` | `spinifex/gateway/spx` (`spx.go`, `storage.go`, `promote_image.go` and their tests) | `spinifex/operator/admin` (package `admin`) | The version, node, VM and storage-status gather helpers and the image-promote NATS request move unchanged with no shim; the `spinifex` service dispatch that calls them stays in `gateway/spinifex.go` until the `spinifex` registration slice. `operator/admin` already imports `domains/ec2/image` and `contracts/operator/v1`, so the move adds no edge and the layering ratchet needed no allowlist row. The `gateway/spx` frozen-legacy entry is removed. |
+| `spinifex#1334` | `spinifex/gateway` (`CreateAccountRequest`/`Response`, `DeleteAccountRequest`, `DescribeAccountDeletionRequest`, `DeletionState*`, `AccountSummary`, `ListAccountsResponse`, `AdminMethodNames`) | `contracts/operator/v1` (`admin.go`) | The `/admin` bodies the CLI shares with the gateway move unchanged in field names, JSON tags and omitempty, pinned by golden JSON written against the gateway types first and moved with them. `AdminMethodNames` is now owned by the contract and the gateway builds its `adminMethods` accept set from it, with a test that the router accepts exactly that list. `AccountQuotaRequest`/`AccountQuotaResponse` (embed `quota.Overrides` from `domains/admission/quota`) and `DeleteAccountResponse` (embeds `accountlifecycle.Result`) stay in `gateway`: `contracts` may import only `foundation` and `contracts`, so they are settled when the `/admin` handlers move to `operator/admin`. `accountlifecycle.Stage` serialises as an integer, not its name; the golden test pins that. |
+| `spinifex#1345` | `spinifex/gateway/admin*.go` (the `/admin/{method}` handlers, gates and body cap) and the `AccountQuotaRequest`/`AccountQuotaResponse`/`DeleteAccountResponse` wire types | `spinifex/operator/admin` (`rest.go`, `account_*.go`), served by the same `spinifex` registration | `/admin/{method}` stays a chi route in `gateway`, ahead of the service selector, but `Admin_Request` now only mints the request ID and header and hands the request to the registered entry through `dispatchRegistered`, the path every Query request takes. `dispatch.Invocation` gains `Service`, the credential-scope service, so the scope gate is the first identity gate in the dispatcher and a key signed for another service is denied exactly as before. The order is unchanged: verb and method, then the five fail-closed gates (scope, super-admin account, IAM user, identity, `spinifex:<Method>` policy), then the cluster check, then the 8 KiB body cap. The dispatcher returns errors and the route renders them as the JSON envelope with `requestId`, then logs once from the real result, so a denial or a 5xx is never logged as a completed request; the registration's `ResolveAction` labels the method, which removes the `auth.go` special case. The caller is read from `dispatch.Actor` (`UserName` for the gate and `updatedBy`). `BucketStore` and `SignupMaxAccounts` leave `GatewayConfig` for `operator/admin.Deps`, composed in `awsgw` beside `IAM`, `Quota`, `ExpectedNodes` and `NodeIDs`. The three wire types that could not enter `contracts` move to `operator/admin` with their handlers, `operator/cli` imports them from there, and the last `operator/cli` → `gateway` allowlist entry is deleted. |
+| `spinifex#1346` | `spinifex/gateway/iam.go` (`IAM_Request`, the IAM action table, `getUserForCaller`, the PassRole check) | `spinifex/domains/iam/awsapi` (`NewRegistration`, `Deps`, `ActionNames`), composed by `spinifex/runtime/roles/awsgw` | IAM registers through the ADR-0002 seam with the XML error envelope it had. The check order is kept: query parse, action, service and account checks, then one parse of the input, the `GetUser` default for a missing `UserName`, scope resolution, authorization, the `iam:PassRole` check for `AddRoleToInstanceProfile`, then the handler. The caller is read from `dispatch.Actor` (`PrincipalType`, `UserName`), with no ARN parsing. The dependencies are `Deps.IAM` and `Deps.CountInstanceProfileAssociations`, composed in `awsgw`, and the gateway keeps no IAM fallback. The `cmd/aws-model-coverage` named exception gains an `iam` row. |
+| `spinifex#1346` | `spinifex/gateway/sts.go` (`STS_Request`, the STS action table, the anonymous STS dispatch) | `spinifex/domains/sts/awsapi` (`NewRegistration`, `Deps`, `ActionNames`, `NewAnonymousAssumeRoleWithWebIdentity`), composed by `spinifex/runtime/roles/awsgw` | STS registers through the ADR-0002 seam with the XML error envelope it had. The check order is kept: query parse, action, service check, caller, the `AssumeRole` identity gate scoped to the stored role ARN, then the handler. The caller is read from `dispatch.Actor` (`CallerARN`, `UserName`, `SessionName`, `UserID`, `AccessKeyID`), with no ARN parsing. The gateway's authorization failure wraps a generic `dispatch.PolicyDenialError`, and STS matches it to answer with the caller, action and resource, so the gateway holds no STS-specific denial code. An unsigned `AssumeRoleWithWebIdentity` stays ahead of SigV4 in the gateway and reaches the handler through `GatewayConfig.AnonymousSTS`, a func composed in `awsgw`, never the STS service. The `cmd/aws-model-coverage` named exception gains an `sts` row. |
+| `spinifex#1348` | `spinifex/gateway/ecs.go` (`ECS_Request`) and `spinifex/gateway/eks.go` (`EKS_Request`, the EKS route table) | `spinifex/domains/ecs/awsapi` and `spinifex/domains/eks/awsapi` (`NewRegistration`, `Deps`, `OperationInventory`), composed by `spinifex/runtime/roles/awsgw` | ECS (AWS JSON 1.1) and EKS (REST JSON) register through the ADR-0002 seam with the JSON error envelopes they had, both declared as AWS JSON 1.1 (see the envelope divergence below). The check order and every response are kept, pinned by the pipeline characterisation tests committed first. EKS and ECS resolve roles through `foundation/aws/passrole` `Check` with `domains/iam` `RoleResolver`: a malformed or foreign role is refused first, a missing role is authorized as written before it is reported, and a resolver outage is returned unchanged. Role handling now follows the AWS capture of 2026-10-30. ECS RegisterTaskDefinition refuses a foreign role with `ClientException` "Role is not valid" before PassRole and accepts a nonexistent, invented-path or non-ARN role from a caller who can pass roles. ECS RunTask refuses a foreign `taskRoleArn` or `executionRoleArn` override the same way and answers a nonexistent or invented-path one with "ECS was unable to assume the role". EKS CreateCluster and CreateNodegroup answer a non-ARN or foreign role with `AccessDeniedException` "Cross-account pass role is not allowed.". EKS public routes now report every parameter fault as `InvalidParameterException`, the code the EKS model declares, while the internal control-plane routes keep `InvalidParameterValue` because the guest agents match on it. Two deliberate EKS divergences remain. First, CreateCluster still refuses a nonexistent or invented-path role with `InvalidParameterException` "role ... does not exist", where AWS accepts it and the cluster later ends FAILED. Our EKS only stores, validates and echoes the cluster role and never assumes it, so accepting would leave an ACTIVE cluster bound to a role that does not exist. Second, CreateNodegroup keeps its existing refusal of a nonexistent role because AWS was not captured for it. A third divergence is the error envelope. EKS is restJson1 but registers `ErrorEnvelopeJSON`, so errors carry `application/x-amz-json-1.1` and a `__type` body where AWS sends `application/json` and a `message` body. The `X-Amzn-Errortype` header is correct and SDKs read the code from it, so impact is low. Switching to `ErrorEnvelopeRESTJSON` also changes the throttle code and the default internal-error code (`ServiceException` where the EKS model declares `ServerException`), so it is left for its own change. Verified live on env14: the ECS and EKS role cases above, and a valid CreateCluster reaching ACTIVE with the control-plane agent publishing through the gateway. The EKS route table and its tests move with the registration, and the gateway keeps no ECS or EKS fallback. The `cmd/aws-model-coverage` named exception gains an `ecs` and an `eks` row, and the two `gateway` allowlist rows are removed. |
+| `spinifex#1350` | `spinifex/gateway` (`ecr.go`, `ecrauth_middleware.go`, the `/v2/token` exchange, `ecr_principal.go` with `ResolveECRPrincipal`, the per-operation `/v2` authorization, the registry host matcher, `ctxECRPrincipal`) | `spinifex/domains/ecr/registry` (`Bridge`, `NewBridge`, `BridgeConfig`, and its own `Principal`, `Rehydrator`, `Authorizer`, `Admission` types with `ErrInvalidCredential` and `ErrUnavailable`), `spinifex/ingress/http` (`HostMatcher`, `TargetAccount`, `RequestClientIP`; names no service), `spinifex/ingress/aws/credential` (`Rehydrator`, `Principal.CallerARN`), and under `spinifex/runtime/roles/awsgw`: `credverify` (`Rehydrate`, sharing the SigV4 key and session lookups), `policyauth`, `accountadmission`, `registryadapt` | ADR-0001 amendment 2026-10-29 and ADR-0005:S2: the OCI `/v2` bridge and its token auth move out of the gateway, and ECR resolves its principal through the credential capability instead of reading `IAMService` and `STSService` directly. ADR-0002:S5 keeps `/v2` outside the seam, so the gateway keeps only a `RegistryHandler` mount that answers 501 when unset. `domains/ecr/registry` imports no ingress package (`TestADR0001_Layering` unchanged): the principal, rehydrator and authorizer are registry-local types and `registryadapt` maps `credential.Failure` codes onto `ErrInvalidCredential` (401) and `ErrUnavailable` (503). No widening of `dispatch.Actor` or `dispatch.Invocation`. The SigV4 path and the bridge share one policy evaluator (`policyauth`) and one account-admission check. Deliberate change: the open-registry mode for a nil token verifier is deleted and `NewBridge` refuses a nil Verifier, Issuer, Rehydrator, Authorizer or Admission, so tests that want an unauthenticated registry mount `registry.Handler` directly. The pre-move pipeline characterisation (status codes, error bodies, the `WWW-Authenticate` challenge, referrers, token exchange, revoked and deleted principals) passes unchanged apart from setup, and the retired `ResolveECRPrincipal` parity rows live on as `credverify` `Rehydrate` rows and `Bridge` claim-check rows. Live on env14 (DHCP external IP): crane, skopeo and docker push and pull (`tests/e2e/ecr`), a cosign sign and verify in the legacy tag mode and in OCI 1.1 referrers mode with the signature returned by `GET /v2/{repo}/referrers/{digest}`, `/v2/token` refresh from fresh logins, and a deactivated access key answering 401 on `/v2/token`, manifest GET, crane pull and crane push, then 200 again once reactivated. The ECR signing-key deadlock bead (`mulga-6wqce`) is untouched. |
+| `spinifex#1351` | `spinifex/gateway/{bedrock,bedrockruntime,bedrockagent,bedrockagentruntime}.go` (`Bedrock_Request`, `BedrockRuntime_Request`, `BedrockAgent_Request`, `BedrockAgentRuntime_Request`, the four route tables), `spinifex/domains/ochre/authz.go`, and the 13 `Bedrock*` fields of `GatewayConfig` | `spinifex/domains/ochre/awsapi` (`NewRegistration`, `Deps`, `Quota`, `OperationInventory`, `ServiceName`), composed by `spinifex/runtime/roles/awsgw` | The four Bedrock services register through the ADR-0002 seam as one registration, per ADR-0002:S1 (one per signing service) and the ADR-0001 amendment of 2026-10-06: the service is `bedrock` and `bedrock-runtime`, `bedrock-agent` and `bedrock-agent-runtime` are aliases. A request signed with a sub-service scope is dispatched to that sub-service whatever its path, so a path owned by another sub-service answers `InvalidActionException` as before; only the shared `bedrock` scope is split by path. The pipeline characterisation tests committed first (status, error type and envelope per service, metering order, nil-capability answers) pass unchanged apart from setup. The 13 gateway fields become `awsapi.Deps`: `Credentials`, `Endpoints`, `Access`, `Recorder`, `Logging`, `Provisioned`, `Guardrails`, `Embedder`, `KB`, `DataSources`, `Vector` and `Quota`, with the documented nil fallbacks (no grants denies every model, a nil store errors rather than panics); `BedrockAccessAdmin` had no reader outside one test and is gone, and `Deps.NATS` is dropped because the nil-NATS answer is given by the gateway-wide gate before dispatch. `Logging`, `Provisioned`, `Guardrails`, `KB` and `DataSources` are the concrete exported store types, a temporary edge until Ochre exposes capability interfaces for them. The wire-typed action functions stay in the `domains/ochre` root and the SDK-typed adapters move; `authz.go` moves with them, which removes the `domains/ochre` to `ingress/aws/bodyscope` allowlist row, the only allowlist change. The legacy switch cases, the `supportedServices` and `jsonErrorService` entries and the path-based sub-service rewrite in `GetService` are removed, so `ec2` is the only service left on the legacy table; the rewrite's rows are pinned in `awsapi` as `resolveSubservice` and dispatch tests. The registration declares the AWS JSON 1.1 envelope it had and `cmd/aws-model-coverage` is not wired, so no coverage allowlist row is added. Recorded divergences, not changed by this move: Bedrock is restJson1 in AWS but answers with the JSON 1.1 envelope (`application/x-amz-json-1.1`, `__type` body), and CreateGuardrail answers 200 where AWS answers 202; both are tracked by `mulga-xug8b`. No change to `dispatch.Actor` or `dispatch.Invocation`. |
+| `spinifex#1354` | `spinifex/gateway/ec2.go` (the EC2 action table, `ec2Handler`, `ec2Actions`, the dispatch pipeline), `ec2_idempotency.go`, `ec2_validation.go`, and the EC2 token-bucket fields of `GatewayConfig` | `spinifex/domains/ec2/awsapi` (`Handler`, `NewHandler`, `Deps`, `Quota`, `Dispatch`, `ActionNames`, `InputHasField`), composed by `spinifex/gateway/ec2.go` | Slice 9a of the EC2 move: the capabilities first, with no behaviour change. The EC2 action table, idempotency and validation move to `domains/ec2/awsapi` behind `Deps{NATS, IAM, Quota, ActiveNodes, ExpectedNodes, NodeIDs, AZ, Endpoint, InstanceStatus}`; the gateway keeps the composition and `EC2_Request`, still on the legacy switch until slice 9b registers EC2 through the ADR-0002 seam. `Quota` is declared in `awsapi` because `domains/admission/quota` imports the EC2 action packages and cannot be imported back, and the gateway wraps `quota.Service` to supply the NATS instance-type resolver its retype gate takes. The instance and spot-instance adapters take the two-method `InstanceProfiles` interface (`ResolveInstanceProfile`, `CanonicalResourceARN`) instead of `iamdomain.IAMService`, which pays off the `spotinstance` to `domains/admission/quota` and `domains/iam` allowlist rows. Allowlist changes are 20 deletions and no additions: the 18 `gateway` to `domains/ec2/awsapi/*` rows and those 2. The pipeline characterisation tests committed first (error envelopes, check order, gateway-local actions, idempotent replay, PassRole, ImportKeyPair padding, launch-template expansion before authorization, RunInstances instance-profile refusal order) pass unchanged. Package-local tests drive every routed action through `Dispatch` against a stub daemon. No change to `dispatch.Actor` or `dispatch.Invocation`. Live on env14: DescribeRegions, DescribeAvailabilityZones and DescribeAccountAttributes, an idempotent CreateVolume replay and its parameter mismatch, the DryRun and MaxResults errors, a no-policy user refused with UnauthorizedOperation (including on the gateway-local DescribeRegions), an assumed-role session allowed and refused per its policy, and one RunInstances to running then terminated. |
+| `spinifex#1355` | `spinifex/gateway/ec2.go` (`EC2_Request` in production, the legacy service switch, its `supportedServices` and JSON tables) and the `Quota` field of `GatewayConfig` | `spinifex/domains/ec2/awsapi` (`NewRegistration`, `OperationInventory`, `ServiceName`), composed by `spinifex/runtime/roles/awsgw` (`ec2.go`: `newEC2Registration`, `advertisedEndpoint`; `ec2quota`) | Slice 9b of the EC2 move: EC2 registers through the ADR-0002 seam as service `ec2` and the legacy gateway switch is empty. The EC2 query-protocol error envelope is kept through `ErrorEnvelopeXML`, and the policy denial match is now `*dispatch.PolicyDenialError`. The characterisation tests committed first also pin the Authorize service name and action, the trace span and audit action, the throttle key (account plus action) and the cluster-unavailable gate, which runs before policy and action lookup for gateway-local and NATS-backed actions alike. They and the 9a tests pass unchanged apart from setup: STS and EC2 test setups now register the services they sign as, and the two tests that claimed a legacy service name are removed because no legacy name remains. `EC2_Request` survives only as a test shim. Allowlist changes are two deletions (the `gateway` to `domains/ec2/awsapi` and `domains/ec2/awsapi/instance` rows) and one addition, the per-service model-coverage row `cmd/aws-model-coverage` to `domains/ec2/awsapi`. `startInstanceStatus` stays in `awsgw.go` because ADR-0007:S6 records the raw instance-state uses by file. No change to `dispatch.Actor` or `dispatch.Invocation`. Live on env14: DescribeRegions, DescribeAvailabilityZones and DescribeAccountAttributes, an idempotent CreateVolume replay and its parameter mismatch, the DryRun and InvalidVolume.NotFound errors, a no-policy user refused with UnauthorizedOperation (including on the gateway-local DescribeRegions), an assumed-role session allowed and refused per its policy, and one RunInstances to running then terminated. The gateway request logs carry `service=ec2` and the per-action label. Not exercised live: a throttled request (the default limits are not reachable from one client; the throttle key is pinned by the characterisation tests) and an agent-signed EC2 call. |
+| `spinifex#1363` | `spinifex/gateway` context keys (`ctxService`, `ctxRegion`, `ctxAction`, the principal and access key id keys, and the unread `ctxQueryArgs`), the duplicate `callerPrincipal` and `principalContext` types, `gateway/caller_arn.go` (`buildCallerARN`), and the direct `policyauth`, `accountadmission` and NATS-connection reads in the auth pipeline | `spinifex/ingress/aws/reqctx` (the request record), `credverify.CallerARN`, and the gateway-consumed capabilities `PolicyEvaluator`, `AccountAdmission` and `Readiness`, composed by `spinifex/runtime/roles/awsgw` (`clusterready`) | Slice 11a of the gateway removal: the request-context record and the ADR-0002:S8 capabilities, with no behaviour change. The record is a copy-on-write value under one context key; the principal is sealed once and a second `WithPrincipal` keeps the first caller. The layering rule `S7_RequestRecordPrivateToIngress` lets only `gateway` and `ingress/aws/*` import `reqctx`, with a fault-injection test, and slice 11c moves the package to `ingress/aws/internal/reqctx` and drops the rule. The caller ARN is built once in `credverify` (root cases kept) and filled on `Principal.CallerARN`. The policy evaluator, account admission (`credential.ErrAccountNotActive`, with `accountadmission.ErrNotActive` an alias) and cluster readiness (`Available() bool`) are interfaces the gateway consumes and fails closed on when nil. The step 1 characterisation tests (throttle against both readiness gates and the policy gate for ECS and IAM, `/admin` skipping `GetService` and the second gate, the nil-connection difference between the two gates, five error writers each minting a request id that differs from the audit record, and the credential-scope region reaching only the span) pass unchanged apart from setup. No allowlist rows change, and there is no change to `dispatch.Actor` or `dispatch.Invocation`. Debt for slice 11c: the test-only `defaultCapabilities` seam (a package variable set by a test `init` so about 60 gateway test files can build a config from `IAMService` alone), the nil-connection case of the early readiness gate, the remaining `NATSConn` readers (`DiscoverActiveNodes` and the field that `awsgw` and the integration harness set), and the `IAMService`, `STSService`, `Config`, `RootCAs` and `InternalSuffix` fields of `GatewayConfig`, which no production code reads. Live on env14: DescribeRegions, DescribeAvailabilityZones and DescribeAccountAttributes, an idempotent CreateVolume replay and its parameter mismatch, the DryRun and InvalidVolume.NotFound errors, a bad access key refused with InvalidClientTokenId on a Query service and InvalidClientTokenIdException on a JSON service, a no-policy user refused with UnauthorizedOperation, and an assumed-role session allowed and refused per its policy. Not exercised live: a suspended-account refusal, a throttled request and the NATS-down 503, which the step 1 tests pin. |
+| `spinifex#1378` | `spinifex/gateway` `oidc_discovery.go` and `registry.go` (the `/oidc/eks` routes and the `/v2` mount), the `DiscoverActiveNodes` method and its cache, and the `GatewayConfig` fields `Config`, `RootCAs`, `InternalSuffix`, `Version`, `Commit`, `IAMService`, `STSService`, `NATSConn` and `ExpectedNodes` | `spinifex/runtime/roles/awsgw` (`PublicRoutes`, `NodeDiscovery.ActiveNodes`, `composeGateway`); the gateway keeps one `PublicRoutes` mount hook and the `Admin` registration name supplied by the role |
+| `spinifex#1384` | `spinifex/gateway` (`GatewayConfig`, the SigV4 and policy middleware, `Request`, the admin dispatch, `AWSOperationInventory`) | `spinifex/ingress/aws/pipeline` (`Config`) and `spinifex/ingress/aws/dispatch` (`AWSOperationInventory`); the gateway layering rows are retired and R2 binds the pipeline |
+| `spinifex#1334` | `spinifex/gateway/oidc_discovery.go` (`eksAccountKV`, the discovery-document and JWKS read and construction) | `spinifex/domains/eks/cluster` (`OIDCPublisher`, `OIDCDiscoveryDocument`), composed by `spinifex/runtime/roles/awsgw` | The EKS domain now reads its own account bucket and builds the per-cluster discovery document and JWKS responses (ADR-0005:S4), and the gateway only maps the unauthenticated `/oidc/eks/...` routes onto an `OIDCPublisher` interface it is handed. Responses are byte-identical, pinned by `oidc_discovery_current_behaviour_test.go`, whose assertions are unchanged. The gateway no longer imports `domains/eks/cluster` in production code. `STS` keeps its own `ClusterJWKSResolver` (`OIDCJWKSReader`), untouched. Recorded debt, not repaired here: a stored JWKS is served verbatim with 200 even when it is not valid JSON. |
+| `spinifex#1245`, `spinifex#1390` | — | `contracts/admission/v1` | The quota override seal route (`admission.v1.SealAccountQuotaOverride`, served by `awsgw`) and the per-gateway read-only capability probe (`admission.v1.AccountQuotaOverrideSealCapability.<node>`) are an explicit versioned cross-process contract between the quota owner and `operator/accountlifecycle`. The consumer wiring is new: `NewClusterEngine` takes a `ClusterNodes` value with separate daemon and gateway node sets, `ClusterConfig.NodesRunning` is the single definition of which nodes run a service, and the seal reaper runs in `StageAccount` after the EC2 settings seal. A termination whose gateway probe is incomplete seals nothing and leaves the account `TERMINATING`. Contract, responder, probe and reaper tests pin the subjects, JSON shapes and fail-closed cases; `TestAccountTermination_QuotaOverrideEndsSealed` replaces the former characterization that overrides survive teardown. |
 The EC2 contract row is intentionally different from the directory moves: it creates
 a compatibility boundary. `ec2.cmd.*` retains its deployed one-token NATS
 subscription and permission shape, while callers use
@@ -387,9 +406,9 @@ the canonical IAM/STS principal before dispatch, because ECR must not depend on
 gateway identity context or ARN construction.
 
 All implemented ECR control-plane actions now dispatch through explicit ECR
-capabilities. The remaining ECR-named gateway code is generic `/v2/*` route
-assembly, SigV4/registry authentication and policy enforcement; it remains in
-gateway rather than inverting those dependencies into the ECR domain.
+capabilities. The OCI `/v2` bridge, its token exchange and its principal
+resolution now live in `domains/ecr/registry`; the gateway keeps only the
+`RegistryHandler` mount, and `awsgw` composes the bridge.
 
 ### ACM relocation residuals
 
@@ -438,7 +457,7 @@ These residuals were recorded by individual slices and must be resolved once the
 - `runtime/roles/vpcd` (placement): the role still mixes process/role supervision with network-controller responsibilities. `lease_reaper.go` (orphaned external-pool DHCP lease sweep), `ovn_controller_watchdog.go` (OVN controller liveness and restart), `host_eip_loop.go` (host-EIP reconcile loop) and `lease_datapath_reconcile.go` (lease datapath reconcile) are periodic network-control loops, not role wiring; `vpcd.go` itself also carries bridge-mode detection and verification, chassis-list parsing, external-CIDR resolution, DHCP manager startup and lease/IP-change handling, and gateway/host-EIP allocator and neighbour flush/prime construction alongside `Service.Start`. The future placement question is whether these belong in `domains/network`. Removal condition: an accepted decision on network-controller placement, and extraction of those loops and helpers behind the role.
 - ECR control-plane registration: `awsapi.Deps.NATS` is a raw `*nats.Conn`, not a narrow capability. Every NATS-backed `Handler` passes it to `domains/ecr.NewNATSMetaStore`, which calls `natsmsg.NATSRequest`, so narrowing it means changing the exported `Handler` type, every handler and the meta store; it must not be broadened further meanwhile.
 - ACM control-plane registration: `domains/acm/awsapi.Deps.NATS` (`registration.go`) is likewise a raw `*nats.Conn` carried by every `Handler`, for the same reason; it must not be broadened further meanwhile.
-- `cmd/aws-model-coverage` imports `ingress/aws/dispatch`, `domains/ecr/awsapi`, `domains/acm/awsapi` and `gateway` to read ECR's and ACM's declared inventories. It is a non-serving tool and the only named exception to the ADR-0002 composition invariant; it must not construct a registration.
+- `cmd/aws-model-coverage` imports `ingress/aws/dispatch`, `gateway` and the `awsapi` package of each registered service (`domains/ecr/awsapi`, `domains/acm/awsapi`, `domains/route53/awsapi`, `domains/elbv2/awsapi`, `domains/rds/awsapi`, `domains/ecs/awsapi`, `domains/eks/awsapi`) to read their declared inventories. It is a non-serving tool and the only named exception to the ADR-0002 composition invariant; it must not construct a registration.
 - Ingress pipeline (ADR-0002 phase 4): the request-audit record, the trace action enricher, the throttle key functions and the gateway request-context keys stay in `gateway` until the auth/IAM and request-context boundary is designed. SigV4 authentication writes the audit record and context keys from four places, so moving them now would export exactly the setters ADR-0002 says must not be exported solely to make a directory move possible.
 - NAT projection delivery is inconsistent and is preserved as found: EIP `AssociateAddress` sends `vpc.add-nat` unacknowledged (`AnnounceNATAdd`), skipping the 45s OVN barrier every other add path waits on, and `DeleteNetworkInterface` sends `vpc.delete-nat` unacknowledged (`AnnounceNATDelete`), skipping the ack and the ErrNoResponders retry. The reconciler's orphan sweep is the only backstop for both. Clears when those call sites use `AddNAT`/`AddNATBestEffort` and `RemoveNAT`, a reliability change of its own.
 - `vpc.add-system-egress` has a `vpcd` subscriber and a contract entry but no producer (its former `domains/ec2/eip` publisher is gone); `domains/eks/cluster` still publishes the delete. Either a producer is missing or the add route is dead; it needs an owner decision.
@@ -446,7 +465,7 @@ These residuals were recorded by individual slices and must be resolved once the
 - `foundation/messaging/nats/nats_trace.go` still names its OpenTelemetry tracer `github.com/mulgadc/spinifex/spinifex/utils`. It is an emitted instrumentation-scope name that dashboards may filter on, so renaming it is a telemetry change, not a path fix.
 - `runtime/compute/cache.New`: two construction-time panics carry a lint exemption pending a constructor that returns an error.
 - `domains/ec2/systeminstance`: managed-service parent metadata is only the `ManagedBy` tag value.
-- `domains/ochre` (action adapters): since `spinifex#1266` the Bedrock action layer (handlers, action authorization, event-stream framing and the CRUD, catalogue and Converse/Invoke entry points) shares package `ochre` with the domain core, stores and inference engine, and imports `ingress/aws/bodyscope`. Moving it to `domains/ochre/awsapi` is not mechanical: the adapters reach package-private store internals (the guardrail and provisioned-throughput store handles and get/put/update methods), and adapter tests override the package-private model catalogue. It needs core capabilities first, then the move; the top-level `gateway/bedrock*.go` HTTP handlers additionally wait for the ingress registration seam.
+- `domains/ochre` (action adapters): cleared by the Bedrock registration in `spinifex#1351`. The SDK-typed adapters and `authz.go` moved to `domains/ochre/awsapi`, the `ingress/aws/bodyscope` allowlist row is deleted, and the adapters take the concrete exported Ochre stores through `awsapi.Deps` as a temporary edge until Ochre exposes capability interfaces for them.
 - `domains/network`: `reconcile/intent.go` builds desired network state by reading EC2's KV buckets and decoding EC2's private VPC, subnet, ENI, route-table, gateway, EIP, NAT and security-group records, and `reconcile/drift.go` watches those buckets. ADR-0001 names this pair explicitly: EC2 and network communicate through a capability/projection or versioned contract, never by one being the other's KV reader. Step 7 fix: an EC2-owned `domains/ec2/projection` exposing network intent with list and watch (it may stay KV-backed, so no data moves); a narrow intent interface declared by `reconcile`; wiring at the composition root; and the duplicated `PurposeEIP`/`PurposeIGWLRP` constants and mirrored `SGRule`/`SGEvent` folded into the projection or `contracts/ec2/v1`. The drift-watch coverage test and reconcile scenario tests are the evidence the projection must keep. `domains/network` also imports `runtime/compute/vm` and `runtime/host/command` (domain-to-runtime).
 - `domains/ec2/instance`: imports `runtime/compute/vm` and `runtime/compute/gpu` (domain-to-runtime) and `domains/dns` (cross-domain implementation import).
 - `domains/ec2/instance/state` (dependency): the vpcd IMDS local reader imports `runtime/compute/vm` to return `vm.VM`. Removal condition: that type moves into a domain-owned package that `runtime/compute/vm` consumes (runtime to domain is allowed).
@@ -456,14 +475,12 @@ These residuals were recorded by individual slices and must be resolved once the
 - `operator/installer/firstboot` (security): `Config.InstallCallback` is rendered into `spinifex-firstboot.sh`, which is world-readable (0755). The field is currently unused (`fireInstallCallback` reads `SPINIFEX_INSTALL_CALLBACK` from the environment instead), but any future wiring must not place secrets or tokenised callback URLs in the generated script; sensitive callback credentials should reach first boot through a protected channel such as a 0600 file or a credential store.
 - `operator/cli` (known security risk): `spx admin join` contacts the formation server with `InsecureSkipVerify`, sends the join token as a bearer credential and receives the cluster CA certificate and key, master key, Viperblock key and shared credentials, so an on-path attacker during formation can capture the token and the cluster root secrets or substitute a CA. This predates the move; lint surfaced it when the tree left `cmd/`. It is accepted for now and carries a line-level gosec annotation stating it is not a false positive. Removal condition: authenticate formation TLS with a pinned key or certificate delivered with the join material, or an out-of-band CA, with tests proving a mismatched server certificate fails before credentials are sent.
 - `domains/eks/cluster` (potential security risk): `ClusterMeta.ControlPlaneTemplate` (`domains/eks/cluster/service_impl.go`) clears the template's credentials but keeps `OIDCPrivateKeyPEM` and the k3s `JoinToken`, so both are persisted in the `eks-account-{acct}` cluster record, which the gateway also reads. Whether that record is encrypted at rest is unconfirmed; if it is not, a reader of the bucket can forge service-account tokens for the cluster's OIDC issuer and join nodes to it. This predates the refactor and is not fixed in it. Removal condition: the key and token are held by a dedicated secret store (a future KMS or Secrets Manager capability is the likely owner) and the cluster record keeps only a reference, with a test proving the persisted record contains neither secret.
-- `operator/cli` (dependency): the command tree still imports domain and service implementations directly (`handlers/{iam,eks}`, `domains/admission/quota`, `gateway`, `domains/ochre`, `vpcd`, the Viperblock adapter and runtime roles). The ADR's `cmd/spinifex` current-to-target map row requires command packages to reach domains through public capabilities; splitting supported use cases into `operator/*` packages is later work.
+- `operator/cli` (dependency): the command tree still imports domain and service implementations directly (`handlers/{iam,eks}`, `domains/admission/quota`, `domains/ochre`, `vpcd`, the Viperblock adapter and runtime roles). The ADR's `cmd/spinifex` current-to-target map row requires command packages to reach domains through public capabilities; splitting supported use cases into `operator/*` packages is later work.
 - `providers/ebs/viperblock` → `runtime/compute/nbd` and `runtime/host/process` (dependency, moved unchanged from `services/viperblockd`): the adapter formats NBD endpoints and manages nbdkit processes through runtime packages. ADR-0001 does not list providers → runtime as forbidden and the layering test does not enforce it (same category as `providers/ebs/conformance` → `runtime/compute/nbd`), but a provider should not depend on node runtime. Clears when NBD endpoint naming has a provider-facing home (a candidate for `contracts/ebs/v1`) and process supervision is injected by the role. The adapter also imports `providers/objectstore`, a provider-to-provider edge that is allowed.
-- `gateway/ec2.go` (registration): the EC2 dispatch map stays a `GatewayConfig` method in `gateway` and imports every `domains/ec2/awsapi/*` package; `gateway/ec2_idempotency.go` also imports `domains/ec2/awsapi/idem` (`KVBucket`, `TTL`, `TokenAndParams`) for the dispatch-time idempotency wrapper. ADR-0001 makes the domain own its action registration; moving the map waits on the generic ingress registration seam and must not be done as a package move.
-- `gateway` (service registration): the root `gateway` dispatch for IAM, STS, EKS, RDS, ECS and ELBv2 imports `domains/{iam,sts,eks,rds,ecs,elbv2}/awsapi`, which only AWSGW may wire. It clears when each service registers through the ADR-0002 ingress registration seam; as with EC2, moving the dispatch must not be done as a package move.
+- `gateway` (service registration): the root `gateway` dispatch for IAM, STS, EKS, RDS and ECS imports `domains/{iam,sts,eks,rds,ecs}/awsapi`, which only AWSGW may wire. It clears when each service registers through the ADR-0002 ingress registration seam; as with EC2, moving the dispatch must not be done as a package move.
 - `operator/accountlifecycle` (dependency): its EC2 reapers call `domains/ec2/awsapi/*` action adapters directly, as they did under `gateway/ec2`. The amended ADR-0001 `accountteardown` row keeps replacing these with domain-owned cleanup capabilities as deferred debt; it was not a relocation prerequisite. It also holds a read-only consumer of the EC2 platform-inventory contract (`contracts/ec2/v1`, `ec2.v1.PlatformInventory`), served by `domains/ec2/accountinventory`; only the explicit dry run (`Engine.DryRun`) calls it, to log a shadow comparison, and the key-pair, placement-group and launch-template reapers still use the `awsapi` adapters. A domain-owned cleanup capability for the proven-safe subset of those three kinds (key pairs and empty placement groups no instance, spot request or live launch-template version names, and launch-template residue; never live launch templates) exists as `domains/ec2/accountcleanup` (a Go interface with injected owners and reference readers, no NATS subject); nothing calls it yet. It also holds the StageAccount settings-seal reaper, a consumer of `contracts/ec2/v1`'s `ec2.v1.SealAccountSettings` and per-node `ec2.v1.AccountSettingsSealCapability.<node>` subjects, served by `domains/ec2/account` through the daemon; the reaper reads no KV and imports no domain package, reaching both subjects only through NATS and the configured node set `NewClusterEngine` already receives.
 - `domains/admission/quota` (dependency, moved from `handlers/quota` unchanged): `live.go` imports `domains/elbv2/awsapi` and `domains/rds/instance` for its live ELBv2 and RDS counters; `records.go` imports `runtime/compute/vm` for the instance-record vCPU sweep; `live.go`, `vcpu.go` and `storage.go` call `domains/ec2/awsapi/{vpc,eip,volume}` describe adapters to count VPCs, subnets, addresses and volumes, as they did under `gateway/ec2`. The ADR's `handlers/quota` current-to-target map row replaces these with reader capabilities once their final locations exist.
-- `domains/ec2/awsapi/instance` (dependency): imports `domains/iam` (`IAMService` in `DescribeInstances.go`, `RunInstances.go` and `IamInstanceProfileAssociation.go`; `InstanceProfile` in the latter two), `runtime/compute/cache` (`NodeState`, `NodeUnknown`, `NodeLive`) and `runtime/compute/vm` (`VM`), both in `status_synthesis.go`. The IAM edge needs an IAM capability; the runtime edges are domain-to-runtime and need the instance record and node liveness exposed through an EC2-owned projection.
-- `domains/ec2/awsapi/spotinstance` (dependency): imports `domains/iam.IAMService` and `domains/admission/quota.Service` in `RequestSpotInstances.go`.
+- `domains/ec2/awsapi/instance` (dependency): imports `domains/iam` for the `InstanceProfile` type that `InstanceProfiles.ResolveInstanceProfile` returns (`RunInstances.go`, `IamInstanceProfileAssociation.go`), `runtime/compute/cache` (`NodeState`, `NodeUnknown`, `NodeLive`) and `runtime/compute/vm` (`VM`), both in `status_synthesis.go`. The IAM edge is now the two-method `InstanceProfiles` interface plus that type; the runtime edges are domain-to-runtime and need the instance record and node liveness exposed through an EC2-owned projection.
 - Instance adapter consumers outside EC2: `domains/rds/awsapi/enginecatalog.go`, `runtime/daemon/eks_cp_control.go`, `runtime/daemon/rds_deps.go`, `domains/admission/quota`'s `vcpu.go` and `gateway/iam.go` (`CountInstanceProfileAssociations`) call `domains/ec2/awsapi/instance` directly; each needs an EC2 capability rather than the AWS action adapter.
 - `runtime/daemon` (Q-28 ownership and deployment): the daemon moved whole, keeping its current ownership, deployment and `daemon → gateway → handlers` dependencies. Q-28's write-side consolidation and any daemon decomposition or split remain deferred by ADR-0001 and are not paid by the move.
 
@@ -472,8 +489,8 @@ Enforcement: the owner decided the layering test is blocking now, with a ratchet
 - `domains/rds/instance` → `domains/rds/parametergroup` (legacy edges): `parameterGroupReferences` binds `Dependants` by scanning instance records; this clears when the RDS instance gains its own owner and answers `Dependants` from it. `tags.go` writes the meta record through its generic tag registry; this clears when tagging invokes a parameter-group-owned tag mutation capability, and moving the tagging adapter to `domains/rds/awsapi` alone does not clear it. The Modify handler still calls `propagateParameterGroup` after the owner's write; propagation is instance-side realization and stays legacy until it is designed as asynchronous realization (group generation, instance staleness, retry). `parameterGroupValues` returns the persisted `Record` and overrides to instance code; when the instance owner is extracted, replace them with a consumer-owned resolved-parameters projection.
 - `domains/ecs` → `domains/ecs/taskdefinition` (owner bypass): `tags.go` reads and writes the revision record through its generic tag registry; this clears when ECS tagging invokes an owner-provided tag mutation capability, and moving `tags.go` to `domains/ecs/awsapi` alone does not clear it. `taskDefinitionResolver` returns the persisted `Record` to task and service launch code; when the task and service owners are extracted, replace the record crossing that boundary with a consumer-owned resolved task-definition (launch-spec) projection. Moving packages has not made the record private.
 - `domains/ecs/taskdefinition` (deliberate duplication, not debt): private `getJSON`, `putJSON` and `keysWithPrefix` copy the raw-KV helpers in `domains/ecs` so the extraction keeps ECS's get/put/list error and recovery semantics. Revisit only if a later, tested state abstraction proves it preserves those semantics; do not migrate ECS to `kvstore` merely to remove them.
-- `domains/eks/cluster` → `domains/eks/access` (legacy edges): `ResolveTokenReview` opens the EKS bucket in the gateway process and reads records through the exported `access.Get`; this clears when token review asks the owner across a contract rather than opening the bucket itself. `DeleteClusterPrefix` erases access-entry keys as part of its blanket cluster sweep; this clears when cluster delete invokes an owner-provided removal of the cluster's entries. The owner returns the persisted `Record` to the AWS adapter and token review; replace it with consumer-owned projections when those consumers are extracted.
-- `domains/eks/cluster` → `domains/eks/addon` (legacy edges): `DeleteClusterPrefix` erases add-on records and manifests in its blanket cluster sweep; this clears when cluster delete invokes an add-on-owned teardown capability. `stageGPUDeviceAddon` is a temporary node-group-to-add-on orchestration edge: best-effort, errors logged and swallowed, never undone on node-group delete; it clears when the node-group owner is extracted and the dependency is designed. `ClusterReconciler` hosts the add-on status subscription; it clears when add-on delivery observation has its own host under the lifecycle correction. Cluster existence is checked in the handler, not through an owner-declared capability; it moves into the owner when cluster admission is defined (today status is ignored). Guest wire types (`AddonStatusReport`, `AddonDeliveryPhase`, `AddonStatusSubject`, `StagedAddonManifest`, `InternalAddonsResponse`) are owned by `contracts/eks/v1`, which `domains/eks/cluster`, `gateway/eks` and `agents/eks/gatewayfetch` import directly.
+- `domains/eks/cluster` → `domains/eks/access` (legacy edges): `ResolveTokenReview` opens the EKS bucket in the gateway process and reads records through the exported `access.Get`; this clears when token review asks the owner across a contract rather than opening the bucket itself. `DeleteClusterChildren` erases access-entry keys as part of its blanket cluster sweep; this clears when cluster delete invokes an owner-provided removal of the cluster's entries. The owner returns the persisted `Record` to the AWS adapter and token review; replace it with consumer-owned projections when those consumers are extracted.
+- `domains/eks/cluster` → `domains/eks/addon` (legacy edges): `DeleteClusterChildren` erases add-on records and manifests in its blanket cluster sweep; this clears when cluster delete invokes an add-on-owned teardown capability. `stageGPUDeviceAddon` is a temporary node-group-to-add-on orchestration edge: best-effort, errors logged and swallowed, never undone on node-group delete; it clears when the node-group owner is extracted and the dependency is designed. `ClusterReconciler` hosts the add-on status subscription; it clears when add-on delivery observation has its own host under the lifecycle correction. Cluster existence is checked in the handler, not through an owner-declared capability; it moves into the owner when cluster admission is defined (today status is ignored). Guest wire types (`AddonStatusReport`, `AddonDeliveryPhase`, `AddonStatusSubject`, `StagedAddonManifest`, `InternalAddonsResponse`) are owned by `contracts/eks/v1`, which `domains/eks/cluster`, `gateway/eks` and `agents/eks/gatewayfetch` import directly.
 - `domains/eks/cluster` and `domains/eks/awsapi` → `domains/eks/addon` (further edges): the AWS adapter reads the persisted `Record` through `addon.Get`/`addon.List`; this clears when the owner returns a view and the add-on adapter moves to `domains/eks/awsapi`. The guest route reaches the owner through the `handlers_eks` NATS RPC (`ListStagedAddonManifests`); this clears with per-resource commands behind EKS registration. The add-on ARN is derived in both the owner and `domains/eks/awsapi/authz.go`; this clears when authorization obtains the ARN from the owner, which an incarnation suffix would require. The owner's `Manifest`/`Report` beside the `contracts/eks/v1` types is deliberate duplication, not debt.
 - `agents/rds/agent` (dependency, narrowed): engine metadata now arrives through the agent-declared `EngineCatalog`, but the agent still imports `domains/rds/instance` for the command protocol, wire and record types (`Command*`, `Parameter`, `EngineHealth*`, `BootstrapMode*`) and API output types. This clears when those move behind a guest/controller wire contract.
 - `domains/rds/awsapi` → `domains/rds/instance` (`InstanceSizing`): the gateway obtains its `Sizing` through the RDS handler package's adapter rather than its own composition root. This clears when the gateway's RDS composition moves to `runtime/roles/awsgw` and builds `Sizing` there.
@@ -672,12 +689,12 @@ series above; "needs decision" is open.
 | `handlers/rds` | — | Done. The remaining RDS instance resource moved to `domains/rds/instance` (package `instance`) in `spinifex#1251` with no API, NATS subject, storage or behaviour change, fulfilling ADR-0001's organise-by-resource condition alongside the already-extracted subnet group, parameter group and engine catalogue. The ADR-0004 blocked-dependency imports (quota admission, the duplicated expected-node fan-out, the IAM ensurer and the Bluebottle master key) remain dependency-rule debt, with their allowlist entries now sourced from `domains/rds/instance`. |
 | `handlers/elbv2` | — | Done. The implementation moved to `domains/elbv2` (package `elbv2`) with no API, NATS subject, storage or behaviour change. Q-30's data-plane boundary remains deferred debt; the amended ADR-0001 row makes it not a relocation prerequisite. Its cross-domain, IAM and `agents/elbv2` imports remain dependency-rule debt, with their allowlist entries now sourced from `domains/elbv2`. |
 | `handlers/iam/mock` | — | Done. Moved with the implementation to `domains/iam/mock`. |
-| `gateway` (root) | Generic HTTP route assembly, SigV4/registry authentication, the policy gate, request-audit middleware, request-context keys, and the legacy per-service dispatch switch for every service not yet registered. | Deferred: the ADR-0002 phases above (auth/IAM and request-context boundary design) gate moving the audit, context-key and legacy-dispatch mechanics. |
-| `gateway/bedrock` | — | Done. The whole package moved into `domains/ochre` in `spinifex#1266` with its package-private core and tests intact and no behaviour change. Separating the action adapters into `domains/ochre/awsapi` remains deferred, recorded as `domains/ochre` (action adapters) debt above. |
+| `gateway` (root) | Generic HTTP route assembly, SigV4 authentication, the policy gate, request-audit middleware and request-context keys. | Done and deleted. The request pipeline moved into `ingress/aws/pipeline` (as `pipeline.Config`) and the operation inventory into `ingress/aws/dispatch` in `spinifex#1384`; the `/oidc`, `/v2` and node-discovery code had already moved to `runtime/roles/awsgw` in `spinifex#1378`. The layering test binds the pipeline with R2 and `gateway` is a retired legacy root. |
+| `gateway/bedrock` | — | Done. The whole package moved into `domains/ochre` in `spinifex#1266` with its package-private core and tests intact and no behaviour change. Separating the action adapters into `domains/ochre/awsapi` remains deferred, recorded as `domains/ochre` (action adapters) debt above. The action adapters then moved to `domains/ochre/awsapi` behind one ADR-0002 registration in `spinifex#1351`. |
 | `gateway/{iam,sts,eks,rds}` | — | Done. The action adapters moved to `domains/{iam,sts,eks,rds}/awsapi` in `spinifex#1265` with no API, NATS subject, storage or behaviour change. ADR-0001 does not make ADR-0002 registration a prerequisite for relocating a domain's adapters, as the EC2 adapter moves already showed; the root `gateway` dispatch that still imports them is recorded `gateway` (service registration) debt. |
 | `gateway/{ecs,elbv2}` | — | Done. The action adapters moved to `domains/{ecs,elbv2}/awsapi` in `spinifex#1273` with no API, NATS subject, storage or behaviour change; the root `gateway` dispatch that still imports them is recorded `gateway` (service registration) debt. |
-| `gateway/spx` | `promote_image.go` and `spx.go` dispatch (`storage.go` already consumes `contracts/operator/v1`). | Needs decision: which package owns operator-plane action dispatch registration — an `ingress/aws`-style seam for the operator plane, or `gateway` stays its home. |
-| `gateway/tagging` | — | Done. `domains/tagging` owns the read-only cross-resource `GetResources` aggregation and its API semantics (`0d830e901`, `aedd0b2c4`); AWSGW composes the owner listers, the EC2 tag index and ELBv2 (`f3545e3ea`); the root `gateway` keeps only the legacy dispatch in `gateway/tagging.go`. The package is removed, and its two allowlist rows and frozen-legacy entry are gone. Owner tag mutation capabilities remain future direction, so the RDS and ECS tag-writing edges above stay open. |
+| `gateway/spx` | — | Done. The gather helpers, storage-status config and image-promote request moved to `operator/admin` with no behaviour change and the package is removed. The `spinifex` Query dispatch and the `/admin/{method}` handlers that call them now live in `operator/admin` as one seam registration. |
+| `gateway/tagging` | — | Done. `domains/tagging` owns the read-only cross-resource `GetResources` aggregation and its API semantics (`0d830e901`, `aedd0b2c4`); AWSGW composes the owner listers, the EC2 tag index and ELBv2 (`f3545e3ea`); the dispatch moved to `domains/tagging/awsapi` and the root `gateway` no longer serves Tagging. The package is removed, and its two allowlist rows and frozen-legacy entry are gone. Owner tag mutation capabilities remain future direction, so the RDS and ECS tag-writing edges above stay open. |
 | `services/viperblockd` | — | Done. The adapter is `providers/ebs/viperblock` (with the read-only `dirty` reader), the shared vocabulary is `contracts/viperblockd/legacy/v1`, and the role is `runtime/roles/viperblockd`. `services/` is empty. |
 | `utils` | — | Done and deleted. `ec2_subject.go` went to `contracts/ec2/v1`, the vpcd protocol to `contracts/network/v1` with the typed client in `domains/network/projection`, and the master-key loader to `domains/ec2/ebs/encryption`. |
 | `vpcd` | — | Done. The IMDS local-file reader and record-space opener moved to `domains/ec2/instance/state` and are injected through `guestmetadata.LocalStateReader`/`RecordLoader` (recorded above), so `vpcd` no longer imports `daemon` directly; its inter-process boundary now exists (`contracts/network/v1`, with `domains/network/subscribers` as the consumer and `domains/network/projection` as the producer client). The package moved unchanged to `runtime/roles/vpcd` (recorded below), and the legacy directory is removed. |
@@ -696,7 +713,7 @@ ADR-0001 completion is structural, so none of these is cleared by remediating se
 | Class | Paths | Status |
 |---|---|---|
 | No missing boundary (candidate moves) | `handlers/ecs/bus` → `contracts/ecs/v1` (`a130f6b8e`); `admin/images_promote.go` → `domains/ec2/image` (`263c58167`); `admin` bootstrap account identifiers → `foundation/aws/identifiers` (`3fdaa54d9`); `admin` credential-format generators → `foundation/aws/credentials` `0368e861a`, retiring `admin` | Done. |
-| One narrow missing boundary | Done: `handlers/sts` EKS JWKS lookup → `handlers/eks` `ClusterJWKSResolver` (`254d898a3`); `gateway/tagging` → `domains/tagging` with AWSGW-composed listers (`0d830e901`, `f3545e3ea`, `aedd0b2c4`). Open: `gateway/spx` (owner of operator-plane dispatch registration) | Each needs one decision or one capability, then a mechanical move. |
+| One narrow missing boundary | Done: `handlers/sts` EKS JWKS lookup → `handlers/eks` `ClusterJWKSResolver` (`254d898a3`); `gateway/tagging` → `domains/tagging` with AWSGW-composed listers (`0d830e901`, `f3545e3ea`, `aedd0b2c4`); `gateway/spx` → `operator/admin`. Done: the `spinifex` service and `/admin` dispatch registration, owned by `operator/admin` | Each needs one decision or one capability, then a mechanical move. |
 | Broader boundary or follows another package | `gateway` root (ADR-0002 auth and request-context phase) | Recorded debt with removal conditions above. |
 
 Per-consumer IAM capabilities are the widest unblocker: they gate STS's IAM reads, Ochre's IAM reads, the ELBv2 role path and the ECS, EKS and RDS ensurers.

docs/admin/account-management/README.md

@@ -236,12 +236,12 @@ Each stage runs to empty before the next begins, so nothing is deleted while som
 A large account takes minutes to tear down. The API does not hold the connection open for it: `DeleteAccount` starts the job, returns a `deletionId` immediately, and progress is read from `DescribeAccountDeletion`.
 
 ```bash
-AWS_PROFILE=spinifex-operator spx admin account delete 000000000042 --remote \
+AWS_PROFILE=spinifex-operator spx admin account delete 000000000042 \
   --name customer@example.com --yes \
   --endpoint https://node1.example.com:9999 --region us-west-1
 ```
 
-`spx --remote` starts the job and then polls, printing each stage as it lands. Your own client can do the same, or start the job and check back later — the deletion record outlives the account, so `DescribeAccountDeletion` stays answerable after the account is gone.
+`spx admin account delete` always goes through the admin API, so it signs with the standard AWS credential chain. On a cluster node that means `AWS_PROFILE=spinifex` (there is no default profile, so without it the command cannot find credentials), or equivalent credentials for an IAM user holding `spinifex:DeleteAccount` and `spinifex:DescribeAccountDeletion`, and a reachable gateway. The older `--remote` flag is accepted and ignored. The command starts the job and then polls, printing each stage as it lands. Your own client can do the same, or start the job and check back later — the deletion record outlives the account, so `DescribeAccountDeletion` stays answerable after the account is gone.
 
 One job runs per account. A second `DeleteAccount` with a different token while one is running returns `OperationInProgress`; with the *same* token it returns the running job, which is what makes a retry safe.
 

docs/admin/host-firewall/README.md

@@ -155,8 +155,11 @@ Restarting the daemon is what re-arms it: the node rebuilds its peer list from t
 
 `disable` removes Spinifex's own table and the peer file and leaves every other nftables table alone. It is idempotent, and it is the recovery path — it runs even on a node with no policy installed, so it cannot leave you with a node you can neither arm nor disarm.
 
+`disable` leaves `table inet spinifex_conntrack` loaded, and the boot-time unit loads it on every node whether or not the policy is armed. It accepts everything and exists only to keep connection tracking running. Without it, a connection opened while the policy is off, such as an OVN RAFT link formed during that window, is first seen mid-stream when the policy comes back, and its replies are dropped as new connections. Do not delete it.
+
 > [!WARNING]
 > Do not flush the whole nftables ruleset to clear the policy. The VPC daemon writes SNAT and per-elastic-IP forward rules into the `ip nat` and `ip filter` tables and reinstalls them only when it starts, so `nft flush ruleset` breaks elastic IPs silently until the next `systemctl restart spinifex-vpcd`. Use `disable`, which only ever touches `inet spinifex_filter`.
+> On Oracle Linux with IPsec on, a flush also removes `inet spinifex_ipsec_guard`, so Geneve to a peer without an SA goes out in clear until the daemon puts the table back, within 60 seconds.
 
 ## The Files
 
@@ -170,6 +173,7 @@ Everything lives under `/etc/spinifex/firewall/`. Which files are safe to edit f
 | `open-ports.nft` | The apply helper — the formation window only | No. Reset closed on install and upgrade |
 | `custom.nft` | **You** — `trusted_ssh_peers` and anything else you add | **Yes.** Created once, never rewritten |
 | `mode` | `setup.sh` — whether the install path armed the policy | No |
+| `conntrack.nft` | `setup.sh` — the tracking table, loaded armed or not | No. Edits are overwritten |
 
 The daemon cannot write `/etc` itself, and a blanket `nft` grant would be root-equivalent, so changes go through `/usr/local/lib/spinifex/spinifex-firewall-apply` — a root helper that accepts a fixed set of verbs and re-validates every address handed to it.
 
@@ -189,10 +193,10 @@ Dropped packets are logged, rate-limited to five a minute, so this tells you whe
 sudo journalctl -k | grep 'spinifex-fw drop'
 ```
 
-An empty log with a connection still failing means the policy is not the cause. A node with no table loaded at all is not armed:
+An empty log with a connection still failing means the policy is not the cause. A node with no peer file is not armed, and its boot-time unit loads only the tracking table:
 
 ```bash
-systemctl is-enabled spinifex-firewall.service
+sudo test -e /etc/spinifex/firewall/peers.nft && echo armed || echo not armed
 ```
 
 ## Troubleshooting

docs/admin/setting-up-your-cluster/README.md

@@ -106,7 +106,7 @@ done
 | `spinifex-rds-postgres` | RDS, `--engine postgres`         | Alpine 3.24.1 + PostgreSQL 18 + `rds-init`      |
 | `spinifex-rds-mariadb`  | RDS, `--engine mariadb`          | Alpine 3.24.1 + MariaDB 11.8 + `rds-init`       |
 | `spinifex-ecs-node`     | ECS container instances          | Alpine 3.21.7 + containerd + ecs-agent          |
-| `spinifex-eks-node`     | EKS control plane and nodegroups | Alpine 3.21.7 + K3s v1.32.5 + eks-token-webhook |
+| `spinifex-eks-node`     | EKS control plane and nodegroups | Alpine 3.21.7 + K3s v1.32.5 + eks-token-webhook + eks-etcd-member |
 
 All four boot **BIOS**, unlike the distro images, which boot UEFI. That is a property of how they are built and nothing you need to pass.
 

docs/compute/gpu-passthrough/README.md

@@ -51,12 +51,24 @@ sudo spx admin gpu setup
 What it does:
 
 1. Detects installed GPUs and their PCI IDs.
-2. Enables IOMMU in GRUB (`intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt`).
+2. Adds the IOMMU kernel arguments (`intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt`): to `GRUB_CMDLINE_LINUX_DEFAULT` with `update-grub`, or with `grubby` to every boot entry and to `/etc/kernel/cmdline`, which kernels installed later inherit.
 3. Writes the vfio udev rule (`/etc/udev/rules.d/99-spinifex-vfio.rules`).
 4. Blacklists `nouveau` (NVIDIA) and `amdgpu` (AMD) so the host kernel does not claim the device.
 5. Configures `vfio-pci` early binding in `/etc/modprobe.d/vfio-pci.conf`.
-6. Adds vfio modules to initramfs.
+6. Loads the vfio modules from the initramfs: through `/etc/initramfs-tools/modules` and `update-initramfs -u`, or through `/etc/dracut.conf.d/spinifex-vfio.conf` and `dracut -f --regenerate-all`. With dracut, each rebuilt image is checked with `lsinitrd` for `vfio_pci`, its early-load line and the `modprobe.d` files, and setup stops naming whatever is missing.
 
+The command finds its boot tools before changing anything, and stops naming them if the host has neither or both of `update-grub` and `grubby`, or of `update-initramfs` and `dracut`.
+
+**Oracle Linux hosts** take the `grubby` and `dracut` path: Oracle Linux keeps kernel arguments in BLS boot entries and has no `GRUB_CMDLINE_LINUX_DEFAULT`.
+`grubby --update-kernel=ALL` writes the IOMMU arguments to every entry and to `/etc/kernel/cmdline`, from which a kernel installed later builds its own entry, and dracut applies `spinifex-vfio.conf` to every initramfs it builds.
+After installing a new `kernel-uek`, check its entry before rebooting into it:
+
+```bash
+sudo grubby --info=ALL | grep -E '^(kernel|args)'
+sudo lsinitrd -k <new-kernel-version> | grep vfio
+```
+
+Whole-GPU passthrough is supported on Oracle Linux; NVIDIA MIG is not.
 If any change requires a reboot, the command prints instructions and exits:
 
 ```
@@ -175,7 +187,7 @@ The command signals the daemon and waits for confirmation. The GPU is released b
 
 ### IOMMU not active after reboot
 
-Verify IOMMU is enabled in BIOS/UEFI (`Intel VT-d` or `AMD-Vi`). Then check GRUB picked up the parameters:
+Verify IOMMU is enabled in BIOS/UEFI (`Intel VT-d` or `AMD-Vi`). Then check the boot loader picked up the parameters (on Oracle Linux, `sudo grubby --info=DEFAULT` lists them for the default entry):
 
 ```bash
 cat /proc/cmdline | grep iommu

docs/compute/launching-instances/README.md

@@ -150,13 +150,17 @@ aws ec2 describe-instance-types \
   --query "InstanceTypes[?InstanceType=='t3.small']"
 ```
 
-Show capacity (how many of each type can still be launched):
+List only the types that can still be launched:
 
 ```bash
 aws ec2 describe-instance-types \
   --filters Name=capacity,Values=true
 ```
 
+Each type appears once, whatever its free capacity.
+This filter is a Spinifex extension and AWS does not declare it, so it is ignored by a region that does not implement it.
+AWS reports exhaustion at launch instead, with `InsufficientInstanceCapacity`, and so does Spinifex.
+
 ## SSH
 
 To SSH into instances via their public IPs, see [Setting Up Your Cluster](/docs/setting-up-your-cluster).

docs/compute/vpc-networking/README.md

@@ -144,6 +144,27 @@ The guest MTU moves with the setting, but a running guest holds its DHCP lease f
 
 **Expect launches to fail for a minute or two after re-enabling.** `RunInstances` returns `ServerInternal` with `vpc.add-nat` timing out, because the OVN flows-ready barrier cannot confirm while `ovn-controller` reprograms every tunnel for the new SAs. It clears once `ip xfrm state` settles. The error is reported as a capacity problem, which it is not.
 
+### Containers and nested overlays inside a guest
+
+A Docker bridge, Kubernetes pod network or other overlay inside a guest defaults to an MTU of 1500, which is larger than the guest's 1408 or 1442.
+Downloads from the internet still work for TCP on the veth and routed uplinks, because vpcd clamps the TCP MSS on the host hop (see [Performance Tuning → MTU](/docs/performance-tuning#4-mtu)).
+That was measured on a single node (env12) with a Docker container; it was not measured on a physical uplink or on a multi-node cluster.
+
+On a physical uplink there is no host hop to clamp, and UDP is never clamped, so set the inner MTU to the guest MTU.
+For Docker, in `/etc/docker/daemon.json`:
+
+```json
+{
+  "mtu": 1408,
+  "default-network-opts": {
+    "bridge": { "com.docker.network.driver.mtu": "1408" }
+  }
+}
+```
+
+Use 1442 when `ipsec_enabled = false`, and restart the Docker daemon so the default bridge is recreated.
+The gateway MTU follows `ipsec_enabled` and converges on the next reconcile pass, so change the inner MTU together with the guest MTU.
+
 ## Private vs Public Subnets
 
 A subnet's behavior depends on three things: whether the VPC has an Internet Gateway, whether the subnet's route table has a default route to that IGW, and whether the subnet has `MapPublicIpOnLaunch` enabled.

docs/containers/ecr/README.md

@@ -45,7 +45,8 @@ Each repository holds image manifests and their layers. Images are stored in obj
 
 - Repository lifecycle — `CreateRepository`, `DeleteRepository`, `DescribeRepositories`.
 - OCI push and pull — blobs, manifests, and tags, including cross-repository blob mounts.
-- Token auth — `GetAuthorizationToken` plus bearer/basic auth on `/v2/`.
+- OCI 1.1 referrers — the referrers API, so cosign v3 signatures and other artifacts attached to an image can be listed (see [OCI referrers and image signing](#oci-referrers-and-image-signing)).
+- Token auth — `GetAuthorizationToken` plus bearer/basic auth on `/v2/`. A token is valid for 12 hours. One minted by an assumed-role session (an EKS node, for example) stays valid after that session expires and is checked against the live role on every request, so it stops working when the role is deleted or loses its ECR permissions.
 - Repository policies — set, get, and delete a repository policy document.
 
 **Current limitations**
@@ -176,6 +177,71 @@ Then authenticate Docker and push using the `repository_url` output, following s
 
 :::
 
+## EKS workers and multiple gateways
+
+Each gateway puts its own address in the `repositoryUri` it returns, so repositories created through different gateways can carry different hosts.
+When an EKS worker launches, the daemon that launches it writes every cluster gateway into the worker's registry mirrors, TLS trust and credential-provider config.
+A pod can then pull from any `repositoryUri` host in the cluster.
+The worker tries the gateway named in the image reference first and falls back to the others in cluster order.
+
+Limits:
+
+- The gateway list comes from the cluster config the launching daemon loaded at start.
+  A gateway added later is not trusted by existing workers; restart the launching daemon and replace the worker.
+- Peer gateways are assumed to listen on port 9999.
+  A hand-edited `awsgw` bind on a peer is not covered.
+- A gateway address that does not answer costs about 30 seconds on each cold pull while the worker's connection attempt gives up.
+  A stopped gateway on a host that is still up refuses the connection, and the worker moves to the next gateway at once.
+
+Measured on a single-node environment: pulls by `repositoryUri` and by the account's ECR DNS name, a pull rejected with `x509: certificate signed by unknown authority` when the gateway host was not in the worker config, and a 31 second pull through an unreachable first endpoint.
+Measured on a 3-node environment: a pod pulling each of the three gateways' `repositoryUri` on a worker launched with this change (two of the three failed with `x509` on a worker launched without it), and a pull of the stopped gateway's own `repositoryUri` that succeeded through another gateway within one retry (the pull itself took 147 ms after the first endpoint refused).
+
+## OCI referrers and image signing
+
+A manifest can name another manifest as its `subject`, which is how tools such as cosign v3 attach signatures and SBOMs to an image.
+The registry records the subject when the manifest is pushed and answers with an `OCI-Subject` header on the `201`.
+The `subject` does not have to exist in the repository when the referrer is pushed.
+
+`GET /v2/<name>/referrers/<digest>` lists the manifests that name `<digest>` as their subject.
+The request is authorised as `ecr:BatchGetImage` on the repository.
+The response is `200` with `Content-Type: application/vnd.oci.image.index.v1+json` and an index whose `manifests` hold each referrer's `mediaType`, `digest`, `size`, `artifactType` and annotations.
+A digest with no referrers, or one that is not stored, returns an empty index.
+A malformed digest returns `400` with `DIGEST_INVALID`.
+
+Adding `?artifactType=<type>` returns only referrers of that type and sets `OCI-Filters-Applied: artifactType`.
+The header is also set when nothing matches.
+An image manifest without an `artifactType` is listed under its config `mediaType`; an index without one has none.
+
+Deleting an image by digest (`DELETE /v2/<name>/manifests/<digest>`, `aws ecr batch-delete-image` with `imageDigest`) also deletes its referrers, and their referrers in turn.
+Deleting by tag removes only the tag and leaves the image and its referrers.
+
+cosign v3 sign and verify work against the registry, including on a child manifest of an image index copied with `cosign copy`.
+cosign v2 keeps writing its `sha256-<digest>.sig` tags and is unaffected.
+`GET` on a manifest still returns `406` when the request's `Accept` header does not include the stored `mediaType`.
+
+Limits:
+
+- `cosign copy` does not copy v3 signatures to the destination repository.
+  This is client behaviour.
+  To keep the original signature, copy with `oras cp -r <source> <mirror>`, which copies the image and its referrers in one command, or `crane copy` the signature manifest by digest after the image.
+  Otherwise, sign the copy again.
+- Referrers pushed before this version are not listed.
+  Push them again to add them to the index.
+- A crash between deleting an image and deleting its referrers can leave referrers behind that nothing lists.
+- The referrers listing is not paginated.
+- `HEAD` on the referrers route is not served.
+
+Measured on a 3-node environment with cosign v3.1.4 and v2.6.5:
+
+- cosign v3.1.4 signed a fresh image index, with one `GET /referrers` answered `200`, no `sha256-<digest>` tag written and no `406`.
+- Verify passed.
+- A copied index's child manifest signed and verified, which returned `406` before this change.
+- The listing returned the signature with its `artifactType`, the filter returned one entry for the signature's type and none for another type.
+- `aws ecr batch-delete-image` of the signed image left an empty listing and `404` for both the signature and the image.
+- cosign v2.6.5 signed and verified a fresh image.
+
+Not measured: `oras`, `notation`, a listing with many referrers, and a crash in the middle of a delete.
+
 ## Troubleshooting
 
 **`docker login` fails with an authorization error.** The token is account-scoped and short-lived. Re-run `aws ecr get-login-password` to mint a fresh token, and confirm the registry host in `docker login` matches your account's endpoint (`aws ecr describe-repositories` returns it).

docs/containers/eks/README.md

@@ -35,6 +35,8 @@ resources:
 
 An EKS cluster on Spinifex has two parts: a **control plane** (the managed Kubernetes API server, provisioned on a Spinifex-managed VM and fronted by a network load balancer) and one or more **managed node groups** of worker VMs that run your pods. You provide the VPC, subnets, and IAM roles; Spinifex creates and reconciles the cluster, its load balancer, and its security groups.
 
+Control-plane hosts are not Kubernetes Nodes, as on AWS: `kubectl get nodes` lists only your node group workers, so a cluster with no node group lists none, and no pod runs on a control-plane host. Add-ons that used to tolerate the control-plane `CriticalAddonsOnly` taint, such as CoreDNS, metrics-server and konnectivity-agent, run on workers, so a Ready worker is required before they become available.
+
 Authentication uses **EKS access entries** (the API authentication mode) — IAM principals are mapped to Kubernetes RBAC groups, and `kubectl` authenticates with `aws eks get-token` exactly as it does on AWS.
 
 <p align="center">
@@ -56,10 +58,110 @@ Authentication uses **EKS access entries** (the API authentication mode) — IAM
 **Spinifex specifics**
 
 - **Access entries are the only auth path.** `authentication_mode` accepts `API` (the default when unset) and `API_AND_CONFIG_MAP`; bare `CONFIG_MAP` is rejected. Spinifex has no `aws-auth` ConfigMap path, so `API_AND_CONFIG_MAP` behaves as `API`: grant access with access entries.
-- **Security groups are auto-managed.** Spinifex creates the cluster and node-group security groups deterministically (e.g. `eks-cluster-<name>-nodegroup-sg`); `vpc_config.security_group_ids` is ignored. To expose a workload you add an ingress rule to the auto-managed node-group SG.
+- **The cluster security group is auto-managed.** As on EKS, Spinifex creates one `eks-cluster-sg-<name>-<id>` in your VPC, tagged `kubernetes.io/cluster/<name>=owned` and `aws:eks:cluster-name`, allowing all traffic from itself. It is attached to the workers and the private-endpoint network interface, and `DescribeCluster` reports it as `clusterSecurityGroupId`. A launch template that names security groups replaces it on the workers, as on AWS. Clusters created before this change keep their `-control-plane-sg` and `-nodegroup-sg` until you recreate them. Deleting a cluster deletes the cluster security group and removes only the rules on your other security groups that reference it. `vpc_config.security_group_ids` is ignored.
 - **The worker image is fixed.** Node groups always boot Spinifex's `eks-node` image. `ami_type` is recorded but does not select the image, and the image must be registered on the cluster before you create a node group.
+- **Workers launch with IMDSv2 required and an IMDS hop limit of 2.**
+  This matches AWS managed node groups and lets a pod without IRSA reach the node role through the SDK default credential chain.
+  A node group launch template that sets `metadata_options` overrides each field it sets, and any field it leaves unset keeps the default.
+  Existing workers keep their hop limit until they are replaced or changed with `ModifyInstanceMetadataOptions`.
 - **Default Kubernetes version is `1.32`.**
 
+### Control plane readiness and kubectl authentication
+
+Each control-plane VM authenticates `kubectl` tokens through a webhook that calls the AWS gateway on the host the VM runs on.
+When that gateway is stopped, the webhook cannot validate tokens and `kubectl` reports `Unauthorized` for the share of requests that the load balancer sends to that control plane.
+
+The webhook serves `/readyz` on port 8444 of each control-plane VM.
+It returns 200 only when the host gateway answers a TLS handshake and the local API server accepts a TCP connection, and 503 otherwise.
+The API server target group health-checks `HTTP :8444/readyz` every 5 seconds with a healthy and unhealthy threshold of 2.
+The target group still forwards TCP on 6443.
+A control plane that cannot authenticate leaves rotation, and returns when its gateway is back.
+
+Measured on a three-node cluster, one control plane per host, with `kubectl get ns` run continuously after stopping one gateway:
+
+| Gateway stopped | Result |
+| --- | --- |
+| Host without the cluster load balancer VM, first run | 4 of 20 calls failed in the first round, 0 of 20 in the next round 54 s after the stop |
+| Host without the cluster load balancer VM, second run | 2 failures in 100 s, at 3 s and 8 s after the stop |
+| Host running the cluster load balancer VM | Not fixed: 20 of 60 calls failed over 100 s |
+
+Before this change the same test failed 3, 4 and 3 calls of 10 per stopped gateway.
+
+Limits:
+
+- Only clusters created with the current `eks-node` image get the readiness listener, and only clusters created from now on get the health check on the target group.
+  Older clusters keep the TCP check on 6443 and are not changed.
+- A cluster created with an older image gets the health check on its target group but no listener on 8444, so every control plane fails the check and the load balancer fails open to all of them.
+- A stopped gateway on the host that runs the cluster load balancer VM is not covered.
+  The load balancer VM reports target health through its own host's gateway, so while that gateway is down the unhealthy control plane stays in rotation.
+- Not measured: more than one control plane per host, a physical-node failure instead of a stopped gateway, and the target health state as `DescribeTargetHealth` reports it, because the load balancer and control-plane VMs belong to the system account.
+
+### IAM roles for service accounts in your workloads
+
+A pod gets an IAM role by running under a ServiceAccount that carries the role ARN in an annotation.
+No pod-spec wiring is needed.
+
+```yaml
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: app
+  annotations:
+    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/app-role
+```
+
+Every control plane runs a mutating webhook that acts on pod creation.
+When the pod's ServiceAccount has the annotation, the webhook adds these to each container and init container:
+
+- `AWS_ROLE_ARN` and `AWS_WEB_IDENTITY_TOKEN_FILE`, with a projected token volume named `aws-iam-token` mounted at `/var/run/secrets/eks.amazonaws.com/serviceaccount`.
+- `AWS_REGION`, `AWS_DEFAULT_REGION` and `AWS_STS_REGIONAL_ENDPOINTS`.
+- `AWS_ENDPOINT_URL_STS`, so the SDK calls the Spinifex STS endpoint.
+- `AWS_CA_BUNDLE`, pointing at `/etc/spinifex/ca-bundle/ca.pem`, a ConfigMap named `spinifex-ca-bundle` that is created in every namespace and holds the system roots plus the gateway CA.
+- `SSL_CERT_FILE` with the same path, only when the container does not set it. Clients such as the AWS Load Balancer Controller read this variable instead of `AWS_CA_BUNDLE`.
+
+Your workload still sets `AWS_ENDPOINT_URL` (or the per-service variable) for every service other than STS.
+
+The token audience defaults to `sts.amazonaws.com`.
+The expiry defaults to a value between 79200 and 86400 seconds and is clamped to 600 to 86400 seconds.
+Set them with the `eks.amazonaws.com/audience` and `eks.amazonaws.com/token-expiration` annotations on the ServiceAccount.
+The `token-expiration` annotation on the pod wins over the ServiceAccount.
+
+A container is left alone when it already has the role, region and regional STS settings, which is how the managed add-ons and your own explicit wiring are skipped.
+When some are present, the missing ones and the mount are added.
+Containers named in the pod's `eks.amazonaws.com/skip-containers` annotation (comma separated) are not touched.
+A pod whose ServiceAccount has no role annotation is not changed.
+
+The webhook fails open: if it cannot decide, the pod is created without injection.
+Each injection, skip and failure is logged with the namespace, pod and reason in the control-plane console log.
+Because it fails open, a pod that needs a role but starts without credentials is the symptom to look for, and the console log says why.
+
+The webhook listens on `127.0.0.1:8445` of each control plane, and each API server calls its own.
+Port 8444 is a different listener, the readiness check described above.
+
+Existing clusters get the webhook only when their control planes are replaced by the current `eks-node` image.
+No user-facing operation replaces a control plane, and automatic replacement does not currently work when the replacement reuses the failed member's IP.
+Recreate the cluster to adopt it.
+User-workload injection through a cluster created with an older image is not measured.
+
+### Add-on versions
+
+Add-on versions use the AWS form `v<x.y.z>-eksbuild.<n>`, for example `v1.40.1-eksbuild.1`.
+`DescribeAddonVersions` returns versions in this form, newest first, and every other add-on response reports the version the same way.
+Spinifex ships one EKS build of each upstream version, so `<n>` is `1` today.
+
+`DescribeAddonVersions` honours `addonName`, `kubernetesVersion`, `types`, `publishers`, `owners`, `maxResults` and `nextToken`.
+Each version carries `compatibilities` with the `clusterVersion`, the `platformVersions`, and a `defaultVersion` flag that is true for exactly one version per add-on and cluster version.
+A `kubernetesVersion` that Spinifex does not run returns no add-ons.
+
+`CreateAddon` and `UpdateAddon` accept the AWS form of a version in the catalogue.
+They also accept the bare upstream form, such as `1.40.1`, and Spinifex logs each use at Info.
+This is a Spinifex leniency that AWS does not offer, and it exists so that older scripts keep working.
+Any other value is refused with a message that names `addonVersion` and lists the supported versions, for example `v1.40.1-eksbuild.2`, `1.40` or `v1.40.1`.
+
+In Terraform, pass the AWS form to `aws_eks_addon.addon_version`, either as a literal or from `data.aws_eks_addon_version`.
+The AWS provider validates the format before it calls the API, so it rejects the bare form.
+Leaving `addon_version` unset still installs the catalogue default.
+
 ## Prerequisites
 
 > [!IMPORTANT]
@@ -123,7 +225,7 @@ aws eks describe-cluster --name demo --query 'cluster.status'
 aws eks create-nodegroup \
   --cluster-name demo \
   --nodegroup-name default \
-  --node-role-arn arn:aws:iam::000000000000:role/eks-node-role \
+  --node-role arn:aws:iam::000000000000:role/eks-node-role \
   --subnets subnet-aaaa subnet-bbbb \
   --scaling-config minSize=1,maxSize=2,desiredSize=1 \
   --instance-types t3.medium
@@ -138,14 +240,15 @@ aws eks update-kubeconfig --name demo
 kubectl get nodes
 ```
 
+The list shows the node group workers only. The control-plane VMs never appear as Nodes.
+
 ### 4. Expose a workload
 
-Workers join an auto-managed security group that only allows intra-cluster traffic. To reach a NodePort service, add one ingress rule to that SG (look it up by its deterministic name):
+Workers join the cluster security group, which only allows intra-cluster traffic. To reach a NodePort service, add one ingress rule to it (`DescribeCluster` reports its id):
 
 ```bash
-SG=$(aws ec2 describe-security-groups \
-  --filters Name=group-name,Values=eks-cluster-demo-nodegroup-sg \
-  --query 'SecurityGroups[0].GroupId' --output text)
+SG=$(aws eks describe-cluster --name demo \
+  --query 'cluster.resourcesVpcConfig.clusterSecurityGroupId' --output text)
 
 aws ec2 authorize-security-group-ingress \
   --group-id "$SG" --protocol tcp --port 30080 --cidr 0.0.0.0/0
@@ -187,7 +290,7 @@ The form notes that the authentication mode is API and that the legacy aws-auth
 
 - The **Access** tab lists IAM principals and their cluster-access policies — add an access entry to let another user or role reach the cluster.
 - The **Addons** tab installs and shows the health of managed addons.
-- The **Networking** tab shows the cluster's VPC, subnets, endpoint access, and the auto-managed security groups.
+- The **Networking** tab shows the cluster's VPC, subnets, endpoint access, and the auto-managed cluster security group.
 
 To run `kubectl` against the cluster, follow the AWS CLI tab's `update-kubeconfig` step.
 
@@ -250,21 +353,31 @@ The workbooks form a ladder. **eks-quickstart** is a NodePort demo; **eks-https-
 
 :::
 
+### Upgrade notes
+
+**Control-plane hosts no longer register as Nodes.** `kubectl get nodes` shows workers only, so node counts drop by the number of control planes, for example 6 to 3 on a three-control-plane cluster with three workers. Add-ons that tolerated `CriticalAddonsOnly` now run on workers.
+
+**Existing clusters keep the old behaviour.** A cluster created before this change keeps registering its control planes as tainted Nodes until it is recreated. Recreate the cluster to adopt the new behaviour: automatic control-plane replacement does not currently work when the replacement reuses the failed member's IP, so it is not a supported way to upgrade.
+
 ## Troubleshooting
 
 **Cluster stuck in `CREATING`.** The control plane is still booting. Check `aws eks describe-cluster --name <name> --query 'cluster.health'` for reported issues. The control plane needs egress to bootstrap, so confirm its subnet has an Internet Gateway route.
 
 **`kubectl` cannot connect.** Re-run `aws eks update-kubeconfig --name <name>` to refresh the endpoint and CA, and confirm the cluster is `ACTIVE`. `aws sts get-caller-identity` verifies your credentials reach the Spinifex endpoint.
 
+**`kubectl` returns `Unauthorized` for some requests and works for others.**
+One control plane cannot reach its host's AWS gateway, so check `systemctl status spinifex-awsgw` on each node.
+Control plane readiness and kubectl authentication above lists which clusters recover on their own.
+
 **Workers stay `NotReady`.** Almost always an egress or IAM problem:
 - Confirm the workers can pull images — a default route to an Internet Gateway (public) or NAT Gateway (private).
 - Confirm the node role has `AmazonEKSWorkerNodePolicy`, `AmazonEKS_CNI_Policy`, and `AmazonEC2ContainerRegistryReadOnly`.
 
-**Workload unreachable from outside the cluster.** The node-group security group only allows intra-cluster traffic by default. Add an ingress rule for your NodePort to `eks-cluster-<name>-nodegroup-sg`, as shown in the CLI tab.
+**Workload unreachable from outside the cluster.** The cluster security group only allows intra-cluster traffic by default. Add an ingress rule for your NodePort to it, as shown in the CLI tab. An Ingress served by the AWS Load Balancer Controller needs no rule: the controller adds it to the cluster security group.
 
 **`create-cluster` rejected with an authentication-mode error.** `CONFIG_MAP` is the one mode Spinifex rejects, because it has no `aws-auth` ConfigMap path. Set `authentication_mode` to `API` or `API_AND_CONFIG_MAP`, or leave it unset, and grant access with access entries.
 
-**Addon install fails.** Only addons bundled into the registered `eks-node` image install successfully. If you pin an addon version in Terraform, leave `addon_version` unset so Spinifex selects the catalog default.
+**Addon install fails.** Only addons bundled into the registered `eks-node` image install successfully. If you pin an addon version in Terraform, use the AWS form from `DescribeAddonVersions` (see Add-on versions above), or leave `addon_version` unset so Spinifex selects the catalog default.
 
 ## Reference Architectures
 

docs/coverage/README.md

@@ -17,7 +17,7 @@ tags:
 
 ## Overview
 
-The platform serves **392 operations** across the AWS APIs below. Every page names the operations its service implements from the pinned model, generated from the dispatch tables on each build rather than written by hand.
+The platform serves **406 operations** across the AWS APIs below. Every page names the operations its service implements from the pinned model, generated from the dispatch tables on each build rather than written by hand.
 
 | Service | Operations |
 |---|---:|
@@ -27,8 +27,9 @@ The platform serves **392 operations** across the AWS APIs below. Every page nam
 | [ECS](/coverage/ecs) | 31 |
 | [EKS](/coverage/eks) | 27 |
 | [ELBv2](/coverage/elbv2) | 36 |
-| [IAM](/coverage/iam) | 81 |
+| [IAM](/coverage/iam) | 82 |
 | [RDS](/coverage/rds) | 27 |
+| [Route 53](/coverage/route53) | 13 |
 | [S3](/coverage/s3) | 26 |
 | [STS](/coverage/sts) | 5 |
-| **Total** | **392** |
+| **Total** | **406** |

docs/coverage/coverage.json

@@ -2,8 +2,8 @@
   "schema_version": 1,
   "model_source": {
     "repository": "github.com/aws/api-models-aws",
-    "commit": "9b1a4929274443d8671ea8acfda7afbe856e29e9",
-    "commit_date": "2026-09-25"
+    "commit": "e10d6a1c5ac36af7c0c61d1f67b3c3ebbe41de49",
+    "commit_date": "2026-10-28"
   },
   "services": [
     {
@@ -300,6 +300,7 @@
           "DeleteCapacityManagerDataExport",
           "DeleteCarrierGateway",
           "DeleteClientVpnEndpoint",
+          "DeleteClientVpnEndpointAuthorizationPolicy",
           "DeleteClientVpnRoute",
           "DeleteCoipCidr",
           "DeleteCoipPool",
@@ -681,7 +682,9 @@
           "GetCapacityManagerMetricData",
           "GetCapacityManagerMetricDimensions",
           "GetCapacityManagerMonitoredTagKeys",
+          "GetCapacityManagerSupportabilityScores",
           "GetCapacityReservationUsage",
+          "GetClientVpnEndpointAuthorizationPolicy",
           "GetCoipPoolUsage",
           "GetConsoleOutput",
           "GetConsoleScreenshot",
@@ -695,6 +698,7 @@
           "GetHostReservationPurchasePreview",
           "GetImageAncestry",
           "GetImageBlockPublicAccessState",
+          "GetImageFindings",
           "GetInstanceMetadataDefaults",
           "GetInstanceTpmEkPub",
           "GetInstanceTypesFromInstanceRequirements",
@@ -734,6 +738,7 @@
           "GetSnapshotBlockPublicAccessState",
           "GetSpotPlacementScores",
           "GetSubnetCidrReservations",
+          "GetTransitGatewayAttachmentNatConfiguration",
           "GetTransitGatewayAttachmentPropagations",
           "GetTransitGatewayMeteringPolicyEntries",
           "GetTransitGatewayMulticastDomainAssociations",
@@ -766,6 +771,7 @@
           "ModifyCapacityReservation",
           "ModifyCapacityReservationFleet",
           "ModifyClientVpnEndpoint",
+          "ModifyClientVpnEndpointAuthorizationPolicy",
           "ModifyDefaultCreditSpecification",
           "ModifyEbsDefaultKmsKeyId",
           "ModifyFleet",
@@ -814,6 +820,7 @@
           "ModifyTrafficMirrorFilterRule",
           "ModifyTrafficMirrorSession",
           "ModifyTransitGateway",
+          "ModifyTransitGatewayAttachmentNatConfiguration",
           "ModifyTransitGatewayMeteringPolicy",
           "ModifyTransitGatewayPolicyTableEntry",
           "ModifyTransitGatewayPrefixListReference",
@@ -1335,6 +1342,7 @@
           "DeleteCapacityManagerDataExport",
           "DeleteCarrierGateway",
           "DeleteClientVpnEndpoint",
+          "DeleteClientVpnEndpointAuthorizationPolicy",
           "DeleteClientVpnRoute",
           "DeleteCoipCidr",
           "DeleteCoipPool",
@@ -1653,7 +1661,9 @@
           "GetCapacityManagerMetricData",
           "GetCapacityManagerMetricDimensions",
           "GetCapacityManagerMonitoredTagKeys",
+          "GetCapacityManagerSupportabilityScores",
           "GetCapacityReservationUsage",
+          "GetClientVpnEndpointAuthorizationPolicy",
           "GetCoipPoolUsage",
           "GetConsoleScreenshot",
           "GetDeclarativePoliciesReportSummary",
@@ -1665,6 +1675,7 @@
           "GetHostReservationPurchasePreview",
           "GetImageAncestry",
           "GetImageBlockPublicAccessState",
+          "GetImageFindings",
           "GetInstanceMetadataDefaults",
           "GetInstanceTpmEkPub",
           "GetInstanceTypesFromInstanceRequirements",
@@ -1701,6 +1712,7 @@
           "GetSnapshotBlockPublicAccessState",
           "GetSpotPlacementScores",
           "GetSubnetCidrReservations",
+          "GetTransitGatewayAttachmentNatConfiguration",
           "GetTransitGatewayAttachmentPropagations",
           "GetTransitGatewayMeteringPolicyEntries",
           "GetTransitGatewayMulticastDomainAssociations",
@@ -1732,6 +1744,7 @@
           "ModifyCapacityReservation",
           "ModifyCapacityReservationFleet",
           "ModifyClientVpnEndpoint",
+          "ModifyClientVpnEndpointAuthorizationPolicy",
           "ModifyDefaultCreditSpecification",
           "ModifyEbsDefaultKmsKeyId",
           "ModifyFleet",
@@ -1773,6 +1786,7 @@
           "ModifyTrafficMirrorFilterRule",
           "ModifyTrafficMirrorSession",
           "ModifyTransitGateway",
+          "ModifyTransitGatewayAttachmentNatConfiguration",
           "ModifyTransitGatewayMeteringPolicy",
           "ModifyTransitGatewayPolicyTableEntry",
           "ModifyTransitGatewayPrefixListReference",
@@ -1897,6 +1911,7 @@
           "DescribeImages",
           "DescribePullThroughCacheRules",
           "DescribeRegistry",
+          "DescribeReplicationBackfillResults",
           "DescribeRepositories",
           "DescribeRepositoryCreationTemplates",
           "GetAccountSetting",
@@ -1912,6 +1927,7 @@
           "ListImageReferrers",
           "ListImages",
           "ListPullTimeUpdateExclusions",
+          "ListReplicationBackfills",
           "ListTagsForResource",
           "PutAccountSetting",
           "PutImage",
@@ -1926,6 +1942,8 @@
           "SetRepositoryPolicy",
           "StartImageScan",
           "StartLifecyclePolicyPreview",
+          "StartReplicationBackfill",
+          "StopReplicationBackfill",
           "TagResource",
           "UntagResource",
           "UpdateImageStorageClass",
@@ -2031,14 +2049,18 @@
           "DescribeImageReplicationStatus",
           "DescribeImageSigningStatus",
           "DescribePullThroughCacheRules",
+          "DescribeReplicationBackfillResults",
           "DescribeRepositoryCreationTemplates",
           "GetAccountSetting",
           "GetSigningConfiguration",
           "ListImageReferrers",
           "ListPullTimeUpdateExclusions",
+          "ListReplicationBackfills",
           "PutAccountSetting",
           "PutSigningConfiguration",
           "RegisterPullTimeUpdateExclusion",
+          "StartReplicationBackfill",
+          "StopReplicationBackfill",
           "UpdateImageStorageClass",
           "UpdatePullThroughCacheRule",
           "UpdateRepositoryCreationTemplate",
@@ -2058,11 +2080,13 @@
       "note": "",
       "operations": {
         "modelled": [
+          "AssociateGatewayWithCluster",
           "ContinueServiceDeployment",
           "CreateCapacityProvider",
           "CreateCluster",
           "CreateDaemon",
           "CreateExpressGatewayService",
+          "CreateGateway",
           "CreateService",
           "CreateTaskSet",
           "DeleteAccountSetting",
@@ -2072,6 +2096,7 @@
           "DeleteDaemon",
           "DeleteDaemonTaskDefinition",
           "DeleteExpressGatewayService",
+          "DeleteGateway",
           "DeleteService",
           "DeleteTaskDefinitions",
           "DeleteTaskSet",
@@ -2085,12 +2110,16 @@
           "DescribeDaemonRevisions",
           "DescribeDaemonTaskDefinition",
           "DescribeExpressGatewayService",
+          "DescribeGateway",
+          "DescribeGatewayResources",
+          "DescribeGatewayRoute",
           "DescribeServiceDeployments",
           "DescribeServiceRevisions",
           "DescribeServices",
           "DescribeTaskDefinition",
           "DescribeTaskSets",
           "DescribeTasks",
+          "DisassociateGatewayFromCluster",
           "DiscoverPollEndpoint",
           "ExecuteCommand",
           "GetTaskProtection",
@@ -2101,6 +2130,9 @@
           "ListDaemonDeployments",
           "ListDaemonTaskDefinitions",
           "ListDaemons",
+          "ListGatewayClusterAssociations",
+          "ListGatewayRoutes",
+          "ListGateways",
           "ListServiceDeployments",
           "ListServices",
           "ListServicesByNamespace",
@@ -2131,6 +2163,7 @@
           "UpdateContainerInstancesState",
           "UpdateDaemon",
           "UpdateExpressGatewayService",
+          "UpdateGateway",
           "UpdateService",
           "UpdateServicePrimaryTaskSet",
           "UpdateTaskProtection",
@@ -2219,15 +2252,18 @@
         ],
         "unsupported": [],
         "missing": [
+          "AssociateGatewayWithCluster",
           "ContinueServiceDeployment",
           "CreateDaemon",
           "CreateExpressGatewayService",
+          "CreateGateway",
           "CreateTaskSet",
           "DeleteAccountSetting",
           "DeleteAttributes",
           "DeleteDaemon",
           "DeleteDaemonTaskDefinition",
           "DeleteExpressGatewayService",
+          "DeleteGateway",
           "DeleteTaskDefinitions",
           "DeleteTaskSet",
           "DescribeDaemon",
@@ -2235,9 +2271,13 @@
           "DescribeDaemonRevisions",
           "DescribeDaemonTaskDefinition",
           "DescribeExpressGatewayService",
+          "DescribeGateway",
+          "DescribeGatewayResources",
+          "DescribeGatewayRoute",
           "DescribeServiceDeployments",
           "DescribeServiceRevisions",
           "DescribeTaskSets",
+          "DisassociateGatewayFromCluster",
           "DiscoverPollEndpoint",
           "ExecuteCommand",
           "GetTaskProtection",
@@ -2245,6 +2285,9 @@
           "ListDaemonDeployments",
           "ListDaemonTaskDefinitions",
           "ListDaemons",
+          "ListGatewayClusterAssociations",
+          "ListGatewayRoutes",
+          "ListGateways",
           "ListServiceDeployments",
           "PutAccountSettingDefault",
           "PutAttributes",
@@ -2257,6 +2300,7 @@
           "UpdateContainerAgent",
           "UpdateDaemon",
           "UpdateExpressGatewayService",
+          "UpdateGateway",
           "UpdateServicePrimaryTaskSet",
           "UpdateTaskProtection",
           "UpdateTaskSet"
@@ -2894,7 +2938,8 @@
           "UpdateAccessKey",
           "UpdateAssumeRolePolicy",
           "UpdateRole",
-          "UpdateRoleDescription"
+          "UpdateRoleDescription",
+          "UpdateUser"
         ],
         "implemented": [
           "AddRoleToInstanceProfile",
@@ -2977,7 +3022,8 @@
           "UpdateAccessKey",
           "UpdateAssumeRolePolicy",
           "UpdateRole",
-          "UpdateRoleDescription"
+          "UpdateRoleDescription",
+          "UpdateUser"
         ],
         "stubbed": [],
         "unsupported": [],
@@ -3077,7 +3123,6 @@
           "UpdateServerCertificate",
           "UpdateServiceSpecificCredential",
           "UpdateSigningCertificate",
-          "UpdateUser",
           "UploadSSHPublicKey",
           "UploadServerCertificate",
           "UploadSigningCertificate"
@@ -3099,6 +3144,7 @@
           "ApplyPendingMaintenanceAction",
           "AuthorizeDBSecurityGroupIngress",
           "BacktrackDBCluster",
+          "CancelDBCapacityReservation",
           "CancelExportTask",
           "CopyDBClusterParameterGroup",
           "CopyDBClusterSnapshot",
@@ -3107,6 +3153,7 @@
           "CopyOptionGroup",
           "CreateBlueGreenDeployment",
           "CreateCustomDBEngineVersion",
+          "CreateDBCapacityReservation",
           "CreateDBCluster",
           "CreateDBClusterEndpoint",
           "CreateDBClusterParameterGroup",
@@ -3150,6 +3197,7 @@
           "DescribeAccountAttributes",
           "DescribeBlueGreenDeployments",
           "DescribeCertificates",
+          "DescribeDBCapacityReservations",
           "DescribeDBClusterAutomatedBackups",
           "DescribeDBClusterBacktracks",
           "DescribeDBClusterEndpoints",
@@ -3204,6 +3252,7 @@
           "ModifyCertificates",
           "ModifyCurrentDBClusterCapacity",
           "ModifyCustomDBEngineVersion",
+          "ModifyDBCapacityReservation",
           "ModifyDBCluster",
           "ModifyDBClusterEndpoint",
           "ModifyDBClusterParameterGroup",
@@ -3354,6 +3403,7 @@
           "ApplyPendingMaintenanceAction",
           "AuthorizeDBSecurityGroupIngress",
           "BacktrackDBCluster",
+          "CancelDBCapacityReservation",
           "CancelExportTask",
           "CopyDBClusterParameterGroup",
           "CopyDBClusterSnapshot",
@@ -3362,6 +3412,7 @@
           "CopyOptionGroup",
           "CreateBlueGreenDeployment",
           "CreateCustomDBEngineVersion",
+          "CreateDBCapacityReservation",
           "CreateDBClusterEndpoint",
           "CreateDBClusterParameterGroup",
           "CreateDBClusterSnapshot",
@@ -3392,6 +3443,7 @@
           "DescribeAccountAttributes",
           "DescribeBlueGreenDeployments",
           "DescribeCertificates",
+          "DescribeDBCapacityReservations",
           "DescribeDBClusterAutomatedBackups",
           "DescribeDBClusterBacktracks",
           "DescribeDBClusterEndpoints",
@@ -3433,6 +3485,7 @@
           "ModifyCertificates",
           "ModifyCurrentDBClusterCapacity",
           "ModifyCustomDBEngineVersion",
+          "ModifyDBCapacityReservation",
           "ModifyDBClusterEndpoint",
           "ModifyDBClusterParameterGroup",
           "ModifyDBClusterSnapshotAttribute",
@@ -3483,6 +3536,180 @@
         ]
       }
     },
+    {
+      "service": "route53",
+      "api_version": "2013-04-01",
+      "opaque": false,
+      "note": "",
+      "operations": {
+        "modelled": [
+          "ActivateKeySigningKey",
+          "AssociateVPCWithHostedZone",
+          "ChangeCidrCollection",
+          "ChangeResourceRecordSets",
+          "ChangeTagsForResource",
+          "CreateCidrCollection",
+          "CreateHealthCheck",
+          "CreateHostedZone",
+          "CreateKeySigningKey",
+          "CreateQueryLoggingConfig",
+          "CreateReusableDelegationSet",
+          "CreateTrafficPolicy",
+          "CreateTrafficPolicyInstance",
+          "CreateTrafficPolicyVersion",
+          "CreateVPCAssociationAuthorization",
+          "DeactivateKeySigningKey",
+          "DeleteCidrCollection",
+          "DeleteHealthCheck",
+          "DeleteHostedZone",
+          "DeleteKeySigningKey",
+          "DeleteQueryLoggingConfig",
+          "DeleteReusableDelegationSet",
+          "DeleteTrafficPolicy",
+          "DeleteTrafficPolicyInstance",
+          "DeleteVPCAssociationAuthorization",
+          "DisableHostedZoneDNSSEC",
+          "DisassociateVPCFromHostedZone",
+          "EnableHostedZoneDNSSEC",
+          "GetAccountLimit",
+          "GetChange",
+          "GetCheckerIpRanges",
+          "GetDNSSEC",
+          "GetGeoLocation",
+          "GetHealthCheck",
+          "GetHealthCheckCount",
+          "GetHealthCheckLastFailureReason",
+          "GetHealthCheckStatus",
+          "GetHostedZone",
+          "GetHostedZoneCount",
+          "GetHostedZoneLimit",
+          "GetQueryLoggingConfig",
+          "GetReusableDelegationSet",
+          "GetReusableDelegationSetLimit",
+          "GetTrafficPolicy",
+          "GetTrafficPolicyInstance",
+          "GetTrafficPolicyInstanceCount",
+          "ListCidrBlocks",
+          "ListCidrCollections",
+          "ListCidrLocations",
+          "ListGeoLocations",
+          "ListHealthChecks",
+          "ListHostedZones",
+          "ListHostedZonesByName",
+          "ListHostedZonesByVPC",
+          "ListQueryLoggingConfigs",
+          "ListResourceRecordSets",
+          "ListReusableDelegationSets",
+          "ListTagsForResource",
+          "ListTagsForResources",
+          "ListTrafficPolicies",
+          "ListTrafficPolicyInstances",
+          "ListTrafficPolicyInstancesByHostedZone",
+          "ListTrafficPolicyInstancesByPolicy",
+          "ListTrafficPolicyVersions",
+          "ListVPCAssociationAuthorizations",
+          "TestDNSAnswer",
+          "UpdateHealthCheck",
+          "UpdateHostedZoneComment",
+          "UpdateHostedZoneFeatures",
+          "UpdateTrafficPolicyComment",
+          "UpdateTrafficPolicyInstance"
+        ],
+        "registered": [
+          "ChangeResourceRecordSets",
+          "ChangeTagsForResource",
+          "CreateHostedZone",
+          "DeleteHostedZone",
+          "GetChange",
+          "GetHostedZone",
+          "GetHostedZoneCount",
+          "ListHostedZones",
+          "ListHostedZonesByName",
+          "ListResourceRecordSets",
+          "ListTagsForResource",
+          "ListTagsForResources",
+          "UpdateHostedZoneComment"
+        ],
+        "implemented": [
+          "ChangeResourceRecordSets",
+          "ChangeTagsForResource",
+          "CreateHostedZone",
+          "DeleteHostedZone",
+          "GetChange",
+          "GetHostedZone",
+          "GetHostedZoneCount",
+          "ListHostedZones",
+          "ListHostedZonesByName",
+          "ListResourceRecordSets",
+          "ListTagsForResource",
+          "ListTagsForResources",
+          "UpdateHostedZoneComment"
+        ],
+        "stubbed": [],
+        "unsupported": [],
+        "missing": [
+          "ActivateKeySigningKey",
+          "AssociateVPCWithHostedZone",
+          "ChangeCidrCollection",
+          "CreateCidrCollection",
+          "CreateHealthCheck",
+          "CreateKeySigningKey",
+          "CreateQueryLoggingConfig",
+          "CreateReusableDelegationSet",
+          "CreateTrafficPolicy",
+          "CreateTrafficPolicyInstance",
+          "CreateTrafficPolicyVersion",
+          "CreateVPCAssociationAuthorization",
+          "DeactivateKeySigningKey",
+          "DeleteCidrCollection",
+          "DeleteHealthCheck",
+          "DeleteKeySigningKey",
+          "DeleteQueryLoggingConfig",
+          "DeleteReusableDelegationSet",
+          "DeleteTrafficPolicy",
+          "DeleteTrafficPolicyInstance",
+          "DeleteVPCAssociationAuthorization",
+          "DisableHostedZoneDNSSEC",
+          "DisassociateVPCFromHostedZone",
+          "EnableHostedZoneDNSSEC",
+          "GetAccountLimit",
+          "GetCheckerIpRanges",
+          "GetDNSSEC",
+          "GetGeoLocation",
+          "GetHealthCheck",
+          "GetHealthCheckCount",
+          "GetHealthCheckLastFailureReason",
+          "GetHealthCheckStatus",
+          "GetHostedZoneLimit",
+          "GetQueryLoggingConfig",
+          "GetReusableDelegationSet",
+          "GetReusableDelegationSetLimit",
+          "GetTrafficPolicy",
+          "GetTrafficPolicyInstance",
+          "GetTrafficPolicyInstanceCount",
+          "ListCidrBlocks",
+          "ListCidrCollections",
+          "ListCidrLocations",
+          "ListGeoLocations",
+          "ListHealthChecks",
+          "ListHostedZonesByVPC",
+          "ListQueryLoggingConfigs",
+          "ListReusableDelegationSets",
+          "ListTrafficPolicies",
+          "ListTrafficPolicyInstances",
+          "ListTrafficPolicyInstancesByHostedZone",
+          "ListTrafficPolicyInstancesByPolicy",
+          "ListTrafficPolicyVersions",
+          "ListVPCAssociationAuthorizations",
+          "TestDNSAnswer",
+          "UpdateHealthCheck",
+          "UpdateHostedZoneFeatures",
+          "UpdateTrafficPolicyComment",
+          "UpdateTrafficPolicyInstance"
+        ],
+        "extra": []
+      }
+    },
     {
       "service": "s3",
       "api_version": "2006-03-01",
@@ -3752,6 +3979,86 @@
         "extra": []
       }
     },
+    {
+      "service": "sqs",
+      "api_version": "2012-11-05",
+      "opaque": false,
+      "note": "",
+      "operations": {
+        "modelled": [
+          "AddPermission",
+          "CancelMessageMoveTask",
+          "ChangeMessageVisibility",
+          "ChangeMessageVisibilityBatch",
+          "CreateQueue",
+          "DeleteMessage",
+          "DeleteMessageBatch",
+          "DeleteQueue",
+          "GetQueueAttributes",
+          "GetQueueUrl",
+          "ListDeadLetterSourceQueues",
+          "ListMessageMoveTasks",
+          "ListQueueTags",
+          "ListQueues",
+          "PurgeQueue",
+          "ReceiveMessage",
+          "RemovePermission",
+          "SendMessage",
+          "SendMessageBatch",
+          "SetQueueAttributes",
+          "StartMessageMoveTask",
+          "TagQueue",
+          "UntagQueue"
+        ],
+        "registered": [
+          "ChangeMessageVisibility",
+          "ChangeMessageVisibilityBatch",
+          "CreateQueue",
+          "DeleteMessage",
+          "DeleteMessageBatch",
+          "GetQueueAttributes",
+          "GetQueueUrl",
+          "ListQueueTags",
+          "ListQueues",
+          "ReceiveMessage",
+          "SendMessage",
+          "SendMessageBatch",
+          "SetQueueAttributes",
+          "TagQueue",
+          "UntagQueue"
+        ],
+        "implemented": [
+          "ChangeMessageVisibility",
+          "ChangeMessageVisibilityBatch",
+          "CreateQueue",
+          "DeleteMessage",
+          "DeleteMessageBatch",
+          "GetQueueAttributes",
+          "GetQueueUrl",
+          "ListQueueTags",
+          "ListQueues",
+          "ReceiveMessage",
+          "SendMessage",
+          "SendMessageBatch",
+          "SetQueueAttributes",
+          "TagQueue",
+          "UntagQueue"
+        ],
+        "stubbed": [],
+        "unsupported": [],
+        "missing": [
+          "AddPermission",
+          "CancelMessageMoveTask",
+          "DeleteQueue",
+          "ListDeadLetterSourceQueues",
+          "ListMessageMoveTasks",
+          "PurgeQueue",
+          "RemovePermission",
+          "StartMessageMoveTask"
+        ],
+        "extra": []
+      }
+    },
     {
       "service": "sts",
       "api_version": "2011-06-15",

docs/coverage/ec2/README.md

@@ -28,6 +28,21 @@ Spot Instance Requests are a mock over the on-demand `RunInstances` path. A requ
 
 An AMI cannot be shared with another account or made public. `ModifyImageAttribute` refuses `LaunchPermission`, `UserIds`, `UserGroups`, `OrganizationArns`, `OrganizationalUnitArns` and `OperationType`, and `ResetImageAttribute` resets only `description`, not `launchPermission`. Each refusal is `InvalidParameterValue`. Terraform's `aws_ami_launch_permission` therefore cannot be created. `description` is the only image attribute that can be modified or reset.
 
+### RegisterImage Registers From a Snapshot
+
+`RegisterImage` takes a root EBS snapshot in `BlockDeviceMappings` and writes an image that points at it.
+`EnaSupport` is supported, with `true` and `false` both accepted and stored on the image.
+`DescribeImages` returns `EnaSupport` when the image has one and leaves the member out when it was never set, as AWS does.
+`CopyImage` carries the value to the copy, and `CreateImage` sets it to `true` because every instance reports ENA support.
+`RunInstances` does not read the flag, so an image registered with `EnaSupport=false` is not held back from instance types that need ENA on AWS, since guests use virtio networking either way.
+
+`RegisterImage` refuses `ImageLocation`, `KernelId`, `RamdiskId`, `TpmSupport`, `ImdsSupport` and `SriovNetSupport`, and any `VirtualizationType` other than `hvm`.
+Each refusal is `InvalidParameterValue`, and the message names the parameter and the value, for example `Value (aki-123) for parameter KernelId is invalid. KernelId is not supported.`
+The same form is used for a bad `Architecture`, `BootMode` or `Name` length.
+A missing `Name`, `BlockDeviceMappings` or root snapshot answers `MissingParameter` and names what is missing.
+Registering with `--ena-support`, `--no-ena-support` and neither, copying the image, and three refusals were run on a three-node cluster with the AWS CLI.
+Launching an instance from an `EnaSupport` image was not run, so the launch path is unchanged by construction and not by test.
+
 ### Operations
 
 | Operation |

docs/coverage/ec2/intro.md

@@ -5,3 +5,18 @@ Spot Instance Requests are a mock over the on-demand `RunInstances` path. A requ
 ### AMI Launch Permissions Are Refused
 
 An AMI cannot be shared with another account or made public. `ModifyImageAttribute` refuses `LaunchPermission`, `UserIds`, `UserGroups`, `OrganizationArns`, `OrganizationalUnitArns` and `OperationType`, and `ResetImageAttribute` resets only `description`, not `launchPermission`. Each refusal is `InvalidParameterValue`. Terraform's `aws_ami_launch_permission` therefore cannot be created. `description` is the only image attribute that can be modified or reset.
+
+### RegisterImage Registers From a Snapshot
+
+`RegisterImage` takes a root EBS snapshot in `BlockDeviceMappings` and writes an image that points at it.
+`EnaSupport` is supported, with `true` and `false` both accepted and stored on the image.
+`DescribeImages` returns `EnaSupport` when the image has one and leaves the member out when it was never set, as AWS does.
+`CopyImage` carries the value to the copy, and `CreateImage` sets it to `true` because every instance reports ENA support.
+`RunInstances` does not read the flag, so an image registered with `EnaSupport=false` is not held back from instance types that need ENA on AWS, since guests use virtio networking either way.
+
+`RegisterImage` refuses `ImageLocation`, `KernelId`, `RamdiskId`, `TpmSupport`, `ImdsSupport` and `SriovNetSupport`, and any `VirtualizationType` other than `hvm`.
+Each refusal is `InvalidParameterValue`, and the message names the parameter and the value, for example `Value (aki-123) for parameter KernelId is invalid. KernelId is not supported.`
+The same form is used for a bad `Architecture`, `BootMode` or `Name` length.
+A missing `Name`, `BlockDeviceMappings` or root snapshot answers `MissingParameter` and names what is missing.
+Registering with `--ena-support`, `--no-ena-support` and neither, copying the image, and three refusals were run on a three-node cluster with the AWS CLI.
+Launching an instance from an `EnaSupport` image was not run, so the launch path is unchanged by construction and not by test.

docs/coverage/elbv2/README.md

@@ -24,6 +24,74 @@ Spinifex implements **36 operations** in the ELBv2 `2015-12-01` API model.
 
 The data plane is a system-managed load balancer VM, launched automatically during `CreateLoadBalancer`. Application Load Balancers run HAProxy for the L7 surface — rules, fixed responses and redirects over HTTP and HTTPS. Network Load Balancers run nginx `stream` for the L4 surface — TCP, UDP, TLS and TCP_UDP — because HAProxy cannot load-balance UDP. The agent selects the engine from the configuration the control plane delivers, so the choice follows the load balancer type and is not separately configurable.
 
+### Target health and routing
+
+Traffic goes only to targets that are healthy or still `initial`.
+A target that has failed its health check is removed from the data plane on the next render and put back when it passes again.
+When every target in a group is unhealthy, the load balancer fails open and sends traffic to all of them rather than refusing it.
+A Network Load Balancer renders every registered server in that case, and an Application Load Balancer falls back to unchecked backup servers that HAProxy uses only when every checked server is down.
+`DescribeTargetHealth` keeps reporting the targets as unhealthy while this happens.
+Draining targets and targets without an address never receive traffic.
+
+An `initial` target receives traffic before its first successful check.
+This differs from AWS, where a target takes no traffic until it passes.
+Health state is reset to `initial` for every target when the control plane restarts, so excluding `initial` targets would drop all traffic after a restart.
+
+Changing the health check of a target group with `ModifyTargetGroup` takes effect on the next render of every load balancer that uses it.
+The time from a backend failing to traffic stopping is the health check interval times the unhealthy threshold, plus the agent poll of about 5 seconds and the heartbeat that reports the result.
+Recovery takes the interval times the healthy threshold, plus the same delays.
+
+Measured on a three-node cluster with a 5 second interval and thresholds of 2:
+
+| Event | Time |
+| --- | --- |
+| Backend starts failing to reported `unhealthy` | 21 s |
+| Backend recovers to reported `healthy` | 15 s, serving again 6 s later |
+| Every backend failing to reported `unhealthy` | 12 s, fail open immediate |
+
+A TCP connection held through a Network Load Balancer survived four configuration reloads while the other backend changed state, because the agent applies nginx with a graceful reload and HAProxy with a graceful restart.
+The HAProxy configuration with the backup servers was accepted by a live Application Load Balancer, but its rendered text was not dumped and checked by hand.
+
+### Forwarding headers
+
+An Application Load Balancer sets three headers on every HTTP and HTTPS request it forwards.
+`X-Forwarded-For` carries the client address, `X-Forwarded-Proto` is `http` or `https` for the listener that accepted the request, and `X-Forwarded-Port` is that listener's port.
+`X-Forwarded-Proto` and `X-Forwarded-Port` are always overwritten, so a value sent by the client never reaches the target.
+Network Load Balancers do not touch the stream.
+
+`routing.http.xff_header_processing.mode` decides what happens to `X-Forwarded-For`:
+
+| Mode | Behaviour |
+| --- | --- |
+| `append` (default) | Keeps what the client sent and adds the client address last, as one comma-separated line such as `203.0.113.9, 192.0.2.10`. |
+| `preserve` | Forwards the client's value unchanged and adds nothing. The header is absent when the client sent none. |
+| `remove` | Drops the header, including any client value. |
+
+`routing.http.xff_client_port.enabled` (default `false`) appends the client's source port in `append` mode, as `192.0.2.10:54321`.
+An IPv6 client is written `[2001:db8::1]:54321`.
+The setting has no effect in `preserve` and `remove` modes.
+
+`ModifyLoadBalancerAttributes` accepts only `append`, `preserve` or `remove` for the mode and `true` or `false` for the client port, and rejects anything else with `ValidationError`.
+A change re-renders the load balancer configuration, and a call that leaves both values as they were does not.
+In a live test each change reached the target within one 3 second poll.
+
+A client that sends `X-Forwarded-For` more than once has its lines joined into the single comma-separated value.
+At most eight client lines are joined, and any beyond the eighth are dropped, while the client address is still added last.
+
+Every other `routing.http.*` attribute is stored and returned by `DescribeLoadBalancerAttributes` but has no effect on the data plane.
+
+Measured on a three-node cluster, with the caller on the first node:
+
+| Listener | `X-Forwarded-For` the target saw |
+| --- | --- |
+| Internet-facing HTTP | The node's real address, `192.168.0.120` |
+| Internet-facing HTTPS | The same address, with `https` and port 443 |
+| Internal HTTP | The calling instance's private address |
+| Internet-facing HTTP with a client value of `203.0.113.9` | `203.0.113.9, 192.168.0.120` on one line |
+
+A packet capture on the load balancer's interface showed HAProxy receiving the real client address, so nothing translates it first.
+Behaviour against AWS was not compared, and the cases above were not run with an IPv6 client.
+
 ### Operations
 
 | Operation |

docs/coverage/elbv2/intro.md

@@ -1,3 +1,71 @@
 ### Two data planes
 
 The data plane is a system-managed load balancer VM, launched automatically during `CreateLoadBalancer`. Application Load Balancers run HAProxy for the L7 surface — rules, fixed responses and redirects over HTTP and HTTPS. Network Load Balancers run nginx `stream` for the L4 surface — TCP, UDP, TLS and TCP_UDP — because HAProxy cannot load-balance UDP. The agent selects the engine from the configuration the control plane delivers, so the choice follows the load balancer type and is not separately configurable.
+
+### Target health and routing
+
+Traffic goes only to targets that are healthy or still `initial`.
+A target that has failed its health check is removed from the data plane on the next render and put back when it passes again.
+When every target in a group is unhealthy, the load balancer fails open and sends traffic to all of them rather than refusing it.
+A Network Load Balancer renders every registered server in that case, and an Application Load Balancer falls back to unchecked backup servers that HAProxy uses only when every checked server is down.
+`DescribeTargetHealth` keeps reporting the targets as unhealthy while this happens.
+Draining targets and targets without an address never receive traffic.
+
+An `initial` target receives traffic before its first successful check.
+This differs from AWS, where a target takes no traffic until it passes.
+Health state is reset to `initial` for every target when the control plane restarts, so excluding `initial` targets would drop all traffic after a restart.
+
+Changing the health check of a target group with `ModifyTargetGroup` takes effect on the next render of every load balancer that uses it.
+The time from a backend failing to traffic stopping is the health check interval times the unhealthy threshold, plus the agent poll of about 5 seconds and the heartbeat that reports the result.
+Recovery takes the interval times the healthy threshold, plus the same delays.
+
+Measured on a three-node cluster with a 5 second interval and thresholds of 2:
+
+| Event | Time |
+| --- | --- |
+| Backend starts failing to reported `unhealthy` | 21 s |
+| Backend recovers to reported `healthy` | 15 s, serving again 6 s later |
+| Every backend failing to reported `unhealthy` | 12 s, fail open immediate |
+
+A TCP connection held through a Network Load Balancer survived four configuration reloads while the other backend changed state, because the agent applies nginx with a graceful reload and HAProxy with a graceful restart.
+The HAProxy configuration with the backup servers was accepted by a live Application Load Balancer, but its rendered text was not dumped and checked by hand.
+
+### Forwarding headers
+
+An Application Load Balancer sets three headers on every HTTP and HTTPS request it forwards.
+`X-Forwarded-For` carries the client address, `X-Forwarded-Proto` is `http` or `https` for the listener that accepted the request, and `X-Forwarded-Port` is that listener's port.
+`X-Forwarded-Proto` and `X-Forwarded-Port` are always overwritten, so a value sent by the client never reaches the target.
+Network Load Balancers do not touch the stream.
+
+`routing.http.xff_header_processing.mode` decides what happens to `X-Forwarded-For`:
+
+| Mode | Behaviour |
+| --- | --- |
+| `append` (default) | Keeps what the client sent and adds the client address last, as one comma-separated line such as `203.0.113.9, 192.0.2.10`. |
+| `preserve` | Forwards the client's value unchanged and adds nothing. The header is absent when the client sent none. |
+| `remove` | Drops the header, including any client value. |
+
+`routing.http.xff_client_port.enabled` (default `false`) appends the client's source port in `append` mode, as `192.0.2.10:54321`.
+An IPv6 client is written `[2001:db8::1]:54321`.
+The setting has no effect in `preserve` and `remove` modes.
+
+`ModifyLoadBalancerAttributes` accepts only `append`, `preserve` or `remove` for the mode and `true` or `false` for the client port, and rejects anything else with `ValidationError`.
+A change re-renders the load balancer configuration, and a call that leaves both values as they were does not.
+In a live test each change reached the target within one 3 second poll.
+
+A client that sends `X-Forwarded-For` more than once has its lines joined into the single comma-separated value.
+At most eight client lines are joined, and any beyond the eighth are dropped, while the client address is still added last.
+
+Every other `routing.http.*` attribute is stored and returned by `DescribeLoadBalancerAttributes` but has no effect on the data plane.
+
+Measured on a three-node cluster, with the caller on the first node:
+
+| Listener | `X-Forwarded-For` the target saw |
+| --- | --- |
+| Internet-facing HTTP | The node's real address, `192.168.0.120` |
+| Internet-facing HTTPS | The same address, with `https` and port 443 |
+| Internal HTTP | The calling instance's private address |
+| Internet-facing HTTP with a client value of `203.0.113.9` | `203.0.113.9, 192.168.0.120` on one line |
+
+A packet capture on the load balancer's interface showed HAProxy receiving the real client address, so nothing translates it first.
+Behaviour against AWS was not compared, and the cases above were not run with an IPv6 client.

docs/coverage/iam/README.md

@@ -18,7 +18,7 @@ tags:
 
 ## Overview
 
-Spinifex implements **81 operations** in the IAM `2010-05-08` API model.
+Spinifex implements **82 operations** in the IAM `2010-05-08` API model.
 
 ### Scope
 
@@ -109,3 +109,4 @@ All IAM operations are account-scoped. The root user of an account bypasses poli
 | `UpdateAssumeRolePolicy` |
 | `UpdateRole` |
 | `UpdateRoleDescription` |
+| `UpdateUser` |

docs/coverage/internal-report.md

@@ -4,7 +4,7 @@ Generated by `make aws-model-coverage`. This file is not published: it carries t
 
 **Not implemented** is the candidate work. **Not applicable** is declared in `pages.json` and is not work: those operations describe something this platform will not have.
 
-Pinned model source: `github.com/aws/api-models-aws@9b1a49292744, 2026-09-25`.
+Pinned model source: `github.com/aws/api-models-aws@e10d6a1c5ac3, 2026-10-28`.
 
 ## acm
 
@@ -46,7 +46,7 @@ Pinned model source: `github.com/aws/api-models-aws@9b1a49292744, 2026-09-25`.
 
 ## ec2
 
-`2016-11-15` — 127 of 804 modelled operations implemented (15.8%), 0 stubbed, 39 not applicable, 0 registered outside the pinned model.
+`2016-11-15` — 127 of 811 modelled operations implemented (15.7%), 0 stubbed, 39 not applicable, 0 registered outside the pinned model.
 
 ### Not implemented
 
@@ -190,6 +190,7 @@ Pinned model source: `github.com/aws/api-models-aws@9b1a49292744, 2026-09-25`.
 - `DeleteApplicationStatusCheck`
 - `DeleteCapacityManagerDataExport`
 - `DeleteClientVpnEndpoint`
+- `DeleteClientVpnEndpointAuthorizationPolicy`
 - `DeleteClientVpnRoute`
 - `DeleteCoipCidr`
 - `DeleteCoipPool`
@@ -484,7 +485,9 @@ Pinned model source: `github.com/aws/api-models-aws@9b1a49292744, 2026-09-25`.
 - `GetCapacityManagerMetricData`
 - `GetCapacityManagerMetricDimensions`
 - `GetCapacityManagerMonitoredTagKeys`
+- `GetCapacityManagerSupportabilityScores`
 - `GetCapacityReservationUsage`
+- `GetClientVpnEndpointAuthorizationPolicy`
 - `GetCoipPoolUsage`
 - `GetConsoleScreenshot`
 - `GetDeclarativePoliciesReportSummary`
@@ -496,6 +499,7 @@ Pinned model source: `github.com/aws/api-models-aws@9b1a49292744, 2026-09-25`.
 - `GetHostReservationPurchasePreview`
 - `GetImageAncestry`
 - `GetImageBlockPublicAccessState`
+- `GetImageFindings`
 - `GetInstanceMetadataDefaults`
 - `GetInstanceTpmEkPub`
 - `GetInstanceTypesFromInstanceRequirements`
@@ -532,6 +536,7 @@ Pinned model source: `github.com/aws/api-models-aws@9b1a49292744, 2026-09-25`.
 - `GetSnapshotBlockPublicAccessState`
 - `GetSpotPlacementScores`
 - `GetSubnetCidrReservations`
+- `GetTransitGatewayAttachmentNatConfiguration`
 - `GetTransitGatewayAttachmentPropagations`
 - `GetTransitGatewayMeteringPolicyEntries`
 - `GetTransitGatewayMulticastDomainAssociations`
@@ -563,6 +568,7 @@ Pinned model source: `github.com/aws/api-models-aws@9b1a49292744, 2026-09-25`.
 - `ModifyCapacityReservation`
 - `ModifyCapacityReservationFleet`
 - `ModifyClientVpnEndpoint`
+- `ModifyClientVpnEndpointAuthorizationPolicy`
 - `ModifyDefaultCreditSpecification`
 - `ModifyEbsDefaultKmsKeyId`
 - `ModifyFleet`
@@ -601,6 +607,7 @@ Pinned model source: `github.com/aws/api-models-aws@9b1a49292744, 2026-09-25`.
 - `ModifyTrafficMirrorFilterRule`
 - `ModifyTrafficMirrorSession`
 - `ModifyTransitGateway`
+- `ModifyTransitGatewayAttachmentNatConfiguration`
 - `ModifyTransitGatewayMeteringPolicy`
 - `ModifyTransitGatewayPolicyTableEntry`
 - `ModifyTransitGatewayPrefixListReference`
@@ -762,7 +769,7 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
 
 ## ecr
 
-`2015-09-21` — 23 of 58 modelled operations implemented (39.7%), 10 stubbed, 0 not applicable, 3 registered outside the pinned model.
+`2015-09-21` — 23 of 62 modelled operations implemented (37.1%), 10 stubbed, 0 not applicable, 3 registered outside the pinned model.
 
 ### Not implemented
 
@@ -776,14 +783,18 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
 - `DescribeImageReplicationStatus`
 - `DescribeImageSigningStatus`
 - `DescribePullThroughCacheRules`
+- `DescribeReplicationBackfillResults`
 - `DescribeRepositoryCreationTemplates`
 - `GetAccountSetting`
 - `GetSigningConfiguration`
 - `ListImageReferrers`
 - `ListPullTimeUpdateExclusions`
+- `ListReplicationBackfills`
 - `PutAccountSetting`
 - `PutSigningConfiguration`
 - `RegisterPullTimeUpdateExclusion`
+- `StartReplicationBackfill`
+- `StopReplicationBackfill`
 - `UpdateImageStorageClass`
 - `UpdatePullThroughCacheRule`
 - `UpdateRepositoryCreationTemplate`
@@ -819,19 +830,22 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
 
 ## ecs
 
-`2014-11-13` — 31 of 77 modelled operations implemented (40.3%), 5 stubbed, 0 not applicable, 3 registered outside the pinned model.
+`2014-11-13` — 31 of 88 modelled operations implemented (35.2%), 5 stubbed, 0 not applicable, 3 registered outside the pinned model.
 
 ### Not implemented
 
+- `AssociateGatewayWithCluster`
 - `ContinueServiceDeployment`
 - `CreateDaemon`
 - `CreateExpressGatewayService`
+- `CreateGateway`
 - `CreateTaskSet`
 - `DeleteAccountSetting`
 - `DeleteAttributes`
 - `DeleteDaemon`
 - `DeleteDaemonTaskDefinition`
 - `DeleteExpressGatewayService`
+- `DeleteGateway`
 - `DeleteTaskDefinitions`
 - `DeleteTaskSet`
 - `DescribeDaemon`
@@ -839,9 +853,13 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
 - `DescribeDaemonRevisions`
 - `DescribeDaemonTaskDefinition`
 - `DescribeExpressGatewayService`
+- `DescribeGateway`
+- `DescribeGatewayResources`
+- `DescribeGatewayRoute`
 - `DescribeServiceDeployments`
 - `DescribeServiceRevisions`
 - `DescribeTaskSets`
+- `DisassociateGatewayFromCluster`
 - `DiscoverPollEndpoint`
 - `ExecuteCommand`
 - `GetTaskProtection`
@@ -849,6 +867,9 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
 - `ListDaemonDeployments`
 - `ListDaemonTaskDefinitions`
 - `ListDaemons`
+- `ListGatewayClusterAssociations`
+- `ListGatewayRoutes`
+- `ListGateways`
 - `ListServiceDeployments`
 - `PutAccountSettingDefault`
 - `PutAttributes`
@@ -861,6 +882,7 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
 - `UpdateContainerAgent`
 - `UpdateDaemon`
 - `UpdateExpressGatewayService`
+- `UpdateGateway`
 - `UpdateServicePrimaryTaskSet`
 - `UpdateTaskProtection`
 - `UpdateTaskSet`
@@ -965,7 +987,7 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
 
 ## iam
 
-`2010-05-08` — 81 of 180 modelled operations implemented (45.0%), 0 stubbed, 19 not applicable, 0 registered outside the pinned model.
+`2010-05-08` — 82 of 180 modelled operations implemented (45.6%), 0 stubbed, 19 not applicable, 0 registered outside the pinned model.
 
 ### Not implemented
 
@@ -1047,7 +1069,6 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
 - `UpdateOpenIDConnectProviderThumbprint`
 - `UpdateSAMLProvider`
 - `UpdateServerCertificate`
-- `UpdateUser`
 - `UploadServerCertificate`
 
 ### Not applicable — do not build
@@ -1085,7 +1106,7 @@ signing-certificates: X.509 signing certificates are an EC2-Classic SOAP credent
 
 ## rds
 
-`2014-10-31` — 27 of 164 modelled operations implemented (16.5%), 0 stubbed, 4 not applicable, 5 registered outside the pinned model.
+`2014-10-31` — 27 of 168 modelled operations implemented (16.1%), 0 stubbed, 4 not applicable, 5 registered outside the pinned model.
 
 ### Not implemented
 
@@ -1095,6 +1116,7 @@ signing-certificates: X.509 signing certificates are an EC2-Classic SOAP credent
 - `ApplyPendingMaintenanceAction`
 - `AuthorizeDBSecurityGroupIngress`
 - `BacktrackDBCluster`
+- `CancelDBCapacityReservation`
 - `CancelExportTask`
 - `CopyDBClusterParameterGroup`
 - `CopyDBClusterSnapshot`
@@ -1103,6 +1125,7 @@ signing-certificates: X.509 signing certificates are an EC2-Classic SOAP credent
 - `CopyOptionGroup`
 - `CreateBlueGreenDeployment`
 - `CreateCustomDBEngineVersion`
+- `CreateDBCapacityReservation`
 - `CreateDBClusterEndpoint`
 - `CreateDBClusterParameterGroup`
 - `CreateDBClusterSnapshot`
@@ -1133,6 +1156,7 @@ signing-certificates: X.509 signing certificates are an EC2-Classic SOAP credent
 - `DescribeAccountAttributes`
 - `DescribeBlueGreenDeployments`
 - `DescribeCertificates`
+- `DescribeDBCapacityReservations`
 - `DescribeDBClusterAutomatedBackups`
 - `DescribeDBClusterBacktracks`
 - `DescribeDBClusterEndpoints`
@@ -1174,6 +1198,7 @@ signing-certificates: X.509 signing certificates are an EC2-Classic SOAP credent
 - `ModifyCertificates`
 - `ModifyCurrentDBClusterCapacity`
 - `ModifyCustomDBEngineVersion`
+- `ModifyDBCapacityReservation`
 - `ModifyDBClusterEndpoint`
 - `ModifyDBClusterParameterGroup`
 - `ModifyDBClusterSnapshotAttribute`
@@ -1243,6 +1268,71 @@ option-groups: Option groups configure engine add-ons for engines this platform
 - `DescribeOptionGroups`
 - `ModifyOptionGroup`
 
+## route53
+
+`2013-04-01` — 13 of 71 modelled operations implemented (18.3%), 0 stubbed, 0 not applicable, 0 registered outside the pinned model.
+
+### Not implemented
+
+- `ActivateKeySigningKey`
+- `AssociateVPCWithHostedZone`
+- `ChangeCidrCollection`
+- `CreateCidrCollection`
+- `CreateHealthCheck`
+- `CreateKeySigningKey`
+- `CreateQueryLoggingConfig`
+- `CreateReusableDelegationSet`
+- `CreateTrafficPolicy`
+- `CreateTrafficPolicyInstance`
+- `CreateTrafficPolicyVersion`
+- `CreateVPCAssociationAuthorization`
+- `DeactivateKeySigningKey`
+- `DeleteCidrCollection`
+- `DeleteHealthCheck`
+- `DeleteKeySigningKey`
+- `DeleteQueryLoggingConfig`
+- `DeleteReusableDelegationSet`
+- `DeleteTrafficPolicy`
+- `DeleteTrafficPolicyInstance`
+- `DeleteVPCAssociationAuthorization`
+- `DisableHostedZoneDNSSEC`
+- `DisassociateVPCFromHostedZone`
+- `EnableHostedZoneDNSSEC`
+- `GetAccountLimit`
+- `GetCheckerIpRanges`
+- `GetDNSSEC`
+- `GetGeoLocation`
+- `GetHealthCheck`
+- `GetHealthCheckCount`
+- `GetHealthCheckLastFailureReason`
+- `GetHealthCheckStatus`
+- `GetHostedZoneLimit`
+- `GetQueryLoggingConfig`
+- `GetReusableDelegationSet`
+- `GetReusableDelegationSetLimit`
+- `GetTrafficPolicy`
+- `GetTrafficPolicyInstance`
+- `GetTrafficPolicyInstanceCount`
+- `ListCidrBlocks`
+- `ListCidrCollections`
+- `ListCidrLocations`
+- `ListGeoLocations`
+- `ListHealthChecks`
+- `ListHostedZonesByVPC`
+- `ListQueryLoggingConfigs`
+- `ListReusableDelegationSets`
+- `ListTrafficPolicies`
+- `ListTrafficPolicyInstances`
+- `ListTrafficPolicyInstancesByHostedZone`
+- `ListTrafficPolicyInstancesByPolicy`
+- `ListTrafficPolicyVersions`
+- `ListVPCAssociationAuthorizations`
+- `TestDNSAnswer`
+- `UpdateHealthCheck`
+- `UpdateHostedZoneFeatures`
+- `UpdateTrafficPolicyComment`
+- `UpdateTrafficPolicyInstance`
+
 ## s3
 
 `2006-03-01` — 26 of 112 modelled operations implemented (23.2%), 0 stubbed, 6 not applicable, 0 registered outside the pinned model.
@@ -1350,6 +1440,21 @@ request-payment: Requester Pays shifts transfer charges to the caller, which nee
 - `GetBucketRequestPayment`
 - `PutBucketRequestPayment`
 
+## sqs
+
+`2012-11-05` — 15 of 23 modelled operations implemented (65.2%), 0 stubbed, 0 not applicable, 0 registered outside the pinned model.
+
+### Not implemented
+
+- `AddPermission`
+- `CancelMessageMoveTask`
+- `DeleteQueue`
+- `ListDeadLetterSourceQueues`
+- `ListMessageMoveTasks`
+- `PurgeQueue`
+- `RemovePermission`
+- `StartMessageMoveTask`
+
 ## sts
 
 `2011-06-15` — 5 of 11 modelled operations implemented (45.5%), 0 stubbed, 0 not applicable, 0 registered outside the pinned model.

docs/coverage/pages.json

@@ -247,6 +247,20 @@
         "option-groups": "Option groups configure engine add-ons for engines this platform does not offer, such as Oracle and SQL Server."
       }
     },
+    "route53": {
+      "slug": "route53",
+      "name": "Route 53",
+      "title": "Route 53 API Coverage",
+      "seoTitle": "Amazon Route 53 API Coverage on Spinifex — Spinifex Docs",
+      "description": "The Amazon Route 53 API operations Spinifex implements, covering the hosted zones it serves, their resource record sets, change status and tags on each zone.",
+      "tags": [
+        "aws",
+        "compatibility",
+        "coverage",
+        "route53",
+        "dns"
+      ]
+    },
     "s3": {
       "slug": "s3",
       "name": "S3",

docs/coverage/route53/README.md

@@ -0,0 +1,38 @@
+---
+title: "Route 53 API Coverage"
+seoTitle: "Amazon Route 53 API Coverage on Spinifex — Spinifex Docs"
+description: "The Amazon Route 53 API operations Spinifex implements, covering the hosted zones it serves, their resource record sets, change status and tags on each zone."
+category: "Coverage"
+sections:
+  - overview
+tags:
+  - aws
+  - compatibility
+  - coverage
+  - route53
+  - dns
+---
+
+# Route 53 API Coverage
+
+## Overview
+
+Spinifex implements **13 operations** in the Route 53 `2013-04-01` API model.
+
+### Operations
+
+| Operation |
+|---|
+| `ChangeResourceRecordSets` |
+| `ChangeTagsForResource` |
+| `CreateHostedZone` |
+| `DeleteHostedZone` |
+| `GetChange` |
+| `GetHostedZone` |
+| `GetHostedZoneCount` |
+| `ListHostedZones` |
+| `ListHostedZonesByName` |
+| `ListResourceRecordSets` |
+| `ListTagsForResource` |
+| `ListTagsForResources` |
+| `UpdateHostedZoneComment` |

docs/install/install-multi-node/README.md

@@ -163,11 +163,14 @@ Open between all hosts:
 | Protocol and port | Used by |
 |---|---|
 | UDP 6081 | Geneve tunnels |
+| UDP 500, 4500 and IP protocol 50 (ESP) | IPsec for the Geneve tunnels, while IPsec is on |
 | TCP 4222, 4248 | NATS |
 | TCP 6641, 6642 | OVN northbound and southbound |
 | TCP 8443 | Predastore S3 gate |
 | UDP 6660, 7660 | Object shards, metadata consensus |
 
+UDP 4500 carries the tunnel traffic only where IKE detects NAT between two nodes; otherwise it travels as ESP, on Oracle Linux as on Debian.
+
 Open between the three OVN database servers (servers 1 to 3) only:
 
 | Protocol and port | Used by |
@@ -207,7 +210,10 @@ Spinifex's own **host** firewall is separate, and already carries this policy
 
 Choose one method and apply it to **every** server in the cluster.
 
-**Option A — existing OS.** On a server already running Ubuntu 26.04 or Debian 13:
+**Option A — existing OS.** Use this on servers already running Ubuntu 26.04, Debian 13 or Oracle Linux 10, and prepare an Oracle Linux server with [Oracle Linux Hosts](/docs/install-oracle-linux) first.
+Every server in a cluster must run the same OS: Oracle Linux never shares a cluster with Debian or Ubuntu, and `spx admin join` does not check this for you.
+
+Run the installer on each server:
 
 ```bash
 curl -fsSL https://install.mulgadc.com | bash

docs/install/install-oracle-linux/README.md

@@ -0,0 +1,201 @@
+---
+title: "Oracle Linux Hosts"
+seoTitle: "Install Spinifex on Oracle Linux 10 — Spinifex Docs"
+description: "Install Spinifex on Oracle Linux 10: prepare the host with UEK R8, SELinux permissive and a NetworkManager br-wan bridge, then run the single-node install."
+category: "Install"
+tags:
+  - install
+  - oracle linux
+  - single node
+  - multi node
+resources:
+  - title: "Oracle Linux 10 Documentation"
+    url: "https://docs.oracle.com/en/operating-systems/oracle-linux/10/"
+  - title: "CentOS NFV SIG"
+    url: "https://sigs.centos.org/nfv/"
+---
+
+# Oracle Linux Hosts
+
+> Install Spinifex on Oracle Linux 10.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Instructions](#instructions)
+- [What the Installer Changes](#what-the-installer-changes)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Overview
+
+Spinifex runs on Oracle Linux 10.2, on the UEK R8 kernel, on x86_64 with an x86-64-v3 CPU (Intel Haswell, AMD Excavator or later).
+
+The host must be a dedicated Spinifex host: Spinifex claims every `tap*` device and owns libreswan's `ipsec.service`.
+
+**Not supported yet:** clusters mixing Oracle Linux with Debian or Ubuntu, SELinux enforcing, aarch64, Oracle Linux 9 or earlier, the Red Hat Compatible Kernel, a ZFS root, NVIDIA MIG and air-gapped installs.
+
+## Instructions
+
+Work through both tabs in order.
+
+:::tabs
+@tab 1. Prepare Oracle Linux
+
+### Update and check the host
+
+Update to the current minor release and reboot:
+
+```bash
+sudo dnf update -y
+sudo reboot
+```
+
+Then check the release, the running kernel and the CPU:
+
+```bash
+cat /etc/oracle-release        # Oracle Linux Server release 10.2 or later
+uname -r                       # 6.12.0-...el10uek.x86_64
+grep -c -w avx2 /proc/cpuinfo  # non-zero
+```
+
+### Set SELinux permissive
+
+```bash
+sudo setenforce 0
+sudo sed -i 's/^SELINUX=.*/SELINUX=permissive/' /etc/selinux/config
+getenforce                     # Permissive
+```
+
+### Create the br-wan bridge
+
+Run this from the console, not over SSH on the NIC being moved.
+Replace `eno1` with the uplink NIC and `"System eno1"` with its current profile (`nmcli connection show`):
+
+```bash
+sudo nmcli connection add type bridge ifname br-wan con-name br-wan \
+  bridge.stp no ipv4.method auto ipv6.method disabled
+sudo nmcli connection add type bridge-slave ifname eno1 master br-wan con-name br-wan-port
+sudo nmcli connection down "System eno1"
+sudo nmcli connection up br-wan
+```
+
+For a static address, use `ipv4.method manual` with `ipv4.addresses`, `ipv4.gateway` and `ipv4.dns`.
+Keep `bridge.stp no`, or DHCP on the bridge can time out.
+
+```bash
+ip -br link show br-wan        # UP
+ip route                       # default via ... dev br-wan
+```
+
+@tab 2. Install Single Node
+
+### Install Spinifex
+
+```bash
+curl -fsSL https://install.mulgadc.com | bash
+```
+
+The installer stops and masks firewalld; see [firewalld](#firewalld) before relying on any rule it held.
+
+### Set up OVN networking
+
+```bash
+sudo /usr/local/share/spinifex/setup-ovn.sh --management
+```
+
+### Initialize
+
+```bash
+sudo spx admin init --node node1 --nodes 1
+```
+
+### Start services
+
+```bash
+sudo systemctl start spinifex.target
+```
+
+### Verify
+
+```bash
+export AWS_PROFILE=spinifex
+aws ec2 describe-instance-types
+```
+
+If this returns a list of instance types, Spinifex is installed.
+Continue to [Setting Up Your Cluster](/docs/setting-up-your-cluster) to import an AMI and launch your first instance.
+:::
+
+**For a cluster**, prepare and install every node with the steps above, then continue with [Multi-Node Install](/docs/install-multi-node) from Step 2.
+Every node must run Oracle Linux.
+IPsec between nodes runs on libreswan, so open UDP 500, UDP 4500 and ESP between nodes on any firewall between them.
+
+## What the Installer Changes
+
+On Oracle Linux the installer also:
+
+- Enables `ol10_UEKR8` and `ol10_kvm_utils`, and excludes `qemu-kvm*` and `qemu-img*` from `ol10_appstream`.
+- Adds the NFV SIG repository as `/etc/yum.repos.d/spinifex-nfv-sig.repo`, limited to the OVS and OVN packages.
+- Installs `kernel-uek-modules` for the running kernel, about 800 MB with `linux-firmware`.
+- Switches DNS to systemd-resolved behind NetworkManager where resolved is installed but not running.
+- Links `/usr/bin/spx` to `/usr/local/bin/spx`, so `sudo spx` works.
+
+### firewalld
+
+The installer stops and masks firewalld and turns on the Spinifex nftables policy in its place; firewalld's rules are not migrated.
+The Spinifex policy opens 53, 443, 3000, 8443, 9999 and SSH to any source.
+Restate any restriction you relied on, such as SSH limited to a management subnet, in `/etc/spinifex/firewall/custom.nft`; see [Host Firewall](/docs/host-firewall).
+
+### Updates
+
+Do not enable `dnf-automatic`: an Open vSwitch upgrade restarts OVS and interrupts guest networking, so update `openvswitch3.5`, `ovn25.03*` and `qemu-kvm*` in a maintenance window.
+Before rebooting into a new kernel, check `rpm -q kernel-uek-core kernel-uek-modules` shows matching versions, or OVS cannot start.
+
+### Uninstall
+
+`/usr/local/share/spinifex/uninstall-spx.sh` removes Spinifex and lists what it leaves in place.
+`--purge-deps` is refused on Oracle Linux; remove packages with `dnf remove` yourself.
+
+## Troubleshooting
+
+### Installer stops on SELinux, the kernel or kernel modules
+
+Set SELinux permissive both now and in `/etc/selinux/config`, or update and boot into the current UEK R8 kernel, then run the installer again:
+
+```bash
+sudo dnf update -y kernel-uek
+sudo reboot
+```
+
+### Installer stops: a repository is unreachable
+
+Check the host can reach `yum.oracle.com` and `mirror.stream.centos.org` over HTTPS, including through any proxy dnf uses.
+
+### DHCP on br-wan times out
+
+STP is on. Turn it off:
+
+```bash
+sudo nmcli connection modify br-wan bridge.stp no
+sudo nmcli connection up br-wan
+```
+
+### Cluster nodes form no IPsec tunnels
+
+```bash
+systemctl is-active ipsec.service openvswitch-ipsec.service
+sudo ipsec trafficstatus                       # one line per peer tunnel
+```
+
+If the services are inactive, check `journalctl -u spinifex-daemon`.
+If no tunnel is listed, check UDP 500 and ESP pass between the nodes in both directions.
+Guest traffic between two nodes is dropped until their tunnel forms.
+
+### CA certificate not trusted
+
+```bash
+sudo cp /etc/spinifex/ca.pem /etc/pki/ca-trust/source/anchors/spinifex-ca.crt
+sudo update-ca-trust
+```