feat: build an Oracle Linux appliance gold image that sets each unit up on first boot

* feat: add the store-neutral SQS message store contract and conformance suite

Defines message.Store over opaque shard and queue ID namespaces with no
create call, plus a reusable conformance runner exercised against a
test-only in-memory reference store.

* feat: add the SQS message store's JetStream layout, append and state CAS

Provisions the fixed set of shared message streams and state buckets,
publishes each queue on sqs.m.<shard>.<queue>, dedupes idempotent appends
with a queue-scoped Nats-Msg-Id, and maps state revision CAS onto KV.

* fix: give each SQS queue immutable message and state shard allocations

A changed stream or bucket count applies only to new queues: the namespace
names its message shard and state shard directly, retained streams and
buckets stay addressable, and the store refuses cluster sizes other than
one node (R1) or three (R3).

* fix: enumerate complete SQS namespaces and backstop state buckets

Enumeration now returns each namespace with both allocations, so a
state-only namespace can be counted and cleaned without its queue record,
and state buckets expire records after 14 days plus 1 hour.

* feat: add the SQS JetStream store's reads, counts and cleanup

Get and Delete are scoped to the namespace's subject, DeleteThrough and
DeleteNamespace are subject-filtered purges that leave shard neighbours
intact, and Count reads the per-subject message total and live state keys.

* fix: count distinct SQS state keys and validate before namespace purge

The key lister can repeat a key under concurrent writes, so Count tallies
distinct names; DeleteNamespace resolves both the message stream and the
state bucket before purging either, so a wrong state shard deletes nothing.

* feat: add the SQS JetStream store's ordered message-shard feed

Append records the namespace's state shard in an internal header that is
never returned as metadata. Feed reads one shared message stream through
an ephemeral, headers-only ordered consumer with no acknowledgement state,
rebuilds each change's complete namespace, refuses a malformed header with
ErrCorruptRecord, and deletes its consumer on Close.

* feat: enumerate SQS namespaces and run conformance on the JetStream store

Namespaces joins per-subject body counts, whose internal header names the
state shard, with live state keys from every state bucket that exists,
found by listing so a historical bucket counts and none is created. An
allocation conflict is reported as ErrCorruptRecord. The store now
implements message.Store and runs the conformance suite on R1, per case
with crash reopen and on one shared instance.

* fix: check every SQS body's state shard when enumerating namespaces

Sampling the first and last body let a middle body naming another state
shard pass. Namespaces now reads every body's header in one headers-only
ordered pass over the shard's stream, counting each queue's bodies from
that pass, and deletes its consumer when done.

* test: run SQS store conformance on R3 and guard against consumer state

TestR3Conformance runs the conformance suite on a real three-node
JetStream cluster in process, with Reopen restarting every node over the
same data, and a check that every stream and state bucket is R3.
TestS5_NoConsumerDeliveryStateInSQS fails on any acknowledgement,
redelivery, durable-consumer or ack-retention mechanism in SQS source.

* test: bar legacy NATS acknowledgement helpers from SQS source

Adds AckSync, ManualAck, AckAll and AckExplicit to the ADR-0011:S5
consumer-state guard.

* feat: add a bounded SQS store scan and the tombstoned-body sweeper

Store gains Scan: a paginated, headers-only pass over a message shard,
bounded at the shard's last position when it starts, skipping deleted
positions and never returning a later append. The conformance suite and
fake cover continuation, deleted runs and the bound.

The JetStream adapter reads every bounded range (scan, body enumeration
and state keys) through one reader whose completion rests on a direct get
of the next match, not on consumer state; the client's key lister and
pending counts are no longer trusted to end a pass.

message.Sweep walks a shard with Scan and, for each body whose record the
domain reports terminal, deletes the body then the record by revision.

* test: add an opt-in SQS store conformance run on an external R3 cluster

TestR3ExternalConformance runs the conformance suite against the
dedicated three-node SQS JetStream cluster named by SQS_STORE_R3_URLS,
checks every stream is R3 with two peers, and uses SQS_STORE_R3_RESTART
to restart every node for the crash cases. It skips when unset.

* test: run the SQS sweeper against the reference store fake

Covers live records kept, tombstones swept, another shard untouched, a
no-op rerun, and a retry after the body delete succeeds and the state
delete fails.

* test: harden the SQS store R3 harnesses against restart and placement races

The external R3 harness reused one connection across restarts, so a case's
cleanup closed it for every later case; each case and restart now opens its
own connection and waits for stream leaders.

The in-process cluster released its free ports before the servers bound them,
and R3 placement can fail until the meta leader has every peer's config and
stats. Client ports are now server-chosen, route ports are reserved together,
and provisioning retries the idempotent New until placement succeeds.

* fix: stop the SQS store feed idling for a whole pull expiry

Under load an ordered feed on a three-node cluster sat for nats.go's 30s
default pull expiry, once 60s, with stored changes pending and no pull
waiting on its consumer, both on feeds opened after a full restart and on
feeds kept open across one. Feed now sets a 2s pull expiry.

Adds restart tests for a feed opened after a full-cluster restart and one
open across it, each replaying a shared-shard backlog then a live append and
failing on any 5s gap. The in-process cluster keeps client ports across
restarts so open connections reconnect, and can write node logs.

* fix: read SQS message state from the stream leader

GetState used nats.go's KV Get, which switches to a direct get whenever
the bucket allows it. On a replicated bucket a follower can answer that
from state it has not applied yet, so a record the leader had already
deleted or purged could still be returned.

Read the key's last message with a stream message get instead, which
only the stream leader answers, and apply KV Get's delete and purge
marker rules to the result. Bucket configuration is unchanged. Missing,
deleted and purged records still return ErrNotFound; values, revisions,
context errors and the API prefix or domain of the caller's JetStream
context are preserved.

* docs: narrow the SQS GetState leader-read comment to what it avoids

* fix: enumerate SQS state keys from the stream leader

Count and Namespaces found live state keys with a direct-get probe and a
headers-only ordered consumer on the state bucket's stream. On a
replicated bucket a follower can answer either from state it has not
applied, so keys the leader had already deleted could still be counted.

Walk the key prefix with stream message gets by next subject, which
only the stream leader answers, through the stream's last sequence at
the call. Delete and purge markers drop a key, as GetState reads them.
This costs one leader request per record under the prefix. Bucket
configuration, historical state shards, the message-body scans and the
feed are unchanged.

* test: keep SQS API recorder deliveries in arrival order

The recorder fed API subjects and its sentinel to two async
subscriptions, which nats.go delivers on separate goroutines, so drain
could return before the last request was recorded. Both now share one
channel that the connection fills in arrival order.

* fix: read SQS body enumerations from the stream leader

Namespaces and Scan read bodies through an ephemeral ordered consumer,
which the server may place on a follower that has not applied a purge,
so a deleted queue could still be listed or scanned. Each stored match
through the bound is now a leader stream message get. That transfers
every full body, where the consumer read headers only.

* test: stop the SQS API recorder goroutine with its test

The recorder ranged over a caller-owned ChanSubscribe channel, which
nats.go never closes, so its goroutine outlived every test. Cleanup now
unsubscribes both subscriptions, then signals the goroutine and waits
for it to return; the sentinel send also watches that signal.

* fix: recover a stalled SQS feed from its cursor

An ordered feed was seen to idle with stored changes pending after a
cluster restart. Next now waits at most an idle check on the consumer,
then asks the stream leader whether a change follows the cursor; if one
does, the consumer is deleted and reopened just after the cursor, so no
change is skipped or repeated. The root cause of the stall is unknown.

* fix: make closing an SQS feed terminal and safe against Next

Close now marks the feed closed under a lock, cancels any in-flight
Next, stops the iterator and deletes the consumer. A later or in-flight
Next returns ErrFeedClosed and never opens a consumer; one opened by a
recovery racing Close is deleted rather than adopted. Repeated Close
returns nil.

* fix: define SQS feed close in the store contract

message.Feed now documents one Next reader with Close callable at any
time and more than once, and message.ErrFeedClosed replaces the
adapter's own error. Once Close has begun, Next returns ErrFeedClosed
and withholds any change it had received; a feed from the last cursor
still delivers it. Shared conformance cases and the reference fake
enforce this.

The JetStream feed's Close also waits, within a bound, for a consumer
open already in progress, so a consumer created during Close is deleted
before Close returns nil; past the bound, or if that delete fails, Close
returns an error and a repeated Close waits again.

* fix: track SQS feed consumer cleanup until Close succeeds

Every consumer open and release is now tracked work recorded under the
lock that takes the consumer, and a delete that fails is kept for Close
to retry. Close returns nil only once no such work is in progress and no
failed delete remains, waiting within a bound and otherwise returning an
error; concurrent and repeated Close calls wait the same way.

Next claims a change under that lock and advances its cursor there, so a
change is either returned before Close begins or withheld for a feed
resumed from the last cursor.

A consumer create no longer inherits the caller's cancellation, which
was seen to leave most cancelled creates on the server with no owner;
it has its own timeout, and the name of one that fails is deleted and
kept for Close to delete again.

* fix: release SQS feed consumers opened after the caller's context ends

Feed and recovery now check the caller's context before starting a
consumer open and after it completes. A create that finishes after the
context ended is released as tracked work and the context's error is
returned, or ErrFeedClosed when Close won; no change is returned.

The feed contract and adapter docs now describe cleanup as complete for
consumers the feed knows of, with the server's one-minute inactive
threshold removing one an ambiguous create lands after its last delete.

* fix: report cancellation when an SQS feed open fails after it

A first consumer open that fails after the caller's context ended now
returns the context's error joined with the failure, so cancellation
stays detectable. The feed contract's cancellation text is narrowed to
opening and reopening, whose cleanup is best effort.

* docs: describe the internal SQS message store and its limits

DESIGN.md gains a section on the store-neutral SQS message store, its
JetStream adapter and conformance suite: an internal primitive with no
service wired to it, its shared stream and bucket layout, leader-directed
reads and their full-body cost, feed recovery and its best-effort
consumer cleanup, and the absence of end-to-end or multi-host
validation. The Scan contract says metadata only, not headers only.

.github/e2e-cells.json

@@ -190,5 +190,60 @@
     "extra": "",
     "suite": "kvquorum",
     "timeout_minutes": 55
+  },
+  {
+    "cell": 34,
+    "install": "binary",
+    "topo": "single",
+    "net": "dhcp",
+    "host": "oracle-10",
+    "arch": "x86_64",
+    "runner": "ci-single",
+    "extra": "",
+    "suite": ""
+  },
+  {
+    "cell": 35,
+    "install": "binary",
+    "topo": "single",
+    "net": "static",
+    "host": "oracle-10",
+    "arch": "x86_64",
+    "runner": "ci-single",
+    "extra": "reboot-single",
+    "suite": ""
+  },
+  {
+    "cell": 36,
+    "install": "binary",
+    "topo": "single",
+    "net": "nat",
+    "host": "oracle-10",
+    "arch": "x86_64",
+    "runner": "ci-single",
+    "extra": "nat-single",
+    "suite": ""
+  },
+  {
+    "cell": 37,
+    "install": "binary",
+    "topo": "real-multi",
+    "net": "dhcp",
+    "host": "oracle-10",
+    "arch": "x86_64",
+    "runner": "ci-multi",
+    "extra": "",
+    "suite": ""
+  },
+  {
+    "cell": 38,
+    "install": "binary",
+    "topo": "real-multi",
+    "net": "nat",
+    "host": "oracle-10",
+    "arch": "x86_64",
+    "runner": "ci-multi",
+    "extra": "nat-multi",
+    "suite": ""
   }
 ]

.github/workflows/apt-packages.yml

@@ -2,7 +2,8 @@ name: Apt Packages
 
 # Path-filtered like image-scripts.yml: this resolves setup.sh's runtime package
 # list against each supported suite's real indices, so a renamed, dropped, or
-# ambiguously virtual package fails here rather than on a user's machine.
+# ambiguously virtual package fails here rather than on a user's machine. The
+# dnf job does the same for Oracle Linux, through setup.sh's own repo setup.
 on:
   push:
     branches:
@@ -11,6 +12,7 @@ on:
     paths:
       - scripts/setup.sh
       - scripts/lint-apt-packages.sh
+      - scripts/lint-dnf-packages.sh
       - .github/workflows/apt-packages.yml
   pull_request:
     branches:
@@ -19,6 +21,7 @@ on:
     paths:
       - scripts/setup.sh
       - scripts/lint-apt-packages.sh
+      - scripts/lint-dnf-packages.sh
       - .github/workflows/apt-packages.yml
   workflow_dispatch:
 
@@ -44,3 +47,14 @@ jobs:
 
       - name: Resolve runtime package list
         run: ./scripts/lint-apt-packages.sh ${{ matrix.image }}
+
+  dnf_packages:
+    name: Resolve dnf list (oraclelinux:10)
+    runs-on: ubuntu-26.04
+
+    steps:
+      - name: Checkout Code
+        uses: actions/checkout@v7.0.1
+
+      - name: Resolve runtime package list
+        run: ./scripts/lint-dnf-packages.sh oraclelinux:10

.github/workflows/dev-release.yml

@@ -102,7 +102,7 @@ jobs:
         env:
           GH_TOKEN: ${{ github.token }}
 
-      - name: Build microVM artifacts (lb-agent + kernel + initramfs)
+      - name: Build microVM artifacts (lb-agent + kernel + initramfs + qboot)
         if: steps.version.outputs.exists != 'true'
         run: cd spinifex && make build-microvm-image
 
@@ -144,6 +144,7 @@ jobs:
               --notes-file "$RUNNER_TEMP/notes.md" \
               "dist/spinifex-$TAG-linux-amd64.tar.gz" \
               "dist/spinifex-$TAG-linux-amd64.tar.gz.sha256" \
+              dist/qboot-*.tar.gz \
               "scripts/setup.sh"
 
       # The assertion that matters. A prerelease must never become the latest

.github/workflows/e2e-nightly.yml

@@ -10,7 +10,7 @@ on:
   workflow_dispatch:
     inputs:
       cells:
-        description: "Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection, 30 routed NAT on three nodes, 31 instance recovery on three nodes, 32 load balancer recovery on three nodes, 33 KV bucket quorum under node loss on three nodes"
+        description: "Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 34-36 Oracle Linux single node (basic, reboot, routed NAT), 37-38 Oracle Linux on three nodes (multi-node subset with IPsec, routed NAT), 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection, 30 routed NAT on three nodes, 31 instance recovery on three nodes, 32 load balancer recovery on three nodes, 33 KV bucket quorum under node loss on three nodes"
         required: false
 
 permissions:
@@ -33,6 +33,7 @@ concurrency:
 env:
   GIT_BRANCH: ${{ github.ref_name }}
   UBUNTU_GOLD_IMAGE: /mnt/disk1/gold-images/spinifex-image-builder-ubuntu-26.04.qcow2
+  ORACLE_GOLD_IMAGE: /mnt/disk1/gold-images/spinifex-image-builder-oracle-10.qcow2
   SPINIFEX_OTLP_ENDPOINT: ${{ vars.SPINIFEX_OTLP_ENDPOINT || 'http://192.168.1.13:8200' }}
   SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }}
 
@@ -195,7 +196,8 @@ jobs:
           # back onto the debian gold.
           TOFU_VAR_ARGS=""
           case "$MATRIX_HOST" in
-            ubuntu-*) TOFU_VAR_ARGS="-var=host_image_override=${UBUNTU_GOLD_IMAGE}" ;;
+            ubuntu-*) TOFU_VAR_ARGS="-var=host_image_override=${UBUNTU_GOLD_IMAGE} -var=host_os=ubuntu" ;;
+            oracle-*) TOFU_VAR_ARGS="-var=host_image_override=${ORACLE_GOLD_IMAGE} -var=host_os=oracle" ;;
           esac
 
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
@@ -687,7 +689,7 @@ jobs:
               'echo "host: $(hostname)"; echo "loadavg: $(cat /proc/loadavg)";
                grep -E "^(MemTotal|MemAvailable|SwapFree):" /proc/meminfo;
                echo "cpu(steal is field 8): $(grep "^cpu " /proc/stat)";
-               echo "running qemu: $(pgrep -c qemu-system-x86 || echo 0)"' 2>&1
+               echo "running qemu: $(pgrep -c "qemu-system-x86|qemu-kvm" || echo 0)"' 2>&1
           } >> "${ARTIFACT_DIR}/node-load.txt"
           exit 0
 
@@ -858,7 +860,8 @@ jobs:
           # different image path than apply did.
           TOFU_VAR_ARGS=""
           case "$MATRIX_HOST" in
-            ubuntu-*) TOFU_VAR_ARGS="-var=host_image_override=${UBUNTU_GOLD_IMAGE}" ;;
+            ubuntu-*) TOFU_VAR_ARGS="-var=host_image_override=${UBUNTU_GOLD_IMAGE} -var=host_os=ubuntu" ;;
+            oracle-*) TOFU_VAR_ARGS="-var=host_image_override=${ORACLE_GOLD_IMAGE} -var=host_os=oracle" ;;
           esac
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1

.github/workflows/release.yml

@@ -57,7 +57,7 @@ jobs:
           go-version-file: spinifex/go.mod
           cache: false
 
-      - name: Build microVM artifacts (lb-agent + kernel + initramfs)
+      - name: Build microVM artifacts (lb-agent + kernel + initramfs + qboot)
         run: cd spinifex && make build-microvm-image
 
       - name: Build amd64 Tarball
@@ -72,6 +72,7 @@ jobs:
           path: |
             spinifex/dist/spinifex-${{ github.ref_name }}-linux-amd64.tar.gz
             spinifex/dist/spinifex-${{ github.ref_name }}-linux-amd64.tar.gz.sha256
+            spinifex/dist/qboot-*.tar.gz
             spinifex/scripts/setup.sh
 
   distro-arm64:

.gitignore

@@ -19,6 +19,9 @@ bin/
 dist/
 build/microvm/vmlinuz
 build/microvm/initramfs.cpio.gz
+build/microvm/qboot.rom
+build/microvm/qboot.LICENSE
+build/qboot/
 !spinifex/runtime/roles/spinifexui/frontend/dist/
 tmp/
 *.test

Makefile

@@ -140,12 +140,23 @@ $(MICROVM_ARTIFACTS) &: $(MICROVM_INPUTS)
 bin/lb-agent:
 	$(MAKE) build-lb-agent
 
-build-microvm-image: $(MICROVM_ARTIFACTS) ## Build microVM kernel + initramfs (incremental — skips when up to date)
+# qboot is the microvm firmware. The source tarball ships with each release
+# because qboot is GPL-2.0.
+QBOOT_COMMIT := $(shell sed -n 's/^QBOOT_COMMIT="\(.*\)"$$/\1/p' scripts/build-qboot.sh)
+QBOOT_SOURCE := build/qboot/qboot-$(QBOOT_COMMIT).tar.gz
+QBOOT_ARTIFACTS := $(MICROVM_OUT_DIR)/qboot.rom $(MICROVM_OUT_DIR)/qboot.LICENSE $(QBOOT_SOURCE)
 
-install-microvm: $(MICROVM_ARTIFACTS) ## Install microVM artifacts to /usr/share/spinifex/microvm/
+$(QBOOT_ARTIFACTS) &: scripts/build-qboot.sh
+	./scripts/build-qboot.sh
+
+build-microvm-image: $(MICROVM_ARTIFACTS) $(QBOOT_ARTIFACTS) ## Build microVM kernel, initramfs and qboot firmware (incremental — skips when up to date)
+
+install-microvm: $(MICROVM_ARTIFACTS) $(QBOOT_ARTIFACTS) ## Install microVM artifacts to /usr/share/spinifex/microvm/
 	sudo install -d /usr/share/spinifex/microvm
 	sudo install -m 0644 $(MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/vmlinuz
 	sudo install -m 0644 $(MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/initramfs.cpio.gz
+	sudo install -m 0644 $(MICROVM_OUT_DIR)/qboot.rom /usr/share/spinifex/microvm/qboot.rom
+	sudo install -m 0644 $(MICROVM_OUT_DIR)/qboot.LICENSE /usr/share/spinifex/microvm/qboot.LICENSE
 
 # the pre-commit gate: manifest checks, lint, vuln, and the unit and e2e-harness tiers.
 # integration and race tests skipped to keep quick, they run in CI.
@@ -167,7 +178,7 @@ preflight:
 # reason: between them they install the software, wire OVN, erase the state and
 # remove the install, so a shellcheck warning in one is an outage or a wrong
 # deletion, not a style note.
-NODE_SCRIPTS := scripts/setup.sh scripts/setup-ovn.sh scripts/install-node.sh scripts/smoke-test.sh scripts/node-reset.sh scripts/uninstall-spx.sh
+NODE_SCRIPTS := scripts/setup.sh scripts/setup-ovn.sh scripts/install-node.sh scripts/smoke-test.sh scripts/node-reset.sh scripts/uninstall-spx.sh scripts/lib/distro.sh
 
 test-build-scripts:
 	@echo -e "\n....Running build/scripts/**/*_test.sh...."
@@ -175,13 +186,14 @@ test-build-scripts:
 		echo "-- $$t"; \
 		sh "$$t" || exit 1; \
 	done
-	@echo -e "\n....Running scripts/*_test.sh...."
-	@for t in $$(find scripts -maxdepth 1 -name '*_test.sh' | sort); do \
+	@echo -e "\n....Running scripts/*_test.sh, scripts/lib/*_test.sh and scripts/appliance/*_test.sh...."
+	@for t in $$( (find scripts -maxdepth 1 -name '*_test.sh'; find scripts/lib scripts/appliance -name '*_test.sh') | sort); do \
 		echo "-- $$t"; \
 		sh "$$t" || exit 1; \
 	done
 	@echo -e "\n....Running shellcheck over build/scripts/**/*.sh + the node scripts...."
 	shellcheck -S warning $$(find build/scripts -name '*.sh' | sort) $(NODE_SCRIPTS)
+	shellcheck -S warning -x scripts/build-oracle-appliance.sh $$(find scripts/appliance -name '*.sh' | sort)
 	@echo "  test-build-scripts ok"
 
 # E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
@@ -333,14 +345,20 @@ fuzz:
 # Microvm artifacts are reinstalled when they already exist on disk — the rule's
 # input timestamps drive a rebuild only if anything actually changed. On a fresh
 # checkout (no build/microvm/vmlinuz yet) the install step is skipped; run
-# `make install-microvm` explicitly the first time.
+# `make install-microvm` explicitly the first time. They go in before the
+# preflight, which checks the microvm firmware.
 deploy: build
 	sudo install -m 755 bin/spx /usr/local/bin/spx
+	@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ]; then \
+		$(MAKE) install-microvm; \
+	else \
+		echo "[deploy] microvm artifacts absent — run 'make install-microvm' for first-time setup"; \
+	fi
 	@if [ "$${SKIP_ASSET_PREFLIGHT:-}" != "1" ]; then \
 		if ! sudo /usr/local/bin/spx admin preflight; then \
 			echo ""; \
-			echo "[deploy] host-asset preflight failed: this node's helper scripts or sudoers"; \
-			echo "  grants don't match the binary just installed. Remediate with:"; \
+			echo "[deploy] host-asset preflight failed: this node's helper scripts, sudoers"; \
+			echo "  grants, QEMU or microvm firmware don't match the binary just installed. Remediate with:"; \
 			echo "    scripts/update-nodes.sh   (from the mulga root)"; \
 			echo "  or:"; \
 			echo "    make reinstall"; \
@@ -350,11 +368,6 @@ deploy: build
 	else \
 		echo "[deploy] SKIP_ASSET_PREFLIGHT=1 — skipping host-asset preflight"; \
 	fi
-	@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ]; then \
-		$(MAKE) install-microvm; \
-	else \
-		echo "[deploy] microvm artifacts absent — run 'make install-microvm' for first-time setup"; \
-	fi
 	sudo systemctl daemon-reload
 	sudo systemctl restart spinifex.target
 
@@ -367,16 +380,25 @@ clean:
 	rm -f ./bin/$(GO_PROJECT_NAME)
 	rm -rf spinifex/runtime/roles/spinifexui/frontend/dist
 
+# On dnf hosts the runtime packages and repositories come from setup.sh, so the
+# two cannot drift; nbdkit-devel, for building the plugin, is only in CodeReady.
 install-system:
 	@echo -e "\n....Installing system dependencies for $(ARCH)...."
-	sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
-		-o Dpkg::Options::="--force-confdef" \
-		-o Dpkg::Options::="--force-confold" \
-		nbdkit nbdkit-plugin-dev pkg-config qemu-system-x86 qemu-system-arm qemu-utils \
-		ovmf qemu-efi-aarch64 \
-		libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
-		iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
-		ovn-central ovn-host openvswitch-switch dhcpcd-base
+	@pm=$$(INSTALL_SPINIFEX_LIB_ONLY=1 bash -c '. scripts/setup.sh && distro_pkg_manager') || exit 1; \
+	if [ "$$pm" = dnf ]; then \
+		INSTALL_SPINIFEX_LIB_ONLY=1 bash -c '. scripts/setup.sh && setup_sudo && detect_os && install_dnf_deps' && \
+		sudo dnf config-manager --set-enabled ol10_codeready_builder && \
+		sudo dnf install -y nbdkit-devel pkgconf-pkg-config make gcc openssh-clients git sudo; \
+	else \
+		sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
+			-o Dpkg::Options::="--force-confdef" \
+			-o Dpkg::Options::="--force-confold" \
+			nbdkit nbdkit-plugin-dev pkg-config qemu-system-x86 qemu-system-arm qemu-utils \
+			ovmf qemu-efi-aarch64 \
+			libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
+			iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
+			ovn-central ovn-host openvswitch-switch dhcpcd-base; \
+	fi
 
 install-go:
 	@echo -e "\n....Installing Go 1.27.3 for $(ARCH) ($(GO_ARCH))...."
@@ -438,10 +460,11 @@ distro-amd64:
 		-f build/Dockerfile.distro \
 		--output type=local,dest=dist/amd64/ \
 		../
-	@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ] && [ -f $(MICROVM_OUT_DIR)/initramfs.cpio.gz ]; then \
+	@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ] && [ -f $(MICROVM_OUT_DIR)/initramfs.cpio.gz ] && [ -f $(MICROVM_OUT_DIR)/qboot.rom ] && [ -f $(QBOOT_SOURCE) ]; then \
 		echo "[distro-amd64] staging microvm artifacts into tarball"; \
 		mkdir -p dist/amd64/microvm; \
-		cp $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz dist/amd64/microvm/; \
+		cp $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz $(MICROVM_OUT_DIR)/qboot.rom $(MICROVM_OUT_DIR)/qboot.LICENSE dist/amd64/microvm/; \
+		cp $(QBOOT_SOURCE) dist/; \
 	else \
 		echo "[distro-amd64] WARNING: microvm artifacts missing — tarball will not include them"; \
 		echo "[distro-amd64]          run 'make build-microvm-image' before 'make distro-amd64'"; \

architecture/layering/allowlist_test.go

@@ -29,8 +29,9 @@ var allowlist = []allowance{
 
 	{From: "agents/rds/agent", To: "domains/rds/instance", Debt: "agents/rds/agent (dependency, narrowed)", Reason: "command protocol, wire and record types; needs a guest/controller wire contract"},
 
-	{From: "cmd/aws-model-coverage", To: "domains/acm/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared ECR/ACM inventories; named exception, must not construct a registration"},
-	{From: "cmd/aws-model-coverage", To: "domains/ecr/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared ECR/ACM inventories; named exception, must not construct a registration"},
+	{From: "cmd/aws-model-coverage", To: "domains/acm/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared ECR/ACM/Route 53 inventories; named exception, must not construct a registration"},
+	{From: "cmd/aws-model-coverage", To: "domains/ecr/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared ECR/ACM/Route 53 inventories; named exception, must not construct a registration"},
+	{From: "cmd/aws-model-coverage", To: "domains/route53/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared ECR/ACM/Route 53 inventories; named exception, must not construct a registration"},
 
 	{From: "runtime/daemon", To: "domains/ec2/awsapi/instance", Debt: "Instance adapter consumers outside EC2", Reason: "runtime/daemon/eks_cp_control.go and rds_deps.go call the instance adapter; needs an EC2 capability"},
 

build/Dockerfile.distro

@@ -42,9 +42,16 @@ RUN COMMIT=$(cd spinifex && git rev-parse --short HEAD 2>/dev/null || echo "unkn
 # Build nbdkit viperblock plugin (.so)
 RUN cd viperblock && GOFIPS140=v1.0.0 go build -o /out/nbdkit-viperblock-plugin.so -buildmode=c-shared nbd/viperblock.go
 
+# One tarball ships to every OS. The ceiling is what both binaries need today,
+# so a rise is a decision rather than an accident; every supported OS has 2.39+.
+RUN spinifex/scripts/check-glibc-ceiling.sh 2.34 /out/spx /out/nbdkit-viperblock-plugin.so
+
 # Copy setup scripts and systemd units
 RUN cp spinifex/scripts/setup.sh /out/setup.sh 2>/dev/null || true
 RUN cp spinifex/scripts/setup-ovn.sh /out/setup-ovn.sh
+RUN cd spinifex/scripts && cp install-node.sh smoke-test.sh node-reset.sh uninstall-spx.sh /out/
+RUN mkdir -p /out/udev && cp spinifex/build/udev/* /out/udev/
+RUN mkdir -p /out/lib && cp spinifex/scripts/lib/distro.sh /out/lib/distro.sh
 RUN mkdir -p /out/systemd && cp spinifex/build/systemd/* /out/systemd/
 RUN mkdir -p /out/scripts && cp spinifex/build/scripts/* /out/scripts/
 RUN cp spinifex/build/logrotate/spinifex /out/logrotate-spinifex

build/systemd/spinifex-daemon.service

@@ -1,10 +1,10 @@
-# spinifex-unit-version: 3
+# spinifex-unit-version: 5
 [Unit]
 Description=Spinifex Daemon (VM Orchestration)
 PartOf=spinifex.target
-After=spinifex-nats.service spinifex-viperblock.service openvswitch-switch.service ovn-controller.service openvswitch-ipsec.service
-# Wants (not Requires): stay bootable when openvswitch-ipsec is absent
-Wants=openvswitch-ipsec.service
+After=spinifex-nats.service spinifex-viperblock.service ovsdb-server.service ovs-vswitchd.service ovn-controller.service openvswitch-ipsec.service spinifex-hostnet.service
+# Wants (not Requires): boot without openvswitch-ipsec, survive OVS restarts via hostnet
+Wants=openvswitch-ipsec.service spinifex-hostnet.service
 
 [Service]
 EnvironmentFile=-/etc/spinifex/telemetry.env
@@ -14,6 +14,7 @@ Type=simple
 Slice=spinifex-system.slice
 User=spinifex-daemon
 Group=spinifex
+ExecStartPre=/usr/local/bin/spx admin hostnet check
 ExecStart=/usr/local/bin/spx service spinifex start
 Restart=on-failure
 RestartSec=5

build/systemd/spinifex-hostnet.service

@@ -0,0 +1,25 @@
+# spinifex-unit-version: 2
+[Unit]
+Description=Spinifex host network wiring (veth pairs, mgmt address, encap route)
+# PartOf/WantedBy ovs-vswitchd: re-applied whenever OVS starts; daemon and vpcd only Want it
+Requires=ovs-vswitchd.service
+After=ovs-vswitchd.service NetworkManager-wait-online.service
+PartOf=ovs-vswitchd.service
+# Stopping OVS stops ovn-controller through its Requires= chain and starting OVS
+# does not bring it back, so the start that re-applies hostnet restarts it too
+Wants=ovn-controller.service
+Before=spinifex.target
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/usr/local/bin/spx admin hostnet apply
+# Covers the 120s wait behind setup-ovn.sh plus the 60s wait for links
+TimeoutStartSec=240
+TimeoutStopSec=10
+ProtectSystem=full
+ProtectHome=yes
+PrivateTmp=yes
+
+[Install]
+WantedBy=spinifex.target ovs-vswitchd.service

build/systemd/spinifex-sqs-store.service

@@ -0,0 +1,57 @@
+# spinifex-unit-version: 1
+[Unit]
+Description=Spinifex SQS Message Store (dedicated JetStream)
+PartOf=spinifex.target
+ConditionPathExists=/etc/spinifex/sqs-store/sqs-store.conf
+
+[Service]
+EnvironmentFile=-/etc/spinifex/telemetry.env
+Type=simple
+Slice=spinifex-system.slice
+User=spinifex-sqsstore
+Group=spinifex
+ExecStart=/usr/local/bin/spx service sqs-store start
+Restart=on-failure
+RestartSec=5
+# JetStream store shutdown and raft stepdown have no internal deadline.
+TimeoutStopSec=20
+OOMScoreAdjust=-500
+# Soft Go heap limit for the store; MemoryHigh/Max bound a runaway on top.
+Environment=GOMEMLIMIT=512MiB
+MemoryHigh=10%
+MemoryMax=20%
+Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/sqs-store/sqs-store.conf
+
+# Filesystem access
+NoNewPrivileges=yes
+ProtectSystem=strict
+PrivateDevices=yes
+ReadOnlyPaths=/etc/spinifex/sqs-store /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
+ReadWritePaths=/var/lib/spinifex/sqs-store /var/log/spinifex
+
+# Hardening
+ProtectHome=yes
+PrivateTmp=yes
+RestrictSUIDSGID=yes
+LockPersonality=yes
+RestrictRealtime=yes
+RestrictNamespaces=yes
+ProtectClock=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+ProtectControlGroups=yes
+ProtectProc=invisible
+SystemCallArchitectures=native
+MemoryDenyWriteExecute=yes
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
+CapabilityBoundingSet=
+ProtectHostname=yes
+KeyringMode=private
+RemoveIPC=yes
+LimitCORE=0
+LimitNOFILE=65536
+LimitNPROC=4096
+
+[Install]
+WantedBy=spinifex.target

build/systemd/spinifex-vpcd.service

@@ -1,9 +1,9 @@
-# spinifex-unit-version: 1
+# spinifex-unit-version: 3
 [Unit]
 Description=Spinifex VPC Daemon (OVN Networking)
 PartOf=spinifex.target
-After=spinifex-nats.service spinifex-daemon.service systemd-networkd.service openvswitch-switch.service ovn-controller.service
-Wants=spinifex-nats.service ovn-controller.service
+After=spinifex-nats.service spinifex-daemon.service systemd-networkd.service ovsdb-server.service ovs-vswitchd.service ovn-controller.service spinifex-hostnet.service
+Wants=spinifex-nats.service ovn-controller.service spinifex-hostnet.service
 
 [Service]
 EnvironmentFile=-/etc/spinifex/telemetry.env
@@ -11,6 +11,7 @@ Type=simple
 Slice=spinifex-system.slice
 User=spinifex-vpcd
 Group=spinifex
+ExecStartPre=/usr/local/bin/spx admin hostnet check
 ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
 ExecStart=/usr/local/bin/spx service vpcd start
 Restart=on-failure

build/systemd/spinifex.target

@@ -1,9 +1,9 @@
-# spinifex-unit-version: 1
+# spinifex-unit-version: 3
 [Unit]
 Description=Spinifex Infrastructure Platform
 After=network-online.target
 Wants=network-online.target
-Wants=spinifex-nats.service spinifex-predastore.service spinifex-northstar.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service spinifex-qmp-collector.service spinifex-shutdown.service spinifex-nats-watchdog.timer openvswitch-switch.service ovn-controller.service
+Wants=spinifex-nats.service spinifex-sqs-store.service spinifex-predastore.service spinifex-northstar.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service spinifex-qmp-collector.service spinifex-shutdown.service spinifex-nats-watchdog.timer ovsdb-server.service ovs-vswitchd.service ovn-controller.service
 
 [Install]
 WantedBy=multi-user.target

cmd/aws-model-coverage/main.go

@@ -4,11 +4,13 @@ import (
 	"flag"
 	"fmt"
 	"os"
+	"slices"
 
 	"github.com/mulgadc/predastore/s3api"
 	"github.com/mulgadc/spinifex/internal/awsmodel"
 	acmawsapi "github.com/mulgadc/spinifex/spinifex/domains/acm/awsapi"
 	awsapi "github.com/mulgadc/spinifex/spinifex/domains/ecr/awsapi"
+	route53awsapi "github.com/mulgadc/spinifex/spinifex/domains/route53/awsapi"
 	"github.com/mulgadc/spinifex/spinifex/gateway"
 	"github.com/mulgadc/spinifex/spinifex/ingress/aws/dispatch"
 
@@ -49,9 +51,17 @@ func compareAll() ([]awsmodel.OperationCoverage, error) {
 	// Registered services contribute their declared inventory, which needs no
 	// live dependencies; nothing here serves a request.
 	inventories := gateway.AWSOperationInventory(map[string]dispatch.Inventory{
-		awsapi.ServiceName:    awsapi.OperationInventory(),
-		acmawsapi.ServiceName: acmawsapi.OperationInventory(),
+		awsapi.ServiceName:        awsapi.OperationInventory(),
+		acmawsapi.ServiceName:     acmawsapi.OperationInventory(),
+		route53awsapi.ServiceName: route53awsapi.OperationInventory(),
 	})
+	// The loop below walks the embedded models, so a served service with no
+	// model would otherwise drop out of the report without a word.
+	for name := range inventories {
+		if !slices.Contains(awsmodel.Services(), awsmodel.Service(name)) {
+			return nil, fmt.Errorf("no embedded AWS model for served service %s", name)
+		}
+	}
 	coverages := make([]awsmodel.OperationCoverage, 0, len(awsmodel.Services()))
 	for _, service := range awsmodel.Services() {
 		inventory, ok := inventories[string(service)]

cmd/eks-token-webhook/main.go

@@ -29,6 +29,7 @@ func loadConfig() (tokenwebhook.Config, error) {
 		CertPath:    envOr("EKS_WEBHOOK_CERT", "/etc/spinifex-eks/token-webhook.crt"),
 		KeyPath:     envOr("EKS_WEBHOOK_KEY", "/etc/spinifex-eks/token-webhook.key"),
 		Kubeconfig:  envOr("EKS_WEBHOOK_KUBECONFIG", "/etc/spinifex-eks/token-webhook.kubeconfig"),
+		ReadyAddr:   os.Getenv("EKS_WEBHOOK_READY_ADDR"),
 	}
 	// Static SigV4 creds are optional: when absent, eksgw.New signs with the
 	// AWS SDK chain (IMDS instance-role creds), the same path the CP VM's

docs/COMMANDS.md

@@ -95,7 +95,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 
 | Command | Flags | Description |
 |---------|-------|-------------|
-| `spx admin init` | `--nodes`, `--node`, `--bind`, `--advertise` (external IP off-host clients dial; auto-detected from the default route), `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--predastore-compaction-interval` (seconds; test clusters set a short one), `--services`, `--formation-timeout`, `--token-ttl`, `--force`, `--discard-jetstream` (with `--nodes` >= 2, remove this node's existing JetStream streams once every node has joined; without it a multi-node init over a store that holds streams is refused, and it never proceeds while NATS is running), `--skip-host-dns`, `--email` (operator address for update and security notices), `--no-telemetry`, `--gpu-passthrough`, `--ipsec` (default true; encrypt intra-AZ Geneve via OVN native IPsec cluster-wide — disable only for a trusted single-rack lab), `--host` (leader to join; falls back to multicast discovery), `--external-mode` (`pool` \| `nat` routed NAT for non-bridgeable uplinks, single-node; pair with `setup-ovn.sh --nat-uplink`; note: system instances (ECS/EKS/load-balancer agents) are unsupported in nat v1 — they require `pool` mode), `--external-iface` (WAN NIC for br-external, auto-detected), `--external-source` (`dhcp` \| `static`), `--external-pool` (`start-end`), `--external-prefix-len` (default 24), `--external-gateway` (auto-detected), `--external-dns` (comma-separated IPv4 addresses, at most 3; the DNS servers written to the external pool's `dns_servers` and handed to guests, verbatim; when omitted they are detected from the host resolver at init time, so set it to give every node of a cluster the same value), `--external-bind-bridge` (default `br-wan` when `--external-source=dhcp`), `--gateway-ip` (OVN gateway router's external IP for SNAT; defaults to the pool's range start, required for `nat` mode without DHCP) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` → points the host resolver at this node's local northstar so the Spinifex zones (ELB/EKS names) resolve from the node itself (opt out with `--skip-host-dns`). `--force` on an already-initialized node is idempotent for crypto: it preserves the existing keys (master.key, predastore/viperblock encryption keys), system + admin credentials, and CA, and refreshes only the config files and the CA-signed server certificate (so a changed bind IP / SANs is picked up without breaking already-joined nodes or CA-baked AMIs). A genuine clean slate requires removing the data dirs first. |
+| `spx admin init` | `--nodes`, `--node`, `--bind`, `--advertise` (external IP off-host clients dial; auto-detected from the default route), `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--predastore-compaction-interval` (seconds; test clusters set a short one), `--services`, `--formation-timeout`, `--token-ttl`, `--force`, `--discard-jetstream` (with `--nodes` >= 2, remove this node's existing JetStream streams once every node has joined; without it a multi-node init over a store that holds streams is refused, and it never proceeds while NATS is running), `--skip-host-dns`, `--email` (operator address for update and security notices), `--no-telemetry`, `--gpu-passthrough`, `--ipsec` (default true; encrypt intra-AZ Geneve via OVN native IPsec cluster-wide — disable only for a trusted single-rack lab; on a host whose IPsec stack is libreswan it defaults to false and `--ipsec=true` is refused), `--host` (leader to join; falls back to multicast discovery), `--external-mode` (`pool` \| `nat` routed NAT for non-bridgeable uplinks, single-node; pair with `setup-ovn.sh --nat-uplink`; note: system instances (ECS/EKS/load-balancer agents) are unsupported in nat v1 — they require `pool` mode), `--external-iface` (WAN NIC for br-external, auto-detected), `--external-source` (`dhcp` \| `static`), `--external-pool` (`start-end`), `--external-prefix-len` (default 24), `--external-gateway` (auto-detected), `--external-dns` (comma-separated IPv4 addresses, at most 3; the DNS servers written to the external pool's `dns_servers` and handed to guests, verbatim; when omitted they are detected from the host resolver at init time, so set it to give every node of a cluster the same value), `--external-bind-bridge` (default `br-wan` when `--external-source=dhcp`), `--gateway-ip` (OVN gateway router's external IP for SNAT; defaults to the pool's range start, required for `nat` mode without DHCP) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` → points the host resolver at this node's local northstar so the Spinifex zones (ELB/EKS names) resolve from the node itself (opt out with `--skip-host-dns`). `--force` on an already-initialized node is idempotent for crypto: it preserves the existing keys (master.key, predastore/viperblock encryption keys), system + admin credentials, and CA, and refreshes only the config files and the CA-signed server certificate (so a changed bind IP / SANs is picked up without breaking already-joined nodes or CA-baked AMIs). A genuine clean slate requires removing the data dirs first. |
 | `spx admin join` | `--host` (required), `--node` (required), `--token` (required), `--bind`, `--advertise` (defaults to `--bind`, or the auto-detected WAN IP when `--bind` is 0.0.0.0), `--port`, `--region`, `--az`, `--cluster-bind`, `--cluster-routes`, `--data-dir`, `--services`, `--skip-host-dns`, `--email`, `--no-telemetry`, `--predastore-compaction-interval`, `--join-timeout` (how long to keep retrying while the formation server is unreachable, default 20m), `--force` (join even though this node is already initialized or holds JetStream streams, **discarding its own CA, master key and JetStream store**; the store is removed only after formation succeeds, and the join is refused while NATS is running even with `--force`) | Connects to leader node with join token (Authorization: Bearer header) → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations → points the host resolver at this node's local northstar so the Spinifex zones resolve from the node itself (opt out with `--skip-host-dns`) |
 
 ### Version
@@ -127,7 +127,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 
 | Command | Flags | Description |
 |---------|-------|-------------|
-| `spx admin preflight` | `--quiet` (print only non-OK rows) | Reports whether this host has the privileged helper scripts and sudoers grants the installed build depends on. `spx version` knows the build but not the host: `setup.sh` and `setup-ovn.sh` install those assets, and a binary-swap deploy (`make deploy`, or a manual copy) skips them. It also reports any running service still executing a replaced binary — overwriting `/usr/local/bin/spx` while `spinifex.target` is up does not move services onto it, so each keeps executing the replaced inode until it restarts and a node can serve a mixture of two builds indefinitely. Checks compare content, not existence, so a present-but-stale asset is caught. Exits non-zero if any asset is Missing, Stale or Ungranted. |
+| `spx admin preflight` | `--quiet` (print only non-OK rows) | Reports whether this host has the privileged helper scripts and sudoers grants the installed build depends on. `spx version` knows the build but not the host: `setup.sh` and `setup-ovn.sh` install those assets, and a binary-swap deploy (`make deploy`, or a manual copy) skips them. It also reports any running service still executing a replaced binary — overwriting `/usr/local/bin/spx` while `spinifex.target` is up does not move services onto it, so each keeps executing the replaced inode until it restarts and a node can serve a mixture of two builds indefinitely. Checks compare content, not existence, so a present-but-stale asset is caught. On x86_64 it also checks that the QEMU binary the daemon will run lists the `microvm` machine type and that the microvm firmware `/usr/share/spinifex/microvm/qboot.rom` is installed, since load balancers boot on both. It runs the host network check `spinifex-daemon` and `spinifex-vpcd` run before they start, and fails while firewalld or ufw is active, since the Spinifex nftables policy is the only supported node firewall. It reports a Warning, without failing, when systemd-resolved is active but `/etc/resolv.conf` is not its stub, since lookups then bypass the route that sends Spinifex names to northstar. Exits non-zero if any asset is Missing, Stale, Ungranted, Unsupported or Failed. |
 | `spx admin node js-probe` | `--timeout` (default: 10s) | Round-trips a canary key through the cluster-state KV bucket and exits non-zero if the write or the read-back fails. JetStream can latch a write error per message block and keep serving reads — nats-server's filestore records the failure on the block and returns it on every later write, and nothing clears it short of a restart. The HTTP health endpoint reports the server-wide JetStream state, which that latch never touches, so only an actual write observes it. |
 | `spx admin banner` | `--boot-check` | Writes the node information banner to `/etc/issue` (shown before login on the physical/serial console) and `/etc/motd` (shown after SSH login). With `--boot-check` it also detects a management IP that has changed since last boot and updates `/etc/spinifex/node.conf` accordingly. |
 
@@ -181,7 +181,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 | `spx admin gpu status` | `--node` (default: local node) | Queries `spinifex.node.status` fan-out → finds the target node response → prints Node, GPU hardware (model list or "none detected"), IOMMU state, vfio-pci state, passthrough enabled/disabled, and GPU pool allocation (`allocated/total`). Also lists GPU-capable instance types when passthrough is active. |
 | `spx admin gpu enable` | — | Checks current passthrough state via NATS → errors if already enabled or prerequisites not met (directs to `setup`) → writes `gpu_passthrough = true` to `spinifex.toml` via `admin.SetGPUPassthrough` → sends SIGHUP to `spinifex-daemon` via `systemctl kill -s HUP` → polls node status for up to 30 s until daemon confirms new state → prints final `gpu status` output. Must be run directly on the target host. |
 | `spx admin gpu disable` | — | Checks current passthrough state via NATS → errors if already disabled or if `AllocGPUs > 0` (must terminate GPU instances first) → writes `gpu_passthrough = false` → sends SIGHUP to `spinifex-daemon` → polls for up to 30 s → prints final `gpu status` output. Must be run directly on the target host. |
-| `spx admin gpu setup` | — | Idempotent host configuration for GPU passthrough. Steps (skipped if already applied): detect GPUs via `gpu.Discover` → collect PCI IDs for all IOMMU-group siblings → check/enable IOMMU in GRUB (`intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt`) → write vfio udev rule (`/etc/udev/rules.d/99-spinifex-vfio.rules`) → blacklist nouveau (`/etc/modprobe.d/blacklist-nouveau.conf`) → blacklist amdgpu if AMD GPU present (`/etc/modprobe.d/blacklist-amdgpu.conf`) → write vfio-pci early binding config (`/etc/modprobe.d/vfio-pci.conf`) → add vfio modules to initramfs. If any change requires a reboot: runs `update-initramfs -u` and exits with reboot instructions. After reboot: verifies `vfio_pci` module is loaded → verifies each GPU is bound to `vfio-pci` (binds explicitly via `driver_override` if unbound) → calls `gpu enable` to activate passthrough. |
+| `spx admin gpu setup` | — | Idempotent host configuration for GPU passthrough. Resolves its boot tools first (`update-grub` or `grubby`, `update-initramfs` or `dracut`) and stops before any change if either pair has neither or both. Steps (skipped if already applied): detect GPUs via `gpu.Discover` → collect PCI IDs for all IOMMU-group siblings → if IOMMU is inactive, add `intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt` to `GRUB_CMDLINE_LINUX_DEFAULT` and run `update-grub`, or run `grubby --update-kernel=ALL --args=...` and check the default entry and `/etc/kernel/cmdline` carry them → write vfio udev rule (`/etc/udev/rules.d/99-spinifex-vfio.rules`) → blacklist nouveau (`/etc/modprobe.d/blacklist-nouveau.conf`) → blacklist amdgpu if AMD GPU present (`/etc/modprobe.d/blacklist-amdgpu.conf`) → write vfio-pci early binding config (`/etc/modprobe.d/vfio-pci.conf`) → add vfio modules to `/etc/initramfs-tools/modules`, or write `/etc/dracut.conf.d/spinifex-vfio.conf` (`force_drivers` for the vfio modules, `install_items` for the `modprobe.d` files). If any change requires a reboot: runs `update-initramfs -u`, or `dracut -f --regenerate-all` and checks every kernel's image with `lsinitrd`, then exits with reboot instructions. After reboot: verifies `vfio_pci` module is loaded → verifies every PCI function in each GPU's slot is bound to `vfio-pci` (binds explicitly via `driver_override` if unbound, fails naming any function on another driver) → calls `gpu enable` to activate passthrough. |
 | `spx admin gpu mig status` | — (must be run on target host) | Runs `gpu.Discover()` locally → for each GPU prints: PCI address, model, MIG capability, MIG mode (enabled/disabled/N/A); for GPUs with MIG enabled lists active MIG slices with GI ID, profile name, and mdev path. |
 | `spx admin gpu mig enable` | `--profile <name>` (required, e.g. `1g.10gb`), `--gpu <pci-addr>` (optional, default: all MIG-capable GPUs) | Checks no GPU instances running (NATS) → discovers MIG-capable GPUs via `gpu.Discover()` (filtered by `--gpu` if set) → enables MIG mode on each target (`gpu.EnableMIGMode`) → lists available profiles (`gpu.ListProfiles`) and validates requested profile name → destroys any existing instances (`gpu.DestroyAllInstances`) → creates new instances filling GPU capacity (`gpu.CreateInstances`) → writes `mig_profile` to `spinifex.toml` via `admin.SetMIGProfile` → sends SIGHUP to `spinifex-daemon`. Must be run directly on the target host. |
 | `spx admin gpu mig disable` | `--gpu <pci-addr>` (optional, default: all MIG-capable GPUs) | Checks no GPU instances running (NATS) → discovers MIG-capable GPUs via `gpu.Discover()` (filtered by `--gpu` if set) → destroys all GPU instances (`gpu.DestroyAllInstances`) → disables MIG mode (`gpu.DisableMIGMode`) → clears `mig_profile` in `spinifex.toml` via `admin.SetMIGProfile` → sends SIGHUP to `spinifex-daemon`. Must be run directly on the target host. |

docs/DESIGN.md

@@ -229,6 +229,20 @@ type EBSRequest struct {
 
 Viperblock responds with an NBD URI that QEMU uses to access the block device.
 
+#### Direct-boot microVM data volumes
+
+A direct-boot microVM keeps the launcher's saved `vm.Config` as a clean base, and `startQEMU` adds data volumes only to a launch-only copy of it (`runtime/compute/vm/lifecycle.go`).
+At each start the daemon takes one snapshot of the instance's already-mounted data requests, probes each NBD or host block export from that snapshot, and builds the command from the same snapshot, so the launched endpoint is the one that was checked and is never written back into the saved base.
+Each data volume becomes a `virtio-blk-device` on the MMIO bus with the same id, node name, I/O thread, serial and `werror=stop,rerror=stop` policy as the PCI device, and takes no PCIe hot-plug port.
+
+Boot and EFI requests are rejected on this path, and a direct-boot instance with no data requests launches its base unchanged.
+Disk-boot (q35) instances keep the existing PCI hot-plug path.
+
+Hot-attaching a volume to a running direct-boot microVM is not supported, because the MMIO bus has no PCIe root port to plug into.
+`AttachVolume` refuses it before mounting the volume or changing any instance or volume state, and the EC2 API returns `UnsupportedOperation`.
+
+This covers only requests that are already mounted when the instance starts; nothing yet supplies data volumes to a system instance, and the path is verified with fake-QEMU unit tests, not with a real EBS volume in a guest.
+
 ### Predastore (S3)
 
 Object storage used for:
@@ -256,6 +270,29 @@ file. It does not connect to Predastore to do so, and no Predastore node
 exposes a status endpoint. This is an operator status interface, not a generic
 storage-provider contract.
 
+### SQS message store (internal)
+
+`domains/sqs/message` defines the store-neutral interface the SQS domain reaches its message store through (ADR-0011:S6), with a JetStream adapter in `domains/sqs/message/jetstream` and a shared conformance suite in `domains/sqs/message/conformance`.
+It is an internal primitive: nothing outside the SQS domain uses it, no daemon or service is wired to it, and it does not itself deliver an SQS API.
+The dedicated JetStream server ADR-0011 describes is not deployed by this code.
+
+The interface covers durable and idempotent append, body and metadata reads, conditional state writes and deletes, per-namespace counts, namespace enumeration, an ordered change feed, a bounded paginated scan, namespace deletion and a sweep of tombstoned bodies.
+A namespace carries an immutable message-shard and state-shard allocation chosen when the queue is created, so a later change in shard counts never moves an existing queue.
+
+The adapter provisions a fixed set of shared message streams (`SQS_M_<shard>`, one subject per queue) and shared state buckets (`sqs_state_<shard>`), keeps historical ones, and applies a 14-day-plus-one-hour retention backstop to both.
+It supports one node (R1) and three nodes (R3) only and refuses any other declared cluster size before provisioning.
+
+Reads that must observe a completed write or delete go to the stream leader: `GetState`, the state-key pass behind `Count` and `Namespaces`, and the body pass behind `Namespaces` and `Scan`.
+This avoids the follower-served direct gets and follower-placed consumers that returned purged data in local three-node fault-injection tests; it is not a general linearizability guarantee.
+Each leader read is one request per record, and the body pass transfers each full body internally even though `Scan` returns metadata only, which is a material cost for a large backlog.
+A state-key pass is bounded but is not a snapshot when a key is overwritten during it.
+
+The feed reads through an in-memory, headers-only ordered consumer.
+If that consumer delivers nothing for about two seconds while the stream holds a change after the feed's cursor, the feed replaces it from the cursor, so a stored change does not wait on a stalled consumer; the cause of those stalls is not established.
+`Close` is terminal, waits for consumer opens and deletes in progress and retries failed deletes, but cleanup is best effort for a create whose outcome is unknown: a consumer that lands after its last delete is removed by the server's one-minute inactive threshold.
+
+Coverage is unit and conformance tests on in-process R1 and R3 servers; there is no end-to-end or multi-host validation of this code, and earlier multi-host evidence predates the final feed.
+
 ## Configuration
 
 Cluster configuration (`spinifex/bootstrap/config/config.go`):

docs/PACKAGE_BOUNDARY_MIGRATION.md

@@ -438,7 +438,7 @@ These residuals were recorded by individual slices and must be resolved once the
 - `runtime/roles/vpcd` (placement): the role still mixes process/role supervision with network-controller responsibilities. `lease_reaper.go` (orphaned external-pool DHCP lease sweep), `ovn_controller_watchdog.go` (OVN controller liveness and restart), `host_eip_loop.go` (host-EIP reconcile loop) and `lease_datapath_reconcile.go` (lease datapath reconcile) are periodic network-control loops, not role wiring; `vpcd.go` itself also carries bridge-mode detection and verification, chassis-list parsing, external-CIDR resolution, DHCP manager startup and lease/IP-change handling, and gateway/host-EIP allocator and neighbour flush/prime construction alongside `Service.Start`. The future placement question is whether these belong in `domains/network`. Removal condition: an accepted decision on network-controller placement, and extraction of those loops and helpers behind the role.
 - ECR control-plane registration: `awsapi.Deps.NATS` is a raw `*nats.Conn`, not a narrow capability. Every NATS-backed `Handler` passes it to `domains/ecr.NewNATSMetaStore`, which calls `natsmsg.NATSRequest`, so narrowing it means changing the exported `Handler` type, every handler and the meta store; it must not be broadened further meanwhile.
 - ACM control-plane registration: `domains/acm/awsapi.Deps.NATS` (`registration.go`) is likewise a raw `*nats.Conn` carried by every `Handler`, for the same reason; it must not be broadened further meanwhile.
-- `cmd/aws-model-coverage` imports `ingress/aws/dispatch`, `domains/ecr/awsapi`, `domains/acm/awsapi` and `gateway` to read ECR's and ACM's declared inventories. It is a non-serving tool and the only named exception to the ADR-0002 composition invariant; it must not construct a registration.
+- `cmd/aws-model-coverage` imports `ingress/aws/dispatch`, `domains/ecr/awsapi`, `domains/acm/awsapi`, `domains/route53/awsapi` and `gateway` to read ECR's, ACM's and Route 53's declared inventories. It is a non-serving tool and the only named exception to the ADR-0002 composition invariant; it must not construct a registration.
 - Ingress pipeline (ADR-0002 phase 4): the request-audit record, the trace action enricher, the throttle key functions and the gateway request-context keys stay in `gateway` until the auth/IAM and request-context boundary is designed. SigV4 authentication writes the audit record and context keys from four places, so moving them now would export exactly the setters ADR-0002 says must not be exported solely to make a directory move possible.
 - NAT projection delivery is inconsistent and is preserved as found: EIP `AssociateAddress` sends `vpc.add-nat` unacknowledged (`AnnounceNATAdd`), skipping the 45s OVN barrier every other add path waits on, and `DeleteNetworkInterface` sends `vpc.delete-nat` unacknowledged (`AnnounceNATDelete`), skipping the ack and the ErrNoResponders retry. The reconciler's orphan sweep is the only backstop for both. Clears when those call sites use `AddNAT`/`AddNATBestEffort` and `RemoveNAT`, a reliability change of its own.
 - `vpc.add-system-egress` has a `vpcd` subscriber and a contract entry but no producer (its former `domains/ec2/eip` publisher is gone); `domains/eks/cluster` still publishes the delete. Either a producer is missing or the add route is dead; it needs an owner decision.

docs/admin/host-firewall/README.md

@@ -73,6 +73,7 @@ The cluster plane is peer-scoped. On a single-NIC node the planes collapse onto
 | Ports | Serving |
 |---|---|
 | TCP 4222, 4248 | NATS client and cluster routes |
+| TCP 4223, 4249 | SQS message store client and cluster routes |
 | TCP 4432 | Formation, then the daemon cluster manager |
 | TCP and UDP 5300 | Internal DNS |
 | TCP 6641, 6642 | OVN northbound and southbound |
@@ -157,6 +158,7 @@ Restarting the daemon is what re-arms it: the node rebuilds its peer list from t
 
 > [!WARNING]
 > Do not flush the whole nftables ruleset to clear the policy. The VPC daemon writes SNAT and per-elastic-IP forward rules into the `ip nat` and `ip filter` tables and reinstalls them only when it starts, so `nft flush ruleset` breaks elastic IPs silently until the next `systemctl restart spinifex-vpcd`. Use `disable`, which only ever touches `inet spinifex_filter`.
+> On Oracle Linux with IPsec on, a flush also removes `inet spinifex_ipsec_guard`, so Geneve to a peer without an SA goes out in clear until the daemon puts the table back, within 60 seconds.
 
 ## The Files
 

docs/compute/gpu-passthrough/README.md

@@ -51,12 +51,24 @@ sudo spx admin gpu setup
 What it does:
 
 1. Detects installed GPUs and their PCI IDs.
-2. Enables IOMMU in GRUB (`intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt`).
+2. Adds the IOMMU kernel arguments (`intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt`): to `GRUB_CMDLINE_LINUX_DEFAULT` with `update-grub`, or with `grubby` to every boot entry and to `/etc/kernel/cmdline`, which kernels installed later inherit.
 3. Writes the vfio udev rule (`/etc/udev/rules.d/99-spinifex-vfio.rules`).
 4. Blacklists `nouveau` (NVIDIA) and `amdgpu` (AMD) so the host kernel does not claim the device.
 5. Configures `vfio-pci` early binding in `/etc/modprobe.d/vfio-pci.conf`.
-6. Adds vfio modules to initramfs.
+6. Loads the vfio modules from the initramfs: through `/etc/initramfs-tools/modules` and `update-initramfs -u`, or through `/etc/dracut.conf.d/spinifex-vfio.conf` and `dracut -f --regenerate-all`. With dracut, each rebuilt image is checked with `lsinitrd` for `vfio_pci`, its early-load line and the `modprobe.d` files, and setup stops naming whatever is missing.
 
+The command finds its boot tools before changing anything, and stops naming them if the host has neither or both of `update-grub` and `grubby`, or of `update-initramfs` and `dracut`.
+
+**Oracle Linux hosts** take the `grubby` and `dracut` path: Oracle Linux keeps kernel arguments in BLS boot entries and has no `GRUB_CMDLINE_LINUX_DEFAULT`.
+`grubby --update-kernel=ALL` writes the IOMMU arguments to every entry and to `/etc/kernel/cmdline`, from which a kernel installed later builds its own entry, and dracut applies `spinifex-vfio.conf` to every initramfs it builds.
+After installing a new `kernel-uek`, check its entry before rebooting into it:
+
+```bash
+sudo grubby --info=ALL | grep -E '^(kernel|args)'
+sudo lsinitrd -k <new-kernel-version> | grep vfio
+```
+
+Whole-GPU passthrough is supported on Oracle Linux; NVIDIA MIG is not.
 If any change requires a reboot, the command prints instructions and exits:
 
 ```
@@ -175,7 +187,7 @@ The command signals the daemon and waits for confirmation. The GPU is released b
 
 ### IOMMU not active after reboot
 
-Verify IOMMU is enabled in BIOS/UEFI (`Intel VT-d` or `AMD-Vi`). Then check GRUB picked up the parameters:
+Verify IOMMU is enabled in BIOS/UEFI (`Intel VT-d` or `AMD-Vi`). Then check the boot loader picked up the parameters (on Oracle Linux, `sudo grubby --info=DEFAULT` lists them for the default entry):
 
 ```bash
 cat /proc/cmdline | grep iommu

docs/compute/vpc-networking/README.md

@@ -144,6 +144,27 @@ The guest MTU moves with the setting, but a running guest holds its DHCP lease f
 
 **Expect launches to fail for a minute or two after re-enabling.** `RunInstances` returns `ServerInternal` with `vpc.add-nat` timing out, because the OVN flows-ready barrier cannot confirm while `ovn-controller` reprograms every tunnel for the new SAs. It clears once `ip xfrm state` settles. The error is reported as a capacity problem, which it is not.
 
+### Containers and nested overlays inside a guest
+
+A Docker bridge, Kubernetes pod network or other overlay inside a guest defaults to an MTU of 1500, which is larger than the guest's 1408 or 1442.
+Downloads from the internet still work for TCP on the veth and routed uplinks, because vpcd clamps the TCP MSS on the host hop (see [Performance Tuning → MTU](/docs/performance-tuning#4-mtu)).
+That was measured on a single node (env12) with a Docker container; it was not measured on a physical uplink or on a multi-node cluster.
+
+On a physical uplink there is no host hop to clamp, and UDP is never clamped, so set the inner MTU to the guest MTU.
+For Docker, in `/etc/docker/daemon.json`:
+
+```json
+{
+  "mtu": 1408,
+  "default-network-opts": {
+    "bridge": { "com.docker.network.driver.mtu": "1408" }
+  }
+}
+```
+
+Use 1442 when `ipsec_enabled = false`, and restart the Docker daemon so the default bridge is recreated.
+The gateway MTU follows `ipsec_enabled` and converges on the next reconcile pass, so change the inner MTU together with the guest MTU.
+
 ## Private vs Public Subnets
 
 A subnet's behavior depends on three things: whether the VPC has an Internet Gateway, whether the subnet's route table has a default route to that IGW, and whether the subnet has `MapPublicIpOnLaunch` enabled.

docs/containers/ecr/README.md

@@ -45,6 +45,7 @@ Each repository holds image manifests and their layers. Images are stored in obj
 
 - Repository lifecycle — `CreateRepository`, `DeleteRepository`, `DescribeRepositories`.
 - OCI push and pull — blobs, manifests, and tags, including cross-repository blob mounts.
+- OCI 1.1 referrers — the referrers API, so cosign v3 signatures and other artifacts attached to an image can be listed (see [OCI referrers and image signing](#oci-referrers-and-image-signing)).
 - Token auth — `GetAuthorizationToken` plus bearer/basic auth on `/v2/`.
 - Repository policies — set, get, and delete a repository policy document.
 
@@ -176,6 +177,71 @@ Then authenticate Docker and push using the `repository_url` output, following s
 
 :::
 
+## EKS workers and multiple gateways
+
+Each gateway puts its own address in the `repositoryUri` it returns, so repositories created through different gateways can carry different hosts.
+When an EKS worker launches, the daemon that launches it writes every cluster gateway into the worker's registry mirrors, TLS trust and credential-provider config.
+A pod can then pull from any `repositoryUri` host in the cluster.
+The worker tries the gateway named in the image reference first and falls back to the others in cluster order.
+
+Limits:
+
+- The gateway list comes from the cluster config the launching daemon loaded at start.
+  A gateway added later is not trusted by existing workers; restart the launching daemon and replace the worker.
+- Peer gateways are assumed to listen on port 9999.
+  A hand-edited `awsgw` bind on a peer is not covered.
+- A gateway address that does not answer costs about 30 seconds on each cold pull while the worker's connection attempt gives up.
+  A stopped gateway on a host that is still up refuses the connection, and the worker moves to the next gateway at once.
+
+Measured on a single-node environment: pulls by `repositoryUri` and by the account's ECR DNS name, a pull rejected with `x509: certificate signed by unknown authority` when the gateway host was not in the worker config, and a 31 second pull through an unreachable first endpoint.
+Measured on a 3-node environment: a pod pulling each of the three gateways' `repositoryUri` on a worker launched with this change (two of the three failed with `x509` on a worker launched without it), and a pull of the stopped gateway's own `repositoryUri` that succeeded through another gateway within one retry (the pull itself took 147 ms after the first endpoint refused).
+
+## OCI referrers and image signing
+
+A manifest can name another manifest as its `subject`, which is how tools such as cosign v3 attach signatures and SBOMs to an image.
+The registry records the subject when the manifest is pushed and answers with an `OCI-Subject` header on the `201`.
+The `subject` does not have to exist in the repository when the referrer is pushed.
+
+`GET /v2/<name>/referrers/<digest>` lists the manifests that name `<digest>` as their subject.
+The request is authorised as `ecr:BatchGetImage` on the repository.
+The response is `200` with `Content-Type: application/vnd.oci.image.index.v1+json` and an index whose `manifests` hold each referrer's `mediaType`, `digest`, `size`, `artifactType` and annotations.
+A digest with no referrers, or one that is not stored, returns an empty index.
+A malformed digest returns `400` with `DIGEST_INVALID`.
+
+Adding `?artifactType=<type>` returns only referrers of that type and sets `OCI-Filters-Applied: artifactType`.
+The header is also set when nothing matches.
+An image manifest without an `artifactType` is listed under its config `mediaType`; an index without one has none.
+
+Deleting an image by digest (`DELETE /v2/<name>/manifests/<digest>`, `aws ecr batch-delete-image` with `imageDigest`) also deletes its referrers, and their referrers in turn.
+Deleting by tag removes only the tag and leaves the image and its referrers.
+
+cosign v3 sign and verify work against the registry, including on a child manifest of an image index copied with `cosign copy`.
+cosign v2 keeps writing its `sha256-<digest>.sig` tags and is unaffected.
+`GET` on a manifest still returns `406` when the request's `Accept` header does not include the stored `mediaType`.
+
+Limits:
+
+- `cosign copy` does not copy v3 signatures to the destination repository.
+  This is client behaviour.
+  To keep the original signature, copy with `oras cp -r <source> <mirror>`, which copies the image and its referrers in one command, or `crane copy` the signature manifest by digest after the image.
+  Otherwise, sign the copy again.
+- Referrers pushed before this version are not listed.
+  Push them again to add them to the index.
+- A crash between deleting an image and deleting its referrers can leave referrers behind that nothing lists.
+- The referrers listing is not paginated.
+- `HEAD` on the referrers route is not served.
+
+Measured on a 3-node environment with cosign v3.1.4 and v2.6.5:
+
+- cosign v3.1.4 signed a fresh image index, with one `GET /referrers` answered `200`, no `sha256-<digest>` tag written and no `406`.
+- Verify passed.
+- A copied index's child manifest signed and verified, which returned `406` before this change.
+- The listing returned the signature with its `artifactType`, the filter returned one entry for the signature's type and none for another type.
+- `aws ecr batch-delete-image` of the signed image left an empty listing and `404` for both the signature and the image.
+- cosign v2.6.5 signed and verified a fresh image.
+
+Not measured: `oras`, `notation`, a listing with many referrers, and a crash in the middle of a delete.
+
 ## Troubleshooting
 
 **`docker login` fails with an authorization error.** The token is account-scoped and short-lived. Re-run `aws ecr get-login-password` to mint a fresh token, and confirm the registry host in `docker login` matches your account's endpoint (`aws ecr describe-repositories` returns it).

docs/containers/eks/README.md

@@ -60,6 +60,36 @@ Authentication uses **EKS access entries** (the API authentication mode) — IAM
 - **The worker image is fixed.** Node groups always boot Spinifex's `eks-node` image. `ami_type` is recorded but does not select the image, and the image must be registered on the cluster before you create a node group.
 - **Default Kubernetes version is `1.32`.**
 
+### Control plane readiness and kubectl authentication
+
+Each control-plane VM authenticates `kubectl` tokens through a webhook that calls the AWS gateway on the host the VM runs on.
+When that gateway is stopped, the webhook cannot validate tokens and `kubectl` reports `Unauthorized` for the share of requests that the load balancer sends to that control plane.
+
+The webhook serves `/readyz` on port 8444 of each control-plane VM.
+It returns 200 only when the host gateway answers a TLS handshake and the local API server accepts a TCP connection, and 503 otherwise.
+The API server target group health-checks `HTTP :8444/readyz` every 5 seconds with a healthy and unhealthy threshold of 2.
+The target group still forwards TCP on 6443.
+A control plane that cannot authenticate leaves rotation, and returns when its gateway is back.
+
+Measured on a three-node cluster, one control plane per host, with `kubectl get ns` run continuously after stopping one gateway:
+
+| Gateway stopped | Result |
+| --- | --- |
+| Host without the cluster load balancer VM, first run | 4 of 20 calls failed in the first round, 0 of 20 in the next round 54 s after the stop |
+| Host without the cluster load balancer VM, second run | 2 failures in 100 s, at 3 s and 8 s after the stop |
+| Host running the cluster load balancer VM | Not fixed: 20 of 60 calls failed over 100 s |
+
+Before this change the same test failed 3, 4 and 3 calls of 10 per stopped gateway.
+
+Limits:
+
+- Only clusters created with the current `eks-node` image get the readiness listener, and only clusters created from now on get the health check on the target group.
+  Older clusters keep the TCP check on 6443 and are not changed.
+- A cluster created with an older image gets the health check on its target group but no listener on 8444, so every control plane fails the check and the load balancer fails open to all of them.
+- A stopped gateway on the host that runs the cluster load balancer VM is not covered.
+  The load balancer VM reports target health through its own host's gateway, so while that gateway is down the unhealthy control plane stays in rotation.
+- Not measured: more than one control plane per host, a physical-node failure instead of a stopped gateway, and the target health state as `DescribeTargetHealth` reports it, because the load balancer and control-plane VMs belong to the system account.
+
 ## Prerequisites
 
 > [!IMPORTANT]
@@ -123,7 +153,7 @@ aws eks describe-cluster --name demo --query 'cluster.status'
 aws eks create-nodegroup \
   --cluster-name demo \
   --nodegroup-name default \
-  --node-role-arn arn:aws:iam::000000000000:role/eks-node-role \
+  --node-role arn:aws:iam::000000000000:role/eks-node-role \
   --subnets subnet-aaaa subnet-bbbb \
   --scaling-config minSize=1,maxSize=2,desiredSize=1 \
   --instance-types t3.medium
@@ -256,6 +286,10 @@ The workbooks form a ladder. **eks-quickstart** is a NodePort demo; **eks-https-
 
 **`kubectl` cannot connect.** Re-run `aws eks update-kubeconfig --name <name>` to refresh the endpoint and CA, and confirm the cluster is `ACTIVE`. `aws sts get-caller-identity` verifies your credentials reach the Spinifex endpoint.
 
+**`kubectl` returns `Unauthorized` for some requests and works for others.**
+One control plane cannot reach its host's AWS gateway, so check `systemctl status spinifex-awsgw` on each node.
+Control plane readiness and kubectl authentication above lists which clusters recover on their own.
+
 **Workers stay `NotReady`.** Almost always an egress or IAM problem:
 - Confirm the workers can pull images — a default route to an Internet Gateway (public) or NAT Gateway (private).
 - Confirm the node role has `AmazonEKSWorkerNodePolicy`, `AmazonEKS_CNI_Policy`, and `AmazonEC2ContainerRegistryReadOnly`.

docs/coverage/README.md

@@ -17,7 +17,7 @@ tags:
 
 ## Overview
 
-The platform serves **392 operations** across the AWS APIs below. Every page names the operations its service implements from the pinned model, generated from the dispatch tables on each build rather than written by hand.
+The platform serves **405 operations** across the AWS APIs below. Every page names the operations its service implements from the pinned model, generated from the dispatch tables on each build rather than written by hand.
 
 | Service | Operations |
 |---|---:|
@@ -29,6 +29,7 @@ The platform serves **392 operations** across the AWS APIs below. Every page nam
 | [ELBv2](/coverage/elbv2) | 36 |
 | [IAM](/coverage/iam) | 81 |
 | [RDS](/coverage/rds) | 27 |
+| [Route 53](/coverage/route53) | 13 |
 | [S3](/coverage/s3) | 26 |
 | [STS](/coverage/sts) | 5 |
-| **Total** | **392** |
+| **Total** | **405** |

docs/coverage/coverage.json

@@ -3483,6 +3483,180 @@
         ]
       }
     },
+    {
+      "service": "route53",
+      "api_version": "2013-04-01",
+      "opaque": false,
+      "note": "",
+      "operations": {
+        "modelled": [
+          "ActivateKeySigningKey",
+          "AssociateVPCWithHostedZone",
+          "ChangeCidrCollection",
+          "ChangeResourceRecordSets",
+          "ChangeTagsForResource",
+          "CreateCidrCollection",
+          "CreateHealthCheck",
+          "CreateHostedZone",
+          "CreateKeySigningKey",
+          "CreateQueryLoggingConfig",
+          "CreateReusableDelegationSet",
+          "CreateTrafficPolicy",
+          "CreateTrafficPolicyInstance",
+          "CreateTrafficPolicyVersion",
+          "CreateVPCAssociationAuthorization",
+          "DeactivateKeySigningKey",
+          "DeleteCidrCollection",
+          "DeleteHealthCheck",
+          "DeleteHostedZone",
+          "DeleteKeySigningKey",
+          "DeleteQueryLoggingConfig",
+          "DeleteReusableDelegationSet",
+          "DeleteTrafficPolicy",
+          "DeleteTrafficPolicyInstance",
+          "DeleteVPCAssociationAuthorization",
+          "DisableHostedZoneDNSSEC",
+          "DisassociateVPCFromHostedZone",
+          "EnableHostedZoneDNSSEC",
+          "GetAccountLimit",
+          "GetChange",
+          "GetCheckerIpRanges",
+          "GetDNSSEC",
+          "GetGeoLocation",
+          "GetHealthCheck",
+          "GetHealthCheckCount",
+          "GetHealthCheckLastFailureReason",
+          "GetHealthCheckStatus",
+          "GetHostedZone",
+          "GetHostedZoneCount",
+          "GetHostedZoneLimit",
+          "GetQueryLoggingConfig",
+          "GetReusableDelegationSet",
+          "GetReusableDelegationSetLimit",
+          "GetTrafficPolicy",
+          "GetTrafficPolicyInstance",
+          "GetTrafficPolicyInstanceCount",
+          "ListCidrBlocks",
+          "ListCidrCollections",
+          "ListCidrLocations",
+          "ListGeoLocations",
+          "ListHealthChecks",
+          "ListHostedZones",
+          "ListHostedZonesByName",
+          "ListHostedZonesByVPC",
+          "ListQueryLoggingConfigs",
+          "ListResourceRecordSets",
+          "ListReusableDelegationSets",
+          "ListTagsForResource",
+          "ListTagsForResources",
+          "ListTrafficPolicies",
+          "ListTrafficPolicyInstances",
+          "ListTrafficPolicyInstancesByHostedZone",
+          "ListTrafficPolicyInstancesByPolicy",
+          "ListTrafficPolicyVersions",
+          "ListVPCAssociationAuthorizations",
+          "TestDNSAnswer",
+          "UpdateHealthCheck",
+          "UpdateHostedZoneComment",
+          "UpdateHostedZoneFeatures",
+          "UpdateTrafficPolicyComment",
+          "UpdateTrafficPolicyInstance"
+        ],
+        "registered": [
+          "ChangeResourceRecordSets",
+          "ChangeTagsForResource",
+          "CreateHostedZone",
+          "DeleteHostedZone",
+          "GetChange",
+          "GetHostedZone",
+          "GetHostedZoneCount",
+          "ListHostedZones",
+          "ListHostedZonesByName",
+          "ListResourceRecordSets",
+          "ListTagsForResource",
+          "ListTagsForResources",
+          "UpdateHostedZoneComment"
+        ],
+        "implemented": [
+          "ChangeResourceRecordSets",
+          "ChangeTagsForResource",
+          "CreateHostedZone",
+          "DeleteHostedZone",
+          "GetChange",
+          "GetHostedZone",
+          "GetHostedZoneCount",
+          "ListHostedZones",
+          "ListHostedZonesByName",
+          "ListResourceRecordSets",
+          "ListTagsForResource",
+          "ListTagsForResources",
+          "UpdateHostedZoneComment"
+        ],
+        "stubbed": [],
+        "unsupported": [],
+        "missing": [
+          "ActivateKeySigningKey",
+          "AssociateVPCWithHostedZone",
+          "ChangeCidrCollection",
+          "CreateCidrCollection",
+          "CreateHealthCheck",
+          "CreateKeySigningKey",
+          "CreateQueryLoggingConfig",
+          "CreateReusableDelegationSet",
+          "CreateTrafficPolicy",
+          "CreateTrafficPolicyInstance",
+          "CreateTrafficPolicyVersion",
+          "CreateVPCAssociationAuthorization",
+          "DeactivateKeySigningKey",
+          "DeleteCidrCollection",
+          "DeleteHealthCheck",
+          "DeleteKeySigningKey",
+          "DeleteQueryLoggingConfig",
+          "DeleteReusableDelegationSet",
+          "DeleteTrafficPolicy",
+          "DeleteTrafficPolicyInstance",
+          "DeleteVPCAssociationAuthorization",
+          "DisableHostedZoneDNSSEC",
+          "DisassociateVPCFromHostedZone",
+          "EnableHostedZoneDNSSEC",
+          "GetAccountLimit",
+          "GetCheckerIpRanges",
+          "GetDNSSEC",
+          "GetGeoLocation",
+          "GetHealthCheck",
+          "GetHealthCheckCount",
+          "GetHealthCheckLastFailureReason",
+          "GetHealthCheckStatus",
+          "GetHostedZoneLimit",
+          "GetQueryLoggingConfig",
+          "GetReusableDelegationSet",
+          "GetReusableDelegationSetLimit",
+          "GetTrafficPolicy",
+          "GetTrafficPolicyInstance",
+          "GetTrafficPolicyInstanceCount",
+          "ListCidrBlocks",
+          "ListCidrCollections",
+          "ListCidrLocations",
+          "ListGeoLocations",
+          "ListHealthChecks",
+          "ListHostedZonesByVPC",
+          "ListQueryLoggingConfigs",
+          "ListReusableDelegationSets",
+          "ListTrafficPolicies",
+          "ListTrafficPolicyInstances",
+          "ListTrafficPolicyInstancesByHostedZone",
+          "ListTrafficPolicyInstancesByPolicy",
+          "ListTrafficPolicyVersions",
+          "ListVPCAssociationAuthorizations",
+          "TestDNSAnswer",
+          "UpdateHealthCheck",
+          "UpdateHostedZoneFeatures",
+          "UpdateTrafficPolicyComment",
+          "UpdateTrafficPolicyInstance"
+        ],
+        "extra": []
+      }
+    },
     {
       "service": "s3",
       "api_version": "2006-03-01",

docs/coverage/elbv2/README.md

@@ -24,6 +24,34 @@ Spinifex implements **36 operations** in the ELBv2 `2015-12-01` API model.
 
 The data plane is a system-managed load balancer VM, launched automatically during `CreateLoadBalancer`. Application Load Balancers run HAProxy for the L7 surface — rules, fixed responses and redirects over HTTP and HTTPS. Network Load Balancers run nginx `stream` for the L4 surface — TCP, UDP, TLS and TCP_UDP — because HAProxy cannot load-balance UDP. The agent selects the engine from the configuration the control plane delivers, so the choice follows the load balancer type and is not separately configurable.
 
+### Target health and routing
+
+Traffic goes only to targets that are healthy or still `initial`.
+A target that has failed its health check is removed from the data plane on the next render and put back when it passes again.
+When every target in a group is unhealthy, the load balancer fails open and sends traffic to all of them rather than refusing it.
+A Network Load Balancer renders every registered server in that case, and an Application Load Balancer falls back to unchecked backup servers that HAProxy uses only when every checked server is down.
+`DescribeTargetHealth` keeps reporting the targets as unhealthy while this happens.
+Draining targets and targets without an address never receive traffic.
+
+An `initial` target receives traffic before its first successful check.
+This differs from AWS, where a target takes no traffic until it passes.
+Health state is reset to `initial` for every target when the control plane restarts, so excluding `initial` targets would drop all traffic after a restart.
+
+Changing the health check of a target group with `ModifyTargetGroup` takes effect on the next render of every load balancer that uses it.
+The time from a backend failing to traffic stopping is the health check interval times the unhealthy threshold, plus the agent poll of about 5 seconds and the heartbeat that reports the result.
+Recovery takes the interval times the healthy threshold, plus the same delays.
+
+Measured on a three-node cluster with a 5 second interval and thresholds of 2:
+
+| Event | Time |
+| --- | --- |
+| Backend starts failing to reported `unhealthy` | 21 s |
+| Backend recovers to reported `healthy` | 15 s, serving again 6 s later |
+| Every backend failing to reported `unhealthy` | 12 s, fail open immediate |
+
+A TCP connection held through a Network Load Balancer survived four configuration reloads while the other backend changed state, because the agent applies nginx with a graceful reload and HAProxy with a graceful restart.
+The HAProxy configuration with the backup servers was accepted by a live Application Load Balancer, but its rendered text was not dumped and checked by hand.
+
 ### Operations
 
 | Operation |

docs/coverage/elbv2/intro.md

@@ -1,3 +1,31 @@
 ### Two data planes
 
 The data plane is a system-managed load balancer VM, launched automatically during `CreateLoadBalancer`. Application Load Balancers run HAProxy for the L7 surface — rules, fixed responses and redirects over HTTP and HTTPS. Network Load Balancers run nginx `stream` for the L4 surface — TCP, UDP, TLS and TCP_UDP — because HAProxy cannot load-balance UDP. The agent selects the engine from the configuration the control plane delivers, so the choice follows the load balancer type and is not separately configurable.
+
+### Target health and routing
+
+Traffic goes only to targets that are healthy or still `initial`.
+A target that has failed its health check is removed from the data plane on the next render and put back when it passes again.
+When every target in a group is unhealthy, the load balancer fails open and sends traffic to all of them rather than refusing it.
+A Network Load Balancer renders every registered server in that case, and an Application Load Balancer falls back to unchecked backup servers that HAProxy uses only when every checked server is down.
+`DescribeTargetHealth` keeps reporting the targets as unhealthy while this happens.
+Draining targets and targets without an address never receive traffic.
+
+An `initial` target receives traffic before its first successful check.
+This differs from AWS, where a target takes no traffic until it passes.
+Health state is reset to `initial` for every target when the control plane restarts, so excluding `initial` targets would drop all traffic after a restart.
+
+Changing the health check of a target group with `ModifyTargetGroup` takes effect on the next render of every load balancer that uses it.
+The time from a backend failing to traffic stopping is the health check interval times the unhealthy threshold, plus the agent poll of about 5 seconds and the heartbeat that reports the result.
+Recovery takes the interval times the healthy threshold, plus the same delays.
+
+Measured on a three-node cluster with a 5 second interval and thresholds of 2:
+
+| Event | Time |
+| --- | --- |
+| Backend starts failing to reported `unhealthy` | 21 s |
+| Backend recovers to reported `healthy` | 15 s, serving again 6 s later |
+| Every backend failing to reported `unhealthy` | 12 s, fail open immediate |
+
+A TCP connection held through a Network Load Balancer survived four configuration reloads while the other backend changed state, because the agent applies nginx with a graceful reload and HAProxy with a graceful restart.
+The HAProxy configuration with the backup servers was accepted by a live Application Load Balancer, but its rendered text was not dumped and checked by hand.

docs/coverage/internal-report.md

@@ -1243,6 +1243,71 @@ option-groups: Option groups configure engine add-ons for engines this platform
 - `DescribeOptionGroups`
 - `ModifyOptionGroup`
 
+## route53
+
+`2013-04-01` — 13 of 71 modelled operations implemented (18.3%), 0 stubbed, 0 not applicable, 0 registered outside the pinned model.
+
+### Not implemented
+
+- `ActivateKeySigningKey`
+- `AssociateVPCWithHostedZone`
+- `ChangeCidrCollection`
+- `CreateCidrCollection`
+- `CreateHealthCheck`
+- `CreateKeySigningKey`
+- `CreateQueryLoggingConfig`
+- `CreateReusableDelegationSet`
+- `CreateTrafficPolicy`
+- `CreateTrafficPolicyInstance`
+- `CreateTrafficPolicyVersion`
+- `CreateVPCAssociationAuthorization`
+- `DeactivateKeySigningKey`
+- `DeleteCidrCollection`
+- `DeleteHealthCheck`
+- `DeleteKeySigningKey`
+- `DeleteQueryLoggingConfig`
+- `DeleteReusableDelegationSet`
+- `DeleteTrafficPolicy`
+- `DeleteTrafficPolicyInstance`
+- `DeleteVPCAssociationAuthorization`
+- `DisableHostedZoneDNSSEC`
+- `DisassociateVPCFromHostedZone`
+- `EnableHostedZoneDNSSEC`
+- `GetAccountLimit`
+- `GetCheckerIpRanges`
+- `GetDNSSEC`
+- `GetGeoLocation`
+- `GetHealthCheck`
+- `GetHealthCheckCount`
+- `GetHealthCheckLastFailureReason`
+- `GetHealthCheckStatus`
+- `GetHostedZoneLimit`
+- `GetQueryLoggingConfig`
+- `GetReusableDelegationSet`
+- `GetReusableDelegationSetLimit`
+- `GetTrafficPolicy`
+- `GetTrafficPolicyInstance`
+- `GetTrafficPolicyInstanceCount`
+- `ListCidrBlocks`
+- `ListCidrCollections`
+- `ListCidrLocations`
+- `ListGeoLocations`
+- `ListHealthChecks`
+- `ListHostedZonesByVPC`
+- `ListQueryLoggingConfigs`
+- `ListReusableDelegationSets`
+- `ListTrafficPolicies`
+- `ListTrafficPolicyInstances`
+- `ListTrafficPolicyInstancesByHostedZone`
+- `ListTrafficPolicyInstancesByPolicy`
+- `ListTrafficPolicyVersions`
+- `ListVPCAssociationAuthorizations`
+- `TestDNSAnswer`
+- `UpdateHealthCheck`
+- `UpdateHostedZoneFeatures`
+- `UpdateTrafficPolicyComment`
+- `UpdateTrafficPolicyInstance`
+
 ## s3
 
 `2006-03-01` — 26 of 112 modelled operations implemented (23.2%), 0 stubbed, 6 not applicable, 0 registered outside the pinned model.

docs/coverage/pages.json

@@ -247,6 +247,20 @@
         "option-groups": "Option groups configure engine add-ons for engines this platform does not offer, such as Oracle and SQL Server."
       }
     },
+    "route53": {
+      "slug": "route53",
+      "name": "Route 53",
+      "title": "Route 53 API Coverage",
+      "seoTitle": "Amazon Route 53 API Coverage on Spinifex — Spinifex Docs",
+      "description": "The Amazon Route 53 API operations Spinifex implements, covering the hosted zones it serves, their resource record sets, change status and tags on each zone.",
+      "tags": [
+        "aws",
+        "compatibility",
+        "coverage",
+        "route53",
+        "dns"
+      ]
+    },
     "s3": {
       "slug": "s3",
       "name": "S3",

docs/coverage/route53/README.md

@@ -0,0 +1,38 @@
+---
+title: "Route 53 API Coverage"
+seoTitle: "Amazon Route 53 API Coverage on Spinifex — Spinifex Docs"
+description: "The Amazon Route 53 API operations Spinifex implements, covering the hosted zones it serves, their resource record sets, change status and tags on each zone."
+category: "Coverage"
+sections:
+  - overview
+tags:
+  - aws
+  - compatibility
+  - coverage
+  - route53
+  - dns
+---
+
+# Route 53 API Coverage
+
+## Overview
+
+Spinifex implements **13 operations** in the Route 53 `2013-04-01` API model.
+
+### Operations
+
+| Operation |
+|---|
+| `ChangeResourceRecordSets` |
+| `ChangeTagsForResource` |
+| `CreateHostedZone` |
+| `DeleteHostedZone` |
+| `GetChange` |
+| `GetHostedZone` |
+| `GetHostedZoneCount` |
+| `ListHostedZones` |
+| `ListHostedZonesByName` |
+| `ListResourceRecordSets` |
+| `ListTagsForResource` |
+| `ListTagsForResources` |
+| `UpdateHostedZoneComment` |

docs/install/install-multi-node/README.md

@@ -163,11 +163,14 @@ Open between all hosts:
 | Protocol and port | Used by |
 |---|---|
 | UDP 6081 | Geneve tunnels |
+| UDP 500, 4500 and IP protocol 50 (ESP) | IPsec for the Geneve tunnels, while IPsec is on |
 | TCP 4222, 4248 | NATS |
 | TCP 6641, 6642 | OVN northbound and southbound |
 | TCP 8443 | Predastore S3 gate |
 | UDP 6660, 7660 | Object shards, metadata consensus |
 
+UDP 4500 carries the tunnel traffic only where IKE detects NAT between two nodes; otherwise it travels as ESP, on Oracle Linux as on Debian.
+
 Open between the three OVN database servers (servers 1 to 3) only:
 
 | Protocol and port | Used by |
@@ -207,7 +210,10 @@ Spinifex's own **host** firewall is separate, and already carries this policy
 
 Choose one method and apply it to **every** server in the cluster.
 
-**Option A — existing OS.** On a server already running Ubuntu 26.04 or Debian 13:
+**Option A — existing OS.** Use this on servers already running Ubuntu 26.04, Debian 13 or Oracle Linux 10, and prepare an Oracle Linux server with [Oracle Linux Hosts](/docs/install-oracle-linux) first.
+Every server in a cluster must run the same OS: Oracle Linux never shares a cluster with Debian or Ubuntu, and `spx admin join` does not check this for you.
+
+Run the installer on each server:
 
 ```bash
 curl -fsSL https://install.mulgadc.com | bash

docs/install/install-oracle-linux/README.md

@@ -0,0 +1,270 @@
+---
+title: "Oracle Linux Hosts"
+seoTitle: "Install Spinifex on Oracle Linux 10 — Spinifex Docs"
+description: "Prepare Oracle Linux 10 hosts for Spinifex: UEK R8, SELinux, a NetworkManager br-wan bridge, the NFV SIG repository, IPsec on libreswan and host updates."
+category: "Install"
+tags:
+  - install
+  - oracle linux
+  - single node
+  - multi node
+resources:
+  - title: "Oracle Linux 10 Documentation"
+    url: "https://docs.oracle.com/en/operating-systems/oracle-linux/10/"
+  - title: "CentOS NFV SIG"
+    url: "https://sigs.centos.org/nfv/"
+---
+
+# Oracle Linux Hosts
+
+> What differs when the Spinifex host runs Oracle Linux 10 instead of Debian or Ubuntu.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Prerequisites](#prerequisites)
+- [Instructions](#instructions)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Overview
+
+Spinifex installs on Oracle Linux 10.2 or a later 10.x release, running the UEK R8 kernel, on x86_64.
+The installer, the release tarball and the install steps are the same as on Debian; this page covers what the host needs first and what the installer changes on it.
+Follow it alongside [Single-Node Install](/docs/install).
+
+**Supported on Oracle Linux:**
+
+- A single node, or a cluster in which every node runs Oracle Linux; servers beyond the third join as compute nodes, as on Debian.
+- Guests, VPCs, routed NAT, Elastic IPs, and Application and Network Load Balancers.
+- Whole-GPU passthrough, on x86_64 GPU hosts (see [GPU Passthrough](/docs/gpu-passthrough)).
+
+**Not supported on Oracle Linux yet:**
+
+- Clusters that mix Oracle Linux with Debian or Ubuntu nodes, on any release.
+- SELinux enforcing, aarch64, Oracle Linux 9 or earlier, the Red Hat Compatible Kernel, a ZFS root, and NVIDIA MIG.
+
+Open vSwitch and OVN come from the CentOS NFV SIG, because Oracle ships neither for Oracle Linux 10.
+The NFV SIG is a community repository, and Oracle does not support the packages it publishes.
+Spinifex installs `openvswitch3.5` and `ovn25.03`, the versions Debian 13 ships, and its repository file lets the SIG supply nothing else.
+QEMU comes from Oracle's KVM Utilities repository, because only that build has the `microvm` machine type Spinifex load balancers boot on.
+
+An Oracle Linux node must be a dedicated Spinifex host.
+Spinifex claims every `tap*` device and owns libreswan's `ipsec.service`, which it keeps stopped on a single node and runs for its own tunnels on a cluster node, so VPNs, hand-run QEMU guests and libreswan tunnels of your own cannot share the host.
+Crypto policies other than `DEFAULT` and `FIPS` are untested, and so is Secure Boot.
+
+## Prerequisites
+
+### Operating system and kernel
+
+Install Oracle Linux 10 with Oracle's ISO or a KVM template, then update it to the current minor release and reboot:
+
+```bash
+sudo dnf update -y
+sudo reboot
+```
+
+After the reboot, check the release, the kernel and the CPU:
+
+```bash
+cat /etc/oracle-release        # Oracle Linux Server release 10.2 or later
+uname -r                       # 6.12.0-...el10uek.x86_64
+grep -c -w avx2 /proc/cpuinfo  # non-zero
+```
+
+The kernel must be UEK R8 (`6.12.0-*uek`), and it must be the running kernel: an installed but not booted `kernel-uek` is refused.
+The CPU must be x86-64-v3 or newer (Intel Haswell, AMD Excavator or later), since the NFV SIG and KVM Utilities builds target it.
+The installer names any missing CPU flag.
+
+The installer adds `kernel-uek-modules` for the running kernel, which carries `openvswitch.ko` and pulls in `linux-firmware`, about 800 MB.
+Oracle publishes modules for each kernel separately, so a host running an old kernel may find its modules gone from the repository; the installer then stops and asks for a kernel update and a reboot.
+
+### SELinux
+
+Spinifex ships no SELinux policy yet, so SELinux must be permissive or disabled, both now and at the next boot.
+Oracle's images boot enforcing.
+
+```bash
+sudo setenforce 0
+sudo sed -i 's/^SELINUX=.*/SELINUX=permissive/' /etc/selinux/config
+getenforce                     # Permissive
+```
+
+The installer and `spx admin preflight` refuse a host that is enforcing in either place.
+
+### WAN bridge with NetworkManager
+
+As on every OS, the host's uplink must already sit on a Linux bridge named `br-wan`, with the host address and default route on the bridge (see [Single-Node Install](/docs/install#prerequisites)).
+On Oracle Linux, NetworkManager owns `br-wan`; Spinifex leaves the bridge and its NIC to it and marks the links it creates itself as unmanaged.
+
+Create the bridge from the console, not over SSH on the NIC being moved, since the host loses its address while the profile changes.
+Replace `eno1` with the uplink NIC and `"System eno1"` with its current profile (`nmcli connection show`):
+
+```bash
+sudo nmcli connection add type bridge ifname br-wan con-name br-wan \
+  bridge.stp no ipv4.method auto ipv6.method disabled
+sudo nmcli connection add type bridge-slave ifname eno1 master br-wan con-name br-wan-port
+sudo nmcli connection down "System eno1"
+sudo nmcli connection up br-wan
+```
+
+For a static address, give the bridge `ipv4.method manual` with `ipv4.addresses`, `ipv4.gateway` and `ipv4.dns` instead of `ipv4.method auto`.
+
+`bridge.stp no` matters: a NetworkManager bridge runs STP by default, which holds the port in its forwarding delay for about 15 seconds after every link up, long enough for DHCP on the bridge to time out.
+Then verify, as on Debian:
+
+```bash
+ip -br link show br-wan        # UP
+ip route                       # default via ... dev br-wan
+```
+
+### Network access during install
+
+Besides `install.mulgadc.com` and the AWS CLI download, the install reaches `yum.oracle.com` for the Oracle Linux repositories, `mirror.stream.centos.org` for the NFV SIG packages, and `www.centos.org` for the NFV SIG signing key.
+An air-gapped install on Oracle Linux is not documented yet.
+
+## Instructions
+
+## Step 1. Install Spinifex
+
+Run the installer as a user with `sudo`, exactly as on Debian:
+
+```bash
+curl -fsSL https://install.mulgadc.com | bash
+```
+
+For a single node, continue with [Single-Node Install](/docs/install) from Step 2: `setup-ovn.sh --management`, `spx admin init` and `systemctl start spinifex.target`.
+IPsec stays off on a single node, because there is no peer to encrypt traffic to.
+
+For a cluster, prepare and install every node this way, then continue with [Multi-Node Install](/docs/install-multi-node) from Step 2.
+
+### Clusters
+
+Every node in an Oracle Linux cluster runs Oracle Linux.
+`install-node.sh` refuses a host list that mixes Oracle Linux with Debian or Ubuntu, but `spx admin join` does not check, so keep to one OS yourself when joining by hand.
+
+IPsec encrypts the Geneve tunnels between nodes and is on by default, as on Debian.
+On Oracle Linux it runs on libreswan, the IKE daemon Oracle ships, rather than strongSwan.
+Once the node's IPsec certificate exists, the daemon unmasks and starts `ipsec.service` and `openvswitch-ipsec.service`.
+`ovs-monitor-ipsec` then loads the node certificate, its key and the cluster CA into libreswan's NSS database at `/var/lib/ipsec/nss`, under nicknames that start with `ovs_cert`.
+
+The tunnels are plain ESP (IP protocol 50), as on Debian, negotiated over UDP 500.
+Open UDP 500, UDP 4500 and ESP between nodes: libreswan moves to UDP 4500 if it detects NAT between two nodes.
+The Spinifex host firewall opens all three to cluster peers; a firewall between the nodes must allow them too.
+While IKE to a peer fails, the node drops its Geneve traffic to that peer rather than send it in clear.
+The nft table `inet spinifex_ipsec_guard` drops outbound UDP 6081 from the encap address whose route carries no SA, which also covers the moments pluto flushes the xfrm policy table as it starts and stops.
+It is not persisted: `ipsec.service` puts it back at boot before ovn-controller builds the tunnels, and the daemon re-syncs it every 60 seconds.
+`spx admin init --ipsec=false` leaves the tunnels unencrypted, as on any OS.
+
+## Step 2. Know What the Installer Changed
+
+On Oracle Linux the installer makes these changes, which Debian hosts do not need:
+
+- **Repositories.** It enables `ol10_UEKR8` and `ol10_kvm_utils`, and adds `qemu-kvm*` and `qemu-img*` to the `excludepkgs` of `ol10_appstream`, keeping any excludes already set, so a later AppStream build can never replace the KVM Utilities QEMU.
+- **NFV SIG.** It writes `/etc/yum.repos.d/spinifex-nfv-sig.repo` with an `includepkgs=` list of the OVS and OVN packages, and imports the SIG key to `/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-NFV` only when its fingerprint is `35154228174901BEFA8E69A621465E289D2A76A7`.
+- **OVS and OVN user.** Both run as root, as on Debian, through `OVS_USER_ID` and `OVN_USER_ID` in `/etc/sysconfig/openvswitch` and `/etc/sysconfig/ovn`.
+- **DNS.** Where systemd-resolved is installed but not running, it saves `/etc/resolv.conf` to `/var/lib/spinifex-reset/resolv.conf.orig`, starts resolved behind NetworkManager and links `/etc/resolv.conf` to the resolved stub. It then checks that names still resolve and stops, printing the commands that put DNS back, if they do not.
+- **chrony.** It appends `confdir /etc/chrony.d` to `/etc/chrony.conf` once, and writes its step policy to `/etc/chrony.d/spinifex.conf`.
+- **`spx` under `sudo`.** Oracle Linux's `sudo` path leaves out `/usr/local/bin`, so the installer links `/usr/bin/spx` to `/usr/local/bin/spx`.
+
+The kernel modules list (`/etc/modules-load.d/spinifex.conf`) and the host firewall below apply on every OS.
+
+## Step 3. Replace firewalld Rules You Depend On
+
+The installer stops and masks firewalld, because with it running a packet must pass both firewalld's table and the Spinifex one, which silently breaks routed NAT, Elastic IPs and instance metadata.
+When firewalld was active, the installer turns on the Spinifex nftables policy in its place; pass `--firewall=on` or `--firewall=off` to choose yourself.
+
+firewalld's rules are not saved or migrated.
+The Spinifex policy filters inbound traffic only and opens 53, 443, 3000, 8443 and 9999 to any source.
+It accepts SSH from every IPv4 source until `$trusted_ssh_peers` in `/etc/spinifex/firewall/custom.nft` narrows it.
+Restate any other restriction you had, such as SSH limited to a management subnet, in that file; [Host Firewall](/docs/host-firewall) shows how.
+
+From the moment firewalld stops until the daemon first loads the Spinifex policy, the host accepts every connection.
+Uninstalling Spinifex leaves firewalld unmasked but disabled; it does not turn it back on.
+
+## Step 4. Keep the Host Updated
+
+Do not enable `dnf-automatic`.
+An Open vSwitch package upgrade restarts OVS, which interrupts guest networking, so update `openvswitch3.5`, `ovn25.03*` and `qemu-kvm*` in a maintenance window.
+Stay on the current Oracle Linux 10 minor release, which is what Spinifex is tested on.
+
+Before rebooting into a new kernel, check that its modules are installed, or OVS cannot start after the reboot:
+
+```bash
+rpm -q kernel-uek-core kernel-uek-modules
+```
+
+Each `kernel-uek-core` version needs a `kernel-uek-modules` of the same version; install a missing one with `sudo dnf install kernel-uek-modules-<version>`.
+
+Track Oracle Linux security advisories (ELSA) for the host, the kernel and QEMU, for example with `dnf updateinfo list --security`.
+Those advisories do not cover the NFV SIG packages, so follow Open vSwitch and OVN security announcements for those.
+
+## Step 5. Uninstall
+
+`/usr/local/share/spinifex/uninstall-spx.sh` removes Spinifex as on Debian.
+On a cluster node it also stops `openvswitch-ipsec.service` and deletes the `ovs_cert*` entries from libreswan's NSS database, so the node can join a different cluster; other NSS identities and your connection files in `/etc/ipsec.d` stay.
+It lists what it leaves in place on Oracle Linux: the repository changes and the SIG key, `OVS_USER_ID` and `OVN_USER_ID`, the OVS and OVN logrotate edits, the chrony include, `/etc/modules-load.d/spinifex.conf`, the SELinux mode, a masked `ipsec.service`, firewalld disabled, and any GPU passthrough configuration.
+`--purge-deps` is refused on Oracle Linux, because dnf cannot tell the packages Spinifex installed from the host's own; remove them with `dnf remove` yourself.
+
+## Troubleshooting
+
+### Installer stops: SELinux is enforcing
+
+Set SELinux permissive now and in `/etc/selinux/config`, as in [SELinux](#selinux), then run the installer again.
+Fixing only one of the two is refused too.
+
+### Installer stops: the running kernel is not UEK R8
+
+Install `kernel-uek`, reboot into it, check `uname -r`, then run the installer again.
+
+### Installer stops: kernel-uek-modules is no longer in the repositories
+
+The running kernel is older than anything the repository still carries modules for.
+
+```bash
+sudo dnf update -y kernel-uek
+sudo reboot
+```
+
+### Installer stops: a repository is unreachable
+
+The message names the repository.
+Check the host can reach `yum.oracle.com` and `mirror.stream.centos.org` over HTTPS, including through any proxy dnf uses, then run the installer again.
+
+### DHCP on br-wan times out
+
+Check STP is off on the bridge:
+
+```bash
+nmcli -g bridge.stp connection show br-wan
+```
+
+If it prints `yes`, run `sudo nmcli connection modify br-wan bridge.stp no` and `sudo nmcli connection up br-wan`.
+
+### `sudo spx` finds no command
+
+Check that `/usr/bin/spx` links to `/usr/local/bin/spx`; run the installer again if it does not, or call `sudo /usr/local/bin/spx` meanwhile.
+
+### Cluster nodes form no IPsec tunnels
+
+Check that both IPsec services run, and that libreswan holds the node's certificate and the cluster CA:
+
+```bash
+systemctl is-active ipsec.service openvswitch-ipsec.service
+sudo certutil -L -d sql:/var/lib/ipsec/nss     # ovs_certkey_<node>, ovs_cert_cacert
+sudo ipsec trafficstatus                       # one line per peer tunnel
+```
+
+If the services are inactive, check the daemon log with `journalctl -u spinifex-daemon` for the IPsec reconcile.
+If the certificates are present but no tunnel is listed, check that UDP 500 and ESP pass between the nodes in both directions, and look for `SEC_ERROR_UNKNOWN_ISSUER` in `journalctl -u ipsec`, which means a peer's certificate is from another CA.
+Until the tunnel forms, guest traffic between those two nodes is dropped, which is the guard in [Clusters](#clusters) working.
+
+### CA certificate not trusted
+
+`sudo spx admin init` installs the CA automatically. To install it again by hand on Oracle Linux:
+
+```bash
+sudo cp /etc/spinifex/ca.pem /etc/pki/ca-trust/source/anchors/spinifex-ca.crt
+sudo update-ca-trust
+```

docs/install/install-source/README.md

@@ -1,7 +1,7 @@
 ---
 title: "Source Install"
 seoTitle: "Build and Install Spinifex from Source — Spinifex Docs"
-description: "Build Spinifex from source on Ubuntu or Debian for development, custom builds, or contributing changes, then install and run the resulting binaries locally."
+description: "Build Spinifex from source on Debian, Ubuntu or Oracle Linux for development, custom builds or contributing changes, then install and run the binaries."
 category: "Install"
 tags:
   - install
@@ -34,6 +34,7 @@ This guide builds Spinifex from source. For production deployments, the [binary
 
 - Ubuntu 26.04
 - Debian 13
+- Oracle Linux 10.2 or a later 10.x, on UEK R8, x86_64 only. Prepare the host with [Oracle Linux Hosts](/docs/install-oracle-linux) first; `make quickinstall` also enables the CodeReady Builder repository for `nbdkit-devel`.
 
 ## Instructions
 

docs/install/install/README.md

@@ -1,7 +1,7 @@
 ---
 title: "Single-Node Install"
 seoTitle: "Install Spinifex on a Single Server — Spinifex Docs"
-description: "Install Spinifex on one Ubuntu or Debian server with the binary installer and get an AWS-compatible EC2, S3, and VPC stack running on your own hardware."
+description: "Install Spinifex on one Debian, Ubuntu or Oracle Linux server with the binary installer and run an AWS-compatible EC2, S3 and VPC stack on your hardware."
 category: "Install"
 tags:
   - install
@@ -39,6 +39,7 @@ This guide installs Spinifex on a single server using the binary installer. For
 
 - Ubuntu 26.04
 - Debian 13
+- Oracle Linux 10.2 or a later 10.x, on UEK R8, x86_64 only, as a single node or, with [Multi-Node Install](/docs/install-multi-node), a cluster. Open vSwitch and OVN come from the CentOS NFV SIG, a community repository that Oracle does not support. Prepare the host with [Oracle Linux Hosts](/docs/install-oracle-linux) first.
 
 **What Gets Installed:**
 
@@ -139,19 +140,33 @@ Continue to [Setting Up Your Cluster](/docs/setting-up-your-cluster) to import a
 
 ### spx Command Not Found
 
+The installer puts `spx` in `/usr/local/bin`.
+Where `sudo`'s `secure_path` leaves that directory out, as on Oracle Linux, the installer also links `/usr/bin/spx` to it, so `sudo spx` works; changing `PATH` in your shell does not reach commands run with `sudo`.
+Check both:
+
 ```bash
-export PATH=$PATH:/usr/local/bin
+ls -l /usr/local/bin/spx /usr/bin/spx
+sudo sh -c 'command -v spx'
 ```
 
+If `sudo` still finds no `spx`, run the installer again, or call `sudo /usr/local/bin/spx` meanwhile.
+
 ### CA Certificate Not Trusted
 
-`sudo spx admin init` installs the CA automatically. If you need to re-install it manually:
+`sudo spx admin init` installs the CA automatically. If you need to re-install it manually on Debian or Ubuntu:
 
 ```bash
 sudo cp /etc/spinifex/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
 sudo update-ca-certificates
 ```
 
+On Oracle Linux:
+
+```bash
+sudo cp /etc/spinifex/ca.pem /etc/pki/ca-trust/source/anchors/spinifex-ca.crt
+sudo update-ca-trust
+```
+
 ### OVN Services Not Starting
 
 ```bash

docs/install/performance-tuning/README.md

@@ -269,6 +269,35 @@ ip link set <dev> mtu 9000        # both ends
 ping -M do -s 8972 -c 3 <peer>    # must succeed
 ```
 
+### The internet path
+
+A guest MTU of 1408 or 1442 sits below the 1500 bytes the internet carries, so a server can send the guest segments it cannot receive.
+The VPC gateway port carries `gateway_mtu = min(guest MTU, 1500)` and answers an oversized packet with ICMP fragmentation-needed (IPv4) or Packet Too Big (IPv6).
+It follows `ipsec_enabled` and `network.underlay_mtu` and converges on the next reconcile pass, with no VPC recreate.
+
+Path MTU discovery depends on that ICMP reaching the server, and an upstream NAT router or a CDN often drops it.
+vpcd therefore also clamps the TCP MSS on the host hop to the gateway MTU minus 40 (IPv4) or minus 60 (IPv6), and re-asserts the rule every 30 seconds so a firewall reload cannot remove it for long.
+
+| Uplink | Clamp | Table |
+| --- | --- | --- |
+| veth (distributed NAT) | yes | `bridge spinifex_mss`, matching the `veth-wan-br` port |
+| routed NAT | yes | `inet spinifex_mss`, matching `spx-nat-host` |
+| physical | no host hop exists | path MTU discovery only |
+
+On a physical uplink the only cover is path MTU discovery, which fails when the upstream drops the ICMP or the peer ignores it.
+
+Measured on one node (env12), IPsec on, Ubuntu 26.04 guest, 16 MB HTTPS download inside a Docker container at the default bridge MTU of 1500:
+
+| Uplink | Without the clamp | With the clamp |
+| --- | --- | --- |
+| veth, instance with a public IP | stalled 3/3 | completed 3/3 |
+| routed NAT | stalled 3/3, no ICMP seen on the host | completed 3/3 |
+
+An instance behind a NAT gateway on the veth uplink also completed 3/3 with the clamp; its unclamped internet run was not measured.
+The guest's own `curl` completed in every case.
+Without the clamp, `gateway_mtu` alone fixed the download only against a server on the same L2 with no NAT in between (completed 3/3), and not against the internet through the lab's upstream NAT router.
+Not measured: a physical uplink, a multi-node cluster, IPv6, and a Debian guest.
+
 ## 5. Measurement notes
 
 Throughput on this stack is **noisy** — repeated identical 20s runs varied between

docs/security/flaw-remediation-policy/README.md

@@ -127,8 +127,8 @@ Operators own remediation for layers Spinifex does not ship:
 | Layer | Responsibility |
 |-------|----------------|
 | Spinifex releases | Apply released patches within the correct-SLA above. |
-| Host OS + kernel | Track distribution advisories (Debian DSA, Ubuntu USN) at matching cadence. |
-| OVN / OVS | Patched via the operator's package channel alongside the OS. |
+| Host OS + kernel | Track distribution advisories (Debian DSA, Ubuntu USN, Oracle Linux ELSA) at matching cadence. |
+| OVN / OVS | Patched via the operator's package channel alongside the OS. On Oracle Linux they come from the CentOS NFV SIG, which ELSA does not cover, so track upstream Open vSwitch and OVN security announcements. |
 | QEMU / KVM / libvirt | As above — hypervisor CVEs affect tenant isolation. |
 | Host AV / EDR agent | See [Malware Protection §2](/docs/malware-protection#2-update-requirements-sil1-3144). |
 

docs/security/malware-protection/README.md

@@ -75,7 +75,7 @@ All Spinifex nodes must include the following in their malware protection scope.
 | `/usr/local/bin/spx` | Spinifex CLI and service entry point. All `spinifex-*.service` units invoke `spx service <name> start`, so every service binary is reached through this one path. |
 | `/usr/bin/ovn-*`, `/usr/bin/ovs-*` | OVN / OVS control-plane and datapath binaries (installed via `apt` from the `ovn-central`, `ovn-host`, `openvswitch-switch` packages). |
 | `/usr/lib/<arch>-linux-gnu/nbdkit/plugins/nbdkit-viperblock-plugin.so` | Viperblock NBD plugin loaded by `nbdkit` to expose EBS volumes as block devices. The exact path comes from `nbdkit --dump-config` (`plugindir=`); on Debian 13 amd64 it is `/usr/lib/x86_64-linux-gnu/nbdkit/plugins/`. |
-| `/usr/local/share/ca-certificates/spinifex-ca.crt` | Cluster CA cert installed into the system trust store |
+| `/usr/local/share/ca-certificates/spinifex-ca.crt` | Cluster CA cert installed into the system trust store. On Oracle Linux it is `/etc/pki/ca-trust/source/anchors/spinifex-ca.crt`. |
 
 ### 1.2 Configuration and Secrets
 

docs/security/network-connections/README.md

@@ -99,6 +99,9 @@ behavior in other words.
 | 4432 | Formation server | HTTPS | Cluster (bootstrap only) | Cluster join coordination; active only while a join token is valid. Binds the node's `--bind` (lan) address. See *Formation port lifecycle* below. | Short-lived bearer token + TLS¹ |
 | 4222 | spinifex-nats (client) | NATS + TLS | Cluster | Internal service bus for EC2/EBS/VPC/S3 handlers | Token + mutual TLS (cluster CA) |
 | 4248 | spinifex-nats (cluster) | NATS + TLS | Cluster | Inter-node NATS federation | Token + mutual TLS (cluster CA) |
+| 4223 | spinifex-sqs-store (client) | NATS + TLS | Cluster | SQS message data, separate from the control-plane bus | Store-only token + TLS |
+| 4249 | spinifex-sqs-store (cluster) | NATS + TLS | Cluster | Inter-node SQS store replication | Store-only token + mutual TLS (cluster CA) |
+| 8223 | spinifex-sqs-store (monitor) | HTTP | Loopback | varz and jsz for the store | Loopback only |
 | 5300 | northstar | DNS (UDP + TCP) | Cluster | Forward target for every node's per-instance DNS shim, dialled cross-node. Binds the wildcard by design. | None — restrict by firewall |
 | 6660 | predastore (blob node) | QUIC / UDP | Cluster | Erasure-coded object shard transport between hosts. Multi-node clusters only — see *Predastore ports* below. | TLS 1.3, server certificate verified against the cluster CA |
 | 7660 | predastore (meta node) | QUIC / UDP | Cluster | Raft consensus over global state — buckets and the object index — between hosts. Multi-node clusters only. | TLS 1.3, server certificate verified against the cluster CA |
@@ -108,8 +111,8 @@ behavior in other words.
 | 6643 | OVN Northbound DB (RAFT) | OVSDB/TCP | Cluster | NB database RAFT replication between the 3 quorum nodes | Cluster network only; TLS planned |
 | 6644 | OVN Southbound DB (RAFT) | OVSDB/TCP | Cluster | SB database RAFT replication between the 3 quorum nodes | Cluster network only; TLS planned |
 | 6081 | OVN (Geneve) | UDP | Encap | Tenant traffic overlay between chassis. A kernel UDP-tunnel socket, so packets are delivered locally and traverse the host's netfilter input hook before OVS sees them — a host firewall must accept them explicitly. The socket is opened by the kernel tunnel driver and takes no bind address, so it binds the wildcard by design and reach must be restricted by firewall. | None — see 500/4500 |
-| 500, 4500 | strongSwan `charon` | IKEv2 / UDP | Encap | IKE and NAT-T for the IPsec protecting Geneve, managed entirely by `ovs-monitor-ipsec`. Binds the wildcard by design (the upstream strongSwan default, accepted rather than overridden), so reach must be restricted by firewall. | Certificate-based, against the cluster CA |
-| — | ESP | IP proto 50 | Encap | The IPsec payload itself, once IKE has negotiated an SA | Cluster CA |
+| 500, 4500 | strongSwan `charon`, or libreswan `pluto` on Oracle Linux | IKEv2 / UDP | Encap | IKE and NAT-T for the IPsec protecting Geneve, managed entirely by `ovs-monitor-ipsec`; 4500 carries the ESP payload only where IKE detects NAT between two nodes. `charon` binds the wildcard by design (the upstream strongSwan default, accepted rather than overridden); `pluto` binds each host address separately, the WAN address included, which is no narrower. Either way reach must be restricted by firewall. | Certificate-based, against the cluster CA |
+| — | ESP | IP proto 50 | Encap | The IPsec payload itself, once IKE has negotiated an SA, on Oracle Linux as on Debian. | Cluster CA |
 | 8222 | spinifex-nats (monitoring) | HTTP | Localhost | `varz`/`subsz` metrics consumed by the daemon | Loopback only |
 | 323 | chronyd | NTP | Localhost | Time sync client control socket | Loopback only |
 | 169.254.169.254:80 | spinifex-vpcd (IMDS) | HTTP | Guest | Instance metadata service. One socket per instance, bound to that instance's `ime-*` interface. Terminates on the host, so it traverses the netfilter input hook. | Instance identity by interface; IMDSv2 tokens |
@@ -145,6 +148,16 @@ Spinifex nodes initiate a small, fixed set of outbound connections.
 | `https://d2yp8ipz5jfqcw.cloudfront.net` | Alpine image for managed HAProxy load-balancer | HTTPS | TLS + checksum verification |
 | `https://install.mulgadc.com/install` | One-shot install telemetry POST on `spx admin init` / `join`. | HTTPS | TLS |
 
+**During install and package updates on Oracle Linux hosts:**
+
+| Destination | Purpose | Protocol | Verification |
+|-------------|---------|----------|--------------|
+| `https://yum.oracle.com` | Oracle Linux BaseOS, AppStream, UEK R8 and KVM Utilities repositories | HTTPS | TLS + Oracle RPM signatures |
+| `https://mirror.stream.centos.org/SIGs/10-stream/nfv/` | Open vSwitch and OVN from the CentOS NFV SIG, limited by `includepkgs=` | HTTPS | TLS + NFV SIG RPM signatures |
+| `https://www.centos.org/keys/RPM-GPG-KEY-CentOS-SIG-NFV` | NFV SIG signing key, fetched once at install | HTTPS | TLS + fingerprint pinned in `setup.sh` |
+
+Debian and Ubuntu hosts reach their distribution's own mirrors for the same purpose.
+
 **To peer nodes (cluster-internal):** NATS federation (4248), Predastore S3 (8443), OVN NB/SB (6641/6642), northstar DNS forwarding (5300), and the Geneve/IPsec overlay (6081, 500, 4500, ESP) — see [§3](#3-cross-node-internal-connections) for encryption and verification of each. The daemon also polls local NATS monitoring at `127.0.0.1:8222/varz` (loopback HTTP). It opens no connection to Predastore for status: the storage topology it reports comes from reading `predastore.toml`, and no Predastore node serves a status endpoint.
 
 **Update checks and metadata.** Spinifex does not check for updates and does not consume a cloud metadata service (`169.254.169.254` is served *by* the cluster to guest VMs). Node software updates come from the operator's OS package channel. The install-telemetry endpoint above is the only vendor-operated destination contacted by a node; closed-egress deployments should disable it and record the opt-out in the security plan.
@@ -164,7 +177,7 @@ Control-plane and data-plane traffic between Spinifex nodes, for completeness an
 | OVN NB/SB (client) | 6641 / 6642 | Cluster network only (TLS planned) | Network control plane; vpcd and ovn-controller dial the quorum |
 | OVN NB/SB (RAFT) | 6643 / 6644 | Cluster network only (TLS planned) | NB/SB database replication across the 3 quorum nodes |
 | OVN tunnels (Geneve) | UDP 6081 | Encapsulated by IPsec when `network.ipsec_enabled` is true, which is the default on multi-node clusters | Tenant traffic overlay between chassis, on the `vpc` plane |
-| OVN IPsec (IKE / NAT-T / ESP) | UDP 500, UDP 4500, IP proto 50 | Certificate-based against the cluster CA, negotiated by `ovs-monitor-ipsec` | Protects the Geneve tunnels above. OVN-native only — no layer manages strongSwan directly. |
+| OVN IPsec (IKE / NAT-T / ESP) | UDP 500, UDP 4500, IP proto 50 | Certificate-based against the cluster CA, negotiated by `ovs-monitor-ipsec` | Protects the Geneve tunnels above. OVN-native only: no layer configures strongSwan or libreswan directly. On Oracle Linux the daemon also blocks outbound Geneve that no SA covers, so a peer failing IKE gets no plaintext. |
 | Instance DNS forwarding | UDP/TCP 5300 | None | Each node's per-instance DNS shim forwards guest queries to peer nodes' northstar `:5300` |
 
 Nodes **must** sit on a network segment that is not routed to tenant/guest VMs or to the internet. The Predastore blob and meta transports and the OVN DBs are cluster-internal and must not be reachable from anywhere else.
@@ -173,7 +186,10 @@ Nodes **must** sit on a network segment that is not routed to tenant/guest VMs o
 
 Default external surface is five listeners — **9999** (AWS API), **3000** (UI), **22** (SSH), **8443** (S3) and **53** (DNS) — plus **4432** transiently during bootstrap. Every other listener is cluster- or encap-scoped and the operator must enforce this with a host firewall or an upstream network ACL.
 
-The nodes ship no firewall policy today. Until one is installed, this is the operator's responsibility and the reference below is the starting point.
+Every node ships a host firewall policy, `table inet spinifex_filter` in `/etc/spinifex/firewall/spinifex.nft`, written by `setup.sh`; `spinifex-daemon` loads it once cluster peers resolve, filling the peer and encap sets itself.
+It is armed by default on ISO installs; a `setup.sh` install arms it with `--firewall=on`, or when firewalld was active, and otherwise installs it unarmed, so on those nodes the scope split is still the operator's to enforce.
+`setup.sh` masks firewalld wherever it is installed, and `spx admin preflight` fails while firewalld or ufw is active, because a second ruleset silently drops routed NAT, elastic IP and metadata traffic.
+The shipped policy is the authority; the reference below explains the rules it needs and is looser than the shipped one in places, such as accepting all `ime-*` traffic where the shipped policy accepts only the metadata and DNS endpoints.
 
 > **Read the notes under the ruleset before applying it.** A default-deny input policy that
 > omits any of the loopback, conntrack, `ime-*` or Geneve rules will break instance
@@ -208,7 +224,7 @@ table inet spinifex_filter {
 
     # Cluster, from peer nodes only. Replace with your nodes' lan-plane
     # addresses — a CIDR does not generalise to nodes on different subnets.
-    ip saddr { 10.0.1.1, 10.0.1.2, 10.0.1.3 } tcp dport { 4222, 4248, 4432, 5300, 6641, 6642, 6643, 6644, 8660 } accept
+    ip saddr { 10.0.1.1, 10.0.1.2, 10.0.1.3 } tcp dport { 4222, 4223, 4248, 4249, 4432, 5300, 6641, 6642, 6643, 6644, 8660 } accept
     ip saddr { 10.0.1.1, 10.0.1.2, 10.0.1.3 } udp dport { 5300, 6660, 7660 } accept
 
     # Encap, from peer chassis only. Replace with your nodes' vpc-plane
@@ -223,6 +239,7 @@ Notes that make the difference between this working and locking you out:
 
 - **Use a dedicated table and never flush the others.** `vpcd` writes MASQUERADE and per-EIP FORWARD rules into the `ip filter` and `ip nat` tables, and only reinstalls them when the service starts. `iptables -F`, `nft flush ruleset`, `ufw enable` and `firewalld` all destroy them silently, and instance networking stays broken until the next restart.
 - **Do not add a `forward` hook.** nftables evaluates every table registered on a hook and any `drop` is final, so a default-deny forward chain here would override `vpcd`'s accepts in the other table and break every routed-NAT instance and every EIP. Filtering forwarded guest traffic is the security group's job, not the host firewall's.
+- **`vpcd` owns a second table named `spinifex_mss`, separate from `inet spinifex_filter`.** It is a `bridge` table on the veth uplink and an `inet` table on the routed uplink, it has a `forward` hook with policy `accept`, and it only rewrites the MSS option of TCP SYN packets on the host hop to the gateway. It never drops or accepts traffic and does not replace the forward-hook advice above. `vpcd` re-installs it every 30 seconds, so a flush is repaired within that interval, and there is none on a physical uplink.
 - **`output` is deliberately untouched.** The metadata service's reply path egresses through per-instance policy routing; filtering `output` breaks it in ways that are hard to attribute.
 - **On a single-NIC node the cluster and encap sets are the node's public addresses**, because the planes collapse. The rules are still correct, but they are no longer a boundary — an upstream ACL is the only real control there.
 
@@ -234,7 +251,7 @@ Every listener and outbound destination is controlled by one of these files. Cha
 
 | File | Keys | Controls |
 |------|------|----------|
-| `/etc/spinifex/spinifex.toml` | `nodes.<node>.{awsgw,nats,predastore,daemon}.host`, `nodes.<node>.vpcd.ovn_{nb,sb}_addr`, `nodes.<node>.daemon.dev_networking`, `network.ipsec_enabled` | Per-service bind addresses/ports; dev-mode QEMU port forwarding. `network.ipsec_enabled` (default `true`) decides whether OVN-native IPsec protects the Geneve tunnels, and therefore whether `charon` listens on 500/4500 — single-node clusters never enable it. |
+| `/etc/spinifex/spinifex.toml` | `nodes.<node>.{awsgw,nats,predastore,daemon}.host`, `nodes.<node>.vpcd.ovn_{nb,sb}_addr`, `nodes.<node>.daemon.dev_networking`, `network.ipsec_enabled` | Per-service bind addresses/ports; dev-mode QEMU port forwarding. `network.ipsec_enabled` (default `true`) decides whether OVN-native IPsec protects the Geneve tunnels, and therefore whether the IKE daemon (`charon`, or `pluto` on Oracle Linux) listens on 500/4500 — single-node clusters never enable it. |
 | `/etc/spinifex/nats.conf` | `listen`, `cluster.listen`, `cluster.routes`, `http`, `tls`, `cluster.authorization` | NATS client/cluster/monitoring listeners, peer routes, TLS, cluster token. |
 | `/etc/spinifex/predastore/predastore.toml` | `[[host]].bind_addr`, `[[host]].addr`, `[[host]].tls_cert`, `[[host]].tls_key`, `[[host.node]].role`, `[[host.node]].port` | Predastore host and node layout: `bind_addr` is the address the host's sockets bind, `addr` is the address peer hosts dial it on, and both carry no port — the nodes pinned to the host supply their own. The service's `--host`/`--port` override the bind address and the gate's S3 port. |
 | `/etc/spinifex/northstar/northstar.toml` | `listen`, `forwarders` | DNS listen addresses (`:53` on the advertise address, `:5300` wildcard) and upstream resolvers. |

docs/testing/e2e-coverage.md

@@ -30,7 +30,7 @@ The largest suite, 27 top-level tests. Notable entry points:
 - **`TestIMDSDatapathSurvivesVswitchdRestart`** restarts `ovs-vswitchd` under a guest with an Elastic IP, once through systemd and once with SIGKILL. After each, the guest must answer over its EIP and fetch its instance ID through IMDSv2 within 60 seconds, and `br-imds` must hold all 7 of the guest's own flows. It is sequential because it takes the whole node's datapath down.
 - `TestStopStart`, `TestEarlyRebootLiveness`, `TestGuestChurnDurability`, `TestSnapshotLifecycle`, `TestSnapshotBackedLaunch`, `TestCreateImage`, `TestVolumeLifecycle`, `TestSpotInstanceLifecycle`, `TestLaunchTemplateBoot`.
 - `TestNegativeErrorPaths` — the `8a`–`8t` AWS error-shape cases (malformed AMI ID, invalid instance type, volume in use, detach-root forbidden, unknown or malformed instance IDs on the lifecycle verbs, and so on).
-- **`TestKVReplication`** — the single-node half of the KV replication policy, and it is not the multi-node assertion with a smaller number in it. On one node one replica per bucket is the correct answer rather than the defect, so what it proves is the opposite: that nothing declared a replica count this deployment cannot place, that `spx admin kv replicas` exits 0 and reports `WANT 1`, and that `--repair` run twice against a cluster with nothing to repair changes no bucket's replica count and leaves every bucket serving. The multi-node suites cannot cover any of that, because on three nodes the unplaceable count is the right one. It lives here rather than in its own suite so it runs in every single-node nightly cell, across all three host and network permutations.
+- **`TestKVReplication`** — the single-node half of the KV replication policy, and it is not the multi-node assertion with a smaller number in it. On one node one replica per bucket is the correct answer rather than the defect, so what it proves is the opposite: that nothing declared a replica count this deployment cannot place, that `spx admin kv replicas` exits 0 and reports `WANT 1`, and that `--repair` run twice against a cluster with nothing to repair changes no bucket's replica count and leaves every bucket serving. The multi-node suites cannot cover any of that, because on three nodes the unplaceable count is the right one. It lives here rather than in its own suite so it runs in every single-node nightly cell, across every host and network permutation.
 
 ### `natuplink` — routed NAT (`external_mode = "nat"`), one node or three
 
@@ -40,7 +40,7 @@ Runs **on** a node, shelling out to `ip`/`iptables`/`ovn-nbctl` locally. Skips u
 
 Phases 7 and 8 are gated on a cluster by what the datapath can actually do: the transit segment is a localnet, so host-to-guest Tier 1 ingress only exists where the VPC's gateway router port and the guest are both on this host. The VPC ingress route itself is installed on every node, so phase 8 checks it unconditionally.
 
-Two nightly cells run it: `nat-single` (cell 19) and `nat-multi` (cell 30). Cell 20 is the bare-metal job and is not in the permutation matrix at all, which is why the multi-node NAT cell is numbered past the end rather than filling a gap.
+Four nightly cells run it: `nat-single` (cell 19) and `nat-multi` (cell 30), and their Oracle Linux twins (cells 36 and 38). Cell 20 is the bare-metal job and is not in the permutation matrix at all, which is why the multi-node NAT cell is numbered past the end rather than filling a gap.
 
 ### `multinode` — behaviour that only exists on more than one node
 
@@ -50,6 +50,17 @@ Two nightly cells run it: `nat-single` (cell 19) and `nat-multi` (cell 30). Cell
 
 **`TestMultinodeDNS` convergence budgets are env-tunable.** `SPINIFEX_MULTINODE_DNS_TOPOLOGY_TIMEOUT` (default `90s`) bounds every node agreeing on the NS topology, `SPINIFEX_MULTINODE_DNS_TIMEOUT` (default `2m`) bounds a record becoming answerable from a guest or a node, and `SPINIFEX_MULTINODE_DNS_POLL` (default `3s`) is the poll interval for both. Values are Go durations; a malformed or non-positive value panics rather than falling back to the default.
 
+**`TestMultinodeOVNRaft` restarts each node's OVN central in turn before it kills the NB leader.**
+A `setup-ovn.sh` rerun restarts a node's NB and SB members together, on EL through the one `ovn-northd.service`, so the test does the same on every database node in turn, waits for both members to be back as cluster members under a known leader, and requires an NB write to land after each step and the full quorum with one leader at the end.
+
+**`TestMultinodeIPSec` proves guest traffic is encrypted on the wire, not only that IPsec is configured.**
+It first requires every node to carry the OVS certificate, key and CA pointers, `NB_Global.ipsec=true` never asserted over an unconfigured chassis, and an AES-GCM SA in `ip xfrm state`.
+None of those fails when SAs exist but traffic bypasses them, so it then launches two instances on distinct nodes and sends 200 pings of 1000 bytes between them.
+For each direction of that pair it fails unless the outbound SAs to the peer's `ovn-encap-ip` carried at least the payload sent, an outbound-only capture on the interface routing to the peer saw ESP (raw or UDP/4500-encapsulated), and that capture saw no plaintext Geneve (UDP/6081).
+The capture is outbound-only because inbound ESP is decrypted and seen again as Geneve on the same interface, which would read as plaintext on a healthy node.
+It proves the node pair the guests landed on; the SA check covers every node.
+It skips when the node config sets `network.ipsec_enabled=false`, or when the scheduler places all four launches on one node.
+
 ### `kvquorum` — the cluster's own state survives losing a node (multi-node, needs three)
 
 **The control plane keeps its state in JetStream KV buckets, and this is the suite that proves those buckets are actually replicated.** A bucket on one replica lives on one JetStream-chosen server recorded in no config, so losing that node makes it return `nats: no responders available for request` from *every* node at once — a cluster-wide outage caused by a single-node event. Leader leases are the worst case: acquiring one is a write, so a lease bucket without a quorum cannot be acquired by anyone, and every reconciler sharing it stops cluster-wide at exactly the moment a node has failed and there is repair work to do.
@@ -104,6 +115,9 @@ CA installed and downloadable, fingerprint matches config, hostnames present in
 
 Internal and internet-facing ALB and NLB, HTTPS listeners, listener rules, `ModifyListener`, and NLB UDP.
 
+Unhealthy-target routing: an NLB with an HTTP health check on a TCP target group stops sending traffic to a target that fails it while still accepting TCP, sends traffic again once it recovers, and fails open when every target is unhealthy.
+An ALB whose targets all fail their health check keeps serving from them instead of answering 503.
+
 ### `eks`, `ecs`, `ecr` — container services
 
 `eks`: cluster create/delete, access entries, addon delivery, `GetToken`, kubeconfig artifacts, the EBS CSI volume path, and an IRSA pod wired with a projected SA token. `ecs`: clusters, container instances, the three network modes (`awsvpc`, `bridge`, `host`), task-role credentials fetched from the container credentials endpoint, a service behind an ELB, and ENI release on terminate. `ecr`: control-plane CRUD plus a real OCI data plane — docker login/build/push/pull round trips, multi-layer images, and raw blob HTTP.

go.mod

@@ -4,12 +4,15 @@ go 1.27.3
 
 require (
 	github.com/aws/aws-sdk-go v1.55.8
-	github.com/aws/aws-sdk-go-v2 v1.47.3
-	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.23
+	github.com/aws/aws-sdk-go-v2 v1.47.4
+	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.24
 	github.com/aws/aws-sdk-go-v2/config v1.33.9
 	github.com/aws/aws-sdk-go-v2/credentials v1.20.9
 	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.20.3
+	github.com/aws/aws-sdk-go-v2/service/lambda v1.113.3
 	github.com/aws/aws-sdk-go-v2/service/s3 v1.114.4
+	github.com/aws/aws-sdk-go-v2/service/sqs v1.52.6
+	github.com/aws/smithy-go v1.28.6
 	github.com/cenkalti/backoff/v4 v4.3.0
 	github.com/charmbracelet/bubbles v1.0.0
 	github.com/charmbracelet/bubbletea v1.3.10
@@ -64,8 +67,8 @@ require (
 	github.com/Microsoft/hcsshim v0.15.0-rc.4 // indirect
 	github.com/antithesishq/antithesis-sdk-go v0.8.0-default-no-op // indirect
 	github.com/atotto/clipboard v0.1.4 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.6 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.6 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.7 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.7 // indirect
 	github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.6 // indirect
 	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.21 // indirect
 	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.11.7 // indirect
@@ -75,7 +78,6 @@ require (
 	github.com/aws/aws-sdk-go-v2/service/sso v1.38.4 // indirect
 	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.43.4 // indirect
 	github.com/aws/aws-sdk-go-v2/service/sts v1.51.4 // indirect
-	github.com/aws/smithy-go v1.28.6 // indirect
 	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/boltdb/bolt v1.3.1 // indirect

go.sum

@@ -22,20 +22,20 @@ github.com/atotto/clipboard v0.1.4 h1:EH0zSVneZPSuFR11BlR9YppQTVDbh5+16AmcJi4g1z
 github.com/atotto/clipboard v0.1.4/go.mod h1:ZY9tmq7sm5xIbd9bOK4onWV4S6X0u6GY7Vn0Yu86PYI=
 github.com/aws/aws-sdk-go v1.55.8 h1:JRmEUbU52aJQZ2AjX4q4Wu7t4uZjOu71uyNmaWlUkJQ=
 github.com/aws/aws-sdk-go v1.55.8/go.mod h1:ZkViS9AqA6otK+JBBNH2++sx1sgxrPKcSzPPvQkUtXk=
-github.com/aws/aws-sdk-go-v2 v1.47.3 h1:HRoT/MkNi3bjO5xF4dCzLzvVmkvRGHj3/cLINDnJejk=
-github.com/aws/aws-sdk-go-v2 v1.47.3/go.mod h1:xSl+OSI5ZUw6bSMupWc218zE5Pl+65MCCu3ySWFY99A=
-github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.23 h1:/fGPpKrXTjzkhsDXExBgNT2cGwxiF9rLrQXh8RlXMhI=
-github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.23/go.mod h1:QWUDuRi9PFjBHIU2iH1Pkldbe7zqLfXJC4wWLbF8nuQ=
+github.com/aws/aws-sdk-go-v2 v1.47.4 h1:bTvk6UdSSSvaQsfLVj1JRDEEqj/PDRwGf1KanaRzau0=
+github.com/aws/aws-sdk-go-v2 v1.47.4/go.mod h1:XkrjAZvr705Nytv6PkPUhoSB2W9+vVHBp5oAWpkh8Ak=
+github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.24 h1:t7FxqJ5KHDeKceyD4HtGaG6z7hMSeVGjs5/4c+L5ArM=
+github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.24/go.mod h1:5XK6NRVlK8s8yj4zKcM43HLluGA/rAz+azLLvWDMRr0=
 github.com/aws/aws-sdk-go-v2/config v1.33.9 h1:Hv6bcQUJVnmuWaKjqD6Ot8sE75NrWsWV20oGT5aWle0=
 github.com/aws/aws-sdk-go-v2/config v1.33.9/go.mod h1:ivowmrZNfBQLcg4QkaItgi5wae4Bzot9RzIdbY1hfTg=
 github.com/aws/aws-sdk-go-v2/credentials v1.20.9 h1:SNO55PnWhaDafsOcv16lCA0kdoFMl/YlegWoVsU02s0=
 github.com/aws/aws-sdk-go-v2/credentials v1.20.9/go.mod h1:+2reOTRtdD3VSBqpRDNwrZIYy8yMCYa7oK3WKTJ/I1I=
 github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.20.3 h1:gr3b87OLB29jymWPzp/CAyXwQRJVE+sC4Wm0qBIKHsk=
 github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.20.3/go.mod h1:BRjf4L/Yx2hY54o+ZibPbLJyMMeSwqYQXwcJfqW66a8=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.6 h1:CqL8NN3p4I0JTwwV5MEoatIl4Y2amZ2X9dRD+0hbRL0=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.6/go.mod h1:exdJ32hMqXl0Wyo6NwqEC5yIhtua3+PDaKE8P29m4qE=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.6 h1:FpBq24TG6WVSObTNhQOv7Q0088rounUUlwn3zlNItkM=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.6/go.mod h1:ONzdPVYC95oBOu0VWNEsKEntcoemP3xVKjT+qWml1s4=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.7 h1:fSNAt6hs4qLjb9tMZc6vj3PHlOCYGXK2JjJZ7AW2tf4=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.7/go.mod h1:TMav+EuXQu5kNpZZz/6rIC8xx4mCsW+6AKHrjvOSAsE=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.7 h1:7NS6kqBjkWIN8P/AEdmtyq6eQGAhN1oHTZdzXclpH6Y=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.7/go.mod h1:1PFqtAUsIgF1nEVojzRJSf/v88zjzqrGW+w+UAtkXyQ=
 github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.6 h1:Qmn60RWuk9+bd2ddvJJqwf3Cpac3VvGMOdbGemtUhX4=
 github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.6/go.mod h1:h/YJhe4AVyQxEUMluph10iqV7mjajH7A9L+BHF5ppis=
 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.21 h1:e6YW5uME30TkrElR+4/xcY7NB6CCj2dcv7vtmGXhfhQ=
@@ -46,10 +46,14 @@ github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.6 h1:LHHdobjo5
 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.6/go.mod h1:CBGB0evAiaMD3Lk4wPO7LQCcCZLuhzL5Qf/wfEUHQ5w=
 github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.20.6 h1:mYViDr2azcgxOpRrObZOl631bvavqILoE/L+Kxgbwys=
 github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.20.6/go.mod h1:OXVdAo9jNhggB+hJRGOyy7WolGTKB6Rg7IIulDeSzPM=
+github.com/aws/aws-sdk-go-v2/service/lambda v1.113.3 h1:oSHldse7GDHpdq81XeAJnhIbsdDKP9bMIt00G8W6uXI=
+github.com/aws/aws-sdk-go-v2/service/lambda v1.113.3/go.mod h1:Y//K046w4uhiR3o77beo/y52Jv2uEY13iJ+3+jicLO8=
 github.com/aws/aws-sdk-go-v2/service/s3 v1.114.4 h1:MrkWRB58Qw5TEkb51QVooVLjlM3IevO2iVXEz74B8pg=
 github.com/aws/aws-sdk-go-v2/service/s3 v1.114.4/go.mod h1:yfIxaKRHasvfy6tNEm3/LpqKsupMzQjMaWj9Y0jMVbo=
 github.com/aws/aws-sdk-go-v2/service/signin v1.10.4 h1:tvpDjuUvgKURf3Yk4hOaGbZ4QdY+070OWcA5d6LUv2A=
 github.com/aws/aws-sdk-go-v2/service/signin v1.10.4/go.mod h1:3ibVKg5Ng82Zby8++jELTD2nHxiPk2DJRLEeW/LKp8M=
+github.com/aws/aws-sdk-go-v2/service/sqs v1.52.6 h1:Suq1LNhMBpsmShEv3vO+J3H8HEeK+GwYMJDYLE4IJSs=
+github.com/aws/aws-sdk-go-v2/service/sqs v1.52.6/go.mod h1:rHGuzFg4AfVnWZa3/533tn5GQd1LzVkTXI9WMT8LqAc=
 github.com/aws/aws-sdk-go-v2/service/sso v1.38.4 h1:ChCtagixSFOxltv0o6E3NT8F7ksvZtZGPwUBynO5Nmk=
 github.com/aws/aws-sdk-go-v2/service/sso v1.38.4/go.mod h1:+FEO9jrVMpc9oN3CdAxJbtlQEu3z8UddDPSedpeM3PY=
 github.com/aws/aws-sdk-go-v2/service/ssooidc v1.43.4 h1:j6c+HA4AoS0EBEjArPjavFEBrucJEryfrQU0TffmACk=

internal/awsmodel/embed.go

@@ -17,6 +17,7 @@ var modelFiles = map[Service]string{
 	ElasticLoadBalancingV2: "models/elastic-load-balancing-v2-2015-12-01.json.gz",
 	IAM:                    "models/iam-2010-05-08.json.gz",
 	RDS:                    "models/rds-2014-10-31.json.gz",
+	Route53:                "models/route-53-2013-04-01.json.gz",
 	S3:                     "models/s3-2006-03-01.json.gz",
 	STS:                    "models/sts-2011-06-15.json.gz",
 }

internal/awsmodel/encode.go

@@ -1,8 +1,10 @@
 package awsmodel
 
 import (
+	"bytes"
 	"encoding/base64"
 	"encoding/json"
+	"encoding/xml"
 	"fmt"
 	"maps"
 	"net/http"
@@ -39,7 +41,7 @@ func (r *EncodedRequest) URL(endpoint string) string {
 }
 
 // EncodeRequest serializes input, as produced by GenerateRequests, for the
-// service's protocol. S3's rest-xml is not supported.
+// service's protocol: query, ec2, json, rest-json or rest-xml.
 func EncodeRequest(service Service, operationName string, input map[string]any) (*EncodedRequest, error) {
 	model, err := Load(service)
 	if err != nil {
@@ -57,6 +59,8 @@ func EncodeRequest(service Service, operationName string, input map[string]any)
 		return encoder.json(operation, input)
 	case "rest-json":
 		return encoder.restJSON(operation, input)
+	case "rest-xml":
+		return encoder.restXML(operation, input)
 	default:
 		return nil, fmt.Errorf("awsmodel: encoding %s requests is not implemented", model.metadata.Protocol)
 	}
@@ -282,18 +286,50 @@ func (e requestEncoder) jsonValue(ref ShapeRef, value any) (any, error) {
 }
 
 func (e requestEncoder) restJSON(operation *Operation, input map[string]any) (*EncodedRequest, error) {
+	request, members, err := e.restBindings(operation, input)
+	if err != nil || len(members) == 0 {
+		return request, err
+	}
+	shape := e.model.shapes[operation.Input.Shape]
+	var document any
+	if payload, ok := members[shape.Payload]; ok && shape.Payload != "" {
+		if document, err = e.jsonValue(shape.Members[shape.Payload], payload); err != nil {
+			return nil, err
+		}
+	} else {
+		body := make(map[string]any, len(members))
+		for member, value := range members {
+			ref := shape.Members[member]
+			encoded, err := e.jsonValue(ref, value)
+			if err != nil {
+				return nil, err
+			}
+			body[elementName(member, ref)] = encoded
+		}
+		document = body
+	}
+	if request.Body, err = json.Marshal(document); err != nil {
+		return nil, err
+	}
+	request.Header.Set("Content-Type", "application/json")
+	return request, nil
+}
+
+// restBindings places the URI, query string and header members of input on a
+// new request, and returns the members left for the body.
+func (e requestEncoder) restBindings(operation *Operation, input map[string]any) (*EncodedRequest, map[string]any, error) {
 	request := &EncodedRequest{Method: operation.HTTP.Method, Query: url.Values{}, Header: http.Header{}}
 	path := operation.HTTP.RequestURI
 	if operation.Input == nil {
 		request.Path = path
-		return request, nil
+		return request, nil, nil
 	}
 	shape := e.model.shapes[operation.Input.Shape]
 	body := map[string]any{}
 	for _, member := range slices.Sorted(maps.Keys(input)) {
 		ref, ok := shape.Members[member]
 		if !ok {
-			return nil, fmt.Errorf("%s has no member %q", operation.Input.Shape, member)
+			return nil, nil, fmt.Errorf("%s has no member %q", operation.Input.Shape, member)
 		}
 		value := input[member]
 		target := e.model.shapes[ref.Shape]
@@ -301,18 +337,18 @@ func (e requestEncoder) restJSON(operation *Operation, input map[string]any) (*E
 		case "uri":
 			text, err := scalarString(value, ref, target, "iso8601")
 			if err != nil {
-				return nil, fmt.Errorf("%s: %w", member, err)
+				return nil, nil, fmt.Errorf("%s: %w", member, err)
 			}
 			path = strings.Replace(path, "{"+member+"+}", escapeGreedyLabel(text), 1)
 			path = strings.Replace(path, "{"+member+"}", url.PathEscape(text), 1)
 		case "querystring":
 			if err := e.restQuery(request.Query, ref, target, value); err != nil {
-				return nil, fmt.Errorf("%s: %w", member, err)
+				return nil, nil, fmt.Errorf("%s: %w", member, err)
 			}
 		case "header":
 			text, err := e.headerValue(ref, target, value)
 			if err != nil {
-				return nil, fmt.Errorf("%s: %w", member, err)
+				return nil, nil, fmt.Errorf("%s: %w", member, err)
 			}
 			request.Header.Set(ref.LocationName, text)
 		case "headers":
@@ -321,41 +357,222 @@ func (e requestEncoder) restJSON(operation *Operation, input map[string]any) (*E
 				request.Header.Set(ref.LocationName+key, fmt.Sprint(entry))
 			}
 		default:
-			encoded, err := e.jsonValue(ref, value)
-			if err != nil {
-				return nil, err
-			}
-			name := member
-			if ref.LocationName != "" {
-				name = ref.LocationName
-			}
-			body[name] = encoded
+			body[member] = value
 		}
 	}
 	request.Path = path
-	if payload := shape.Payload; payload != "" {
-		if document, ok := body[payload]; ok {
-			body = nil
-			encoded, err := json.Marshal(document)
-			if err != nil {
-				return nil, err
-			}
-			request.Body = encoded
-		}
+	return request, body, nil
+}
+
+// restXML encodes the body as AWS SDKs do: a root element named for the input
+// shape, or for the payload member, in the service's namespace. An input with
+// body members sends its root element even when none of them is set.
+func (e requestEncoder) restXML(operation *Operation, input map[string]any) (*EncodedRequest, error) {
+	request, members, err := e.restBindings(operation, input)
+	if err != nil || operation.Input == nil {
+		return request, err
 	}
-	if len(body) > 0 {
-		encoded, err := json.Marshal(body)
-		if err != nil {
+	shape := e.model.shapes[operation.Input.Shape]
+	var root *xmlElement
+	switch {
+	case shape.Payload != "":
+		value, ok := members[shape.Payload]
+		if !ok {
+			return request, nil
+		}
+		ref := shape.Members[shape.Payload]
+		target := e.model.shapes[ref.Shape]
+		if target.Type != "structure" {
+			switch value := value.(type) {
+			case []byte:
+				request.Body = value
+			case string:
+				request.Body = []byte(value)
+			default:
+				return nil, fmt.Errorf("%s: want a blob or string payload, got %T", shape.Payload, value)
+			}
+			return request, nil
+		}
+		name := ref.LocationName
+		if name == "" {
+			name = elementName(ref.Shape, ShapeRef{LocationName: target.LocationName})
+		}
+		if root, err = e.xmlRoot(name, ref, value); err != nil {
 			return nil, err
 		}
-		request.Body = encoded
+	case hasBodyMembers(shape):
+		name := elementName(operation.Input.Shape, ShapeRef{LocationName: shape.LocationName})
+		if root, err = e.xmlRoot(name, *operation.Input, members); err != nil {
+			return nil, err
+		}
+	default:
+		return request, nil
 	}
-	if request.Body != nil {
-		request.Header.Set("Content-Type", "application/json")
+	var body bytes.Buffer
+	encoder := xml.NewEncoder(&body)
+	if err := root.encode(encoder); err != nil {
+		return nil, err
 	}
+	if err := encoder.Flush(); err != nil {
+		return nil, err
+	}
+	request.Body = body.Bytes()
+	request.Header.Set("Content-Type", "application/xml")
 	return request, nil
 }
 
+func hasBodyMembers(shape *Shape) bool {
+	for _, ref := range shape.Members {
+		if ref.Location == "" {
+			return true
+		}
+	}
+	return false
+}
+
+// xmlRoot is the structure value encoded under name, in its shape's namespace
+// or else the service's.
+func (e requestEncoder) xmlRoot(name string, ref ShapeRef, value any) (*xmlElement, error) {
+	holder := &xmlElement{}
+	if err := e.xmlValue(holder, name, ref, value); err != nil {
+		return nil, err
+	}
+	root := holder.children[0]
+	if namespace := e.model.metadata.XMLNamespace; namespace.URI != "" && e.model.shapes[ref.Shape].XMLNamespace.URI == "" {
+		root.attrs = append([]xml.Attr{namespaceAttr(namespace)}, root.attrs...)
+	}
+	return root, nil
+}
+
+func (e requestEncoder) xmlStructure(parent *xmlElement, shapeName string, fields map[string]any) error {
+	shape := e.model.shapes[shapeName]
+	for _, member := range slices.Sorted(maps.Keys(fields)) {
+		ref, ok := shape.Members[member]
+		if !ok {
+			return fmt.Errorf("%s has no member %q", shapeName, member)
+		}
+		name := elementName(member, ref)
+		if ref.XMLAttribute {
+			text, err := scalarString(fields[member], ref, e.model.shapes[ref.Shape], "iso8601")
+			if err != nil {
+				return fmt.Errorf("%s: %w", name, err)
+			}
+			parent.attrs = append(parent.attrs, xml.Attr{Name: xml.Name{Local: name}, Value: text})
+			continue
+		}
+		if err := e.xmlValue(parent, name, ref, fields[member]); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// xmlValue appends the elements value is encoded as to parent. A list or map
+// is wrapped in an element named name unless it is flattened.
+func (e requestEncoder) xmlValue(parent *xmlElement, name string, ref ShapeRef, value any) error {
+	shape := e.model.shapes[ref.Shape]
+	switch shape.Type {
+	case "structure":
+		fields, ok := value.(map[string]any)
+		if !ok {
+			return fmt.Errorf("%s: want a structure, got %T", name, value)
+		}
+		element := parent.add(name)
+		if shape.XMLNamespace.URI != "" {
+			element.attrs = append(element.attrs, namespaceAttr(shape.XMLNamespace))
+		}
+		return e.xmlStructure(element, ref.Shape, fields)
+	case "list":
+		items, ok := value.([]any)
+		if !ok {
+			return fmt.Errorf("%s: want a list, got %T", name, value)
+		}
+		container, itemName := parent, name
+		if !ref.Flattened {
+			container, itemName = parent.add(name), elementName("member", *shape.Member)
+		}
+		for _, item := range items {
+			if err := e.xmlValue(container, itemName, *shape.Member, item); err != nil {
+				return err
+			}
+		}
+		return nil
+	case "map":
+		entries, ok := value.(map[string]any)
+		if !ok {
+			return fmt.Errorf("%s: want a map, got %T", name, value)
+		}
+		container, entryName := parent, name
+		if !ref.Flattened {
+			container, entryName = parent.add(name), "entry"
+		}
+		keyName, valueName := elementName("key", *shape.Key), elementName("value", *shape.Value)
+		for _, key := range slices.Sorted(maps.Keys(entries)) {
+			entry := container.add(entryName)
+			entry.add(keyName).text = key
+			if err := e.xmlValue(entry, valueName, *shape.Value, entries[key]); err != nil {
+				return err
+			}
+		}
+		return nil
+	default:
+		text, err := scalarString(value, ref, shape, "iso8601")
+		if err != nil {
+			return fmt.Errorf("%s: %w", name, err)
+		}
+		parent.add(name).text = text
+		return nil
+	}
+}
+
+// elementName is a member's wire name: its xmlName or jsonName, else its own.
+func elementName(member string, ref ShapeRef) string {
+	if ref.LocationName != "" {
+		return ref.LocationName
+	}
+	return member
+}
+
+func namespaceAttr(namespace XMLNamespace) xml.Attr {
+	name := "xmlns"
+	if namespace.Prefix != "" {
+		name += ":" + namespace.Prefix
+	}
+	return xml.Attr{Name: xml.Name{Local: name}, Value: namespace.URI}
+}
+
+// xmlElement is one element of an encoded rest-xml body.
+type xmlElement struct {
+	name     string
+	attrs    []xml.Attr
+	text     string
+	children []*xmlElement
+}
+
+func (x *xmlElement) add(name string) *xmlElement {
+	child := &xmlElement{name: name}
+	x.children = append(x.children, child)
+	return child
+}
+
+func (x *xmlElement) encode(encoder *xml.Encoder) error {
+	start := xml.StartElement{Name: xml.Name{Local: x.name}, Attr: x.attrs}
+	if err := encoder.EncodeToken(start); err != nil {
+		return err
+	}
+	if x.text != "" {
+		if err := encoder.EncodeToken(xml.CharData(x.text)); err != nil {
+			return err
+		}
+	}
+	for _, child := range x.children {
+		if err := child.encode(encoder); err != nil {
+			return err
+		}
+	}
+	return encoder.EncodeToken(start.End())
+}
+
 func (e requestEncoder) restQuery(query url.Values, ref ShapeRef, shape *Shape, value any) error {
 	switch shape.Type {
 	case "list":

internal/awsmodel/encode_restxml_test.go

@@ -0,0 +1,212 @@
+package awsmodel_test
+
+import (
+	"cmp"
+	"encoding/xml"
+	"io"
+	"net/url"
+	"slices"
+	"strings"
+	"testing"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/aws/credentials"
+	"github.com/aws/aws-sdk-go/aws/request"
+	"github.com/aws/aws-sdk-go/aws/session"
+	"github.com/aws/aws-sdk-go/service/route53"
+	"github.com/aws/aws-sdk-go/service/s3"
+	"github.com/mulgadc/spinifex/internal/awsmodel"
+	"github.com/stretchr/testify/require"
+)
+
+const sdkEndpoint = "https://aws.example"
+
+var sdkSession = session.Must(session.NewSession(&aws.Config{
+	Region:           aws.String("us-east-1"),
+	Endpoint:         aws.String(sdkEndpoint),
+	Credentials:      credentials.AnonymousCredentials,
+	S3ForcePathStyle: aws.Bool(true),
+}))
+
+// TestEncodeRequestRestXMLMatchesSDK checks the encoder against aws-sdk-go's
+// restxml marshaller for the same input, so the expected wire form is the
+// SDK's rather than one written by hand.
+func TestEncodeRequestRestXMLMatchesSDK(t *testing.T) {
+	r53 := route53.New(sdkSession)
+	tests := []struct {
+		name      string
+		service   awsmodel.Service
+		operation string
+		input     map[string]any
+		sdk       *request.Request
+	}{
+		{
+			name:      "CreateHostedZone",
+			service:   awsmodel.Route53,
+			operation: "CreateHostedZone",
+			input: map[string]any{
+				"Name":             "example.com",
+				"CallerReference":  "ref-1",
+				"HostedZoneConfig": map[string]any{"Comment": "a & b", "PrivateZone": true},
+				"VPC":              map[string]any{"VPCRegion": "us-east-1", "VPCId": "vpc-1"},
+			},
+			sdk: requestOf(r53.CreateHostedZoneRequest(&route53.CreateHostedZoneInput{
+				Name:             aws.String("example.com"),
+				CallerReference:  aws.String("ref-1"),
+				HostedZoneConfig: &route53.HostedZoneConfig{Comment: aws.String("a & b"), PrivateZone: aws.Bool(true)},
+				VPC:              &route53.VPC{VPCRegion: aws.String("us-east-1"), VPCId: aws.String("vpc-1")},
+			})),
+		},
+		{
+			name:      "ChangeResourceRecordSets",
+			service:   awsmodel.Route53,
+			operation: "ChangeResourceRecordSets",
+			input: map[string]any{
+				"HostedZoneId": "Z0123456789",
+				"ChangeBatch": map[string]any{
+					"Comment": "two changes",
+					"Changes": []any{
+						map[string]any{"Action": "UPSERT", "ResourceRecordSet": map[string]any{
+							"Name": "www.example.com", "Type": "A", "TTL": int64(300),
+							"ResourceRecords": []any{map[string]any{"Value": "192.0.2.1"}, map[string]any{"Value": "192.0.2.2"}},
+						}},
+						map[string]any{"Action": "DELETE", "ResourceRecordSet": map[string]any{
+							"Name": "old.example.com", "Type": "TXT", "TTL": int64(60),
+							"ResourceRecords": []any{map[string]any{"Value": `"v=1"`}},
+						}},
+					},
+				},
+			},
+			sdk: requestOf(r53.ChangeResourceRecordSetsRequest(&route53.ChangeResourceRecordSetsInput{
+				HostedZoneId: aws.String("Z0123456789"),
+				ChangeBatch: &route53.ChangeBatch{
+					Comment: aws.String("two changes"),
+					Changes: []*route53.Change{
+						{Action: aws.String("UPSERT"), ResourceRecordSet: &route53.ResourceRecordSet{
+							Name: aws.String("www.example.com"), Type: aws.String("A"), TTL: aws.Int64(300),
+							ResourceRecords: []*route53.ResourceRecord{{Value: aws.String("192.0.2.1")}, {Value: aws.String("192.0.2.2")}},
+						}},
+						{Action: aws.String("DELETE"), ResourceRecordSet: &route53.ResourceRecordSet{
+							Name: aws.String("old.example.com"), Type: aws.String("TXT"), TTL: aws.Int64(60),
+							ResourceRecords: []*route53.ResourceRecord{{Value: aws.String(`"v=1"`)}},
+						}},
+					},
+				},
+			})),
+		},
+		{
+			name:      "ChangeTagsForResource",
+			service:   awsmodel.Route53,
+			operation: "ChangeTagsForResource",
+			input: map[string]any{
+				"ResourceType":  "hostedzone",
+				"ResourceId":    "Z0123456789",
+				"AddTags":       []any{map[string]any{"Key": "team", "Value": "dns"}, map[string]any{"Key": "env", "Value": "prod"}},
+				"RemoveTagKeys": []any{"old", "stale"},
+			},
+			sdk: requestOf(r53.ChangeTagsForResourceRequest(&route53.ChangeTagsForResourceInput{
+				ResourceType:  aws.String("hostedzone"),
+				ResourceId:    aws.String("Z0123456789"),
+				AddTags:       []*route53.Tag{{Key: aws.String("team"), Value: aws.String("dns")}, {Key: aws.String("env"), Value: aws.String("prod")}},
+				RemoveTagKeys: []*string{aws.String("old"), aws.String("stale")},
+			})),
+		},
+		{
+			name:      "UpdateHostedZoneComment without a comment",
+			service:   awsmodel.Route53,
+			operation: "UpdateHostedZoneComment",
+			input:     map[string]any{"Id": "Z0123456789"},
+			sdk:       requestOf(r53.UpdateHostedZoneCommentRequest(&route53.UpdateHostedZoneCommentInput{Id: aws.String("Z0123456789")})),
+		},
+		{
+			name:      "ListResourceRecordSets",
+			service:   awsmodel.Route53,
+			operation: "ListResourceRecordSets",
+			input:     map[string]any{"HostedZoneId": "Z0123456789", "StartRecordName": "www.example.com", "StartRecordType": "A", "MaxItems": int64(5)},
+			sdk: requestOf(r53.ListResourceRecordSetsRequest(&route53.ListResourceRecordSetsInput{
+				HostedZoneId: aws.String("Z0123456789"), StartRecordName: aws.String("www.example.com"), StartRecordType: aws.String("A"), MaxItems: aws.String("5"),
+			})),
+		},
+		{
+			name:      "DeleteHostedZone",
+			service:   awsmodel.Route53,
+			operation: "DeleteHostedZone",
+			input:     map[string]any{"Id": "Z0123456789"},
+			sdk:       requestOf(r53.DeleteHostedZoneRequest(&route53.DeleteHostedZoneInput{Id: aws.String("Z0123456789")})),
+		},
+		{
+			name:      "S3 structure payload",
+			service:   awsmodel.S3,
+			operation: "PutBucketTagging",
+			input: map[string]any{
+				"Bucket":  "bucket",
+				"Tagging": map[string]any{"TagSet": []any{map[string]any{"Key": "team", "Value": "storage"}}},
+			},
+			sdk: requestOf(s3.New(sdkSession).PutBucketTaggingRequest(&s3.PutBucketTaggingInput{
+				Bucket:  aws.String("bucket"),
+				Tagging: &s3.Tagging{TagSet: []*s3.Tag{{Key: aws.String("team"), Value: aws.String("storage")}}},
+			})),
+		},
+	}
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			require.NoError(t, test.sdk.Build())
+			want := test.sdk.HTTPRequest
+			wantBody, err := io.ReadAll(test.sdk.GetBody())
+			require.NoError(t, err)
+
+			got, err := awsmodel.EncodeRequest(test.service, test.operation, test.input)
+			require.NoError(t, err)
+			require.Equal(t, want.Method, got.Method)
+			gotURL, err := url.Parse(got.URL(sdkEndpoint))
+			require.NoError(t, err)
+			// The SDK's api-2.json ends ChangeResourceRecordSets' path in a
+			// slash that the Smithy model drops; the gateway accepts both.
+			require.Equal(t, strings.TrimSuffix(want.URL.Path, "/"), gotURL.Path)
+			require.Equal(t, want.URL.Query(), gotURL.Query())
+			if len(wantBody) == 0 {
+				require.Empty(t, got.Body)
+				require.Empty(t, got.Header.Get("Content-Type"))
+				return
+			}
+			require.Equal(t, canonicalXML(t, wantBody), canonicalXML(t, got.Body))
+			require.Equal(t, "application/xml", got.Header.Get("Content-Type"))
+		})
+	}
+}
+
+func requestOf[O any](r *request.Request, _ O) *request.Request { return r }
+
+// xmlTree is a parsed XML element. The SDK emits sibling elements in map
+// order, so canonicalXML sorts them by name, keeping list items in order.
+type xmlTree struct {
+	XMLName  xml.Name   `xml:""`
+	Attrs    []xml.Attr `xml:",any,attr"`
+	Text     string     `xml:",chardata"`
+	Children []xmlTree  `xml:",any"`
+}
+
+func canonicalXML(t *testing.T, body []byte) xmlTree {
+	t.Helper()
+	var tree xmlTree
+	require.NoError(t, xml.Unmarshal(body, &tree), string(body))
+	var canonicalise func(*xmlTree)
+	canonicalise = func(node *xmlTree) {
+		node.Text = strings.TrimSpace(node.Text)
+		slices.SortFunc(node.Attrs, func(a, b xml.Attr) int { return cmp.Compare(a.Name.Space+a.Name.Local, b.Name.Space+b.Name.Local) })
+		slices.SortStableFunc(node.Children, func(a, b xmlTree) int { return cmp.Compare(a.XMLName.Local, b.XMLName.Local) })
+		for i := range node.Children {
+			canonicalise(&node.Children[i])
+		}
+	}
+	canonicalise(&tree)
+	return tree
+}
+
+func TestEncodeRequestRestXMLPutsRootInServiceNamespace(t *testing.T) {
+	request, err := awsmodel.EncodeRequest(awsmodel.Route53, "CreateHostedZone", map[string]any{"Name": "example.com", "CallerReference": "ref"})
+	require.NoError(t, err)
+	tree := canonicalXML(t, request.Body)
+	require.Equal(t, xml.Name{Space: "https://route53.amazonaws.com/doc/2013-04-01/", Local: "CreateHostedZoneRequest"}, tree.XMLName)
+	require.Equal(t, "/2013-04-01/hostedzone", request.Path)
+}

internal/awsmodel/encode_test.go

@@ -117,8 +117,3 @@ func TestEncodeRequestRejectsUnmodelledMember(t *testing.T) {
 	_, err := awsmodel.EncodeRequest(awsmodel.IAM, "GetRole", map[string]any{"RoleName": "app", "Bogus": "x"})
 	require.ErrorContains(t, err, `no member "Bogus"`)
 }
-
-func TestEncodeRequestS3IsNotSupported(t *testing.T) {
-	_, err := awsmodel.EncodeRequest(awsmodel.S3, "ListBuckets", map[string]any{})
-	require.ErrorContains(t, err, "not implemented")
-}

internal/awsmodel/error.go

@@ -119,7 +119,7 @@ func (m *Model) decodeErrorCode(body []byte) (string, error) {
 			return "", fmt.Errorf("JSON error envelope has no code")
 		}
 		return code, nil
-	case "ec2", "query":
+	case "ec2", "query", "rest-xml":
 		root, err := parseXML(body)
 		if err != nil {
 			return "", err

internal/awsmodel/error_test.go

@@ -123,3 +123,16 @@ func TestValidateErrorResponseClassifiesCommonErrorsAsUnmodelled(t *testing.T) {
 		})
 	}
 }
+
+func TestDecodeRequestResultReadsRestXMLErrorEnvelope(t *testing.T) {
+	body := []byte(`<?xml version="1.0"?>
+<ErrorResponse xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><Error><Type>Sender</Type><Code>NoSuchHostedZone</Code><Message>No hosted zone found with ID: Z1</Message></Error><RequestId>r-1</RequestId></ErrorResponse>`)
+	result := DecodeRequestResult(Route53, 404, nil, body)
+	if result.Code != "NoSuchHostedZone" || result.Message != "No hosted zone found with ID: Z1" {
+		t.Fatalf("result = %+v", result)
+	}
+	violations, modelled, err := ValidateErrorResponse(Route53, "GetHostedZone", body)
+	if err != nil || !modelled || len(violations) != 0 {
+		t.Fatalf("ValidateErrorResponse = %v, %v, %v; want a declared code", violations, modelled, err)
+	}
+}

internal/awsmodel/model.go

@@ -34,6 +34,7 @@ const (
 	ElasticLoadBalancingV2 Service = "elasticloadbalancingv2"
 	IAM                    Service = "iam"
 	RDS                    Service = "rds"
+	Route53                Service = "route53"
 	S3                     Service = "s3"
 	STS                    Service = "sts"
 )
@@ -50,6 +51,14 @@ type Metadata struct {
 	ServiceID       string
 	SigningName     string
 	TargetPrefix    string
+	XMLNamespace    XMLNamespace
+}
+
+// XMLNamespace is the namespace an XML element is serialized in. URI is empty
+// when none is declared.
+type XMLNamespace struct {
+	URI    string `json:"uri"`
+	Prefix string `json:"prefix"`
 }
 
 // Operation describes an AWS API operation and the shapes used by its input,
@@ -84,6 +93,7 @@ type Shape struct {
 	Pattern         string
 	LocationName    string
 	TimestampFormat string
+	XMLNamespace    XMLNamespace
 	Payload         string
 	Flattened       bool
 	Sensitive       bool

internal/awsmodel/model_test.go

@@ -8,7 +8,7 @@ import (
 )
 
 func TestServices(t *testing.T) {
-	want := []Service{ACM, EC2, ECR, ECS, EKS, ElasticLoadBalancingV2, IAM, RDS, S3, STS}
+	want := []Service{ACM, EC2, ECR, ECS, EKS, ElasticLoadBalancingV2, IAM, RDS, Route53, S3, STS}
 	if got := Services(); !reflect.DeepEqual(got, want) {
 		t.Fatalf("Services() = %v, want %v", got, want)
 	}
@@ -28,6 +28,7 @@ func TestLoadAllModels(t *testing.T) {
 		ElasticLoadBalancingV2: {"2015-12-01", 51, 418},
 		IAM:                    {"2010-05-08", 180, 624},
 		RDS:                    {"2014-10-31", 164, 791},
+		Route53:                {"2013-04-01", 71, 409},
 		S3:                     {"2006-03-01", 112, 710},
 		STS:                    {"2011-06-15", 11, 101},
 	}

internal/awsmodel/models/README.md

@@ -1,6 +1,6 @@
 # AWS models
 
-The `*.json.gz` files are the Smithy JSON AST service models from [`aws/api-models-aws`](https://github.com/aws/api-models-aws), gzipped and embedded into the binary, so loading them needs no network or Go module cache. All ten come from the one commit recorded in `../model_source.go`.
+The `*.json.gz` files are the Smithy JSON AST service models from [`aws/api-models-aws`](https://github.com/aws/api-models-aws), gzipped and embedded into the binary, so loading them needs no network or Go module cache. All eleven come from the one commit recorded in `../model_source.go`.
 
 Do not edit them by hand. To move to a newer commit, run `scripts/sync-aws-models.sh <commit-sha>`, then `make generate-aws-model-coverage` and review the regenerated `docs/coverage` and the `awsmodel` test expectations: a newer model can add operations, required members and enum values, and each resulting finding is a real gap to triage rather than a reason to stay on an older commit.
 

internal/awsmodel/request.go

@@ -374,6 +374,10 @@ func (g *requestGenerator) structure(name string, path []pathStep, depth int) (m
 	for _, member := range slices.Sorted(maps.Keys(shape.Members)) {
 		ref := shape.Members[member]
 		memberPath := append(slices.Clone(path), member)
+		if !required[member] && slices.Contains(exclusiveMembers[g.model.service][name], member) {
+			g.skip(memberPath, "AWS refuses it alongside the members a valid request carries")
+			continue
+		}
 		memberValue, ok := g.value(ref, memberPath, depth)
 		if !ok {
 			if required[member] {

internal/awsmodel/request_hints.go

@@ -18,7 +18,8 @@ import (
 
 // RequestOptions places generated ARNs in the account and region the
 // requests are sent as. AccessKeyID is the key the requests are signed with.
-// Fixtures maps a member to an existing resource it can name.
+// Fixtures maps a member, or an "Operation.Member" that overrides it for one
+// operation, to an existing resource it can name.
 type RequestOptions struct {
 	AccountID   string
 	Region      string
@@ -73,6 +74,12 @@ var ec2IDMember = regexp.MustCompile(`^(.*?)Ids?$`)
 // with a code the judge cannot tell from a real finding.
 var conditionallyRequired = map[Service]map[string][]string{
 	EC2: {"CreateCapacityReservation": {"AvailabilityZone"}},
+	// A listing that starts at a zone ID needs the name it starts at, and
+	// one that starts at a record type or identifier needs a name and type.
+	Route53: {
+		"ListHostedZonesByName":  {"DNSName"},
+		"ListResourceRecordSets": {"StartRecordName", "StartRecordType"},
+	},
 	ECS: {
 		"CreateCapacityProvider": {"autoScalingGroupProvider"},
 		"CreateService":          {"taskDefinition"},
@@ -123,14 +130,47 @@ var ec2TaggedTypes = map[string]string{
 	"RunInstances":                    "instance",
 }
 
+// exclusiveMembers are optional members a valid request leaves out of the
+// structure that holds them, because AWS refuses them in combination with
+// the rest. Generation skips them, so their constraints are not broken.
+var exclusiveMembers = map[Service]map[string][]string{
+	// A record set takes one routing policy and either records or an alias.
+	Route53: {"ResourceRecordSet": {
+		"AliasTarget", "CidrRoutingConfig", "Failover", "GeoLocation", "GeoProximityLocation", "HealthCheckId",
+		"MultiValueAnswer", "Region", "SetIdentifier", "TrafficPolicyInstanceId", "Weight",
+	}},
+}
+
+// fixture returns the fixture the member at path names, preferring one scoped
+// to the operation.
+func (g *requestGenerator) fixture(member string) (string, bool) {
+	if fixture, ok := g.options.Fixtures[g.operation+"."+member]; ok {
+		return fixture, true
+	}
+	fixture, ok := g.options.Fixtures[member]
+	return fixture, ok
+}
+
 // enumHint returns the enum value a valid request needs at path, where the
 // first value would not do.
 func (g *requestGenerator) enumHint(path []pathStep) (string, bool) {
-	if g.model.service != EC2 || len(path) == 0 || path[0] != "TagSpecifications" || memberName(path) != "ResourceType" {
-		return "", false
+	member := memberName(path)
+	if fixture, ok := g.fixture(member); ok {
+		return fixture, true
 	}
-	value, ok := ec2TaggedTypes[g.operation]
-	return value, ok
+	switch g.model.service {
+	case EC2:
+		if len(path) > 0 && path[0] == "TagSpecifications" && member == "ResourceType" {
+			value, ok := ec2TaggedTypes[g.operation]
+			return value, ok
+		}
+	case Route53:
+		// The first RRType, SOA, is one a zone already has and cannot gain.
+		if member == "Type" {
+			return "A", true
+		}
+	}
+	return "", false
 }
 
 // hint returns a value of the right form for the member at path, if one is
@@ -141,7 +181,7 @@ func (g *requestGenerator) hint(path []pathStep) (string, bool) {
 		return "", false
 	}
 	options := g.options
-	if fixture, ok := options.Fixtures[member]; ok {
+	if fixture, ok := g.fixture(member); ok {
 		return fixture, true
 	}
 	suffix := fmt.Sprintf("%017x", g.saltNumber)
@@ -230,6 +270,17 @@ func (g *requestGenerator) hint(path []pathStep) (string, bool) {
 		if member == "type" && len(path) == 1 {
 			return "STANDARD", true
 		}
+	case Route53:
+		// A generated name is a top-level domain, which cannot be a zone.
+		// Record names fall inside the fixture zone, example.com.
+		switch member {
+		case "Name", "DNSName", "StartRecordName":
+			return name + ".example.com", true
+		case "Value":
+			if slices.Contains(path, pathStep("ResourceRecords")) {
+				return "192.0.2.1", true
+			}
+		}
 	case STS:
 		// GetAccessKeyInfo answers only for a key that exists.
 		if member == "AccessKeyId" && options.AccessKeyID != "" {

internal/awsmodel/request_test.go

@@ -131,6 +131,7 @@ func TestGeneratedRequestsBreakOnlyTheirConstraint(t *testing.T) {
 				for _, request := range plan.Cases {
 					require.False(t, rejecting && request.Acceptance(), "%s acceptance %s follows a rejection", operationName, request.Path)
 					rejecting = !request.Acceptance()
+					// Predastore serves S3, so its requests are never sent.
 					if service != S3 {
 						_, err := EncodeRequest(service, operationName, request.Input)
 						require.NoError(t, err, "%s %s %s encodes", operationName, request.Constraint, request.Path)
@@ -241,6 +242,17 @@ func TestGenerateRequestsTargetsFixturesOutsideCreateAndDelete(t *testing.T) {
 	}
 }
 
+func TestGenerateRequestsPrefersOperationScopedFixtures(t *testing.T) {
+	options := testRequestOptions
+	options.Fixtures = map[string]string{"Id": "Z0FIXTURE", "GetChange.Id": "C0FIXTURE"}
+	for operation, want := range map[string]string{"GetChange": "C0FIXTURE", "GetHostedZone": "Z0FIXTURE"} {
+		plan, err := GenerateRequests(Route53, operation, options)
+		require.NoError(t, err)
+		require.True(t, plan.Cases[0].Acceptance())
+		require.Equal(t, want, plan.Cases[0].Input["Id"], operation)
+	}
+}
+
 func TestGenerateRequestsAttributesRejections(t *testing.T) {
 	plan, err := GenerateRequests(IAM, "CreateRole", testRequestOptions)
 	require.NoError(t, err)

internal/awsmodel/smithy.go

@@ -281,6 +281,9 @@ func (t *smithyTranslator) metadata() (Metadata, error) {
 	if _, err := service.Traits.decode("aws.auth#sigv4", &sigv4); err != nil {
 		return Metadata{}, err
 	}
+	if _, err := service.Traits.decode("smithy.api#xmlNamespace", &metadata.XMLNamespace); err != nil {
+		return Metadata{}, err
+	}
 	metadata.EndpointPrefix = awsService.EndpointPrefix
 	metadata.ServiceID = awsService.SDKID
 	metadata.ServiceFullName = title
@@ -443,6 +446,9 @@ func (t *smithyTranslator) applyShapeTraits(shape *Shape, traits smithyTraits) e
 	if shape.LocationName, err = traits.string("smithy.api#xmlName"); err != nil {
 		return err
 	}
+	if _, err := traits.decode("smithy.api#xmlNamespace", &shape.XMLNamespace); err != nil {
+		return err
+	}
 	if shape.TimestampFormat, err = timestampFormat(traits); err != nil {
 		return err
 	}

scripts/README.md

@@ -6,22 +6,67 @@ Node installation, dev-environment lifecycle, guest image builds, and verificati
 
 | Script | What it does |
 |---|---|
-| `setup.sh` | The production binary installer, served from `install.mulgadc.com`. Installs deps, users, sudoers, files, systemd units, logrotate and udev rules. `SETUP_STAGES` runs a subset; `ISO_BUILD=1` runs it inside the ISO builder's chroot. |
-| `setup-ovn.sh` | Bootstraps a compute node for OVN VPC networking: installs OVN/OVS, creates `br-int`, configures the WAN bridge (bridged, DHCP, or routed-NAT uplink), sets the chassis identity, and applies overlay sysctls. `--management` also starts the OVN central services. |
-| `install-node.sh` | Forms a multi-node cluster across hosts that are already running Spinifex: rebuilds the OVN database in clustered form, runs `spx admin init` on the first host and `spx admin join` on the rest, then verifies. The scripted form of [install-multi-node](../docs/install/install-multi-node/README.md). Destroys the joining nodes' logical network state and crypto — see its `--help`. |
+| `setup.sh` | The production binary installer, served from `install.mulgadc.com`. Installs deps, users, sudoers, the host firewall policy, files, systemd units, logrotate and udev rules, and loads `openvswitch`, `geneve`, `vhost_net` and `tun`, listing them in `/etc/modules-load.d/spinifex.conf` for later boots. It masks firewalld wherever it is installed, since the Spinifex nftables policy is the only supported node firewall. An unset `--firewall` keeps the mode recorded in `/etc/spinifex/firewall/mode`, and otherwise arms the policy when firewalld was active. Where systemd-resolved is installed but not running, resolvconf is absent and NetworkManager runs, it hands host DNS to resolved: it saves `/etc/resolv.conf` once to `/var/lib/spinifex-reset/resolv.conf.orig`, links `/etc/resolv.conf` to the resolved stub, and stops unless resolved has upstream servers and the package repository host still resolves. It links `/usr/bin/spx` to `/usr/local/bin/spx` where sudo's PATH leaves out `/usr/local/bin`, and takes the nbdkit plugin directory from `nbdkit --dump-config`, stopping when nbdkit cannot say. `SETUP_STAGES` runs a subset; `ISO_BUILD=1` runs it inside the ISO builder's chroot. |
+| `setup-ovn.sh` | Bootstraps a compute node for OVN VPC networking: installs OVN/OVS, creates `br-int`, configures the WAN bridge (bridged, DHCP, or routed-NAT uplink), sets the chassis identity, and applies overlay sysctls. It runs OVS and OVN as root, setting `OVS_USER_ID` and `OVN_USER_ID` where `/etc/sysconfig` holds them, and hooks the socket-permission helper onto `ovs-vswitchd`, `ovn-controller`, `openvswitch-ipsec` and, on EL, `ovn-northd`. `--management` also starts the OVN central services, writes their `ovn-ctl` flags to `/etc/default/ovn-central` (Debian) or `/etc/sysconfig/ovn-northd` (EL), and records the role in `/etc/spinifex/ovn-role`. With clustered databases it returns only once this node's NB and SB members are back in their RAFT clusters under a known leader, and fails after `OVN_RAFT_REJOIN_WAIT` seconds (default 60) otherwise, so reruns one node at a time never take two members of a cluster down. A later run without `--management` refuses while that file, or the `OVN_CTL_OPTS` line older nodes carry in `/etc/default/ovn-central`, records the role. It adds `maxsize 100M` to the OVS and OVN logrotate files, plus `rotate 5` where the file sets no count, and rotates as root, matching the daemons. On a host whose IPsec stack is libreswan it enables, stops and masks `openvswitch-ipsec.service` and `ipsec.service` for the daemon to unmask when IPsec is on, unless `openvswitch-ipsec.service` is already running; elsewhere it enables and starts `openvswitch-ipsec.service`, unless the daemon has masked it to keep IPsec off. The links it creates persist through systemd-networkd where that is enabled, and otherwise through `/etc/spinifex/hostnet.json`, which `spinifex-hostnet.service` (`spx admin hostnet apply`) replays at boot and on every OVS start, starting `ovn-controller` again when an OVS stop took it down; the first run records the choice in `/etc/spinifex/hostnet-backend`. Where NetworkManager runs, it leaves the links Spinifex owns unmanaged. |
+| `install-node.sh` | Forms a multi-node cluster across hosts that are already running Spinifex: rebuilds the OVN database in clustered form, runs `spx admin init` on the first host and `spx admin join` on the rest, then verifies. Before touching any host it refuses hosts of different OS families (`distro_os_family`) or `spx` versions, and `--ipsec on` when any host uses libreswan. The scripted form of [install-multi-node](../docs/install/install-multi-node/README.md). Destroys the joining nodes' logical network state and crypto — see its `--help`. |
 | `dev-install.sh` | Full local dev setup through the production installer: builds from source, packs a tarball, runs `setup.sh`, initialises the cluster, and starts services under systemd. |
+| `build-oracle-appliance.sh` | Turns an Oracle Linux 10.2 system reachable over SSH with passwordless sudo into a sealed gold image, then powers it off. It updates OL and reboots into the newest UEK R8, sets SELinux permissive, installs Spinifex without initialising it, and leaves `spinifex.target` and the NATS watchdog disabled. It creates the `spinifex` login with a default password that must be changed at first login, installs `appliance/firstboot.sh` as `spinifex-appliance-firstboot.service`, and wipes the system's identity (machine-id, SSH host keys, OVS chassis, NetworkManager profiles, logs). On each unit's first boot that service puts the first physical NIC with carrier on a DHCP `br-wan`, names the host `spinifex-<mac>`, runs `setup-ovn.sh --management` and `spx admin init --nodes 1`, and shows the IP and login on the console and in `/etc/motd`. `--channel`, `--version` and `--tarball` pick what to install. |
 | `clone-deps.sh` | Clones or updates the viperblock and predastore checkouts alongside spinifex for cross-repo development. |
 
+### `lib/distro.sh`: host-OS capability probes
+
+The node scripts select host-OS behaviour by probing for the capability, not by reading `/etc/os-release`; only `setup.sh`'s supported-OS gate keys on the OS ID.
+Each probe prints its answer, or names every candidate it tried on stderr and returns 1, which it also does when it finds both candidates of a pair.
+
+| Probe | Answers |
+|---|---|
+| `distro_pkg_manager` | `apt` or `dnf` |
+| `distro_ovs_unit` | `openvswitch-switch.service` (Debian, Ubuntu) or `openvswitch.service` (EL) |
+| `distro_ovn_central_layout` | `split` (Debian's `ovn-central` plus separate NB and SB units) or `lone-northd` (EL's single `ovn-northd.service`) |
+| `distro_ca_trust` | the trust-store update command and its anchor directory |
+| `distro_ovs_logrotate` | `/etc/logrotate.d/openvswitch-switch` or `/etc/logrotate.d/openvswitch` |
+| `distro_ovn_logrotate` | `/etc/logrotate.d/ovn-common` or `/etc/logrotate.d/ovn` |
+| `distro_os_family` | the OS family of the `/etc/os-release` text on stdin: `debian` for Debian and Ubuntu, otherwise the `ID` |
+
+`distro_has_libreswan` is a predicate rather than a probe: it succeeds when libreswan's IKE daemon, `/usr/libexec/ipsec/pluto`, is installed.
+EL's `openvswitch-ipsec` drives libreswan even beside strongSwan, so finding both is not an error.
+
+`distro_missing_ovs_deps <unit-file>...` is a check rather than a probe: it prints each OVS or OVN unit the given unit files depend on or order against that is not installed, and fails only when a file cannot be read.
+`setup-ovn.sh` runs it against the installed Spinifex units and stops when one is missing, because systemd silently drops ordering against a unit that does not exist; a missing `openvswitch-ipsec.service` is only a warning.
+
+`setup-ovn.sh`, `uninstall-spx.sh` and `install-node.sh` source it from `lib/` beside themselves, then from `/usr/local/share/spinifex/lib/distro.sh`, where `setup.sh` installs it, and stop naming both paths when neither exists.
+`setup.sh` is piped from the install endpoint before the tarball is downloaded, and `install-node.sh --wipe` copies `node-reset.sh` alone to each host, so those two carry inline copies of the probes they need between `# --- BEGIN distro.sh probes ---` markers.
+`lib/distro_test.sh` fails when an inline copy differs from the library.
+Go code uses `spinifex/runtime/host/hostos` for the same purpose.
+
+### System packages
+
+`setup.sh` installs `APT_RUNTIME_PACKAGES` with apt, or on Oracle Linux `OL_RUNTIME_PACKAGES` with dnf, choosing by `distro_pkg_manager`.
+Before the dnf install it enables `ol10_UEKR8` and `ol10_kvm_utils`, excludes `qemu-kvm*` and `qemu-img*` from `ol10_appstream` so QEMU always comes from KVM Utilities, and writes `/etc/yum.repos.d/spinifex-nfv-sig.repo` for OVS 3.5 and OVN 25.03 from the CentOS NFV SIG.
+That repo carries an `includepkgs=` allowlist, and its key is refused unless its fingerprint matches `NFV_SIG_KEY_FPR`.
+It then installs `kernel-uek-modules` for the running kernel, stopping with a reboot instruction if that version has left the repositories, and loads `openvswitch`, `geneve`, `vhost_net` and `tun`.
+`make install-system` takes the same dnf path, adding the build packages from CodeReady Builder.
+
+| Script | What it does |
+|---|---|
+| `lint-apt-packages.sh <image>` | Resolves the apt list against a Debian or Ubuntu image's real indices with `apt-get install -s`. |
+| `lint-dnf-packages.sh [image]` | Runs `setup.sh`'s Oracle Linux repository setup in an `oraclelinux:10` container, resolves the dnf list, and fails unless QEMU comes from KVM Utilities and OVS and OVN from the NFV SIG. |
+| `check-glibc-ceiling.sh <ceiling> <elf>...` | Fails when an ELF file needs a glibc symbol version newer than the ceiling, naming the symbols. `build/Dockerfile.distro` runs it on `spx` and the nbdkit plugin with 2.34, so the one release tarball keeps loading on every supported OS. |
+
 ## Teardown
 
 Both ship to `/usr/local/share/spinifex/` on every node, so they are available on a host installed with `curl … | bash` and not only from a checkout. They differ in what they take away:
 
 | Script | What it does |
 |---|---|
-| `node-reset.sh` | Clears **state** and leaves the node installed: stops services, destroys guests, wipes `/etc/spinifex`, `/var/lib/spinifex`, `/var/log/spinifex` and the OVN databases, and removes the OVS bridges. The node is then ready for a fresh `spx admin init`. `--keep-data` preserves `/var/lib/spinifex`; `SPX_RESET_CONFIRM=reset` runs it unattended. |
-| `uninstall-spx.sh` | Removes the **software** — the inverse of `setup.sh`. Delegates the state teardown above to `node-reset.sh` rather than reimplementing it, then removes the binary, the nbdkit plugin, the systemd units and slices, the service users, the sudoers grant, the sysctl/logrotate/chrony/udev drop-ins, the `systemd-networkd` units, the host firewall policy and its loaded nft table, and the Spinifex CA from the host trust store. Verifies afterwards and exits non-zero naming anything that survived. |
+| `node-reset.sh` | Clears **state** and leaves the node installed: stops services, destroys guests, wipes `/etc/spinifex`, `/var/lib/spinifex`, `/var/log/spinifex` and the OVN databases, and removes the OVS bridges. On a libreswan host it deletes the `ovs_cert_*` and `ovs_certkey_*` entries `ovs-monitor-ipsec` imported into libreswan's NSS database, leaving other identities and `/etc/ipsec.d/*.conf` alone. It stops OVS and OVN central by the unit names the host's packages use. It leaves the NetworkManager unmanaged-device config, which the next `setup-ovn.sh` run rewrites, so NetworkManager cannot take a NIC back in between. The node is then ready for a fresh `spx admin init`. `--keep-data` preserves `/var/lib/spinifex`; `SPX_RESET_CONFIRM=reset` runs it unattended. |
+| `uninstall-spx.sh` | Removes the **software** — the inverse of `setup.sh`. Delegates the state teardown above to `node-reset.sh` rather than reimplementing it, then removes the binary, the nbdkit plugin, the systemd units and slices, the service users, the sudoers grant, the sysctl/logrotate/chrony/udev drop-ins, the `systemd-networkd` units, the host firewall policy and its loaded nft table, the Spinifex systemd-resolved drop-in, and the Spinifex CA from the host trust store. It unmasks firewalld and leaves it disabled. Where `setup.sh` handed DNS to systemd-resolved, it restores the saved `/etc/resolv.conf` and turns resolved off. Verifies afterwards and exits non-zero naming anything that survived, then lists the host settings `setup.sh` changed that it leaves in place, such as the dnf repositories, `OVS_USER_ID`, masked IPsec units and the SELinux mode. |
 
-`uninstall-spx.sh` keeps `/var/lib/spinifex` and every apt package by default — uninstalling software should not destroy data, and a `curl … | bash` install lands on hosts that were already using openvswitch, libvirt and chrony. `--purge-data` and `--purge-deps` opt into each, and both confirm separately even under `--yes` (`SPX_PURGE_CONFIRM=destroy` for unattended data removal). `--dry-run` prints the whole plan and touches nothing.
+`node-reset.sh` finds guests by process name, `qemu-system-*` or `qemu-kvm`, owned by the daemon's service user, and waits for them to exit before it deletes any state.
+`node-reset_test.sh` checks that match against real processes, and that `reset-dev-env.sh` uses the same pattern.
+
+`uninstall-spx.sh` keeps `/var/lib/spinifex` and every package by default — uninstalling software should not destroy data, and a `curl … | bash` install lands on hosts that were already using openvswitch, libvirt and chrony. `--purge-data` and `--purge-deps` opt into each, and both confirm separately even under `--yes` (`SPX_PURGE_CONFIRM=destroy` for unattended data removal). `--dry-run` prints the whole plan and touches nothing.
+`--purge-deps` removes with apt and is refused on dnf hosts, where its fixed list cannot tell what `setup.sh` installed from what the host already had, and dnf would also remove every package that requires them.
 
 **It refuses on a multi-node cluster, and `--yes` does not override that.** Uninstalling one node of a running cluster strands its guests and leaves the survivors reconciling against a chassis that will never answer. Run `spx admin cluster shutdown` first, then the uninstaller on each node.
 
@@ -36,7 +81,7 @@ Both ship to `/usr/local/share/spinifex/` on every node, so they are available o
 
 | Script | What it does |
 |---|---|
-| `smoke-test.sh` | End-to-end sanity check on a running node: imports an SSH keypair and the Ubuntu AMI, then launches an instance. `--create-vpc` builds the VPC, gateway, subnet and routes first and tears them down after; `--nodes N` launches through a spread placement group so each instance lands on a different physical server. `TEST_GPU=1` adds GPU passthrough checks. |
+| `smoke-test.sh` | End-to-end sanity check on a running node: imports an SSH keypair and the Ubuntu AMI, then launches an instance. `--create-vpc` builds the VPC, gateway, subnet and routes first and tears them down after; `--nodes N` launches through a spread placement group so each instance lands on a different physical server. `TEST_GPU=1` adds GPU passthrough checks. It runs the AWS CLI by absolute path, since sudo's `secure_path` leaves out `/usr/local/bin` on Oracle Linux. |
 | `verify-network.sh` | Asserts a node's datapath matches its declared planes — EIP reachability, which plane carries Geneve, and which interfaces services bind. Catches what `smoke-test.sh` cannot, since none of its traffic leaves the box. |
 | `check-coverage.sh` | Fails the build if total Go coverage falls below the minimum threshold, excluding packages exempt from the requirement. |
 | `diff-coverage.sh` | Coverage check scoped to changed lines only, against an auto-detected base ref (`HEAD~1` on main, `origin/main` on dev, otherwise `origin/dev`). |
@@ -52,6 +97,7 @@ Both ship to `/usr/local/share/spinifex/` on every node, so they are available o
 | `build-system-image.sh` | Builds a minimal system AMI (Alpine or Ubuntu) from a manifest. All customisation happens inside the libguestfs appliance, so the build touches no host block device, no host mount, and needs no sudo. `--import` also registers it as an AMI. |
 | `publish-system-image.sh` | Converts a built raw image to compressed qcow2, writes the `.sha256` sidecar the catalog verifier expects, and uploads both to the R2 bucket behind `iso.mulgadc.com/system-ami/`. |
 | `build-microvm-image.sh` | Builds the minimal Alpine kernel + initramfs used by the QEMU microvm machine type and the lb-agent, carrying haproxy (ALB) and nginx-stream (NLB) data planes. |
+| `build-qboot.sh` | Builds qboot, the firmware the daemon passes to every microvm with `-bios`, from a pinned upstream commit in a digest-pinned Debian container. Fails unless the output matches the recorded sha256, which is also Debian 13's `qboot.rom`. Writes `qboot.rom` and its GPL-2.0 licence beside the microvm kernel, and the matching source tarball, which releases publish, under `build/qboot/`. `make build-microvm-image` runs it. |
 | `mkosi-build.sh` | Runs an mkosi system-image build inside the pinned builder container, so a runner needs only Docker and the toolchain cannot drift. `--image` names an output and expands to the right profile list. |
 | `mkosi-publish.sh` | Publishes an mkosi-built image to R2 from inside the same pinned container — qcow2 conversion, `.sha256` sidecar, and upload. |
 | `mkosi-common.sh` | Shared plumbing sourced by both mkosi scripts. Builds the toolchain image once, with the builder account matched to the invoking user so bind-mounted output is not written back as root. |

scripts/appliance/firstboot.sh

@@ -0,0 +1,146 @@
+#!/bin/bash
+# Spinifex appliance first boot, for Oracle Linux units cloned from the gold
+# image that build-oracle-appliance.sh seals. The OL counterpart of the ISO's
+# spinifex-firstboot.sh: it also does the per-unit work the ISO installer does
+# before first boot (hostname, br-wan on the NIC that is present), then sets
+# up a single node. On success it writes $DONE_MARKER and the unit's
+# ConditionPathExists=! stops it running again. A failed run leaves no marker,
+# so the next boot retries from the top; every step below is safe to repeat.
+set -euo pipefail
+
+DONE_MARKER=/var/lib/spinifex/.appliance-firstboot-done
+SHARE=/usr/local/share/spinifex
+
+# shellcheck source=lib.sh
+. "$SHARE/appliance/lib.sh"
+# shellcheck source=../lib/distro.sh
+. "$SHARE/lib/distro.sh"
+
+log() { echo "[appliance-firstboot] $*"; }
+
+if [ -f "$DONE_MARKER" ]; then
+    log "already complete, skipping"
+    exit 0
+fi
+
+# ── Primary NIC ─────────────────────────────────────────────────────────────
+# Links start down when NetworkManager has no profile for them, so bring every
+# physical NIC up to read its carrier, and give a cable up to 60s to come up.
+nics=$(appliance_physical_nics)
+if [ -z "$nics" ]; then
+    log "no physical Ethernet device found"
+    exit 1
+fi
+for nic in $nics; do
+    ip link set "$nic" up || true
+done
+for _i in $(seq 1 60); do
+    for nic in $nics; do
+        [ "$(cat "/sys/class/net/$nic/carrier" 2>/dev/null)" = 1 ] && break 2
+    done
+    sleep 1
+done
+NIC=$(appliance_pick_nic)
+MAC=$(cat "/sys/class/net/$NIC/address")
+log "primary NIC $NIC ($MAC), carrier $(cat "/sys/class/net/$NIC/carrier" 2>/dev/null || echo 0)"
+
+HOSTNAME_NEW=$(appliance_hostname "$MAC")
+hostnamectl set-hostname "$HOSTNAME_NEW"
+log "hostname $HOSTNAME_NEW"
+
+# ── br-wan on the primary NIC, DHCP ─────────────────────────────────────────
+# The bridge takes the NIC's MAC so a DHCP reservation made for the unit's
+# port keeps working. bridge.stp no: with STP on, the port sits in its
+# forwarding delay long enough for DHCP on the bridge to time out.
+for con in $(nmcli -g UUID,DEVICE connection show | awk -F: -v d="$NIC" '$2 == d {print $1}'); do
+    nmcli connection delete "$con" || true
+done
+nmcli connection delete br-wan br-wan-port >/dev/null 2>&1 || true
+nmcli connection add type bridge ifname br-wan con-name br-wan \
+    bridge.stp no bridge.mac-address "$MAC" \
+    ipv4.method auto ipv6.method disabled connection.autoconnect yes
+nmcli connection add type bridge-slave ifname "$NIC" master br-wan con-name br-wan-port \
+    connection.autoconnect yes
+nmcli connection up br-wan || true
+
+log "waiting for a DHCP address on br-wan"
+WAN_IP=""
+for _i in $(seq 1 120); do
+    WAN_IP=$(ip -4 -o addr show br-wan scope global 2>/dev/null | awk '{print $4}' | cut -d/ -f1 | head -1)
+    if [ -n "$WAN_IP" ] && ip -4 route show default dev br-wan | grep -q .; then
+        break
+    fi
+    WAN_IP=""
+    sleep 1
+done
+if [ -z "$WAN_IP" ]; then
+    log "br-wan got no DHCP address and default route in 120s; check the cable on $NIC and the site's DHCP"
+    exit 1
+fi
+log "br-wan $WAN_IP"
+
+# ── Clock ───────────────────────────────────────────────────────────────────
+# Credentials and TLS certificates are minted below, and a certificate issued
+# against a wrong RTC carries a NotBefore that outlives the correction.
+# Bounded and non-fatal, as on the ISO.
+chronyc -a makestep >/dev/null 2>&1 || true
+if chronyc waitsync 12 0.1 0 5 >/dev/null 2>&1; then
+    log "clock synced: $(date -uIs)"
+else
+    log "WARNING: clock not synced, continuing with $(date -uIs)"
+fi
+
+# ── OVS and OVN ─────────────────────────────────────────────────────────────
+# Start them before setup-ovn.sh so their databases exist when it runs: on
+# physical hardware first-boot DB initialisation outlasts its own timeout.
+OVS_UNIT=$(distro_ovs_unit)
+OVN_LAYOUT=$(distro_ovn_central_layout)
+case $OVN_LAYOUT in
+    split) OVN_CENTRAL_UNIT=ovn-central.service ;;
+    lone-northd) OVN_CENTRAL_UNIT=ovn-northd.service ;;
+esac
+systemctl start "$OVS_UNIT" "$OVN_CENTRAL_UNIT"
+for _i in $(seq 1 120); do
+    if ovn-nbctl --timeout=2 get-connection >/dev/null 2>&1; then
+        log "OVN NB DB ready (${_i}s)"
+        break
+    fi
+    sleep 1
+done
+
+"$SHARE/setup-ovn.sh" --management
+
+# ── Single-node cluster ─────────────────────────────────────────────────────
+# Same arguments as the ISO's single-NIC DHCP path. spx writes credentials to
+# disk and never prints them.
+spx admin init --node "$HOSTNAME_NEW" --nodes 1 2>&1
+
+# spx ran as root, so the files it wrote are root:root. setup.sh's fixown
+# stage knows each file's owner and mode.
+SETUP_STAGES=fixown "$SHARE/setup.sh"
+
+# ── spinifex login ──────────────────────────────────────────────────────────
+if [ ! -f /home/spinifex/.ssh/id_ed25519 ]; then
+    install -d -m 700 -o spinifex -g spinifex /home/spinifex/.ssh
+    ssh-keygen -q -t ed25519 -f /home/spinifex/.ssh/id_ed25519 -N "" -C "spinifex@$HOSTNAME_NEW"
+    chown spinifex:spinifex /home/spinifex/.ssh/id_ed25519 /home/spinifex/.ssh/id_ed25519.pub
+fi
+# init wrote the AWS profile under HOME=/root; the operator logs in as spinifex.
+if [ -f /root/.aws/credentials ]; then
+    install -d -m 700 -o spinifex -g spinifex /home/spinifex/.aws
+    install -m 600 -o spinifex -g spinifex /root/.aws/credentials /home/spinifex/.aws/credentials
+    [ -f /root/.aws/config ] && install -m 600 -o spinifex -g spinifex /root/.aws/config /home/spinifex/.aws/config
+fi
+
+# ── Services ────────────────────────────────────────────────────────────────
+# Enabled here, not in the image, so a clone runs nothing of Spinifex before
+# init has written its config.
+systemctl enable spinifex.target spinifex-banner.service spinifex-firewall.service
+systemctl enable --now spinifex-nats-watchdog.timer
+
+mkdir -p "$(dirname "$DONE_MARKER")"
+touch "$DONE_MARKER"
+
+systemctl start spinifex.target
+/usr/local/bin/spx admin banner --boot-check
+log "complete: https://$WAN_IP:3000, login spinifex"

scripts/appliance/getty-wait.conf

@@ -0,0 +1,4 @@
+# Holds the console login prompt until first boot and the banner have written
+# /etc/issue, so the unit's IP shows before anyone logs in.
+[Unit]
+After=spinifex-appliance-firstboot.service spinifex-banner.service

scripts/appliance/lib.sh

@@ -0,0 +1,37 @@
+#!/bin/bash
+# Helpers shared by the appliance first-boot script and its test. Sourced, not run.
+
+# appliance_physical_nics — physical Ethernet devices, one per line, in PCI
+# order. Skips anything without a backing device (bridges, veths, OVS ports,
+# loopback), non-Ethernet link types and wireless. SYSFS overrides /sys.
+appliance_physical_nics() {
+    local sys="${SYSFS:-/sys}/class/net" dev
+    for dev in "$sys"/*; do
+        [ -e "$dev/device" ] || continue
+        [ "$(cat "$dev/type" 2>/dev/null)" = 1 ] || continue
+        [ -d "$dev/wireless" ] && continue
+        printf '%s %s\n' "$(readlink -f "$dev/device")" "$(basename "$dev")"
+    done | sort | awk '{print $2}'
+}
+
+# appliance_pick_nic — the first physical NIC with carrier, else the first
+# physical NIC. Prints nothing when the host has none.
+appliance_pick_nic() {
+    local sys="${SYSFS:-/sys}/class/net" nic first=""
+    for nic in $(appliance_physical_nics); do
+        [ -n "$first" ] || first=$nic
+        if [ "$(cat "$sys/$nic/carrier" 2>/dev/null)" = 1 ]; then
+            echo "$nic"
+            return 0
+        fi
+    done
+    [ -n "$first" ] && echo "$first"
+    return 0
+}
+
+# appliance_hostname <mac> — spinifex-<last 6 hex digits of the MAC>.
+appliance_hostname() {
+    local hex
+    hex=$(echo "$1" | tr -d ':' | tr '[:upper:]' '[:lower:]')
+    echo "spinifex-${hex: -6}"
+}

scripts/appliance/lib_test.sh

@@ -0,0 +1,63 @@
+#!/bin/sh
+# lib_test.sh — tests for the appliance first-boot NIC and hostname helpers,
+# run against a fixture sysfs tree (SYSFS) so the host's own NICs never count.
+set -eu
+
+DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
+TMP=$(mktemp -d)
+trap 'rm -rf "$TMP"' EXIT
+
+fails=0
+pass() { echo "  ok   - $1"; }
+fail() { echo "  FAIL - $1"; fails=$((fails + 1)); }
+
+echo "-- appliance/lib.sh"
+
+# nic <name> <pci-slot> <carrier> [type] — a physical NIC behind a PCI device.
+nic() {
+    mkdir -p "$SYS/devices/pci0000:00/$2" "$SYS/class/net/$1"
+    ln -s "../../../devices/pci0000:00/$2" "$SYS/class/net/$1/device"
+    echo "${4:-1}" > "$SYS/class/net/$1/type"
+    echo "$3" > "$SYS/class/net/$1/carrier"
+}
+# virt <name> — a link with no backing device (bridge, veth, OVS port).
+virt() {
+    mkdir -p "$SYS/class/net/$1"
+    echo 1 > "$SYS/class/net/$1/type"
+    echo 1 > "$SYS/class/net/$1/carrier"
+}
+fixture() { SYS="$TMP/$1"; mkdir -p "$SYS/class/net"; }
+run() { SYSFS="$SYS" bash -c ". '$DIR/lib.sh'; $1"; }
+
+check() {
+    if [ "$2" = "$3" ]; then pass "$1"; else fail "$1: want '$2', got '$3'"; fi
+}
+
+fixture order
+nic enp3s0 0000:00:03.0 0
+nic enp1s0 0000:00:01.0 0
+virt br-wan
+virt lo
+check "physical NICs in PCI order, virtual links skipped" "enp1s0 enp3s0" "$(run appliance_physical_nics | tr '\n' ' ' | sed 's/ $//')"
+check "no carrier anywhere picks the first by PCI order" enp1s0 "$(run appliance_pick_nic)"
+
+fixture carrier
+nic eno1 0000:00:01.0 0
+nic eno2 0000:00:02.0 1
+nic eno3 0000:00:03.0 1
+check "the first NIC with carrier wins over PCI order" eno2 "$(run appliance_pick_nic)"
+
+fixture wireless
+nic wlp1s0 0000:00:01.0 1
+mkdir -p "$SYS/class/net/wlp1s0/wireless"
+nic ib0 0000:00:02.0 1 32
+nic enp4s0 0000:00:04.0 0
+check "wireless and non-Ethernet links are skipped" enp4s0 "$(run appliance_pick_nic)"
+
+fixture none
+virt br-wan
+check "no physical NIC prints nothing" "" "$(run appliance_pick_nic)"
+
+check "hostname from the MAC's last 6 hex digits" spinifex-c0ffee "$(run 'appliance_hostname 52:54:00:C0:FF:EE')"
+
+[ "$fails" -eq 0 ] || { echo "  $fails failed"; exit 1; }

scripts/appliance/prepare.sh

@@ -0,0 +1,178 @@
+#!/bin/bash
+# Runs as root on the Oracle Linux build system, one stage per call, driven by
+# build-oracle-appliance.sh from the payload directory it copied over.
+#   prepare.sh check      refuse anything but OL 10.2+ on UEK R8, x86_64
+#   prepare.sh update     dnf update, SELinux permissive; prints PREPARE_REBOOT=1 if
+#                         the newest kernel is not the running one
+#   prepare.sh install    install Spinifex, uninitialised and not enabled
+#   prepare.sh appliance  spinifex login, first-boot service, console placeholder
+#   prepare.sh seal       wipe this system's identity, then power off
+# INSTALL_SPINIFEX_CHANNEL, INSTALL_SPINIFEX_VERSION and INSTALL_SPINIFEX_TARBALL
+# pass through to the installer. BUILD_USER is the SSH user to lock at seal.
+set -euo pipefail
+
+# Shared by every unit built from this image; first login forces a change.
+DEFAULT_PASSWORD=spinifex
+
+HERE=$(cd "$(dirname "$0")" && pwd)
+SHARE=/usr/local/share/spinifex
+
+log() { echo "[prepare] $*"; }
+die() { echo "[prepare] ERROR: $*" >&2; exit 1; }
+
+[ "$(id -u)" = 0 ] || die "run as root"
+
+stage_check() {
+    # shellcheck source=/dev/null
+    . /etc/os-release
+    [ "${ID:-}" = ol ] || die "not Oracle Linux (ID=${ID:-unset})"
+    local major=${VERSION_ID%%.*} minor=${VERSION_ID#*.}
+    [ "$minor" = "$VERSION_ID" ] && minor=0
+    if [ "$major" -lt 10 ] || { [ "$major" -eq 10 ] && [ "$minor" -lt 2 ]; }; then
+        # A 10.0 or 10.1 system reaches 10.2 through the update stage, which
+        # calls this again with "strict" once it has run.
+        if [ "$major" -ne 10 ] || [ "${1:-}" = strict ]; then
+            die "Oracle Linux $VERSION_ID; 10.2 or later is required"
+        fi
+        log "Oracle Linux $VERSION_ID; the update stage brings it to the current minor"
+    fi
+    [ "$(uname -m)" = x86_64 ] || die "$(uname -m); only x86_64 is supported"
+    case $(uname -r) in
+        6.12.0-*uek*) ;;
+        *) die "running kernel $(uname -r) is not UEK R8; install kernel-uek and boot it first" ;;
+    esac
+    command -v nmcli >/dev/null || die "NetworkManager (nmcli) is required"
+    log "OK: $PRETTY_NAME, kernel $(uname -r)"
+}
+
+stage_update() {
+    setenforce 0 2>/dev/null || true
+    sed -i 's/^SELINUX=.*/SELINUX=permissive/' /etc/selinux/config
+    [ "$(getenforce)" != Enforcing ] || die "SELinux is still enforcing"
+    dnf update -y
+    stage_check strict >/dev/null
+    local newest
+    newest=$(grubby --default-kernel | sed 's|^/boot/vmlinuz-||')
+    case $newest in
+        6.12.0-*uek*) ;;
+        *) die "the default boot kernel is $newest, not UEK R8; set a kernel-uek default with grubby" ;;
+    esac
+    if [ "$newest" != "$(uname -r)" ]; then
+        echo PREPARE_REBOOT=1
+    fi
+}
+
+stage_install() {
+    [ ! -e /etc/spinifex/spinifex.toml ] || die "/etc/spinifex/spinifex.toml exists; this system has been initialised, start from a clean OL install"
+    local args=()
+    [ -n "${INSTALL_SPINIFEX_CHANNEL:-}" ] && args+=(--channel "$INSTALL_SPINIFEX_CHANNEL")
+    [ -n "${INSTALL_SPINIFEX_VERSION:-}" ] && args+=(--version "$INSTALL_SPINIFEX_VERSION")
+    if [ -n "${INSTALL_SPINIFEX_TARBALL:-}" ]; then
+        # A local build: the payload carries the setup.sh of the checkout the
+        # tarball was built from, and setup.sh reads the tarball from the env.
+        [ -f "$HERE/setup.sh" ] || die "no setup.sh in the payload for INSTALL_SPINIFEX_TARBALL"
+        INSTALL_SPINIFEX_SKIP_NEWGRP=1 bash "$HERE/setup.sh" "${args[@]}"
+    else
+        curl -fsSL -o /tmp/spinifex-setup.sh https://install.mulgadc.com
+        INSTALL_SPINIFEX_SKIP_NEWGRP=1 bash /tmp/spinifex-setup.sh "${args[@]}"
+        rm -f /tmp/spinifex-setup.sh
+    fi
+    # The installer enables these on a live host. The ISO image ships them off
+    # and first boot enables them after init; a clone that started them first
+    # would run every service with no config.
+    systemctl disable --now spinifex.target spinifex-nats-watchdog.timer
+    log "Spinifex $(spx version 2>/dev/null | head -1) installed, not initialised"
+}
+
+stage_appliance() {
+    # As on the ISO: the existing spinifex system group as primary, so the
+    # login can reach the sockets setup.sh group-owns, and passwordless sudo.
+    if ! id spinifex >/dev/null 2>&1; then
+        useradd --create-home --shell /bin/bash --gid spinifex --groups wheel spinifex
+    fi
+    echo "spinifex:$DEFAULT_PASSWORD" | chpasswd
+    chage -d 0 spinifex
+    echo 'spinifex ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/spinifex-admin
+    chmod 0440 /etc/sudoers.d/spinifex-admin
+    visudo -cf /etc/sudoers.d/spinifex-admin >/dev/null
+
+    # sshd takes the first value it reads, so this sorts ahead of the
+    # 50-cloud-init.conf that templates ship with passwords off.
+    printf 'PasswordAuthentication yes\n' > /etc/ssh/sshd_config.d/10-spinifex-appliance.conf
+
+    # No DHCP on every port: first boot puts the primary NIC on br-wan and
+    # leaves the rest alone, so no second port competes for the default route.
+    printf '[main]\nno-auto-default=*\n' > /etc/NetworkManager/conf.d/90-spinifex-appliance.conf
+
+    install -D -m 0644 "$HERE/lib.sh" "$SHARE/appliance/lib.sh"
+    install -D -m 0755 "$HERE/firstboot.sh" "$SHARE/appliance/firstboot.sh"
+    install -m 0644 "$HERE/spinifex-appliance-firstboot.service" "$HERE/spinifex-banner.service" /etc/systemd/system/
+    local getty
+    for getty in getty@tty1 serial-getty@ttyS0; do
+        install -D -m 0644 "$HERE/getty-wait.conf" "/etc/systemd/system/$getty.service.d/spinifex-wait.conf"
+    done
+    systemctl daemon-reload
+    systemctl enable spinifex-appliance-firstboot.service
+
+    # Cloud-init would re-run on each unit with no datasource and rewrite the
+    # network and users first boot owns.
+    if [ -d /etc/cloud ]; then
+        touch /etc/cloud/cloud-init.disabled
+    fi
+    log "spinifex login (default password '$DEFAULT_PASSWORD', change forced) and first-boot service installed"
+}
+
+stage_seal() {
+    dnf clean all
+
+    systemctl stop "$(. "$SHARE/lib/distro.sh" && distro_ovs_unit)" ovn-northd.service ovn-controller.service 2>/dev/null || true
+    # OVS writes the chassis UUID on first start; a shared one makes every unit
+    # the same chassis in OVN.
+    rm -f /etc/openvswitch/system-id.conf /etc/openvswitch/conf.db /etc/openvswitch/.conf.db.~lock~
+    rm -f /var/lib/ovn/*.db /etc/ovn/*.db
+    rm -f /etc/spinifex/node.conf
+
+    # Profiles for the build system's NICs, its DHCP leases and NM's
+    # per-machine secret. First boot writes br-wan for the unit's own NIC.
+    rm -f /etc/NetworkManager/system-connections/*
+    rm -f /var/lib/NetworkManager/*.lease /var/lib/NetworkManager/secret_key \
+        /var/lib/NetworkManager/timestamps /var/lib/NetworkManager/seen-bssids
+    if command -v cloud-init >/dev/null; then
+        cloud-init clean --logs || true
+    fi
+
+    # sshd-keygen regenerates host keys before sshd on the next boot; systemd
+    # generates a machine-id when the file is empty.
+    rm -f /etc/ssh/ssh_host_*
+    truncate -s 0 /etc/machine-id
+    rm -f /var/lib/dbus/machine-id /var/lib/systemd/random-seed
+    echo localhost > /etc/hostname
+
+    # The build user's access goes; the spinifex login is the way in.
+    if [ -n "${BUILD_USER:-}" ] && [ "$BUILD_USER" != spinifex ] && [ "$BUILD_USER" != root ]; then
+        local home
+        home=$(getent passwd "$BUILD_USER" | cut -d: -f6)
+        rm -f "$home/.ssh/authorized_keys" "$home/.bash_history"
+        passwd -l "$BUILD_USER" >/dev/null
+        rm -f "/etc/sudoers.d/90-cloud-init-users"
+    fi
+    rm -f /root/.ssh/authorized_keys /root/.bash_history /home/spinifex/.bash_history
+
+    printf '\nSpinifex first boot is in progress. This screen updates when it finishes.\n\n' > /etc/issue
+    sed -i '/^# --- Spinifex ---$/,$d' /etc/motd 2>/dev/null || true
+
+    find /var/log -type f -exec truncate -s 0 {} + 2>/dev/null || true
+    rm -rf /var/log/journal/* /tmp/* /var/tmp/*
+    sync
+    log "sealed; powering off"
+    systemctl poweroff --no-block
+}
+
+case "${1:-}" in
+    check) stage_check ;;
+    update) stage_update ;;
+    install) stage_install ;;
+    appliance) stage_appliance ;;
+    seal) stage_seal ;;
+    *) die "usage: $0 check|update|install|appliance|seal" ;;
+esac

scripts/appliance/spinifex-appliance-firstboot.service

@@ -0,0 +1,22 @@
+[Unit]
+Description=Spinifex appliance first-boot provisioning
+After=NetworkManager.service network-online.target chronyd.service
+Wants=NetworkManager.service network-online.target
+ConditionPathExists=!/var/lib/spinifex/.appliance-firstboot-done
+
+[Service]
+Type=oneshot
+Environment=HOME=/root
+ExecStart=/usr/local/share/spinifex/appliance/firstboot.sh
+RemainAfterExit=yes
+# The banner is never enabled when first boot fails, so without this the
+# console would keep the "first boot in progress" placeholder.
+ExecStopPost=/bin/sh -c 'if [ "$EXIT_STATUS" != "0" ]; then printf "\n*** Spinifex first boot FAILED ***\nLog in as spinifex, then run:\n  sudo journalctl -u spinifex-appliance-firstboot.service\nCheck the network cable and DHCP, then reboot to retry.\n\n" > /etc/issue; fi'
+# setup-ovn.sh and spx admin init must not wedge multi-user.target and keep
+# getty from ever showing a login prompt.
+TimeoutStartSec=600s
+StandardOutput=journal+console
+StandardError=journal+console
+
+[Install]
+WantedBy=multi-user.target

scripts/appliance/spinifex-banner.service

@@ -0,0 +1,15 @@
+[Unit]
+Description=Spinifex console banner and boot health check
+After=network-online.target
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/spx admin banner --boot-check
+RemainAfterExit=yes
+# Cap it so a stuck boot-check cannot hold getty behind the console drop-in.
+TimeoutStartSec=30s
+StandardOutput=journal
+StandardError=journal
+
+[Install]
+WantedBy=multi-user.target

scripts/build-oracle-appliance.sh

@@ -0,0 +1,104 @@
+#!/bin/bash
+# build-oracle-appliance.sh — turn an Oracle Linux 10.2 system reachable over
+# SSH with sudo into a sealed Spinifex gold image: installed, not initialised,
+# with a first-boot service that sets each cloned unit up as a single node.
+# The system powers off at the end; capture its disk as the image.
+#
+# Usage: build-oracle-appliance.sh [options] <user>@<host>
+#   --channel latest|dev   install channel (default: latest)
+#   --version <tag>        install a specific release
+#   --tarball <path>       install a local release tarball instead
+#
+# Ports, keys and jump hosts come from ~/.ssh/config. <user> needs
+# passwordless sudo.
+set -euo pipefail
+
+HERE=$(cd "$(dirname "$0")" && pwd)
+PAYLOAD_SRC="$HERE/appliance"
+REMOTE_DIR=/tmp/spinifex-appliance
+
+CHANNEL="" VERSION="" TARBALL="" TARGET=""
+
+usage() { sed -n '2,13p' "$0" | sed 's/^# \{0,1\}//'; exit "${1:-0}"; }
+log() { echo "==> $*"; }
+die() { echo "ERROR: $*" >&2; exit 1; }
+
+while [ $# -gt 0 ]; do
+    case "$1" in
+        --channel) CHANNEL="${2:-}"; shift ;;
+        --version) VERSION="${2:-}"; shift ;;
+        --tarball) TARBALL="${2:-}"; shift ;;
+        -h | --help) usage ;;
+        -*) die "unknown option: $1" ;;
+        *) [ -z "$TARGET" ] || die "one target only"; TARGET=$1 ;;
+    esac
+    shift
+done
+[ -n "$TARGET" ] || usage 1
+case $TARGET in *@*) ;; *) die "target must be <user>@<host>" ;; esac
+BUILD_USER=${TARGET%@*}
+[ -z "$TARBALL" ] || [ -f "$TARBALL" ] || die "no such tarball: $TARBALL"
+
+# One multiplexed connection; host key checking is relaxed because the build
+# system is usually a fresh install whose key nobody has recorded yet.
+CTL_DIR=$(mktemp -d)
+trap 'ssh -o ControlPath="$CTL_DIR/c" -O exit "$TARGET" >/dev/null 2>&1 || true; rm -rf "$CTL_DIR"' EXIT
+SSH_OPTS=(-o ControlMaster=auto -o ControlPath="$CTL_DIR/c" -o ControlPersist=60
+    -o StrictHostKeyChecking=accept-new -o ServerAliveInterval=15 -o ConnectTimeout=10)
+
+rsh() { ssh "${SSH_OPTS[@]}" "$TARGET" "$@"; }
+
+# prepare <stage> — run one prepare.sh stage as root on the build system.
+prepare() {
+    rsh "sudo env INSTALL_SPINIFEX_CHANNEL='$CHANNEL' INSTALL_SPINIFEX_VERSION='$VERSION' \
+        INSTALL_SPINIFEX_TARBALL='${TARBALL:+$REMOTE_DIR/$(basename "$TARBALL")}' \
+        BUILD_USER='$BUILD_USER' bash $REMOTE_DIR/prepare.sh $1"
+}
+
+push_payload() {
+    rsh "rm -rf $REMOTE_DIR && mkdir -p $REMOTE_DIR"
+    tar -C "$PAYLOAD_SRC" -cf - . | rsh "tar -C $REMOTE_DIR -xf -"
+    if [ -n "$TARBALL" ]; then
+        # setup.sh from this checkout installs the tarball built from it.
+        scp "${SSH_OPTS[@]}" -q "$HERE/setup.sh" "$TARBALL" "$TARGET:$REMOTE_DIR/"
+    fi
+}
+
+wait_for_ssh() {
+    local _i
+    ssh "${SSH_OPTS[@]}" -O exit "$TARGET" >/dev/null 2>&1 || true
+    sleep 10
+    for _i in $(seq 1 60); do
+        if rsh true 2>/dev/null; then return 0; fi
+        sleep 5
+    done
+    die "$TARGET did not come back over SSH after the reboot"
+}
+
+rsh "sudo -n true" || die "$BUILD_USER needs passwordless sudo on the build system"
+
+log "Copying the payload to $TARGET"
+push_payload
+prepare check
+
+log "Updating Oracle Linux"
+prepare update | tee "$CTL_DIR/update.log"
+if grep -qx 'PREPARE_REBOOT=1' "$CTL_DIR/update.log"; then
+    log "Rebooting into the updated kernel"
+    rsh "sudo systemctl reboot" || true
+    wait_for_ssh
+    # /tmp is cleared on boot.
+    push_payload
+    prepare check
+fi
+
+log "Installing Spinifex (not initialising it)"
+prepare install
+
+log "Installing the spinifex login and the first-boot service"
+prepare appliance
+
+log "Sealing; the system powers off when done"
+prepare seal || true
+
+log "Done. Capture $TARGET's disk once it has powered off; each unit set up from it initialises itself on first boot."

scripts/build-qboot.sh

@@ -0,0 +1,75 @@
+#!/bin/bash
+# build-qboot.sh — build qboot, the firmware the QEMU microvm machine boots
+# system instances (ALB/NLB) with, from a pinned upstream commit.
+#
+# Outputs:
+#   $REPO_ROOT/build/microvm/qboot.rom          firmware, shipped beside the microvm kernel
+#   $REPO_ROOT/build/microvm/qboot.LICENSE      qboot's GPL-2.0 licence text
+#   $REPO_ROOT/build/qboot/qboot-<commit>.tar.gz the matching source, published with releases
+#
+# The build runs in a digest-pinned Debian container, so its output is
+# reproducible and is checked against a recorded hash. That hash is also the
+# sha256 of Debian 13's /usr/share/qemu/qboot.rom, which is built from the same commit.
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+OUT_DIR="$REPO_ROOT/build/microvm"
+SRC_DIR="$REPO_ROOT/build/qboot"
+
+QBOOT_REPO="${QBOOT_REPO:-https://gitlab.com/qemu-project/qboot.git}"
+QBOOT_COMMIT="8ca302e86d685fa05b16e2b208888243da319941"
+QBOOT_SHA256="5c9c4b12c4438e3cf11d087263afe6519c8c05dd5896727b8b169b8f5dbf9877"
+BUILD_IMAGE="debian:trixie@sha256:4c25275446ca371766b8b093c696782eabce06263fccacb8d5d813b3dc44b308"
+
+log() { echo "[build-qboot] $*"; }
+fail() { echo "ERROR: $*" >&2; exit 1; }
+
+CONTAINER_TOOL=""
+for tool in docker podman; do
+    if command -v "$tool" >/dev/null 2>&1 && "$tool" info >/dev/null 2>&1; then
+        CONTAINER_TOOL=$tool
+        break
+    fi
+done
+[ -n "$CONTAINER_TOOL" ] || fail "qboot builds in a container; install docker or podman"
+command -v git >/dev/null 2>&1 || fail "required tool not found: git"
+
+WORK=$(mktemp -d)
+trap 'rm -rf "$WORK"' EXIT
+
+log "fetching qboot $QBOOT_COMMIT from $QBOOT_REPO"
+git init -q "$WORK/src"
+git -C "$WORK/src" fetch -q --depth 1 "$QBOOT_REPO" "$QBOOT_COMMIT"
+git -C "$WORK/src" checkout -q FETCH_HEAD
+got_commit=$(git -C "$WORK/src" rev-parse HEAD)
+[ "$got_commit" = "$QBOOT_COMMIT" ] || fail "fetched qboot $got_commit, not the pinned $QBOOT_COMMIT"
+
+mkdir -p "$SRC_DIR"
+src_tarball="$SRC_DIR/qboot-${QBOOT_COMMIT}.tar.gz"
+git -C "$WORK/src" archive --format=tar.gz --prefix="qboot-${QBOOT_COMMIT}/" -o "$src_tarball" "$QBOOT_COMMIT"
+
+log "building in $BUILD_IMAGE with $CONTAINER_TOOL"
+# The firmware comes back on stdout, so nothing the container writes is left
+# root-owned on the host.
+"$CONTAINER_TOOL" run --rm --platform linux/amd64 -v "$WORK/src:/src:ro" \
+    -e DEBIAN_FRONTEND=noninteractive "$BUILD_IMAGE" sh -euc '
+        {
+            apt-get update -qq
+            apt-get install -y -qq --no-install-recommends gcc libc6-dev meson ninja-build binutils
+            meson setup /tmp/build /src
+            ninja -C /tmp/build
+        } >/dev/null
+        cat /tmp/build/bios.bin
+    ' > "$WORK/qboot.rom"
+
+got_sha=$(sha256sum "$WORK/qboot.rom" | cut -d' ' -f1)
+if [ "$got_sha" != "$QBOOT_SHA256" ]; then
+    fail "qboot.rom has sha256 $got_sha, not the recorded $QBOOT_SHA256.
+       The pinned commit or build image changed; rebuild, boot an ALB on it, then update QBOOT_SHA256."
+fi
+
+mkdir -p "$OUT_DIR"
+install -m 0644 "$WORK/qboot.rom" "$OUT_DIR/qboot.rom"
+install -m 0644 "$WORK/src/LICENSE" "$OUT_DIR/qboot.LICENSE"
+log "qboot.rom sha256 $got_sha"
+log "source: $src_tarball"

scripts/check-glibc-ceiling.sh

@@ -0,0 +1,46 @@
+#!/bin/bash
+# check-glibc-ceiling.sh — fail when an ELF file needs a glibc symbol version
+# newer than a ceiling, so the one release tarball keeps loading on the oldest
+# glibc of every OS it ships to.
+#
+# Usage: scripts/check-glibc-ceiling.sh <ceiling, e.g. 2.34> <elf file>...
+set -euo pipefail
+
+if [ $# -lt 2 ]; then
+    echo "usage: $0 <ceiling> <elf file>..." >&2
+    exit 2
+fi
+ceiling=$1
+shift
+
+failed=0
+for f in "$@"; do
+    if ! syms=$(objdump -T "$f" 2>&1); then
+        case "$syms" in
+            *"not a dynamic object"*)
+                echo "$f: statically linked, needs no glibc"
+                continue
+                ;;
+        esac
+        echo "ERROR: cannot read the dynamic symbols of $f: $syms" >&2
+        failed=1
+        continue
+    fi
+    need=$(grep -o 'GLIBC_[0-9][0-9.]*' <<<"$syms" | sed 's/^GLIBC_//' | sort -uV | tail -1 || true)
+    if [ -z "$need" ]; then
+        echo "$f: needs no versioned glibc symbol"
+        continue
+    fi
+    if [ "$(printf '%s\n%s\n' "$need" "$ceiling" | sort -V | tail -1)" = "$ceiling" ]; then
+        echo "$f: needs at most GLIBC_$need (ceiling GLIBC_$ceiling)"
+        continue
+    fi
+    echo "ERROR: $f needs GLIBC_$need, newer than the GLIBC_$ceiling ceiling. Symbols above it:" >&2
+    while read -r ver sym; do
+        if [ "$(printf '%s\n%s\n' "$ver" "$ceiling" | sort -V | tail -1)" != "$ceiling" ]; then
+            echo "  $sym@GLIBC_$ver" >&2
+        fi
+    done < <(awk '{ for (i = 1; i <= NF; i++) if ($i ~ /^\(?GLIBC_[0-9]/) { v = $i; gsub(/[()]|GLIBC_/, "", v); print v, $NF } }' <<<"$syms" | sort -u)
+    failed=1
+done
+exit "$failed"

scripts/dev-install.sh

@@ -31,6 +31,9 @@ cp "$MULGA_ROOT/viperblock/lib/nbdkit-viperblock-plugin.so" "$STAGING/"
 cp "$PROJECT_ROOT/scripts/setup-ovn.sh" "$STAGING/"
 cp "$PROJECT_ROOT/scripts/node-reset.sh" "$STAGING/"
 cp "$PROJECT_ROOT/scripts/uninstall-spx.sh" "$STAGING/"
+cp "$PROJECT_ROOT/scripts/install-node.sh" "$PROJECT_ROOT/scripts/smoke-test.sh" "$STAGING/"
+mkdir -p "$STAGING/lib"
+cp "$PROJECT_ROOT/scripts/lib/distro.sh" "$STAGING/lib/"
 mkdir -p "$STAGING/systemd"
 cp "$PROJECT_ROOT/build/systemd/"* "$STAGING/systemd/"
 mkdir -p "$STAGING/scripts"

scripts/images/eks-node/eks-token-webhook.initd

@@ -25,7 +25,7 @@ error_log="/dev/console"
 if [ -f /etc/spinifex-eks/first-boot.env ]; then
     # shellcheck disable=SC1091
     . /etc/spinifex-eks/first-boot.env
-    export EKS_GATEWAY_URL EKS_GATEWAY_CA EKS_ACCESS_KEY EKS_SECRET_KEY EKS_REGION EKS_ACCOUNT_ID EKS_CLUSTER_NAME
+    export EKS_GATEWAY_URL EKS_GATEWAY_CA EKS_ACCESS_KEY EKS_SECRET_KEY EKS_REGION EKS_ACCOUNT_ID EKS_CLUSTER_NAME EKS_WEBHOOK_READY_ADDR
 fi
 
 depend() {

scripts/install-node.sh

@@ -7,7 +7,7 @@
 # `spx admin join` on the rest, restarts services and verifies the result.
 #
 # It is the scripted form of docs/install/install-multi-node. It installs no
-# software — every host must already be running the version you want. Use
+# software — every host must already run the version you want, on one OS. Use
 # mulga's scripts/update-nodes.sh, or the release installer, for that first.
 #
 # Usage:
@@ -69,6 +69,19 @@ set -euo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 
+# Host-OS capability probes: beside this script in a checkout and in the
+# installed share dir, else where setup.sh installs them.
+DISTRO_LIB=""
+for candidate in "$SCRIPT_DIR/lib/distro.sh" /usr/local/share/spinifex/lib/distro.sh; do
+    if [ -f "$candidate" ]; then DISTRO_LIB="$candidate"; break; fi
+done
+if [ -z "$DISTRO_LIB" ]; then
+    echo "ERROR: distro.sh not found at $SCRIPT_DIR/lib/distro.sh or /usr/local/share/spinifex/lib/distro.sh; re-run setup.sh to install it" >&2
+    exit 1
+fi
+# shellcheck source=lib/distro.sh
+. "$DISTRO_LIB"
+
 SSH_USER="${SSH_USER:-spinifex}"
 IDENTITY=""
 HOSTS=()
@@ -284,6 +297,7 @@ resolve_plane() {
 log "preflight on $N host(s)"
 
 versions=""
+families=""
 for i in $(seq 0 $((N - 1))); do
     host="${HOSTS[$i]}"
 
@@ -303,6 +317,10 @@ for i in $(seq 0 $((N - 1))); do
     version="${version//[$'\r\n']/}"
     versions+="${version}"$'\n'
 
+    family=$(out "$host" "cat /etc/os-release" | distro_os_family) ||
+        fail "$host: cannot tell its OS from /etc/os-release"
+    families+="${host}: ${family}"$'\n'
+
     # A node that is already in a multi-node cluster is one whose keys and
     # logical network state are load-bearing. Refusing here is deliberate:
     # forming a cluster must never be the command that destroys a live one.
@@ -353,6 +371,14 @@ $(sort -u <<<"$versions" | sed 's/^/         /')
        Bring them all to the same build before forming a cluster."
 fi
 
+# A cluster runs one OS. Debian and Oracle Linux ship different OVN builds and
+# IKE daemons, and nothing qualifies them against each other.
+if [ "$(cut -d' ' -f2 <<<"$families" | sort -u | grep -c .)" -gt 1 ]; then
+    fail "hosts run different operating systems:
+$(sed '/^$/d; s/^/         /' <<<"$families")
+       A cluster must run one OS on every host."
+fi
+
 # --- Confirm ---------------------------------------------------------------
 
 echo ""
@@ -421,7 +447,7 @@ done
 # Matched on the service argv so the ssh shell running this cannot match itself.
 for host in "${HOSTS[@]}"; do
     $DRY_RUN && { echo "  [dry-run] $host: wait for every 'spx service' process to exit"; continue; }
-    on "$host" "for u in spinifex-predastore spinifex-viperblock spinifex-nats; do
+    on "$host" "for u in spinifex-predastore spinifex-viperblock spinifex-sqs-store spinifex-nats; do
             systemctl is-active --quiet \$u && sudo systemctl stop \$u
         done
         for _ in \$(seq 1 45); do
@@ -489,10 +515,10 @@ fi
 
 # --- Wipe ------------------------------------------------------------------
 #
-# node-reset.sh is not part of the release tarball, so it is pushed from this
-# checkout the same way smoke-test.sh is. After this a node has no CA, master
-# key, OVN database or chassis identity left for the formation phase to collide
-# with — which is the whole reason --wipe exists.
+# Pushed from beside this script, as smoke-test.sh is, so the reset matches this
+# script whatever the host has installed; it carries its own probes. After this
+# a node has no CA, master key, OVN database or chassis identity left for the
+# formation phase to collide with — which is the whole reason --wipe exists.
 if $WIPE; then
     log "wiping every host back to its pre-install state"
     for host in "${HOSTS[@]}"; do
@@ -964,9 +990,8 @@ out "${HOSTS[0]}" "sudo sed -n '/^\[\[network.external_pools\]\]/,/^$/p' /etc/sp
 if $RUN_SMOKE; then
     echo ""
     log "running the smoke test on ${HOSTS[0]}"
-    # smoke-test.sh is not part of the release tarball, so it is pushed from
-    # this checkout — which also guarantees the test and this script are the
-    # same version.
+    # Pushed from beside this script, so the test and this script are the same
+    # version whatever the host has installed.
     scp "${SSH_OPTS[@]}" -q "$SCRIPT_DIR/smoke-test.sh" "$SSH_USER@${HOSTS[0]}:/tmp/spx-smoke-test.sh"
     on "${HOSTS[0]}" "chmod 0755 /tmp/spx-smoke-test.sh && /tmp/spx-smoke-test.sh --create-vpc --nodes $N" ||
         fail "smoke test failed on ${HOSTS[0]}"

scripts/ipsec_state_helper_test.sh

@@ -0,0 +1,158 @@
+#!/bin/sh
+# ipsec_state_helper_test.sh — tests for the plaintext Geneve guard in the
+# IPsec state helper that setup.sh installs.
+#
+# The helper is cut out of setup.sh's heredoc and run against stub nft,
+# ovs-vsctl and systemctl commands, with pluto's path pointed at a fixture.
+set -eu
+
+SCRIPT_DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
+SETUP="$SCRIPT_DIR/setup.sh"
+TMP=$(mktemp -d)
+trap 'rm -rf "$TMP"' EXIT
+
+fails=0
+pass() { echo "  ok   - $1"; }
+fail() { echo "  FAIL - $1"; fails=$((fails + 1)); }
+
+echo "-- IPsec state helper: plaintext Geneve guard"
+
+HELPER="$TMP/helper"
+awk '/tee "\$\{IPSEC_STATE_HELPER\}"/ { on = 1; next } on && /^HELPER$/ { exit } on { print }' "$SETUP" |
+    sed "s#/usr/libexec/ipsec/pluto#$TMP/pluto#g" > "$HELPER"
+chmod +x "$HELPER"
+grep -q '^    guard)' "$HELPER" || { echo "could not extract the helper from setup.sh" >&2; exit 1; }
+
+mkdir -p "$TMP/bin"
+cat > "$TMP/bin/nft" <<EOF
+#!/bin/sh
+[ "\$*" = "-f -" ] || { echo "nft \$*" >> "$TMP/nft.log"; exit 0; }
+cat >> "$TMP/nft.log"
+echo "--" >> "$TMP/nft.log"
+EOF
+cat > "$TMP/bin/ovs-vsctl" <<EOF
+#!/bin/sh
+[ -e "$TMP/ovs-down" ] && { echo "ovs-vsctl: database connection failed" >&2; exit 1; }
+case "\$*" in
+    *" find Interface type=geneve") cat "$TMP/ifaces" ;;
+    *" get Interface "*)
+        for last; do :; done
+        f="$TMP/opt.\$5.\${last#options:}"
+        if [ -e "\$f" ]; then cat "\$f"; else echo; fi
+        ;;
+    *" get Open_vSwitch . external_ids:ovn-encap-ip")
+        if [ -e "$TMP/encap" ]; then cat "$TMP/encap"; else echo; fi
+        ;;
+    *) exit 1 ;;
+esac
+EOF
+printf '#!/bin/sh\nexit 0\n' > "$TMP/bin/systemctl"
+chmod +x "$TMP/bin/nft" "$TMP/bin/ovs-vsctl" "$TMP/bin/systemctl"
+
+# reset — no tunnels, OVS up, no pluto.
+reset() {
+    rm -f "$TMP"/opt.* "$TMP/ovs-down" "$TMP/nft.log" "$TMP/pluto" "$TMP/encap"
+    : > "$TMP/ifaces"
+    touch "$TMP/nft.log"
+}
+# tunnel <name> <local_ip|-> [remote_name] — a geneve interface as ovn-controller writes it.
+tunnel() {
+    echo "$1" >> "$TMP/ifaces"
+    [ "$2" = - ] || printf '"%s"\n' "$2" > "$TMP/opt.$1.local_ip"
+    [ -z "${3:-}" ] || echo "$3" > "$TMP/opt.$1.remote_name"
+}
+run() {
+    RC=0
+    OUT=$(PATH="$TMP/bin:$PATH" "$HELPER" "$1" 2>&1) || RC=$?
+    NFTLOG=$(cat "$TMP/nft.log")
+}
+LIFTED="table inet spinifex_ipsec_guard
+delete table inet spinifex_ipsec_guard
+--"
+# expect_guarded <desc> <sources> — one transaction that swaps in a table
+# dropping Geneve without an SA from exactly those sources.
+expect_guarded() {
+    if [ "$RC" = 0 ] &&
+        [ "$(grep -c '^--$' "$TMP/nft.log")" = 1 ] &&
+        [ "$(sed -n 2p "$TMP/nft.log")" = "delete table inet spinifex_ipsec_guard" ] &&
+        grep -q "^        elements = { $2 }$" "$TMP/nft.log" &&
+        grep -q '^        ip saddr @srcs udp dport 6081 rt ipsec missing drop$' "$TMP/nft.log"; then
+        pass "$1"
+    else
+        fail "$1 (rc=$RC): got [$NFTLOG] want sources [$2] $OUT"
+    fi
+}
+expect_lifted() {
+    if [ "$RC" = 0 ] && [ "$NFTLOG" = "$LIFTED" ]; then pass "$1"; else fail "$1 (rc=$RC): got [$NFTLOG] $OUT"; fi
+}
+
+reset
+tunnel ovn-node2-0 10.26.7.1 node2
+tunnel ovn-node3-0 10.26.7.1 node3
+run guard
+expect_guarded "guards the encap address once OVN wants the tunnels encrypted" "10.26.7.1/32"
+
+reset
+tunnel ovn-node2-0 10.26.7.1
+tunnel ovn-node3-0 10.26.7.1
+run guard
+expect_lifted "lifts the guard while OVN runs the tunnels in plaintext"
+
+reset
+tunnel ovn-node2-0 10.26.7.1 node2
+tunnel ovn-node3-0 10.26.8.1
+run guard
+expect_guarded "guards only the addresses of tunnels OVN encrypts" "10.26.7.1/32"
+
+reset
+tunnel ovn-node2-0 - node2
+run guard
+expect_guarded "guards a tunnel without a local_ip from every source" "0.0.0.0/0"
+
+reset
+echo '"10.26.7.1"' > "$TMP/encap"
+run guard
+expect_guarded "guards the encap address before ovn-controller has built the tunnels" "10.26.7.1/32"
+
+reset
+echo '"10.26.7.1,10.26.8.1"' > "$TMP/encap"
+run guard
+expect_guarded "guards each encap address of a multi-address host before the tunnels exist" "10.26.7.1/32,10.26.8.1/32"
+
+reset
+run guard
+expect_guarded "guards every source before the tunnels exist when no encap address is set" "0.0.0.0/0"
+
+reset
+tunnel ovn-node2-0 10.26.7.1 node2
+touch "$TMP/ovs-down"
+run guard
+if [ "$RC" != 0 ] && [ -z "$NFTLOG" ]; then
+    pass "keeps the guard and fails when OVS cannot be read"
+else
+    fail "with OVS down: rc=$RC nft calls [$NFTLOG]"
+fi
+
+reset
+touch "$TMP/pluto"
+chmod +x "$TMP/pluto"
+run off
+expect_lifted "off lifts the guard on a libreswan host"
+
+reset
+run off
+if [ "$RC" = 0 ] && [ -z "$NFTLOG" ]; then pass "off leaves nft alone without libreswan"; else fail "off without pluto: rc=$RC [$NFTLOG]"; fi
+
+reset
+run unguard
+expect_lifted "unguard lifts the guard"
+
+reset
+run bogus
+if [ "$RC" = 2 ] && [ -z "$NFTLOG" ]; then pass "refuses an unknown verb"; else fail "unknown verb: rc=$RC [$NFTLOG]"; fi
+
+if [ "$fails" -gt 0 ]; then
+    echo "$fails failure(s)"
+    exit 1
+fi
+echo "all passed"

scripts/lib/distro.sh

@@ -0,0 +1,202 @@
+# shellcheck shell=bash
+# distro.sh — host-OS capability probes. Each prints its answer, or names every
+# candidate it tried on stderr and returns 1, also when it finds both of a pair.
+# setup.sh and node-reset.sh carry inline copies; distro_test.sh keeps them equal.
+
+# Reads unit files off disk, not through systemctl, so it answers in a chroot.
+# DISTRO_ROOT prefixes every host path the probes read; tests set it.
+distro_unit_exists() {
+    local dir
+    for dir in /etc/systemd/system /run/systemd/system /usr/lib/systemd/system /lib/systemd/system; do
+        if [ -e "${DISTRO_ROOT:-}$dir/$1" ] || [ -L "${DISTRO_ROOT:-}$dir/$1" ]; then
+            return 0
+        fi
+    done
+    return 1
+}
+
+distro_pkg_manager() {
+    local apt=false dnf=false
+    command -v apt-get >/dev/null 2>&1 && apt=true
+    command -v dnf >/dev/null 2>&1 && dnf=true
+    if $apt && $dnf; then
+        echo "distro.sh: package manager: found both apt-get and dnf; cannot tell which one manages this host" >&2
+        return 1
+    fi
+    if $apt; then echo apt; return 0; fi
+    if $dnf; then echo dnf; return 0; fi
+    echo "distro.sh: package manager: found neither apt-get nor dnf on PATH" >&2
+    return 1
+}
+
+# The OVS umbrella unit: openvswitch-switch.service on Debian, openvswitch.service on EL.
+# Ubuntu's openvswitch-switch declares Alias=openvswitch.service; the symlink that
+# enabling it leaves points back at openvswitch-switch.service and is not the EL unit.
+distro_ovs_unit() {
+    local deb=false el=false dir
+    distro_unit_exists openvswitch-switch.service && deb=true
+    for dir in /etc/systemd/system /run/systemd/system /usr/lib/systemd/system /lib/systemd/system; do
+        if [ -e "${DISTRO_ROOT:-}$dir/openvswitch.service" ] || [ -L "${DISTRO_ROOT:-}$dir/openvswitch.service" ]; then
+            case $(readlink "${DISTRO_ROOT:-}$dir/openvswitch.service" || true) in
+                openvswitch-switch.service | */openvswitch-switch.service) ;;
+                *) el=true ;;
+            esac
+        fi
+    done
+    if $deb && $el; then
+        echo "distro.sh: OVS unit: found both openvswitch-switch.service and openvswitch.service; cannot tell which one owns OVS" >&2
+        return 1
+    fi
+    if $deb; then echo openvswitch-switch.service; return 0; fi
+    if $el; then echo openvswitch.service; return 0; fi
+    echo "distro.sh: OVS unit: found neither openvswitch-switch.service nor openvswitch.service; is Open vSwitch installed?" >&2
+    return 1
+}
+
+# "split": Debian's ovn-central plus separate NB and SB DB units.
+# "lone-northd": EL's single ovn-northd.service, which starts both DB servers.
+distro_ovn_central_layout() {
+    local split=false lone=false unit missing=""
+    for unit in ovn-central.service ovn-ovsdb-server-nb.service ovn-ovsdb-server-sb.service; do
+        if distro_unit_exists "$unit"; then
+            split=true
+        else
+            missing="${missing:+$missing, }$unit"
+        fi
+    done
+    if distro_unit_exists ovn-northd.service && [ -f "${DISTRO_ROOT:-}/etc/sysconfig/ovn" ]; then
+        lone=true
+    fi
+    if $split && $lone; then
+        echo "distro.sh: OVN central layout: found both the split units (ovn-central.service) and a lone ovn-northd.service with /etc/sysconfig/ovn; cannot tell which one owns the OVN databases" >&2
+        return 1
+    fi
+    if $split; then
+        if [ -n "$missing" ]; then
+            echo "distro.sh: OVN central layout: ovn-central is installed but $missing is missing" >&2
+            return 1
+        fi
+        echo split
+        return 0
+    fi
+    if $lone; then echo lone-northd; return 0; fi
+    echo "distro.sh: OVN central layout: found neither ovn-central.service nor ovn-northd.service with /etc/sysconfig/ovn; is OVN central installed?" >&2
+    return 1
+}
+
+# The trust-store update command and its local anchor directory, space-separated.
+distro_ca_trust() {
+    local deb=false el=false
+    command -v update-ca-certificates >/dev/null 2>&1 && deb=true
+    command -v update-ca-trust >/dev/null 2>&1 && el=true
+    if $deb && $el; then
+        echo "distro.sh: CA trust: found both update-ca-certificates and update-ca-trust; cannot tell which trust store TLS clients read" >&2
+        return 1
+    fi
+    if $deb; then echo "update-ca-certificates /usr/local/share/ca-certificates"; return 0; fi
+    if $el; then echo "update-ca-trust /etc/pki/ca-trust/source/anchors"; return 0; fi
+    echo "distro.sh: CA trust: found neither update-ca-certificates nor update-ca-trust on PATH" >&2
+    return 1
+}
+
+# chrony's main config file: /etc/chrony/chrony.conf on Debian, /etc/chrony.conf on EL.
+distro_chrony_conf() {
+    local deb=false el=false
+    [ -f "${DISTRO_ROOT:-}/etc/chrony/chrony.conf" ] && deb=true
+    [ -f "${DISTRO_ROOT:-}/etc/chrony.conf" ] && el=true
+    if $deb && $el; then
+        echo "distro.sh: chrony config: found both /etc/chrony/chrony.conf and /etc/chrony.conf; cannot tell which one chronyd reads" >&2
+        return 1
+    fi
+    if $deb; then echo /etc/chrony/chrony.conf; return 0; fi
+    if $el; then echo /etc/chrony.conf; return 0; fi
+    echo "distro.sh: chrony config: found neither /etc/chrony/chrony.conf nor /etc/chrony.conf; is chrony installed?" >&2
+    return 1
+}
+
+distro_ovn_logrotate() {
+    local deb=false el=false
+    [ -f "${DISTRO_ROOT:-}/etc/logrotate.d/ovn-common" ] && deb=true
+    [ -f "${DISTRO_ROOT:-}/etc/logrotate.d/ovn" ] && el=true
+    if $deb && $el; then
+        echo "distro.sh: OVN logrotate: found both /etc/logrotate.d/ovn-common and /etc/logrotate.d/ovn; cannot tell which one the OVN packages own" >&2
+        return 1
+    fi
+    if $deb; then echo /etc/logrotate.d/ovn-common; return 0; fi
+    if $el; then echo /etc/logrotate.d/ovn; return 0; fi
+    echo "distro.sh: OVN logrotate: found neither /etc/logrotate.d/ovn-common nor /etc/logrotate.d/ovn; is OVN installed?" >&2
+    return 1
+}
+
+distro_ovs_logrotate() {
+    local deb=false el=false
+    [ -f "${DISTRO_ROOT:-}/etc/logrotate.d/openvswitch-switch" ] && deb=true
+    [ -f "${DISTRO_ROOT:-}/etc/logrotate.d/openvswitch" ] && el=true
+    if $deb && $el; then
+        echo "distro.sh: OVS logrotate: found both /etc/logrotate.d/openvswitch-switch and /etc/logrotate.d/openvswitch; cannot tell which one the OVS packages own" >&2
+        return 1
+    fi
+    if $deb; then echo /etc/logrotate.d/openvswitch-switch; return 0; fi
+    if $el; then echo /etc/logrotate.d/openvswitch; return 0; fi
+    echo "distro.sh: OVS logrotate: found neither /etc/logrotate.d/openvswitch-switch nor /etc/logrotate.d/openvswitch; is Open vSwitch installed?" >&2
+    return 1
+}
+
+# libreswan's IKE daemon. EL's openvswitch-ipsec drives libreswan even beside
+# strongSwan, so this is a presence check rather than one of a pair.
+distro_has_libreswan() {
+    [ -x "${DISTRO_ROOT:-}/usr/libexec/ipsec/pluto" ]
+}
+
+# The OS family of the os-release text on stdin, for install-node.sh's one-OS
+# cluster check: "debian" for Debian and Ubuntu, so mixing those is not newly
+# refused, and the ID itself for anything else.
+distro_os_family() {
+    local line id="" like=""
+    while IFS= read -r line || [ -n "$line" ]; do
+        case "$line" in
+            ID=*) id=${line#ID=} ;;
+            ID_LIKE=*) like=${line#ID_LIKE=} ;;
+        esac
+    done
+    id=${id//[\"\']/}
+    like=${like//[\"\']/}
+    if [ -z "$id" ]; then
+        echo "distro.sh: OS family: /etc/os-release has no ID" >&2
+        return 1
+    fi
+    case " $id $like " in
+        *" debian "*) echo debian ;;
+        *) echo "$id" ;;
+    esac
+}
+
+# The OVS and OVN units the given unit files depend on or order against that
+# are not installed, one per line; prints nothing when all are. Fails only
+# when a unit file cannot be read.
+distro_missing_ovs_deps() {
+    local file line dep missing=""
+    for file in "$@"; do
+        if [ ! -r "${DISTRO_ROOT:-}$file" ]; then
+            echo "distro.sh: OVS dependencies: cannot read $file" >&2
+            return 1
+        fi
+        while IFS= read -r line || [ -n "$line" ]; do
+            case "$line" in
+                After=* | Before=* | Wants=* | Requires=* | Requisite=* | BindsTo=* | PartOf=* | Upholds=*) ;;
+                *) continue ;;
+            esac
+            for dep in ${line#*=}; do
+                case "$dep" in
+                    ovs* | ovn* | openvswitch*) ;;
+                    *) continue ;;
+                esac
+                case " $missing " in
+                    *" $dep "*) continue ;;
+                esac
+                distro_unit_exists "$dep" || missing="${missing:+$missing }$dep"
+            done
+        done <"${DISTRO_ROOT:-}$file"
+    done
+    for dep in $missing; do echo "$dep"; done
+}

scripts/lib/distro_test.sh

@@ -0,0 +1,311 @@
+#!/bin/sh
+# distro_test.sh — tests for the host-OS probes and the inline copies of them.
+#
+# Every probe runs against a fixture tree (DISTRO_ROOT) and a PATH holding only
+# stub commands, so the result never depends on the distro running the test.
+set -eu
+
+LIB_DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
+SCRIPTS_DIR=$(dirname -- "$LIB_DIR")
+DISTRO="$LIB_DIR/distro.sh"
+TMP=$(mktemp -d)
+trap 'rm -rf "$TMP"' EXIT
+
+fails=0
+pass() { echo "  ok   - $1"; }
+fail() { echo "  FAIL - $1"; fails=$((fails + 1)); }
+
+echo "-- lib/distro.sh"
+
+# fixture <name> — a fresh empty host root and stub bin dir.
+fixture() {
+    ROOT="$TMP/$1/root"
+    BIN="$TMP/$1/bin"
+    mkdir -p "$ROOT" "$BIN"
+}
+unit() { mkdir -p "$ROOT/usr/lib/systemd/system" && : > "$ROOT/usr/lib/systemd/system/$1"; }
+file() { mkdir -p "$ROOT$(dirname "$1")" && : > "$ROOT$1"; }
+cmd() { printf '#!/bin/sh\nexit 0\n' > "$BIN/$1" && chmod +x "$BIN/$1"; }
+# realcmd <name> — the host's real command, for probes that read the fixture with it.
+realcmd() { ln -s "$(command -v "$1")" "$BIN/$1"; }
+
+# probe <fn> [arg...] — runs one probe in bash against the current fixture,
+# with only the stub dir on PATH. Sets OUT, ERR and RC.
+probe() {
+    RC=0
+    OUT=$(DISTRO_ROOT="$ROOT" PATH="$BIN" /bin/bash -c '. "$1"; shift; "$@"' _ "$DISTRO" "$@" 2>"$TMP/err") || RC=$?
+    ERR=$(cat "$TMP/err")
+}
+
+# expect_ok <desc> <want>; expect_fail <desc> <word>... — every word must appear
+# in the error, which is how "names every candidate" is checked.
+expect_ok() {
+    if [ "$RC" = 0 ] && [ "$OUT" = "$2" ]; then pass "$1"; else fail "$1 (rc=$RC out='$OUT' err='$ERR')"; fi
+}
+expect_fail() {
+    desc=$1; shift
+    if [ "$RC" = 0 ]; then fail "$desc (succeeded with '$OUT')"; return; fi
+    for word in "$@"; do
+        case "$ERR" in
+            *"$word"*) ;;
+            *) fail "$desc (error does not name $word: '$ERR')"; return ;;
+        esac
+    done
+    pass "$desc"
+}
+
+# --------------------------------------------------------------------------
+# Package manager.
+# --------------------------------------------------------------------------
+fixture pm-apt; cmd apt-get; probe distro_pkg_manager
+expect_ok "package manager: apt-get alone is apt" apt
+fixture pm-dnf; cmd dnf; probe distro_pkg_manager
+expect_ok "package manager: dnf alone is dnf" dnf
+fixture pm-both; cmd apt-get; cmd dnf; probe distro_pkg_manager
+expect_fail "package manager: both fails naming both" apt-get dnf
+fixture pm-none; probe distro_pkg_manager
+expect_fail "package manager: neither fails naming both" apt-get dnf
+
+# --------------------------------------------------------------------------
+# OVS umbrella unit. Unit files are found under any systemd unit directory, and
+# a masked unit (a symlink to /dev/null) still counts as installed.
+# --------------------------------------------------------------------------
+fixture ovs-deb; unit openvswitch-switch.service; probe distro_ovs_unit
+expect_ok "OVS unit: Debian name" openvswitch-switch.service
+fixture ovs-el; unit openvswitch.service; probe distro_ovs_unit
+expect_ok "OVS unit: EL name" openvswitch.service
+fixture ovs-masked; mkdir -p "$ROOT/etc/systemd/system"
+ln -s /dev/null "$ROOT/etc/systemd/system/openvswitch.service"; realcmd readlink; probe distro_ovs_unit
+expect_ok "OVS unit: a masked unit in /etc counts" openvswitch.service
+fixture ovs-lib; mkdir -p "$ROOT/lib/systemd/system"
+: > "$ROOT/lib/systemd/system/openvswitch-switch.service"; probe distro_ovs_unit
+expect_ok "OVS unit: found under /lib on an unmerged-/usr host" openvswitch-switch.service
+fixture ovs-both; unit openvswitch-switch.service; unit openvswitch.service; probe distro_ovs_unit
+expect_fail "OVS unit: both fails naming both" openvswitch-switch.service openvswitch.service
+fixture ovs-ubuntu-alias; unit openvswitch-switch.service; mkdir -p "$ROOT/etc/systemd/system"
+ln -s /usr/lib/systemd/system/openvswitch-switch.service "$ROOT/etc/systemd/system/openvswitch.service"; realcmd readlink; probe distro_ovs_unit
+expect_ok "OVS unit: Ubuntu's openvswitch.service alias is the Debian unit" openvswitch-switch.service
+fixture ovs-relative-alias; unit openvswitch-switch.service; mkdir -p "$ROOT/etc/systemd/system"
+ln -s openvswitch-switch.service "$ROOT/etc/systemd/system/openvswitch.service"; realcmd readlink; probe distro_ovs_unit
+expect_ok "OVS unit: a relative alias symlink is the Debian unit" openvswitch-switch.service
+fixture ovs-alias-and-el; unit openvswitch-switch.service; unit openvswitch.service; mkdir -p "$ROOT/etc/systemd/system"
+ln -s /usr/lib/systemd/system/openvswitch-switch.service "$ROOT/etc/systemd/system/openvswitch.service"; realcmd readlink; probe distro_ovs_unit
+expect_fail "OVS unit: an alias beside a real EL unit still fails naming both" openvswitch-switch.service openvswitch.service
+fixture ovs-none; probe distro_ovs_unit
+expect_fail "OVS unit: neither fails naming both" openvswitch-switch.service openvswitch.service
+
+# --------------------------------------------------------------------------
+# OVN central layout. Both distros ship ovn-northd.service, so the unit alone
+# must not decide it: Debian's split layout carries one too.
+# --------------------------------------------------------------------------
+split_units() { unit ovn-central.service; unit ovn-ovsdb-server-nb.service; unit ovn-ovsdb-server-sb.service; unit ovn-northd.service; }
+
+fixture ovn-split; split_units; probe distro_ovn_central_layout
+expect_ok "OVN layout: Debian split units" split
+fixture ovn-lone; unit ovn-northd.service; file /etc/sysconfig/ovn; probe distro_ovn_central_layout
+expect_ok "OVN layout: EL lone ovn-northd" lone-northd
+fixture ovn-both; split_units; file /etc/sysconfig/ovn; probe distro_ovn_central_layout
+expect_fail "OVN layout: both fails naming both" ovn-central.service ovn-northd.service /etc/sysconfig/ovn
+fixture ovn-partial; unit ovn-central.service; unit ovn-ovsdb-server-nb.service; probe distro_ovn_central_layout
+expect_fail "OVN layout: a partial split install names the missing unit" ovn-ovsdb-server-sb.service
+fixture ovn-northd-only; unit ovn-northd.service; probe distro_ovn_central_layout
+expect_fail "OVN layout: ovn-northd without /etc/sysconfig/ovn is not EL" ovn-central.service ovn-northd.service /etc/sysconfig/ovn
+fixture ovn-none; probe distro_ovn_central_layout
+expect_fail "OVN layout: neither fails naming both" ovn-central.service ovn-northd.service
+
+# --------------------------------------------------------------------------
+# CA trust.
+# --------------------------------------------------------------------------
+fixture ca-deb; cmd update-ca-certificates; probe distro_ca_trust
+expect_ok "CA trust: Debian command and anchor dir" "update-ca-certificates /usr/local/share/ca-certificates"
+fixture ca-el; cmd update-ca-trust; probe distro_ca_trust
+expect_ok "CA trust: EL command and anchor dir" "update-ca-trust /etc/pki/ca-trust/source/anchors"
+fixture ca-both; cmd update-ca-certificates; cmd update-ca-trust; probe distro_ca_trust
+expect_fail "CA trust: both fails naming both" update-ca-certificates update-ca-trust
+fixture ca-none; probe distro_ca_trust
+expect_fail "CA trust: neither fails naming both" update-ca-certificates update-ca-trust
+
+# --------------------------------------------------------------------------
+# chrony config file.
+# --------------------------------------------------------------------------
+fixture chrony-deb; file /etc/chrony/chrony.conf; probe distro_chrony_conf
+expect_ok "chrony config: Debian file" /etc/chrony/chrony.conf
+fixture chrony-el; file /etc/chrony.conf; probe distro_chrony_conf
+expect_ok "chrony config: EL file" /etc/chrony.conf
+fixture chrony-both; file /etc/chrony/chrony.conf; file /etc/chrony.conf; probe distro_chrony_conf
+expect_fail "chrony config: both fails naming both" /etc/chrony/chrony.conf /etc/chrony.conf
+fixture chrony-none; probe distro_chrony_conf
+expect_fail "chrony config: neither fails naming both" /etc/chrony/chrony.conf /etc/chrony.conf
+
+# --------------------------------------------------------------------------
+# OVN logrotate file.
+# --------------------------------------------------------------------------
+fixture lr-deb; file /etc/logrotate.d/ovn-common; probe distro_ovn_logrotate
+expect_ok "OVN logrotate: Debian file" /etc/logrotate.d/ovn-common
+fixture lr-el; file /etc/logrotate.d/ovn; probe distro_ovn_logrotate
+expect_ok "OVN logrotate: EL file" /etc/logrotate.d/ovn
+fixture lr-both; file /etc/logrotate.d/ovn-common; file /etc/logrotate.d/ovn; probe distro_ovn_logrotate
+expect_fail "OVN logrotate: both fails naming both" /etc/logrotate.d/ovn-common /etc/logrotate.d/ovn
+fixture lr-none; probe distro_ovn_logrotate
+expect_fail "OVN logrotate: neither fails naming both" /etc/logrotate.d/ovn-common /etc/logrotate.d/ovn
+
+# --------------------------------------------------------------------------
+# OVS logrotate file.
+# --------------------------------------------------------------------------
+fixture ovslr-deb; file /etc/logrotate.d/openvswitch-switch; probe distro_ovs_logrotate
+expect_ok "OVS logrotate: Debian file" /etc/logrotate.d/openvswitch-switch
+fixture ovslr-el; file /etc/logrotate.d/openvswitch; probe distro_ovs_logrotate
+expect_ok "OVS logrotate: EL file" /etc/logrotate.d/openvswitch
+fixture ovslr-both; file /etc/logrotate.d/openvswitch-switch; file /etc/logrotate.d/openvswitch; probe distro_ovs_logrotate
+expect_fail "OVS logrotate: both fails naming both" /etc/logrotate.d/openvswitch-switch /etc/logrotate.d/openvswitch
+fixture ovslr-none; probe distro_ovs_logrotate
+expect_fail "OVS logrotate: neither fails naming both" /etc/logrotate.d/openvswitch-switch /etc/logrotate.d/openvswitch
+
+# --------------------------------------------------------------------------
+# libreswan.
+# --------------------------------------------------------------------------
+# expect_rc <desc> <want> — for the predicates, which print nothing.
+expect_rc() {
+    if [ "$RC" = "$2" ]; then pass "$1"; else fail "$1 (rc=$RC, want $2)"; fi
+}
+fixture ls-pluto; file /usr/libexec/ipsec/pluto; chmod +x "$ROOT/usr/libexec/ipsec/pluto"; probe distro_has_libreswan
+expect_rc "libreswan: an executable pluto" 0
+fixture ls-noexec; file /usr/libexec/ipsec/pluto; probe distro_has_libreswan
+expect_rc "libreswan: a pluto that cannot run is not libreswan" 1
+fixture ls-charon; file /usr/lib/ipsec/charon; chmod +x "$ROOT/usr/lib/ipsec/charon"; probe distro_has_libreswan
+expect_rc "libreswan: strongSwan's charon is not libreswan" 1
+
+# --------------------------------------------------------------------------
+# OS family, read from os-release text on stdin.
+# --------------------------------------------------------------------------
+# family <os-release text> — runs distro_os_family on it. Sets OUT, ERR and RC.
+family() {
+    RC=0
+    OUT=$(printf '%s\n' "$1" | /bin/bash -c '. "$1"; distro_os_family' _ "$DISTRO" 2>"$TMP/err") || RC=$?
+    ERR=$(cat "$TMP/err")
+}
+family 'PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
+ID=debian
+VERSION_ID="13"'
+expect_ok "OS family: Debian" debian
+family 'NAME="Ubuntu"
+ID=ubuntu
+ID_LIKE=debian'
+expect_ok "OS family: Ubuntu is Debian's family" debian
+family 'NAME="Oracle Linux Server"
+ID="ol"
+ID_LIKE="fedora"
+VERSION_ID="10.2"'
+expect_ok "OS family: Oracle Linux, quoted, is its own" ol
+family 'ID=rocky
+ID_LIKE="rhel centos fedora"'
+expect_ok "OS family: another EL is not Oracle Linux" rocky
+family 'NAME=Something
+VERSION_ID=1'
+expect_fail "OS family: no ID fails naming the file" /etc/os-release ID
+
+# --------------------------------------------------------------------------
+# OVS and OVN units the shipped Spinifex units name. Run against the real unit
+# files, so a unit that names a Debian-only OVS unit fails on the EL fixture.
+# --------------------------------------------------------------------------
+UNITS_SRC="$(dirname -- "$SCRIPTS_DIR")/build/systemd"
+spinifex_units() {
+    mkdir -p "$ROOT/etc/systemd/system"
+    cp "$UNITS_SRC"/spinifex.target "$UNITS_SRC"/spinifex-*.service "$ROOT/etc/systemd/system/"
+}
+deps_probe() {
+    set --
+    for f in "$ROOT"/etc/systemd/system/spinifex*; do set -- "$@" "${f#"$ROOT"}"; done
+    probe distro_missing_ovs_deps "$@"
+}
+el_ovs_units() {
+    for u in openvswitch.service ovsdb-server.service ovs-vswitchd.service ovn-controller.service ovn-northd.service openvswitch-ipsec.service; do unit "$u"; done
+}
+
+fixture deps-el; spinifex_units; el_ovs_units; deps_probe
+expect_ok "OVS dependencies: every one the shipped units name is installed on EL" ""
+fixture deps-no-ipsec; spinifex_units; el_ovs_units; rm "$ROOT/usr/lib/systemd/system/openvswitch-ipsec.service"; deps_probe
+expect_ok "OVS dependencies: a missing unit is printed once" openvswitch-ipsec.service
+fixture deps-debian-only; el_ovs_units
+mkdir -p "$ROOT/etc/systemd/system"
+printf '[Unit]\n# After=ovs-old.service\nAfter=openvswitch-switch.service ovn-controller.service\nWants=openvswitch-switch.service\n' >"$ROOT/etc/systemd/system/spinifex-x.service"
+printf '[Unit]\nRequires=ovn-central.service' >"$ROOT/etc/systemd/system/spinifex-y.service"
+deps_probe
+expect_ok "OVS dependencies: Debian-only names are missing on EL, comments are not read" "openvswitch-switch.service
+ovn-central.service"
+fixture deps-unreadable; probe distro_missing_ovs_deps /etc/systemd/system/spinifex.target
+expect_fail "OVS dependencies: an unreadable unit file fails naming it" "cannot read /etc/systemd/system/spinifex.target"
+
+# --------------------------------------------------------------------------
+# Inline copies. setup.sh and node-reset.sh cannot source the library, so they
+# carry copies between markers; each copied function must match the library's
+# byte for byte, or a fix to one silently misses the other.
+# --------------------------------------------------------------------------
+# fn_body <file> <name> — the function from "name() {" to the first "}" line.
+fn_body() {
+    awk -v name="$2" '$0 == name "() {" { on = 1 } on { print } on && $0 == "}" { exit }' "$1"
+}
+
+for copy in setup.sh node-reset.sh; do
+    path="$SCRIPTS_DIR/$copy"
+    block=$(sed -n '/^# --- BEGIN distro.sh probes ---$/,/^# --- END distro.sh probes ---$/p' "$path")
+    if [ -z "$block" ]; then
+        fail "$copy: no distro.sh probe block"
+        continue
+    fi
+    names=$(printf '%s\n' "$block" | sed -n 's/^\([a-z_][a-z0-9_]*\)() {$/\1/p')
+    if [ -z "$names" ]; then
+        fail "$copy: the distro.sh probe block holds no functions"
+        continue
+    fi
+    for name in $names; do
+        want=$(fn_body "$DISTRO" "$name")
+        got=$(fn_body "$path" "$name")
+        if [ -z "$want" ]; then
+            fail "$copy: $name is not in lib/distro.sh"
+        elif [ "$got" = "$want" ]; then
+            pass "$copy: $name matches lib/distro.sh"
+        else
+            fail "$copy: $name differs from lib/distro.sh"
+        fi
+    done
+
+    # A copied probe calling a probe helper the block lacks fails only when run.
+    for called in $(printf '%s\n' "$block" | grep -v '^ *#' | grep -o 'distro_[a-z_]*' | sort -u); do
+        case " $(echo $names) " in
+            *" $called "*) ;;
+            *) fail "$copy: copies a probe that calls $called without copying it" ;;
+        esac
+    done
+done
+
+# --------------------------------------------------------------------------
+# A consumer with no library beside it and none installed must stop naming
+# both paths, not run on with the probes undefined. Skipped on a host where
+# setup.sh has installed the library, since the fallback then finds it.
+# --------------------------------------------------------------------------
+if [ -e /usr/local/share/spinifex/lib/distro.sh ]; then
+    echo "  skip - consumers without distro.sh: /usr/local/share/spinifex/lib/distro.sh is installed here"
+else
+    # uninstall-spx.sh sources the library after its --help, so it is run into
+    # its multi-node refusal, which exits 1 without touching the host either way.
+    mkdir -p "$TMP/alone/bin" "$TMP/alone/etc"
+    printf '#!/bin/sh\nexec "$@"\n' > "$TMP/alone/bin/sudo"
+    chmod +x "$TMP/alone/bin/sudo"
+    printf '[nodes.node1]\n[nodes.node2]\n' > "$TMP/alone/etc/spinifex.toml"
+    for consumer in "setup-ovn.sh --help" "uninstall-spx.sh --dry-run"; do
+        script=${consumer%% *}
+        cp "$SCRIPTS_DIR/$script" "$TMP/alone/"
+        PATH="$TMP/alone/bin:$PATH" SPX_ETC_DIR="$TMP/alone/etc" "$TMP/alone/$script" "${consumer#* }" >"$TMP/out" 2>&1 && rc=0 || rc=$?
+        if [ "$rc" != 0 ] && grep -q "distro.sh not found at $TMP/alone/lib/distro.sh or /usr/local/share/spinifex/lib/distro.sh" "$TMP/out"; then
+            pass "$script: stops naming both paths when distro.sh is missing"
+        else
+            fail "$script: without distro.sh exited $rc: $(head -3 "$TMP/out")"
+        fi
+    done
+fi
+
+if [ "$fails" -gt 0 ]; then
+    echo "$fails failure(s)"
+    exit 1
+fi

scripts/lint-dnf-packages.sh

@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+# Runs setup.sh's Oracle Linux repository setup in a container, then resolves
+# its runtime dnf list against the real repositories with dnf install --assumeno.
+# Catches a renamed or dropped package, a SIG dependency the includepkgs
+# allowlist misses, a changed SIG key, and QEMU or OVS resolving from the wrong
+# repository.
+set -euo pipefail
+
+IMAGE="${1:-oraclelinux:10}"
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+
+echo "Resolving ${IMAGE}..."
+# The repository setup and the list both come from setup.sh itself, so the lint
+# cannot drift from what the installer does.
+docker run --rm -v "${SCRIPT_DIR}/setup.sh:/setup.sh:ro" "${IMAGE}" bash -c '
+    set -euo pipefail
+    INSTALL_SPINIFEX_LIB_ONLY=1 . /setup.sh
+    SUDO=""
+    ensure_ol_repos
+
+    # --assumeno always exits 1, so the transaction is judged from its output.
+    # shellcheck disable=SC2086
+    out=$(dnf install --assumeno $OL_RUNTIME_PACKAGES 2>&1) || true
+    if ! grep -q "^Operation aborted" <<< "$out" || grep -qE "^(Error|Problem)|No match for argument" <<< "$out"; then
+        echo "$out"
+        echo "FAIL: the Oracle Linux package list does not resolve" >&2
+        exit 1
+    fi
+
+    # repo_of <package> — the repository dnf would install the package from.
+    repo_of() { awk -v p="$1" "\$1 == p { print \$4; exit }" <<< "$out"; }
+    for want in "qemu-kvm $OL_KVM_UTILS_REPO" "qemu-img $OL_KVM_UTILS_REPO" \
+        "openvswitch3.5 $NFV_SIG_REPO_ID" "ovn25.03-central $NFV_SIG_REPO_ID"; do
+        pkg=${want% *} repo=${want#* }
+        got=$(repo_of "$pkg")
+        if [ "$got" != "$repo" ]; then
+            echo "$out"
+            echo "FAIL: $pkg resolves from ${got:-nowhere}, not $repo" >&2
+            exit 1
+        fi
+    done
+'
+
+echo "OK: every package resolves on ${IMAGE}, QEMU from KVM Utilities and OVS/OVN from the NFV SIG"

scripts/node-reset.sh

@@ -57,6 +57,67 @@ run() {
     "$@"
 }
 
+# --- BEGIN distro.sh probes ---
+# Copied from scripts/lib/distro.sh: install-node.sh --wipe copies this script
+# alone to each host, so it cannot source the library. distro_test.sh keeps them equal.
+distro_unit_exists() {
+    local dir
+    for dir in /etc/systemd/system /run/systemd/system /usr/lib/systemd/system /lib/systemd/system; do
+        if [ -e "${DISTRO_ROOT:-}$dir/$1" ] || [ -L "${DISTRO_ROOT:-}$dir/$1" ]; then
+            return 0
+        fi
+    done
+    return 1
+}
+
+distro_ovs_unit() {
+    local deb=false el=false dir
+    distro_unit_exists openvswitch-switch.service && deb=true
+    for dir in /etc/systemd/system /run/systemd/system /usr/lib/systemd/system /lib/systemd/system; do
+        if [ -e "${DISTRO_ROOT:-}$dir/openvswitch.service" ] || [ -L "${DISTRO_ROOT:-}$dir/openvswitch.service" ]; then
+            case $(readlink "${DISTRO_ROOT:-}$dir/openvswitch.service" || true) in
+                openvswitch-switch.service | */openvswitch-switch.service) ;;
+                *) el=true ;;
+            esac
+        fi
+    done
+    if $deb && $el; then
+        echo "distro.sh: OVS unit: found both openvswitch-switch.service and openvswitch.service; cannot tell which one owns OVS" >&2
+        return 1
+    fi
+    if $deb; then echo openvswitch-switch.service; return 0; fi
+    if $el; then echo openvswitch.service; return 0; fi
+    echo "distro.sh: OVS unit: found neither openvswitch-switch.service nor openvswitch.service; is Open vSwitch installed?" >&2
+    return 1
+}
+
+distro_ca_trust() {
+    local deb=false el=false
+    command -v update-ca-certificates >/dev/null 2>&1 && deb=true
+    command -v update-ca-trust >/dev/null 2>&1 && el=true
+    if $deb && $el; then
+        echo "distro.sh: CA trust: found both update-ca-certificates and update-ca-trust; cannot tell which trust store TLS clients read" >&2
+        return 1
+    fi
+    if $deb; then echo "update-ca-certificates /usr/local/share/ca-certificates"; return 0; fi
+    if $el; then echo "update-ca-trust /etc/pki/ca-trust/source/anchors"; return 0; fi
+    echo "distro.sh: CA trust: found neither update-ca-certificates nor update-ca-trust on PATH" >&2
+    return 1
+}
+# --- END distro.sh probes ---
+
+# Resolved before anything stops, so a host whose OVS unit is ambiguous is
+# refused with nothing changed.
+OVS_UNIT=""
+if command -v ovs-vsctl >/dev/null 2>&1; then
+    OVS_UNIT=$(distro_ovs_unit) || exit 1
+fi
+
+# Also resolved up front. The CA tools live in sbin, which Debian leaves off a
+# non-root PATH, and a probe that fails after the wipe leaves a half-reset node.
+CA_TRUST=$(PATH="/usr/sbin:/sbin:$PATH" distro_ca_trust) \
+    || { echo "ERROR: cannot tell which trust store holds the stale CA" >&2; exit 1; }
+
 WIPE_DIRS=("$ETC_DIR" "$LOG_DIR" "$RUN_DIR")
 $KEEP_DATA || WIPE_DIRS+=("$DATA_DIR")
 
@@ -80,6 +141,20 @@ keep_args() {
               -o -path "$1/firewall/custom.nft")
 }
 
+# --- BEGIN at-stake counts ---
+# Copied between node-reset.sh and uninstall-spx.sh; node-reset_test.sh keeps them equal.
+# Instances are the daemon's local state records. Only mounted volumes have a
+# local directory under viperblock/; the rest live only in Predastore.
+count_instances() {
+    local state="$1/state/instance-state.json"
+    sudo test -e "$state" || { echo 0; return 0; }
+    sudo jq -r '.vms // {} | length' "$state" 2>/dev/null || echo unknown
+}
+count_mounted_volumes() {
+    sudo find "$1/viperblock" -maxdepth 1 -mindepth 1 -type d -name 'vol-*' ! -name '*-efi' 2>/dev/null | wc -l || true
+}
+# --- END at-stake counts ---
+
 # Report what is at stake in figures rather than adjectives. An operator who
 # reads "3 instances, 12 volumes, 840G" makes a better decision than one who
 # reads a warning banner.
@@ -87,21 +162,24 @@ keep_args() {
 # created fails the whole pipeline, and a node with nothing to report is the
 # most ordinary case there is.
 if [ -d "$DATA_DIR" ]; then
-    instances=$(sudo find "$DATA_DIR/instances" -maxdepth 1 -mindepth 1 -type d 2>/dev/null | wc -l || true)
-    volumes=$(sudo find "$DATA_DIR/volumes" -maxdepth 1 -mindepth 1 2>/dev/null | wc -l || true)
+    instances=$(count_instances "$DATA_DIR")
+    volumes=$(count_mounted_volumes "$DATA_DIR")
     size=$(sudo du -sh "$DATA_DIR" 2>/dev/null | cut -f1 || true)
-    log "on $(hostname): $instances instance(s), $volumes volume(s), ${size:-unknown} under $DATA_DIR"
+    log "on $(hostname): $instances instance(s), $volumes mounted volume(s), ${size:-unknown} under $DATA_DIR"
     $KEEP_DATA && log "  --keep-data: $DATA_DIR will be preserved"
 fi
 log "removing: ${WIPE_DIRS[*]}"
 
 if ! $ASSUME_YES && ! $DRY_RUN; then
-    read -r -p "Destroy this node's state? Type 'destroy' to continue: " reply
+    read -r -p "Destroy this node's state? Type 'destroy' to continue: " reply || reply=""
     [ "$reply" = "destroy" ] || { log "aborted"; exit 1; }
 fi
 
 log "stopping services"
 run sudo systemctl stop spinifex.target 2>/dev/null || true
+# spinifex-hostnet starts with OVS, which is started below to delete bridges; it
+# would rebuild the old veth pairs. setup-ovn.sh enables it again.
+run sudo systemctl disable --now spinifex-hostnet.service 2>/dev/null || true
 run sudo systemctl reset-failed 'spinifex-*' 2>/dev/null || true
 
 # Stopping the target is not the same as its services being stopped.
@@ -112,7 +190,7 @@ run sudo systemctl reset-failed 'spinifex-*' 2>/dev/null || true
 # the wipe and before the verification, failing the reset on a file the script
 # had already deleted. update-nodes.sh carries the same wait.
 if ! $DRY_RUN; then
-    for unit in spinifex-predastore spinifex-viperblock spinifex-nats; do
+    for unit in spinifex-predastore spinifex-viperblock spinifex-sqs-store spinifex-nats; do
         if systemctl is-active --quiet "$unit"; then
             sudo systemctl stop "$unit" || true
         fi
@@ -134,8 +212,9 @@ fi
 # `pgrep -x` and `pkill -x` match against comm, which the kernel truncates to 15
 # characters, so `qemu-system-x86_64` is stored as `qemu-system-x86` and an -x
 # match on the full name can never fire. Match the prefix instead: it is 11
-# characters, well under the limit, and covers both x86_64 and aarch64. This is
-# the same rule qemuProcessPrefix applies in spinifex/runtime/compute/vm/orphan_scan.go.
+# characters, well under the limit, and covers both x86_64 and aarch64. EL-family
+# hosts run /usr/libexec/qemu-kvm, whose comm is exactly `qemu-kvm`. This is the
+# rule hostos.IsQEMUProcess applies in spinifex/runtime/host/hostos.
 #
 # Deliberately not -f. That matches the whole command line, where argv[0] may
 # carry a path, so an anchored pattern misses the process it was written for.
@@ -144,7 +223,7 @@ fi
 # ones has both under the same comm and an unscoped kill would take the wrong
 # set down. The daemon runs guests as its own service user on an installed node;
 # a dev install runs them as whoever invoked spx.
-QEMU_PGREP=('^qemu-system')
+QEMU_PGREP=('^(qemu-system|qemu-kvm$)')
 for qemu_user in spinifex-daemon spinifex "${SUDO_USER:-$(id -un)}"; do
     if id -u "$qemu_user" >/dev/null 2>&1; then
         QEMU_PGREP=(-u "$qemu_user" "${QEMU_PGREP[@]}")
@@ -178,8 +257,10 @@ fi
 # node keeping its old system-id across a reset re-registers under the old
 # chassis name and collides with the new one over the encap IP.
 log "removing OVS bridges and identity"
-if command -v ovs-vsctl >/dev/null 2>&1; then
-    run sudo systemctl start openvswitch-switch 2>/dev/null || true
+# A running ovn-controller recreates br-int as soon as it is deleted.
+run sudo systemctl stop ovn-controller 2>/dev/null || true
+if [ -n "$OVS_UNIT" ]; then
+    run sudo systemctl start "$OVS_UNIT" 2>/dev/null || true
     $DRY_RUN || sleep 1
     # Listed even under --dry-run: "which bridges" is the question an operator
     # actually has here, and br-wan is a Linux bridge so it is never in this set.
@@ -188,10 +269,37 @@ if command -v ovs-vsctl >/dev/null 2>&1; then
         run sudo ovs-vsctl --if-exists del-br "$br"
     done
     run sudo ovs-vsctl --if-exists clear Open_vSwitch . external_ids 2>/dev/null || true
-    run sudo systemctl stop openvswitch-switch 2>/dev/null || true
+    run sudo systemctl stop "$OVS_UNIT" 2>/dev/null || true
 fi
 run sudo rm -f /etc/openvswitch/system-id.conf
 
+# --- BEGIN ovs nss nicknames ---
+# ovs_nss_nicknames — the entries ovs-monitor-ipsec imported, read from
+# `certutil -L` on stdin: ovs_certkey_* is this node's certificate and key,
+# ovs_cert_* the CA and any peer certificates. Everything else is someone else's.
+ovs_nss_nicknames() {
+    awk '$1 ~ /^ovs_cert(key)?_/ {print $1}' | LC_ALL=C sort -u
+}
+# --- END ovs nss nicknames ---
+
+# A node rejoining another cluster must not keep trusting this one's CA. Only
+# the NSS entries go: other identities and /etc/ipsec.d/*.conf are left alone.
+if [ -x /usr/libexec/ipsec/pluto ] && command -v certutil >/dev/null 2>&1; then
+    log "removing Spinifex IPsec identities from libreswan's NSS database"
+    # A running ovs-monitor-ipsec imports them again from other_config.
+    run sudo systemctl stop openvswitch-ipsec.service 2>/dev/null || true
+    for nss in /var/lib/ipsec/nss /etc/ipsec.d; do
+        sudo test -e "$nss/cert9.db" || continue
+        for nick in $(sudo certutil -L -d "sql:$nss" 2>/dev/null | ovs_nss_nicknames); do
+            log "  deleting $nick from $nss"
+            delete=-D
+            case "$nick" in ovs_certkey_*) delete=-F ;; esac
+            run sudo certutil "$delete" -d "sql:$nss" -n "$nick" ||
+                log "  NOTE: could not delete $nick from $nss; remove it with certutil by hand"
+        done
+    done
+fi
+
 # Delete the OVN DBs outright — a caller re-running setup-ovn.sh gets fresh
 # empty ones. This is what clears stale chassis rows and port bindings, which
 # otherwise accumulate across resets and wedge ovn-controller in a commit loop.
@@ -233,14 +341,20 @@ if [ -e "$ETC_DIR/firewall/custom.nft" ]; then
 fi
 
 log "removing OVN databases"
-run sudo systemctl stop ovn-central 2>/dev/null || true
+# Debian's ovn-central takes its split units down with it; EL's ovn-northd runs
+# both databases itself. A compute node has neither.
+for unit in ovn-central.service ovn-northd.service; do
+    if distro_unit_exists "$unit"; then
+        run sudo systemctl stop "$unit" 2>/dev/null || true
+    fi
+done
 run sudo systemctl stop ovn-controller 2>/dev/null || true
 run sudo rm -f /var/lib/ovn/ovnnb_db.db /var/lib/ovn/ovnsb_db.db
 
-# The RAFT membership flags live in this file, not in the databases just
-# deleted. Left behind, ovn-central recreates the DB as a cluster member and
-# blocks dialling a peer that has itself been reset.
-run sudo rm -f /etc/default/ovn-central
+# The RAFT membership flags live in these files (Debian, EL), not in the
+# databases just deleted. Left behind, OVN central recreates the DB as a
+# cluster member and blocks dialling a peer that has itself been reset.
+run sudo rm -f /etc/default/ovn-central /etc/sysconfig/ovn-northd
 
 if ip link show veth-wan-br >/dev/null 2>&1; then
     log "  deleting veth pair: veth-wan-br <-> veth-wan-ovs"
@@ -259,6 +373,21 @@ if [ -e /etc/systemd/network/15-spinifex-veth-wan.netdev ] ||
     run sudo networkctl reload 2>/dev/null || true
 fi
 
+# The routed-NAT transit veth keeps 100.127.0.1/24 and the routes behind it.
+if ip link show spx-nat-host >/dev/null 2>&1; then
+    log "  deleting veth pair: spx-nat-host <-> spx-nat-ovs"
+    run sudo ip link del spx-nat-host 2>/dev/null || true
+fi
+if [ -e /etc/systemd/network/17-spinifex-nat.netdev ] ||
+    [ -e /etc/systemd/network/17-spinifex-nat.network ] ||
+    [ -e /etc/systemd/network/18-spinifex-nat-ovs.network ]; then
+    log "  deleting NAT transit persistence units"
+    run sudo rm -f /etc/systemd/network/17-spinifex-nat.netdev \
+        /etc/systemd/network/17-spinifex-nat.network \
+        /etc/systemd/network/18-spinifex-nat-ovs.network
+    run sudo networkctl reload 2>/dev/null || true
+fi
+
 # `rm -rf` is wrong here on two counts. On an ISO-installed node several of
 # these are separate ZFS datasets — rpool/log and rpool/data/{nats,viperblock,
 # predastore,predastore-db,predastore-nodes} — and a mountpoint cannot be
@@ -299,6 +428,10 @@ if ! $KEEP_DATA && [ -d "$DATA_DIR/nats/jetstream" ]; then
     log "removing the JetStream store"
     run sudo rm -rf "$DATA_DIR/nats/jetstream"
 fi
+if ! $KEEP_DATA && [ -d "$DATA_DIR/sqs-store/jetstream" ]; then
+    log "removing the SQS store JetStream data"
+    run sudo rm -rf "$DATA_DIR/sqs-store/jetstream"
+fi
 
 # Directories may survive — a mountpoint cannot be removed, and neither can the
 # path leading down to one. Files may not: a surviving file is state that would
@@ -374,10 +507,11 @@ fi
 
 # The next init writes a fresh CA. Leaving the old one trusted means the host
 # trusts a CA nobody holds the key for.
-if [ -f /usr/local/share/ca-certificates/spinifex-ca.crt ]; then
+CA_ANCHOR="${CA_TRUST#* }/spinifex-ca.crt"
+if [ -f "$CA_ANCHOR" ]; then
     log "removing the stale CA from the trust store"
-    run sudo rm -f /usr/local/share/ca-certificates/spinifex-ca.crt
-    run sudo update-ca-certificates
+    run sudo rm -f "$CA_ANCHOR"
+    run sudo "${CA_TRUST%% *}"
 fi
 
 log "done — node is installed but uninitialized, ready to init or join"

scripts/node-reset_test.sh

@@ -0,0 +1,162 @@
+#!/bin/sh
+# node-reset_test.sh — tests the QEMU process match node-reset.sh uses to stop
+# guests before it deletes Viperblock state, against real processes, and the
+# instance and mounted-volume counts it and uninstall-spx.sh report before deleting,
+# and the libreswan NSS entries it deletes.
+#
+# Each fake guest is a copy of sleep named like a QEMU binary, so the kernel
+# gives it the same comm, including the 15-character truncation.
+set -eu
+
+SCRIPT_DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
+TMP=$(mktemp -d)
+pids=""
+cleanup() {
+    for pid in $pids; do kill "$pid" 2>/dev/null || true; done
+    rm -rf "$TMP"
+}
+trap cleanup EXIT
+
+fails=0
+pass() { echo "  ok   - $1"; }
+fail() { echo "  FAIL - $1"; fails=$((fails + 1)); }
+
+echo "-- node-reset.sh QEMU process match"
+
+# pattern_in <script> — the pgrep pattern the script assigns to QEMU_PGREP.
+pattern_in() {
+    sed -n "s/^QEMU_PGREP=('\(.*\)')$/\1/p" "$1"
+}
+
+PATTERN=$(pattern_in "$SCRIPT_DIR/node-reset.sh")
+if [ -z "$PATTERN" ]; then
+    echo "  FAIL - no QEMU_PGREP=('...') line in node-reset.sh"
+    exit 1
+fi
+
+# reset-dev-env.sh stops guests the same way and must not drift from it.
+dev_pattern=$(pattern_in "$SCRIPT_DIR/reset-dev-env.sh")
+if [ "$dev_pattern" = "$PATTERN" ]; then
+    pass "reset-dev-env.sh uses node-reset.sh's pattern"
+else
+    fail "reset-dev-env.sh pattern '$dev_pattern' differs from node-reset.sh's '$PATTERN'"
+fi
+
+# start <name> — runs a process whose comm is <name>, truncated to 15
+# characters. Its PID is left in $!.
+start() {
+    cp "$(command -v sleep)" "$TMP/$1"
+    "$TMP/$1" 300 &
+    pids="$pids $!"
+}
+start qemu-system-x86_64; pid_x86=$!
+start qemu-system-aarch64; pid_arm=$!
+start qemu-kvm; pid_kvm=$!
+start qemu-img; pid_img=$!
+start qemu-kvm-helper; pid_helper=$!
+
+me=$(id -un)
+found=" $(pgrep -u "$me" "$PATTERN" | tr '\n' ' ' || true) "
+
+check_found() {
+    case "$found" in
+        *" $2 "*) pass "finds a $1 guest owned by the service user" ;;
+        *) fail "misses a $1 guest (pid $2) owned by the service user" ;;
+    esac
+}
+check_found qemu-system-x86_64 "$pid_x86"
+check_found qemu-system-aarch64 "$pid_arm"
+check_found qemu-kvm "$pid_kvm"
+
+check_ignored() {
+    case "$found" in
+        *" $2 "*) fail "matches $1 (pid $2), which is not a guest" ;;
+        *) pass "ignores $1" ;;
+    esac
+}
+check_ignored qemu-img "$pid_img"
+check_ignored qemu-kvm-helper "$pid_helper"
+
+# The owner scope: the same guests, asked for as another user, are not found.
+other=root
+[ "$me" = root ] && other=nobody
+other_found=" $(pgrep -u "$other" "$PATTERN" | tr '\n' ' ' || true) "
+case "$other_found" in
+    *" $pid_kvm "* | *" $pid_x86 "*) fail "a scope of -u $other still matches guests owned by $me" ;;
+    *) pass "ignores guests owned by another user" ;;
+esac
+
+echo "-- node-reset.sh and uninstall-spx.sh at-stake counts"
+
+# counts_in <script> — the shared count helpers, between their markers.
+counts_in() {
+    sed -n '/^# --- BEGIN at-stake counts ---$/,/^# --- END at-stake counts ---$/p' "$1"
+}
+counts_in "$SCRIPT_DIR/node-reset.sh" > "$TMP/counts.sh"
+if [ ! -s "$TMP/counts.sh" ]; then
+    fail "no at-stake counts block in node-reset.sh"
+elif counts_in "$SCRIPT_DIR/uninstall-spx.sh" | cmp -s - "$TMP/counts.sh"; then
+    pass "uninstall-spx.sh carries node-reset.sh's count helpers"
+else
+    fail "uninstall-spx.sh's count helpers differ from node-reset.sh's"
+fi
+
+# count <fn> <data dir> — runs a helper with sudo as a no-op, as root would.
+count() {
+    bash -c 'sudo() { "$@"; }; . "$1"; "$2" "$3"' _ "$TMP/counts.sh" "$1" "$2"
+}
+expect() {
+    if [ "$2" = "$3" ]; then pass "$1"; else fail "$1: got '$2', want '$3'"; fi
+}
+
+# The layout a node with three guests has: one volume with its EFI store, an
+# AMI, and the system instances' records, which have no volume of their own.
+data="$TMP/data"
+mkdir -p "$data/state" "$data/viperblock/vol-a" "$data/viperblock/vol-a-efi" \
+    "$data/viperblock/vol-b" "$data/viperblock/ami-c" "$data/volumes"
+: > "$data/viperblock/viperblock.pid"
+printf '{"schema_version":1,"vms":{"i-1":{},"i-2":{},"i-3":{}}}' > "$data/state/instance-state.json"
+if command -v jq >/dev/null 2>&1; then
+    expect "counts each instance in the local state file" "$(count count_instances "$data")" 3
+    printf '{"schema_version":1,"vms":{' > "$data/state/instance-state.json"
+    expect "reports an unreadable state file as unknown, not 0" "$(count count_instances "$data")" unknown
+else
+    echo "  skip - jq is not installed; the instance counts need it"
+fi
+expect "counts mounted volumes, not EFI stores or AMIs" "$(count count_mounted_volumes "$data")" 2
+
+empty="$TMP/empty"
+mkdir -p "$empty"
+expect "counts no instances without a state file" "$(count count_instances "$empty")" 0
+expect "counts no mounted volumes without a viperblock directory" "$(count count_mounted_volumes "$empty")" 0
+
+echo "-- node-reset.sh libreswan NSS nicknames"
+
+sed -n '/^# --- BEGIN ovs nss nicknames ---$/,/^# --- END ovs nss nicknames ---$/p' \
+    "$SCRIPT_DIR/node-reset.sh" > "$TMP/nss.sh"
+# `certutil -L` as it prints a database holding an operator's own identity and
+# CA beside what ovs-monitor-ipsec imported, one peer listed twice.
+cat > "$TMP/certutil-L" <<'EOF'
+
+Certificate Nickname                                         Trust Attributes
+                                                             SSL,S/MIME,JAR/XPI
+
+ovs_certkey_node1                                            u,u,u
+ovs_cert_cacert                                              CT,,
+ovs_cert_node2                                               P,P,P
+ovs_cert_node2                                               P,P,P
+site-vpn                                                     u,u,u
+corp-ca                                                      CT,C,C
+ovs_certificate_store                                        ,,
+EOF
+if [ ! -s "$TMP/nss.sh" ]; then
+    fail "no ovs nss nicknames block in node-reset.sh"
+else
+    got=$(bash -c '. "$1"; ovs_nss_nicknames < "$2"' _ "$TMP/nss.sh" "$TMP/certutil-L" | tr '\n' ' ')
+    expect "selects only ovs-monitor-ipsec's entries, once each" "$got" "ovs_cert_cacert ovs_cert_node2 ovs_certkey_node1 "
+fi
+
+if [ "$fails" -gt 0 ]; then
+    echo "$fails failure(s)"
+    exit 1
+fi

scripts/reset-dev-env.sh

@@ -50,6 +50,8 @@ CONFIG_FILE="$ETC_DIR/spinifex.toml"
 # --- Script context ---
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+# shellcheck source=lib/distro.sh
+. "$SCRIPT_DIR/lib/distro.sh"
 
 # Resolve invoking user's HOME. $HOME is unsafe under sudo (resolves to /root).
 INVOKING_USER="${SUDO_USER:-$(id -un)}"
@@ -359,17 +361,29 @@ fi
 # --- Shutdown services ---
 echo "==> Stopping services"
 sudo systemctl stop spinifex.target 2>/dev/null || true
+# spinifex-hostnet starts with OVS, which is started below to delete bridges; it
+# would rebuild the old veth pairs. setup-ovn.sh enables it again.
+sudo systemctl disable --now spinifex-hostnet.service 2>/dev/null || true
 sudo systemctl reset-failed 'spinifex-*' 2>/dev/null || true
-sudo pkill -x qemu-system-x86_64 2>/dev/null || true
-sudo pkill -x qemu-system-aarch64 2>/dev/null || true
+# comm truncates to 15 characters, so match qemu-system-* by prefix; EL-family
+# hosts run qemu-kvm. Scoped to the user the daemon runs guests as, as in
+# node-reset.sh, so libvirt guests on the same host are left alone.
+QEMU_PGREP=('^(qemu-system|qemu-kvm$)')
+for qemu_user in spinifex-daemon spinifex "$INVOKING_USER"; do
+    if id -u "$qemu_user" >/dev/null 2>&1; then
+        QEMU_PGREP=(-u "$qemu_user" "${QEMU_PGREP[@]}")
+        break
+    fi
+done
+sudo pkill "${QEMU_PGREP[@]}" 2>/dev/null || true
 
 # Wait for QEMU to fully exit before tearing down viperblock state.
 timeout=30
 elapsed=0
-while pgrep -x 'qemu-system-x86_64|qemu-system-aarch64' > /dev/null 2>&1; do
+while pgrep "${QEMU_PGREP[@]}" > /dev/null 2>&1; do
     if [ "$elapsed" -ge "$timeout" ]; then
         echo "❌ QEMU still running after ${timeout}s:"
-        pgrep -af 'qemu-system-' || true
+        pgrep -a "${QEMU_PGREP[@]}" || true
         echo "   Kill them manually and re-run this script."
         exit 1
     fi
@@ -380,21 +394,26 @@ done
 # --- Clean OVS / OVN ---
 echo "==> Cleaning OVS bridges"
 if command -v ovs-vsctl >/dev/null 2>&1; then
-    sudo systemctl start openvswitch-switch 2>/dev/null || true
+    OVS_UNIT=$(distro_ovs_unit) || exit 1
+    sudo systemctl start "$OVS_UNIT" 2>/dev/null || true
     sleep 1
     for br in $(sudo ovs-vsctl list-br 2>/dev/null); do
         echo "  Deleting bridge: $br"
         sudo ovs-vsctl --if-exists del-br "$br"
     done
     sudo ovs-vsctl --if-exists clear Open_vSwitch . external_ids 2>/dev/null || true
-    sudo systemctl stop openvswitch-switch 2>/dev/null || true
+    sudo systemctl stop "$OVS_UNIT" 2>/dev/null || true
 fi
 
 # Delete OVN DB files outright — setup-ovn.sh will restart ovn-central with
 # fresh empty DBs. Eliminates stale SB state (chassis entries, port bindings)
 # that accumulates across resets and triggers ovn-controller commit loops.
 echo "==> Removing OVN database files"
-sudo systemctl stop ovn-central 2>/dev/null || true
+for unit in ovn-central.service ovn-northd.service; do
+    if distro_unit_exists "$unit"; then
+        sudo systemctl stop "$unit" 2>/dev/null || true
+    fi
+done
 sudo systemctl stop ovn-controller 2>/dev/null || true
 if [ -d /var/lib/ovn ]; then
     sudo rm -f /var/lib/ovn/ovnnb_db.db /var/lib/ovn/ovnsb_db.db
@@ -516,10 +535,16 @@ sleep 5
 echo "==> Building and installing microVM artifacts"
 cd "$PROJECT_ROOT" && make build-lb-agent install-microvm
 
+# By absolute path: sudo's secure_path leaves out /usr/local/bin on some hosts,
+# Oracle Linux among them, and setup.sh installs the AWS CLI there.
+AWS_BIN=$(command -v aws || true)
+[ -z "$AWS_BIN" ] && [ -x /usr/local/bin/aws ] && AWS_BIN=/usr/local/bin/aws
+[ -n "$AWS_BIN" ] || { echo "ERROR: aws CLI not found on PATH or at /usr/local/bin/aws" >&2; exit 1; }
+
 # --- Authorize SSH ingress on the default security group ---
 # Default SG denies all inbound, so smoke-test.sh's SSH probe times out.
 # Open port 22 on the default VPC's default SG before the smoke test runs.
-aws_as_user() { sudo -u "$INVOKING_USER" env HOME="$INVOKING_HOME" AWS_PROFILE=spinifex aws "$@"; }
+aws_as_user() { sudo -u "$INVOKING_USER" env HOME="$INVOKING_HOME" AWS_PROFILE=spinifex "$AWS_BIN" "$@"; }
 
 echo "==> Waiting for EC2 daemon"
 DAEMON_TIMEOUT=60

scripts/run-bench.sh

@@ -66,7 +66,7 @@ MAX_QEMU_ATTEMPTS=60
 QEMU_CMD=""
 
 while [ $QEMU_ATTEMPTS -lt $MAX_QEMU_ATTEMPTS ]; do
-    QEMU_CMD=$(ps auxw | grep "$INSTANCE_ID" | grep qemu-system | grep -v grep)
+    QEMU_CMD=$(ps auxw | grep "$INSTANCE_ID" | grep -E "qemu-system|qemu-kvm" | grep -v grep)
 
     if [ -n "$QEMU_CMD" ]; then
         break

scripts/setup-ovn_test.sh

@@ -131,7 +131,7 @@ esac
 
 # The script and the Go constant must agree: vpcd repairs a veth made before
 # this existed, and a disagreement would have the two fighting every pass.
-script_mac=$(printf '%s\n' "$nat_mode_body" | sed -n 's/.*NAT_TRANSIT_HOST_MAC="\([^"]*\)".*/\1/p' | head -1)
+script_mac=$(sed -n 's/^NAT_TRANSIT_HOST_MAC="\([^"]*\)".*/\1/p' "$SETUP" | head -1)
 go_mac=$(sed -n 's/.*NATTransitHostMAC = "\([^"]*\)".*/\1/p' "$SCRIPT_DIR/../spinifex/domains/network/host/routed.go" | head -1)
 if [ -n "$script_mac" ] && [ "$script_mac" = "$go_mac" ]; then
     pass "the transit MAC matches host.NATTransitHostMAC"
@@ -139,6 +139,413 @@ else
     fail "transit MAC disagrees: script=$script_mac go=$go_mac"
 fi
 
+# --------------------------------------------------------------------------
+# OVS and OVN service users.
+# --------------------------------------------------------------------------
+# EL runs OVS and OVN as non-root users and re-chowns their sockets on every
+# start. The function is run on its own against scratch files, through a sudo
+# that only runs its arguments.
+awk '$0 == "set_root_service_user() {" { on = 1 } on { print } on && $0 == "}" { exit }' "$SETUP" >"$TMP/user_fn.sh"
+mkdir -p "$TMP/bin"
+printf '#!/bin/sh\nexec "$@"\n' >"$TMP/bin/sudo"
+chmod +x "$TMP/bin/sudo"
+
+# set_user <file> — runs the function on <file> for OVS_USER_ID; sets RC.
+set_user() {
+    RC=0
+    PATH="$TMP/bin:$PATH" bash -c '. "$1"; set_root_service_user "$2" OVS_USER_ID' _ "$TMP/user_fn.sh" "$1" >/dev/null || RC=$?
+}
+
+set_user "$TMP/absent"
+if [ "$RC" != 0 ] && [ ! -e "$TMP/absent" ]; then pass "service user: no file, no change"; else fail "service user: absent file gave rc=$RC or was created"; fi
+
+printf '# OVS_USER_ID="openvswitch:openvswitch"\nOPTIONS=""\nOVS_USER_ID="openvswitch:hugetlbfs"\n' >"$TMP/el"
+set_user "$TMP/el"
+want='# OVS_USER_ID="openvswitch:openvswitch"
+OPTIONS=""
+OVS_USER_ID="root:root"'
+if [ "$RC" = 0 ] && [ "$(cat "$TMP/el")" = "$want" ]; then pass "service user: EL default replaced in place, comments and other lines kept"; else fail "service user: EL file rc=$RC: $(cat "$TMP/el")"; fi
+
+set_user "$TMP/el"
+if [ "$RC" != 0 ] && [ "$(cat "$TMP/el")" = "$want" ]; then pass "service user: already root reports no change"; else fail "service user: second run rc=$RC: $(cat "$TMP/el")"; fi
+
+printf 'OPTIONS=""\n' >"$TMP/unset"
+set_user "$TMP/unset"
+if [ "$RC" = 0 ] && [ "$(cat "$TMP/unset")" = 'OPTIONS=""
+OVS_USER_ID="root:root"' ]; then pass "service user: appended when the file does not set it"; else fail "service user: unset file rc=$RC: $(cat "$TMP/unset")"; fi
+
+# --------------------------------------------------------------------------
+# Host network persistence backend.
+# --------------------------------------------------------------------------
+# The backend is picked once and recorded; every later run reads the record.
+# The functions run on their own against scratch paths, with systemctl stubbed.
+for fn in select_hostnet_backend write_nm_unmanaged; do
+    awk -v f="$fn() {" '$0 == f { on = 1 } on { print } on && $0 == "}" { exit }' "$SETUP" >>"$TMP/hostnet_fn.sh"
+done
+
+# backend <networkd-enabled:yes|no> — runs select_hostnet_backend; sets RC, OUT.
+backend() {
+    printf '#!/bin/sh
+[ "$1 $2" = "is-enabled --quiet" ] && [ "%s" = yes ]
+' "$1" >"$TMP/bin/systemctl"
+    chmod +x "$TMP/bin/systemctl"
+    RC=0
+    OUT=$(PATH="$TMP/bin:$PATH" HOSTNET_BACKEND_FILE="$TMP/hn/etc/hostnet-backend" NETWORKD_DIR="$TMP/hn/network" \
+        bash -c '. "$1"; select_hostnet_backend' _ "$TMP/hostnet_fn.sh" 2>&1) || RC=$?
+}
+
+reset_hn() { rm -rf "$TMP/hn"; mkdir -p "$TMP/hn/network"; }
+
+reset_hn
+backend no
+if [ "$RC" = 0 ] && [ "$OUT" = hostnet ] && [ "$(cat "$TMP/hn/etc/hostnet-backend")" = hostnet ]; then pass "backend: no networkd picks and records hostnet"; else fail "backend: no networkd gave rc=$RC out=$OUT"; fi
+
+reset_hn
+backend yes
+if [ "$RC" = 0 ] && [ "$OUT" = networkd ] && [ "$(cat "$TMP/hn/etc/hostnet-backend")" = networkd ]; then pass "backend: networkd enabled picks networkd"; else fail "backend: networkd enabled gave rc=$RC out=$OUT"; fi
+
+reset_hn
+: >"$TMP/hn/network/10-spinifex-mgmt.network"
+backend no
+if [ "$RC" = 0 ] && [ "$OUT" = networkd ]; then pass "backend: legacy Spinifex networkd files keep networkd"; else fail "backend: legacy files gave rc=$RC out=$OUT"; fi
+
+reset_hn
+mkdir -p "$TMP/hn/etc"
+echo hostnet >"$TMP/hn/etc/hostnet-backend"
+backend yes
+if [ "$RC" = 0 ] && [ "$OUT" = hostnet ]; then pass "backend: the record wins over networkd being enabled"; else fail "backend: recorded hostnet gave rc=$RC out=$OUT"; fi
+
+echo ifupdown >"$TMP/hn/etc/hostnet-backend"
+backend yes
+case "$RC:$OUT" in
+    0:*) fail "backend: an unknown record was accepted: $OUT" ;;
+    *"expected networkd or hostnet"*) pass "backend: an unknown record is refused" ;;
+    *) fail "backend: an unknown record gave rc=$RC out=$OUT" ;;
+esac
+
+# --------------------------------------------------------------------------
+# NetworkManager unmanaged config.
+# --------------------------------------------------------------------------
+# NetworkManager globs have no character classes, so the management TAPs go to
+# a udev rule; the keyfile lists the rest. The host uplink bridge stays managed.
+printf '#!/bin/sh
+exit 0
+' >"$TMP/bin/udevadm"
+cp "$TMP/bin/udevadm" "$TMP/bin/nmcli"
+chmod +x "$TMP/bin/udevadm" "$TMP/bin/nmcli"
+glob=$(sed -n "s/^MGMT_TAP_GLOB='\([^']*\)'$/\1/p" "$SETUP")
+RC=0
+PATH="$TMP/bin:$PATH" NM_UNMANAGED_CONF="$TMP/nm/conf.d/00-spinifex-unmanaged.conf" NM_UNMANAGED_RULE="$TMP/nm/rules.d/90.rules" \
+    MGMT_TAP_GLOB="$glob" WAN_BRIDGE=br-ext WAN_BRIDGE_MODE=direct WAN_IFACE=eth3 LINUX_BRIDGE=br-wan \
+    MGMT_BRIDGE_ENABLED=true MGMT_BRIDGE=br-mgmt MGMT_IFACE=eth2 \
+    bash -c '. "$1"; write_nm_unmanaged' _ "$TMP/hostnet_fn.sh" >/dev/null 2>&1 || RC=$?
+conf=$(cat "$TMP/nm/conf.d/00-spinifex-unmanaged.conf" 2>/dev/null || true)
+line=$(printf '%s\n' "$conf" | grep '^unmanaged-devices+=' || true)
+if [ "$RC" = 0 ] && printf '%s\n' "$conf" | grep -qx '\[keyfile\]' && [ -n "$line" ]; then pass "nm: keyfile section appends to unmanaged-devices"; else fail "nm: rc=$RC conf=$conf"; fi
+for want in br-int br-ext 'veth-wan-*' 'spx-nat-*' 'tap*' br-mgmt eth2 eth3; do
+    case ";${line#unmanaged-devices+=};" in
+        *";interface-name:$want;"*) pass "nm: $want unmanaged" ;;
+        *) fail "nm: $want missing from $line" ;;
+    esac
+done
+case "$line" in
+    *br-wan*) fail "nm: the host uplink bridge br-wan was made unmanaged" ;;
+    *'['*) fail "nm: keyfile uses a character class NetworkManager cannot match: $line" ;;
+    *) pass "nm: host uplink left managed and no character classes" ;;
+esac
+rule=$(cat "$TMP/nm/rules.d/90.rules" 2>/dev/null || true)
+case "$rule" in
+    *"KERNEL==\"$glob\", ENV{NM_UNMANAGED}=\"1\""*) pass "nm: udev rule unmanages management TAPs" ;;
+    *) fail "nm: udev rule is $rule" ;;
+esac
+
+# --------------------------------------------------------------------------
+# OVN central options file and management role.
+# --------------------------------------------------------------------------
+# The functions run on their own against scratch paths, through the stub sudo.
+for fn in write_ovn_central_opts record_management_role management_role_markers cap_logrotate; do
+    awk -v f="$fn() {" '$0 == f { on = 1 } on { print } on && $0 == "}" { exit }' "$SETUP" >>"$TMP/ovn_fn.sh"
+done
+
+# ovn_fn <cmd>... — runs the functions against scratch role files; sets RC, OUT.
+ovn_fn() {
+    RC=0
+    OUT=$(PATH="$TMP/bin:$PATH" OVN_ROLE_FILE="$TMP/role/etc/spinifex/ovn-role" \
+        LEGACY_OVN_ROLE_FILE="$TMP/role/etc/default/ovn-central" \
+        bash -c '. "$1"; shift; "$@"' _ "$TMP/ovn_fn.sh" "$@" 2>&1) || RC=$?
+}
+
+ovn_fn env OVN_CENTRAL_OPTS_FILE="$TMP/sysconfig-ovn-northd" OVN_CENTRAL_OPTS_VAR=OVN_NORTHD_OPTS \
+    bash -c '. "$1"; write_ovn_central_opts "--db-nb-addr=10.0.0.1 --db-nb-create-insecure-remote=yes"' _ "$TMP/ovn_fn.sh"
+if [ "$RC" = 0 ] && [ "$(cat "$TMP/sysconfig-ovn-northd")" = 'OVN_NORTHD_OPTS="--db-nb-addr=10.0.0.1 --db-nb-create-insecure-remote=yes"' ]; then
+    pass "opts: EL file holds OVN_NORTHD_OPTS"
+else
+    fail "opts: rc=$RC $(cat "$TMP/sysconfig-ovn-northd" 2>/dev/null)"
+fi
+
+rm -rf "$TMP/role"
+ovn_fn management_role_markers
+if [ "$RC" != 0 ] && [ -z "$OUT" ]; then pass "role: a fresh node has no marker"; else fail "role: fresh node rc=$RC out=$OUT"; fi
+
+ovn_fn record_management_role
+ovn_fn management_role_markers
+if [ "$RC" = 0 ] && [ "$OUT" = "$TMP/role/etc/spinifex/ovn-role" ]; then pass "role: recorded marker is found"; else fail "role: recorded marker rc=$RC out=$OUT"; fi
+
+mkdir -p "$TMP/role/etc/default"
+echo 'OVN_CTL_OPTS=""' >"$TMP/role/etc/default/ovn-central"
+ovn_fn management_role_markers
+if [ "$RC" = 0 ] && [ "$OUT" = "$TMP/role/etc/spinifex/ovn-role
+$TMP/role/etc/default/ovn-central" ]; then pass "role: both markers are named, so the demotion hint removes both"; else fail "role: both markers rc=$RC out=$OUT"; fi
+
+rm -f "$TMP/role/etc/spinifex/ovn-role"
+ovn_fn management_role_markers
+if [ "$RC" = 0 ] && [ "$OUT" = "$TMP/role/etc/default/ovn-central" ]; then pass "role: the legacy Debian marker alone still counts"; else fail "role: legacy marker rc=$RC out=$OUT"; fi
+
+# The package's own file sets only commented defaults; it is not a marker.
+printf '# OVN_CTL_OPTS="--db-nb-addr=0.0.0.0"\n' >"$TMP/role/etc/default/ovn-central"
+echo compute >"$TMP/role/etc/spinifex/ovn-role"
+ovn_fn management_role_markers
+if [ "$RC" != 0 ]; then pass "role: a package-default options file and another role are not markers"; else fail "role: non-markers gave rc=$RC out=$OUT"; fi
+
+# --------------------------------------------------------------------------
+# OVS and OVN log rotation.
+# --------------------------------------------------------------------------
+# Fixtures are the stanzas the Debian 13 and NFV SIG el10s packages ship.
+mkdir -p "$TMP/lr"
+cat >"$TMP/lr/ovn-common" <<'EOF'
+/var/log/ovn/*.log {
+    su root root
+    daily
+    compress
+    sharedscripts
+    missingok
+}
+EOF
+cat >"$TMP/lr/openvswitch-switch" <<'EOF'
+/var/log/openvswitch/*.log {
+    daily
+    compress
+    sharedscripts
+    create 640 root adm
+    missingok
+    rotate 30
+}
+EOF
+cat >"$TMP/lr/ovn" <<'EOF'
+/var/log/ovn/*.log {
+    su openvswitch openvswitch
+    daily
+    compress
+    sharedscripts
+    missingok
+}
+EOF
+printf '/var/log/other.log {\n    daily\n}\n' >"$TMP/lr/odd"
+
+# directives <file> — the stanza's directives, sorted, one per line.
+directives() { sed -n 's/^    //p' "$1" | sort; }
+
+ovn_fn cap_logrotate "$TMP/lr/ovn-common"
+if [ "$RC" = 0 ] && [ "$(directives "$TMP/lr/ovn-common")" = "$(printf 'compress\ndaily\nmaxsize 100M\nmissingok\nrotate 5\nsharedscripts\nsu root root\n')" ]; then
+    pass "logrotate: Debian OVN gets rotate 5 and maxsize, as before"
+else
+    fail "logrotate: Debian OVN rc=$RC: $(cat "$TMP/lr/ovn-common")"
+fi
+
+ovn_fn cap_logrotate "$TMP/lr/openvswitch-switch"
+if [ "$RC" = 0 ] && grep -qx '    maxsize 100M' "$TMP/lr/openvswitch-switch" &&
+    [ "$(grep -c '^ *rotate ' "$TMP/lr/openvswitch-switch")" = 1 ] && grep -qx '    rotate 30' "$TMP/lr/openvswitch-switch"; then
+    pass "logrotate: Debian OVS keeps its rotate 30 and gains maxsize"
+else
+    fail "logrotate: Debian OVS rc=$RC: $(cat "$TMP/lr/openvswitch-switch")"
+fi
+
+ovn_fn cap_logrotate "$TMP/lr/ovn"
+if [ "$RC" = 0 ] && [ "$(directives "$TMP/lr/ovn")" = "$(printf 'compress\ndaily\nmaxsize 100M\nmissingok\nrotate 5\nsharedscripts\nsu root root\n')" ]; then
+    pass "logrotate: EL OVN rotates as root, with rotate 5 and maxsize"
+else
+    fail "logrotate: EL OVN rc=$RC: $(cat "$TMP/lr/ovn")"
+fi
+
+before=$(cat "$TMP/lr/ovn")
+ovn_fn cap_logrotate "$TMP/lr/ovn"
+if [ "$RC" = 0 ] && [ "$(cat "$TMP/lr/ovn")" = "$before" ]; then pass "logrotate: a rerun changes nothing"; else fail "logrotate: rerun changed $TMP/lr/ovn: $(cat "$TMP/lr/ovn")"; fi
+
+ovn_fn cap_logrotate "$TMP/lr/odd"
+case "$RC:$OUT" in
+    0:*"left uncapped"*) pass "logrotate: a file without the expected stanza warns" ;;
+    *) fail "logrotate: unexpected stanza gave rc=$RC out=$OUT" ;;
+esac
+
+# --------------------------------------------------------------------------
+# Clustered DB storage, and what --recreate-db purges.
+# --------------------------------------------------------------------------
+# install-node.sh passes --recreate-db on every database node on every run, so
+# re-forming a cluster has to replace a clustered DB as readily as a standalone
+# one. Left in place, ovn-ctl serves it, ignores the RAFT flags and rejoins the
+# previous cluster under the member list the old DB carries.
+for fn in stop_ovn_databases ensure_clustered_db_storage; do
+    awk -v f="$fn() {" '$0 == f { on = 1 } on { print } on && $0 == "}" { exit }' "$SETUP" >>"$TMP/db_fn.sh"
+done
+
+# db-is-clustered reports on the fixture's own contents, so a test writes
+# "clustered" or "standalone" into the file to choose the format.
+printf '#!/bin/sh\n[ "$1" = db-is-clustered ] || exit 2\ngrep -q clustered "$2"\n' >"$TMP/bin/ovsdb-tool"
+printf '#!/bin/sh\nexit 0\n' >"$TMP/bin/systemctl"
+printf '#!/bin/sh\nexit 1\n' >"$TMP/bin/pgrep"
+printf '#!/bin/sh\necho "pkill $*" >>"%s"\nexit 0\n' "$TMP/pkill.log" >"$TMP/bin/pkill"
+chmod +x "$TMP/bin/ovsdb-tool" "$TMP/bin/systemctl" "$TMP/bin/pgrep" "$TMP/bin/pkill"
+
+# db_fixture <format> — a DB directory in <format> with every leftover present.
+db_fixture() {
+    rm -rf "$TMP/dbdir"
+    mkdir -p "$TMP/dbdir/standalone-backup"
+    for f in ovnnb_db.db ovnsb_db.db; do
+        echo "$1" >"$TMP/dbdir/$f"
+        echo "$1" >"$TMP/dbdir/$f.standalone-disabled"
+        echo "$1" >"$TMP/dbdir/standalone-backup/$f"
+        : >"$TMP/dbdir/.$f.~lock~"
+    done
+}
+
+# ensure_db <recreate:true|false> — runs ensure_clustered_db_storage; sets RC, OUT.
+ensure_db() {
+    RC=0
+    OUT=$(PATH="$TMP/bin:$PATH" bash -c '
+        OVN_DBDIR="$1"; RECREATE_DB="$2"; OVN_CENTRAL_UNITS=(ovn-central)
+        OVN_DB_STOP_WAIT=2; OVN_DB_KILL_WAIT=1
+        . "$3"; ensure_clustered_db_storage' _ "$TMP/dbdir" "$1" "$TMP/db_fn.sh" 2>&1) || RC=$?
+}
+
+db_fixture clustered
+ensure_db true
+if [ "$RC" = 0 ] && [ ! -e "$TMP/dbdir/ovnnb_db.db" ] && [ ! -e "$TMP/dbdir/ovnsb_db.db" ]; then
+    pass "recreate-db: a clustered DB is discarded, so re-forming does not rejoin the old cluster"
+else
+    fail "recreate-db: clustered DB rc=$RC left [$(ls -A "$TMP/dbdir")] out=$OUT"
+fi
+if [ -z "$(ls -A "$TMP/dbdir")" ]; then
+    pass "recreate-db: the lock files, move-asides and standalone backup go with it"
+else
+    fail "recreate-db: leftovers survived: $(ls -A "$TMP/dbdir" | tr '\n' ' ')"
+fi
+
+db_fixture standalone
+ensure_db true
+if [ "$RC" = 0 ] && [ ! -e "$TMP/dbdir/ovnnb_db.db" ]; then
+    pass "recreate-db: a standalone DB is discarded as before"
+else
+    fail "recreate-db: standalone DB rc=$RC left [$(ls -A "$TMP/dbdir")] out=$OUT"
+fi
+
+db_fixture clustered
+ensure_db false
+if [ "$RC" = 0 ] && [ -e "$TMP/dbdir/ovnnb_db.db" ]; then
+    pass "no recreate-db: a clustered DB is the steady state and is kept"
+else
+    fail "no recreate-db: clustered DB rc=$RC left [$(ls -A "$TMP/dbdir")] out=$OUT"
+fi
+
+db_fixture standalone
+ensure_db false
+case "$RC:$OUT" in
+    0:*) fail "no recreate-db: a standalone DB was accepted, so the RAFT flags would be ignored" ;;
+    *:*"standalone format"*) pass "no recreate-db: a standalone DB is refused by name" ;;
+    *) fail "no recreate-db: standalone DB rc=$RC out=$OUT" ;;
+esac
+
+# --------------------------------------------------------------------------
+# A rerun returns only once both of this node's RAFT members have rejoined.
+# --------------------------------------------------------------------------
+# A rerun restarts the node's NB and SB members together, so returning early
+# lets the next database node's rerun take a second member of each down.
+awk '$0 == "wait_for_raft_member() {" { on = 1 } on { print } on && $0 == "}" { exit }' "$SETUP" >"$TMP/raft_fn.sh"
+mkdir -p "$TMP/raftbin"
+printf '#!/bin/sh\nexit 0\n' >"$TMP/raftbin/sleep"
+printf '#!/bin/sh\n[ -f "%s" ] || exit 1\ncat "%s"\n' "$TMP/cluster-status" "$TMP/cluster-status" >"$TMP/raftbin/ovs-appctl"
+chmod +x "$TMP/raftbin/sleep" "$TMP/raftbin/ovs-appctl"
+
+# raft_member <Status:> <Leader:> — runs wait_for_raft_member against a
+# cluster/status in that state, or none when Status is "absent"; sets RC.
+raft_member() {
+    rm -f "$TMP/cluster-status"
+    if [ "$1" != absent ]; then
+        printf 'a1b2\nName: OVN_Northbound\nStatus: %s\nRole: follower\nTerm: 4\nLeader: %s\nVote: c3d4\n\nServers:\n    a1b2 (a1b2 at tcp:10.0.0.1:6643) (self)\n    c3d4 (c3d4 at tcp:10.0.0.2:6643)\n' \
+            "$1" "$2" >"$TMP/cluster-status"
+    fi
+    RC=0
+    PATH="$TMP/raftbin:$PATH" bash -c 'sudo() { "$@"; }; OVN_RAFT_REJOIN_WAIT=2; . "$1"
+        wait_for_raft_member /var/run/ovn/ovnnb_db.ctl OVN_Northbound' _ "$TMP/raft_fn.sh" >/dev/null 2>&1 || RC=$?
+}
+
+if [ ! -s "$TMP/raft_fn.sh" ]; then
+    fail "raft: no wait_for_raft_member in setup-ovn.sh"
+else
+    raft_member "cluster member" c3d4
+    if [ "$RC" = 0 ]; then pass "raft: a member following a known leader has rejoined"; else fail "raft: follower with leader rc=$RC"; fi
+    raft_member "cluster member" self
+    if [ "$RC" = 0 ]; then pass "raft: a member that is the leader has rejoined"; else fail "raft: leader rc=$RC"; fi
+    raft_member "cluster member" unknown
+    if [ "$RC" != 0 ]; then pass "raft: a member that has not found its leader has not rejoined"; else fail "raft: unknown leader accepted"; fi
+    raft_member "disconnected from the cluster (election timeout)" c3d4
+    if [ "$RC" != 0 ]; then pass "raft: a disconnected member that remembers a leader has not rejoined"; else fail "raft: disconnected member accepted"; fi
+    raft_member "joining cluster" c3d4
+    if [ "$RC" != 0 ]; then pass "raft: a member still joining has not rejoined"; else fail "raft: joining member accepted"; fi
+    raft_member absent ""
+    if [ "$RC" != 0 ]; then pass "raft: a server that does not answer has not rejoined"; else fail "raft: unanswered cluster/status accepted"; fi
+fi
+
+rm -rf "$TMP/dbdir"
+mkdir -p "$TMP/dbdir"
+ensure_db true
+if [ "$RC" = 0 ]; then pass "recreate-db: an empty DB directory is already what it wants"; else fail "recreate-db: empty directory rc=$RC out=$OUT"; fi
+
+# A server that outlives its unit must be killed by a pattern anchored on the
+# OVN database directory. Matching ovsdb-server alone would reap the OVS
+# conf.db server, which runs the switch.
+db_fixture clustered
+printf '#!/bin/sh\nexit 0\n' >"$TMP/bin/pgrep"
+: >"$TMP/pkill.log"
+ensure_db true
+if grep -q -- "$TMP/dbdir/ovn" "$TMP/pkill.log" 2>/dev/null; then
+    pass "recreate-db: a surviving server is killed by a pattern scoped to the OVN database directory"
+else
+    fail "recreate-db: pkill was called as [$(cat "$TMP/pkill.log" 2>/dev/null)]"
+fi
+if [ "$RC" != 0 ] && [ -e "$TMP/dbdir/ovnnb_db.db" ]; then
+    pass "recreate-db: a server that survives SIGKILL fails the run rather than unlinking under it"
+else
+    fail "recreate-db: unkillable server gave rc=$RC and left [$(ls -A "$TMP/dbdir")]"
+fi
+printf '#!/bin/sh\nexit 1\n' >"$TMP/bin/pgrep"
+
+# A rerun restarts this node's members, which can hand the leadership to a
+# peer, so the create node's set-connection must still land as a follower.
+for fn in db_clients set_client_connections; do
+    awk -v f="$fn() {" '$0 == f { on = 1 } on { print } on && $0 == "}" { exit }' "$SETUP" >>"$TMP/conn_fn.sh"
+done
+mkdir -p "$TMP/followerbin"
+for ctl in ovn-nbctl ovn-sbctl; do
+    cat >"$TMP/followerbin/$ctl" <<EOF
+#!/bin/sh
+case " \$* " in
+    *" --no-leader-only "*) echo "$ctl \$*" >>"$TMP/conn.log" ;;
+    *) echo "$ctl: unix:/var/run/ovn/db.sock: database connection failed ()" >&2; exit 1 ;;
+esac
+EOF
+    chmod +x "$TMP/followerbin/$ctl"
+done
+: >"$TMP/conn.log"
+RC=0
+PATH="$TMP/followerbin:$PATH" bash -c 'set -e; sudo() { "$@"; }; DB_CLUSTER_LOCAL_ADDR=10.0.0.1; . "$1"
+    db_clients; set_client_connections' _ "$TMP/conn_fn.sh" >/dev/null 2>&1 || RC=$?
+if [ "$RC" = 0 ] && [ "$(cat "$TMP/conn.log")" = "ovn-nbctl --db=tcp:10.0.0.1:6641 --no-leader-only set-connection ptcp:6641:127.0.0.1
+ovn-sbctl --db=tcp:10.0.0.1:6642 --no-leader-only set-connection ptcp:6642:127.0.0.1" ]; then
+    pass "connections: the create node sets the client listeners while a follower"
+else
+    fail "connections: as a follower rc=$RC calls [$(cat "$TMP/conn.log")]"
+fi
+
 if [ "$fails" -ne 0 ]; then
     echo "  setup-ovn.sh: $fails failure(s)"
     exit 1

scripts/setup_firewall_test.sh

@@ -0,0 +1,206 @@
+#!/bin/sh
+# setup_firewall_test.sh — tests for how setup.sh picks the firewall mode and
+# masks firewalld.
+#
+# setup.sh is sourced with INSTALL_SPINIFEX_LIB_ONLY=1 so main never runs. A stub
+# systemctl keeps firewalld's state in fixture files, and the mode file is a
+# fixture path.
+set -eu
+
+SCRIPT_DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
+SETUP="$SCRIPT_DIR/setup.sh"
+TMP=$(mktemp -d)
+trap 'rm -rf "$TMP"' EXIT
+
+fails=0
+pass() { echo "  ok   - $1"; }
+fail() { echo "  FAIL - $1"; fails=$((fails + 1)); }
+
+echo "-- setup.sh firewall mode and firewalld"
+
+mkdir -p "$TMP/bin"
+cat > "$TMP/bin/systemctl" <<EOF
+#!/bin/sh
+state="$TMP/state"
+[ "\$1" = "--quiet" ] && shift
+case "\$1 \$2" in
+    "is-active --quiet") [ -e "\$state/active" ] ;;
+    "is-enabled firewalld.service") [ -e "\$state/enabled" ] || { echo not-found; exit 4; }; cat "\$state/enabled" ;;
+    "mask --now")
+        echo "mask" >> "\$state/calls"
+        [ -e "\$state/fail-mask" ] && exit 1
+        echo masked > "\$state/enabled"; rm -f "\$state/active" ;;
+    *) echo "unexpected systemctl \$*" >&2; exit 2 ;;
+esac
+EOF
+chmod +x "$TMP/bin/systemctl"
+
+# host <firewalld: none|inactive|active|masked> [recorded mode] — sets up the
+# fixture host. The mode file is absent when no mode is given.
+host() {
+    rm -rf "$TMP/state" "$TMP/mode"
+    mkdir -p "$TMP/state"
+    case "$1" in
+        inactive) echo disabled > "$TMP/state/enabled" ;;
+        active) echo enabled > "$TMP/state/enabled"; : > "$TMP/state/active" ;;
+        masked) echo masked > "$TMP/state/enabled" ;;
+    esac
+    if [ -n "${2:-}" ]; then printf '%s\n' "$2" > "$TMP/mode"; fi
+}
+
+# run <function...> [--firewall value via FW] — sets RC and OUT, and MODE to
+# the mode setup.sh resolved.
+run() {
+    RC=0
+    OUT=$(PATH="$TMP/bin:$PATH" SPINIFEX_FIREWALL_MODE_FILE="$TMP/mode" INSTALL_SPINIFEX_FIREWALL="${FW:-}" \
+        INSTALL_SPINIFEX_LIB_ONLY=1 bash -c '. "$1"; shift; for f in "$@"; do "$f"; done; echo "MODE=$INSTALL_SPINIFEX_FIREWALL"' \
+        _ "$SETUP" "$@" 2>&1) || RC=$?
+    MODE=$(printf '%s\n' "$OUT" | sed -n 's/^MODE=//p')
+    unset FW
+}
+
+resolves() { # resolves <desc> <mode>
+    if [ "$RC" = 0 ] && [ "$MODE" = "$2" ]; then pass "$1 resolves to $2"; else fail "$1: rc=$RC mode='$MODE': $OUT"; fi
+}
+
+host none
+run resolve_firewall_mode
+resolves "no firewalld, no recorded mode" off
+
+host inactive
+run resolve_firewall_mode
+resolves "firewalld installed but inactive" off
+
+host active
+run resolve_firewall_mode
+resolves "firewalld active" on
+
+host masked on
+run resolve_firewall_mode
+resolves "a rerun after an install with --firewall=on" on
+
+host active on
+run resolve_firewall_mode
+resolves "a retry after the mode file was written but before firewalld was masked" on
+
+host active off
+run resolve_firewall_mode
+resolves "a recorded 'off' with firewalld active" off
+
+host masked on
+FW=off run resolve_firewall_mode
+resolves "an explicit --firewall=off over a recorded 'on'" off
+
+host none on
+FW=off run resolve_firewall_mode
+resolves "an explicit --firewall=off" off
+
+host none
+FW=maybe run resolve_firewall_mode
+case "$RC:$OUT" in
+    0:*) fail "accepts --firewall=maybe" ;;
+    *"--firewall must be"*) pass "rejects --firewall=maybe" ;;
+    *) fail "rejects --firewall=maybe without saying why: $OUT" ;;
+esac
+
+host none garbage
+run resolve_firewall_mode
+case "$RC:$OUT" in
+    0:*) fail "accepts a mode file holding 'garbage'" ;;
+    *"holds 'garbage'"*) pass "rejects a mode file holding 'garbage', naming it" ;;
+    *) fail "rejects a bad mode file without naming its content: $OUT" ;;
+esac
+
+# The record sits under /etc/spinifex, which only root and the spinifex group
+# can enter, so an installer run by any other sudoer must read it through sudo.
+cat > "$TMP/bin/sudo" <<EOF
+#!/bin/sh
+echo "\$*" >> "$TMP/sudo-calls"
+exec "\$@"
+EOF
+chmod +x "$TMP/bin/sudo"
+host masked on
+rm -f "$TMP/sudo-calls"
+RC=0
+OUT=$(PATH="$TMP/bin:$PATH" SUDO=sudo SPINIFEX_FIREWALL_MODE_FILE="$TMP/mode" INSTALL_SPINIFEX_FIREWALL="" \
+    INSTALL_SPINIFEX_LIB_ONLY=1 bash -c '. "$1"; resolve_firewall_mode; echo "MODE=$INSTALL_SPINIFEX_FIREWALL"' \
+    _ "$SETUP" 2>&1) || RC=$?
+MODE=$(printf '%s\n' "$OUT" | sed -n 's/^MODE=//p')
+if [ "$RC" = 0 ] && [ "$MODE" = on ] && grep -qx "cat $TMP/mode" "$TMP/sudo-calls" 2>/dev/null; then
+    pass "a non-root rerun reads the recorded mode through sudo"
+else
+    fail "a non-root rerun did not read the recorded mode through sudo: rc=$RC mode='$MODE' calls='$(cat "$TMP/sudo-calls" 2>/dev/null)': $OUT"
+fi
+
+# A record that exists but cannot be read must stop the install, never fall
+# through to the firewalld check, which reads a masked firewalld as "off".
+if [ "$(id -u)" != 0 ]; then
+    host masked on
+    chmod 000 "$TMP/mode"
+    run resolve_firewall_mode
+    chmod 644 "$TMP/mode"
+    case "$RC:$OUT" in
+        0:*) fail "an unreadable mode file resolved to '$MODE'" ;;
+        *"Cannot read the recorded firewall mode"*) pass "an unreadable mode file stops the install" ;;
+        *) fail "an unreadable mode file stopped without saying why: $OUT" ;;
+    esac
+fi
+
+masks() { # masks <desc> <expect: masked|untouched>
+    calls=$(cat "$TMP/state/calls" 2>/dev/null || true)
+    if [ "$RC" != 0 ]; then
+        fail "$1: rc=$RC: $OUT"
+    elif [ "$2" = masked ] && [ -n "$calls" ] && [ "$(cat "$TMP/state/enabled")" = masked ] && [ ! -e "$TMP/state/active" ]; then
+        pass "$1: masked and stopped"
+    elif [ "$2" = untouched ] && [ -z "$calls" ]; then
+        pass "$1: left alone"
+    else
+        fail "$1: expected $2, calls='$calls': $OUT"
+    fi
+}
+
+host active
+run resolve_firewall_mode mask_firewalld
+masks "firewalld active" masked
+case "$OUT" in
+    *"Restate any other restriction"*) pass "masking firewalld says its rules were not migrated" ;;
+    *) fail "masking firewalld printed no notice: $OUT" ;;
+esac
+
+host inactive
+run mask_firewalld
+masks "firewalld installed but inactive" masked
+
+host none
+run mask_firewalld
+masks "no firewalld" untouched
+
+host masked
+run mask_firewalld
+masks "firewalld already masked" untouched
+
+host active
+echo masked > "$TMP/state/enabled"
+run mask_firewalld
+masks "firewalld masked but still running" masked
+
+host active
+: > "$TMP/state/fail-mask"
+run mask_firewalld
+case "$RC:$OUT" in
+    0:*) fail "succeeds when firewalld cannot be masked" ;;
+    *"Cannot stop and mask firewalld"*) pass "fails by name when firewalld cannot be masked" ;;
+    *) fail "fails without naming firewalld: $OUT" ;;
+esac
+
+host active
+FW=off run mask_firewalld
+case "$OUT" in
+    *"this host now has no firewall"*) pass "masking firewalld under --firewall=off says the host has no firewall" ;;
+    *) fail "masking under --firewall=off did not say the host has no firewall: $OUT" ;;
+esac
+
+if [ "$fails" -gt 0 ]; then
+    echo "$fails failure(s)"
+    exit 1
+fi

scripts/setup_host_test.sh

@@ -0,0 +1,272 @@
+#!/bin/sh
+# setup_host_test.sh — tests for setup.sh's DNS handover to systemd-resolved,
+# its chrony drop-in and its nbdkit plugin directory.
+#
+# setup.sh is sourced with INSTALL_SPINIFEX_LIB_ONLY=1 so main never runs. PATH
+# holds stub commands and links to a few real tools only, so no host resolver,
+# package manager or service is ever touched.
+# shellcheck disable=SC2016  # stub bodies are single-quoted so they expand when the stub runs
+set -eu
+
+SCRIPT_DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
+SETUP="$SCRIPT_DIR/setup.sh"
+TMP=$(mktemp -d)
+trap 'rm -rf "$TMP"' EXIT
+
+fails=0
+pass() { echo "  ok   - $1"; }
+fail() { echo "  FAIL - $1"; fails=$((fails + 1)); }
+
+echo "-- setup.sh host DNS, chrony and nbdkit"
+
+mkdir -p "$TMP/sys"
+for tool in bash awk sed grep readlink basename dirname cp ln mkdir cat head tail wc tee rm chmod; do
+    ln -s "$(command -v "$tool")" "$TMP/sys/$tool"
+done
+
+# stub <name> <body> — a stub command; $S is the fixture's state directory.
+stub() {
+    printf '#!/bin/sh\nS="%s"\n%s\n' "$TMP/state" "$2" > "$TMP/bin/$1"
+    chmod +x "$TMP/bin/$1"
+}
+
+# host — a fresh fixture: resolved installed and inactive, resolvconf the
+# resolvectl link, NetworkManager running with no DNS settings, a dnf host
+# whose first enabled repository is yum.oracle.com.
+host() {
+    rm -rf "${TMP:?}/state" "${TMP:?}/root" "${TMP:?}/bin"
+    mkdir -p "$TMP/state/active" "$TMP/root/etc" "$TMP/root/usr/lib/systemd/system" "$TMP/bin"
+    : > "$TMP/root/usr/lib/systemd/system/systemd-resolved.service"
+    : > "$TMP/state/active/NetworkManager.service"
+    printf 'nameserver 192.168.1.1\nsearch example.lan\n' > "$TMP/root/etc/resolv.conf"
+    printf '[main]\n# dns=default\nconfigure-and-quit=no\n' > "$TMP/state/nm-config"
+    printf 'Global:\nLink 2 (br-wan): 192.168.1.1\n' > "$TMP/state/resolvectl-dns"
+    printf '[ol10_UEKR8]\nbaseurl = https://mirror.example/uek/\nenabled = 0\n[ol10_baseos_latest]\nbaseurl = https://yum.oracle.com/repo/OracleLinux/OL10/baseos/latest/x86_64/\nenabled = 1\n' > "$TMP/state/dnf-dump"
+    stub systemctl '[ "$1" = is-active ] && { [ -e "$S/active/$3" ]; exit; }
+echo "$*" >> "$S/calls"
+[ "$1 $2" = "enable --now" ] && : > "$S/active/$3"
+exit 0'
+    stub NetworkManager 'cat "$S/nm-config"'
+    stub resolvectl '[ "$1" = dns ] && cat "$S/resolvectl-dns"'
+    stub resolvectl-real ':'
+    ln -s resolvectl "$TMP/bin/resolvconf"
+    stub getent '[ ! -e "$S/getent-fail" ] && echo "$2" >> "$S/looked-up"'
+    stub dnf '[ "$1 $2" = "config-manager --dump" ] && cat "$S/dnf-dump"'
+    stub sleep ':'
+}
+
+# run <function...> — runs setup.sh functions against the fixture. Sets RC and OUT.
+run() {
+    RC=0
+    OUT=$(PATH="$TMP/bin:$TMP/sys" DISTRO_ROOT="$TMP/root" \
+        SPINIFEX_RESOLV_CONF="$TMP/root/etc/resolv.conf" SPINIFEX_RESOLV_CONF_SAVED="$TMP/root/saved/resolv.conf.orig" \
+        INSTALL_SPINIFEX_LIB_ONLY=1 bash -c '. "$1"; shift; for f in "$@"; do $f; done' _ "$SETUP" "$@" 2>&1) || RC=$?
+}
+
+calls() { cat "$TMP/state/calls" 2>/dev/null || true; }
+saved() { cat "$TMP/root/saved/resolv.conf.orig" 2>/dev/null || echo "<none>"; }
+expect_untouched() {
+    if [ "$RC" = 0 ] && [ -z "$(calls)" ] && [ ! -L "$TMP/root/etc/resolv.conf" ] && [ "$(saved)" = "<none>" ]; then
+        pass "$1"
+    else
+        fail "$1: rc=$RC calls='$(calls)' saved='$(saved)': $OUT"
+    fi
+}
+expect_fatal() { # expect_fatal <desc> <text>...
+    desc=$1; shift
+    if [ "$RC" = 0 ]; then fail "$desc (succeeded): $OUT"; return; fi
+    for want in "$@"; do
+        case "$OUT" in *"$want"*) ;; *) fail "$desc (output lacks '$want'): $OUT"; return ;; esac
+    done
+    pass "$desc"
+}
+
+# --------------------------------------------------------------------------
+# DNS handover.
+# --------------------------------------------------------------------------
+host
+run hand_dns_to_resolved
+if [ "$RC" = 0 ] && [ "$(readlink "$TMP/root/etc/resolv.conf")" = /run/systemd/resolve/stub-resolv.conf ] \
+    && [ "$(saved)" = "$(printf 'nameserver 192.168.1.1\nsearch example.lan')" ] \
+    && [ "$(calls)" = "$(printf 'enable --now systemd-resolved.service\nrestart NetworkManager.service')" ] \
+    && [ "$(cat "$TMP/state/looked-up")" = yum.oracle.com ]; then
+    pass "hands DNS to resolved: saves resolv.conf, starts resolved, restarts NM, links the stub, looks up the repo host"
+else
+    fail "handover: rc=$RC calls='$(calls)' saved='$(saved)' link='$(readlink "$TMP/root/etc/resolv.conf")': $OUT"
+fi
+
+: > "$TMP/state/calls"
+rm -f "$TMP/state/looked-up"
+run hand_dns_to_resolved
+if [ "$RC" = 0 ] && [ -z "$(calls)" ] && [ "$(cat "$TMP/state/looked-up")" = yum.oracle.com ]; then
+    pass "a rerun after a handover checks DNS again and changes nothing"
+else
+    fail "rerun after handover: rc=$RC calls='$(calls)': $OUT"
+fi
+
+host
+mkdir -p "$TMP/root/saved"
+echo "the original" > "$TMP/root/saved/resolv.conf.orig"
+run hand_dns_to_resolved
+if [ "$RC" = 0 ] && [ "$(saved)" = "the original" ]; then
+    pass "a rerun never overwrites the saved resolv.conf"
+else
+    fail "saved copy overwritten: rc=$RC saved='$(saved)': $OUT"
+fi
+
+host
+rm "$TMP/root/usr/lib/systemd/system/systemd-resolved.service"
+run hand_dns_to_resolved
+expect_untouched "resolved not installed: DNS is left alone"
+
+host
+: > "$TMP/state/active/systemd-resolved.service"
+run hand_dns_to_resolved
+expect_untouched "resolved already active, with no earlier handover: DNS is left alone"
+
+host
+rm "$TMP/bin/resolvconf"
+stub resolvconf ':'
+run hand_dns_to_resolved
+expect_untouched "a real resolvconf: DNS is left alone"
+
+host
+rm "$TMP/state/active/NetworkManager.service"
+run hand_dns_to_resolved
+case "$OUT" in
+    *"NetworkManager is not running"*) expect_untouched "NetworkManager not running: DNS is left alone, with a warning" ;;
+    *) fail "NetworkManager not running: no warning: $OUT" ;;
+esac
+
+for setting in dns=none dns=dnsmasq systemd-resolved=false; do
+    host
+    printf '[main]\n# dns=default\n%s\n[logging]\ndns=none\n' "$setting" > "$TMP/state/nm-config"
+    run hand_dns_to_resolved
+    expect_fatal "NetworkManager $setting stops the install, naming it" "NetworkManager sets $setting"
+    [ -z "$(calls)" ] || fail "NetworkManager $setting: acted before refusing: $(calls)"
+done
+
+host
+printf '[main]\ndns=systemd-resolved\n[logging]\ndns=none\n' > "$TMP/state/nm-config"
+run hand_dns_to_resolved
+[ "$RC" = 0 ] && pass "dns=systemd-resolved, and dns= outside [main], are accepted" || fail "dns=systemd-resolved refused: $OUT"
+
+host
+printf 'Global:\nLink 2 (br-wan):\n' > "$TMP/state/resolvectl-dns"
+run hand_dns_to_resolved
+expect_fatal "no upstream servers stops the install, naming the check and printing the restore" \
+    "'resolvectl dns' lists no DNS servers" "systemctl disable --now systemd-resolved" "cp -a --remove-destination $TMP/root/saved/resolv.conf.orig"
+
+host
+: > "$TMP/state/getent-fail"
+run hand_dns_to_resolved
+expect_fatal "a repository host that no longer resolves stops the install, naming it" "yum.oracle.com no longer resolves" "systemctl restart NetworkManager"
+
+host
+printf '[local]\nbaseurl = file:///srv/repo/\nenabled = 1\n' > "$TMP/state/dnf-dump"
+run hand_dns_to_resolved
+case "$RC:$OUT" in
+    0:*"not checking that names resolve"*) pass "a file:// repository skips the name check with a warning" ;;
+    *) fail "file:// repository: rc=$RC: $OUT" ;;
+esac
+
+host
+printf '[local]\nbaseurl = http://10.0.0.5/repo/\nenabled = 1\n' > "$TMP/state/dnf-dump"
+run hand_dns_to_resolved
+case "$RC:$OUT" in
+    0:*"10.0.0.5 is an IP address"*) pass "a repository at an IP address skips the name check with a warning" ;;
+    *) fail "IP repository: rc=$RC: $OUT" ;;
+esac
+
+# --------------------------------------------------------------------------
+# Repository URLs and their hosts.
+# --------------------------------------------------------------------------
+host
+printf '[a]\nbaseurl = \nenabled = 1\nmirrorlist = https://mirrors.example/list\n[b]\nbaseurl = https://b.example/\nenabled = 1\n' > "$TMP/state/dnf-dump"
+run first_repo_url
+[ "$OUT" = https://mirrors.example/list ] && pass "dnf: a mirrorlist stands in for an empty baseurl" || fail "dnf mirrorlist: '$OUT'"
+printf '[a]\nbaseurl = https://a1.example/, https://a2.example/\nenabled = 1\n' > "$TMP/state/dnf-dump"
+run first_repo_url
+[ "$OUT" = https://a1.example/ ] && pass "dnf: the first of several baseurls" || fail "dnf baseurl list: '$OUT'"
+
+host
+rm "$TMP/bin/dnf"
+stub apt-get '[ "$1" = indextargets ] && printf "http://deb.debian.org/debian/\nhttp://deb.debian.org/debian-security/\n"'
+run first_repo_url
+[ "$OUT" = http://deb.debian.org/debian/ ] && pass "apt: the first repository URI" || fail "apt: '$OUT'"
+
+for case in "https://yum.oracle.com/repo/=yum.oracle.com" "http://user@mirror.lan:8080/x=mirror.lan" \
+    "http://[2001:db8::1]:80/x=[2001:db8::1]" "file:///srv/repo=" "cdrom:[Debian]/=" "=" ; do
+    url=${case%=*}
+    want=${case##*=}
+    run "url_host $url"
+    [ "$OUT" = "$want" ] && pass "url_host '$url' is '$want'" || fail "url_host '$url' = '$OUT', want '$want'"
+done
+
+# --------------------------------------------------------------------------
+# chrony drop-in.
+# --------------------------------------------------------------------------
+host
+mkdir -p "$TMP/root/etc/chrony"
+printf 'pool 2.debian.pool.ntp.org iburst\nmakestep 1 3\nconfdir /etc/chrony/conf.d\n' > "$TMP/root/etc/chrony/chrony.conf"
+before=$(cat "$TMP/root/etc/chrony/chrony.conf")
+run install_chrony_conf
+if [ "$RC" = 0 ] && grep -qx 'makestep 1 -1' "$TMP/root/etc/chrony/conf.d/spinifex.conf" 2>/dev/null \
+    && [ "$(cat "$TMP/root/etc/chrony/chrony.conf")" = "$before" ]; then
+    case "$OUT" in
+        *WARN*) fail "Debian chrony layout warned: $OUT" ;;
+        *) pass "Debian: the drop-in goes in the confdir, chrony.conf is not edited" ;;
+    esac
+else
+    fail "Debian chrony: rc=$RC: $OUT"
+fi
+
+host
+printf 'pool 2.pool.ntp.org iburst\nmakestep 1.0 3\nrtcsync' > "$TMP/root/etc/chrony.conf"
+run install_chrony_conf
+run install_chrony_conf
+if [ "$RC" = 0 ] && grep -qx 'makestep 1 -1' "$TMP/root/etc/chrony.d/spinifex.conf" 2>/dev/null \
+    && [ "$(grep -c '^confdir' "$TMP/root/etc/chrony.conf")" = 1 ] \
+    && [ "$(tail -n 1 "$TMP/root/etc/chrony.conf")" = "confdir /etc/chrony.d" ] \
+    && grep -qx rtcsync "$TMP/root/etc/chrony.conf"; then
+    pass "EL: one confdir line is appended after the stock makestep, and the drop-in goes there"
+else
+    fail "EL chrony: rc=$RC: $OUT: $(cat "$TMP/root/etc/chrony.conf")"
+fi
+
+host
+printf 'confdir /etc/chrony.d\nmakestep 1.0 3\n' > "$TMP/root/etc/chrony.conf"
+run install_chrony_conf
+case "$RC:$OUT" in
+    0:*"makestep on line 2, after its confdir line"*) pass "a makestep after the confdir line warns that it overrides the drop-in" ;;
+    *) fail "late makestep: rc=$RC: $OUT" ;;
+esac
+
+host
+run install_chrony_conf
+case "$RC:$OUT" in
+    0:*"Skipping the chrony drop-in"*) pass "no chrony config: the drop-in is skipped with a warning" ;;
+    *) fail "no chrony: rc=$RC: $OUT" ;;
+esac
+
+# --------------------------------------------------------------------------
+# nbdkit plugin directory.
+# --------------------------------------------------------------------------
+host
+stub nbdkit '[ "$1" = --dump-config ] && printf "binary=/usr/sbin/nbdkit\nplugindir=/usr/lib64/nbdkit/plugins\n"'
+OUT=$(PATH="$TMP/bin:$TMP/sys" INSTALL_SPINIFEX_LIB_ONLY=1 bash -c '. "$1"; resolve_nbdkit_plugindir; echo "DIR=$NBDKIT_PLUGINDIR"' _ "$SETUP" 2>&1) && RC=0 || RC=$?
+[ "$OUT" = DIR=/usr/lib64/nbdkit/plugins ] && pass "the plugin directory comes from nbdkit --dump-config" || fail "nbdkit plugindir: rc=$RC: $OUT"
+
+host
+stub nbdkit 'exit 1'
+run resolve_nbdkit_plugindir
+expect_fatal "a broken nbdkit stops the install, naming it" "nbdkit --dump-config"
+
+host
+run resolve_nbdkit_plugindir
+expect_fatal "no nbdkit stops the install, naming it" "nbdkit --dump-config"
+
+if [ "$fails" -gt 0 ]; then
+    echo "$fails failure(s)"
+    exit 1
+fi

scripts/setup_repos_test.sh

@@ -0,0 +1,150 @@
+#!/bin/sh
+# setup_repos_test.sh — tests for the Oracle Linux repository trust setup in
+# setup.sh: the NFV SIG key fingerprint check and the AppStream QEMU exclude.
+#
+# setup.sh is sourced with INSTALL_SPINIFEX_LIB_ONLY=1 so main never runs. Stub
+# gpg, curl, rpm and dnf read and log fixture files, and the key path is a
+# fixture path.
+set -eu
+
+SCRIPT_DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
+SETUP="$SCRIPT_DIR/setup.sh"
+TMP=$(mktemp -d)
+trap 'rm -rf "$TMP"' EXIT
+
+fails=0
+pass() { echo "  ok   - $1"; }
+fail() { echo "  FAIL - $1"; fails=$((fails + 1)); }
+
+echo "-- setup.sh NFV SIG key and AppStream exclude"
+
+GOOD=35154228174901BEFA8E69A621465E289D2A76A7
+BAD=0000000000000000000000000000000000000BAD
+
+mkdir -p "$TMP/bin"
+# A fixture key file holds the --with-colons listing gpg would print for it.
+cat > "$TMP/bin/gpg" <<'EOF'
+#!/bin/sh
+for last; do :; done
+cat "$last"
+EOF
+cat > "$TMP/bin/curl" <<EOF
+#!/bin/sh
+echo "curl" >> "$TMP/calls"
+while [ \$# -gt 0 ]; do
+    case "\$1" in -o) cp "$TMP/served" "\$2"; exit 0 ;; esac
+    shift
+done
+exit 1
+EOF
+cat > "$TMP/bin/rpm" <<EOF
+#!/bin/sh
+echo "rpm \$*" >> "$TMP/calls"
+EOF
+cat > "$TMP/bin/dnf" <<EOF
+#!/bin/sh
+case "\$*" in
+    "config-manager --dump "*) cat "$TMP/appstream" ;;
+    "config-manager --save "*) echo "\$3" >> "$TMP/calls" ;;
+    *) echo "unexpected dnf \$*" >&2; exit 2 ;;
+esac
+EOF
+chmod +x "$TMP/bin/"*
+
+key() { # key <file> <pub-fpr> [sub-fpr] [second-pub-fpr]
+    {
+        printf 'pub:-:4096:1:AAAA:1:::-:::scESC::::::23::0:\nfpr:::::::::%s:\n' "$2"
+        if [ -n "${3:-}" ]; then printf 'sub:-:4096:1:BBBB:1::::::e::::::23:\nfpr:::::::::%s:\n' "$3"; fi
+        if [ -n "${4:-}" ]; then printf 'pub:-:4096:1:CCCC:1:::-:::scESC::::::23::0:\nfpr:::::::::%s:\n' "$4"; fi
+    } > "$1"
+}
+
+# run <function> — sets RC, OUT and CALLS, from $TMP as the working directory.
+run() {
+    rm -f "$TMP/calls"
+    RC=0
+    OUT=$(cd "$TMP/cwd" && PATH="$TMP/bin:$PATH" INSTALL_SPINIFEX_LIB_ONLY=1 \
+        bash -c '. "$1"; SUDO=""; NFV_SIG_KEY_FILE="$2/installed.key"; "$3"' _ "$SETUP" "$TMP" "$1" 2>&1) || RC=$?
+    CALLS=$(cat "$TMP/calls" 2>/dev/null || true)
+}
+mkdir -p "$TMP/cwd"
+
+imports_once() { [ "$(printf '%s\n' "$CALLS" | grep -c '^rpm --import')" = 1 ]; }
+
+rm -f "$TMP/installed.key"
+key "$TMP/served" "$GOOD"
+run install_nfv_sig_key
+if [ "$RC" = 0 ] && imports_once && cmp -s "$TMP/served" "$TMP/installed.key"; then
+    pass "the SIG's key is installed and imported once"
+else
+    fail "the SIG's key was not installed and imported once: rc=$RC calls='$CALLS': $OUT"
+fi
+
+refused() { # refused <desc>
+    case "$RC:$OUT" in
+        0:*) fail "$1 was accepted" ;;
+        *"Refusing to trust it"*)
+            if printf '%s\n' "$CALLS" | grep -q '^rpm' || [ -e "$TMP/installed.key" ]; then
+                fail "$1 was refused, but only after it was installed or imported: $CALLS"
+            else
+                pass "$1 is refused before anything is trusted"
+            fi ;;
+        *) fail "$1 stopped without the refusal: $OUT" ;;
+    esac
+}
+
+rm -f "$TMP/installed.key"
+key "$TMP/served" "$BAD"
+run install_nfv_sig_key
+refused "a key with another fingerprint"
+case "$OUT" in
+    *"$BAD"*"$GOOD"*) pass "the refusal names both fingerprints" ;;
+    *) fail "the refusal does not name both fingerprints: $OUT" ;;
+esac
+
+rm -f "$TMP/installed.key"
+key "$TMP/served" "$GOOD" "" "$BAD"
+run install_nfv_sig_key
+refused "a file holding the SIG's key and a second key"
+
+rm -f "$TMP/installed.key"
+key "$TMP/served" "$BAD" "$GOOD"
+run install_nfv_sig_key
+refused "a key whose subkey, not primary key, has the SIG's fingerprint"
+
+key "$TMP/installed.key" "$BAD"
+key "$TMP/served" "$GOOD"
+run install_nfv_sig_key
+if [ "$RC" = 0 ] && printf '%s\n' "$CALLS" | grep -q '^curl' && imports_once && cmp -s "$TMP/served" "$TMP/installed.key"; then
+    pass "a cached key with another fingerprint is downloaded again and replaced"
+else
+    fail "a cached key with another fingerprint was not replaced: rc=$RC calls='$CALLS': $OUT"
+fi
+
+key "$TMP/installed.key" "$GOOD"
+run install_nfv_sig_key
+if [ "$RC" = 0 ] && ! printf '%s\n' "$CALLS" | grep -q '^curl' && imports_once; then
+    pass "a cached SIG key is imported without a download"
+else
+    fail "a cached SIG key was not reused: rc=$RC calls='$CALLS': $OUT"
+fi
+
+# excludes <desc> <dump excludepkgs line, or empty> <expected setopt>
+excludes() {
+    if [ -n "$2" ]; then printf '[ol10_appstream]\nexcludepkgs = %s\nenabled = 1\n' "$2" > "$TMP/appstream"
+    else printf '[ol10_appstream]\nenabled = 1\n' > "$TMP/appstream"; fi
+    run exclude_from_appstream
+    if [ "$RC" = 0 ] && [ "$CALLS" = "--setopt=ol10_appstream.excludepkgs=$3" ]; then
+        pass "$1"
+    else
+        fail "$1: rc=$RC calls='$CALLS': $OUT"
+    fi
+}
+
+excludes "no exclude set gains the QEMU patterns" "" "qemu-kvm*,qemu-img*"
+excludes "an operator's excludes are kept" "foo*, bar" "foo*,bar,qemu-kvm*,qemu-img*"
+excludes "a rerun adds no duplicates" "foo*, bar, qemu-kvm*, qemu-img*" "foo*,bar,qemu-kvm*,qemu-img*"
+: > "$TMP/cwd/qemu-kvm-x"
+excludes "the patterns are never expanded against the working directory" "" "qemu-kvm*,qemu-img*"
+
+[ "$fails" = 0 ] || { echo "$fails failure(s)"; exit 1; }

scripts/setup_test.sh

@@ -0,0 +1,172 @@
+#!/bin/sh
+# setup_test.sh — tests for setup.sh's supported-OS gate.
+#
+# setup.sh is sourced with INSTALL_SPINIFEX_LIB_ONLY=1 so main never runs, and
+# detect_os reads fixture os-release, cpuinfo and SELinux config files, and stub
+# uname and getenforce commands.
+set -eu
+
+SCRIPT_DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
+SETUP="$SCRIPT_DIR/setup.sh"
+TMP=$(mktemp -d)
+trap 'rm -rf "$TMP"' EXIT
+
+fails=0
+pass() { echo "  ok   - $1"; }
+fail() { echo "  FAIL - $1"; fails=$((fails + 1)); }
+
+echo "-- setup.sh OS gate"
+
+V3_FLAGS="fpu cx16 lahf_lm popcnt sse4_1 sse4_2 ssse3 avx avx2 bmi1 bmi2 f16c fma abm movbe xsave"
+V2_FLAGS="fpu cx16 lahf_lm popcnt sse4_1 sse4_2 ssse3"
+UEK_R8="6.12.0-207.111.5.4.el10uek.x86_64"
+
+# gate <os-release body> [machine] [kernel] [cpu flags] — runs detect_os and
+# sets RC and OUT (stdout and stderr together). SELINUX_NOW is what getenforce
+# prints; SELINUX_BOOT is the config file's SELINUX=, or "none" for no file.
+gate() {
+    printf '%s\n' "$1" > "$TMP/os-release"
+    printf 'processor\t: 0\nflags\t\t: %s\n' "${4:-$V3_FLAGS}" > "$TMP/cpuinfo"
+    rm -f "$TMP/selinux-config"
+    if [ "${SELINUX_BOOT:-permissive}" != none ]; then
+        printf '# fixture\nSELINUX=%s\nSELINUXTYPE=targeted\n' "${SELINUX_BOOT:-permissive}" > "$TMP/selinux-config"
+    fi
+    mkdir -p "$TMP/bin"
+    printf '#!/bin/sh\necho %s\n' "${SELINUX_NOW:-Permissive}" > "$TMP/bin/getenforce"
+    chmod +x "$TMP/bin/getenforce"
+    cat > "$TMP/bin/uname" <<EOF
+#!/bin/sh
+case "\$1" in
+    -m) echo "${2:-x86_64}" ;;
+    -r) echo "${3:-$UEK_R8}" ;;
+    *) exit 1 ;;
+esac
+EOF
+    chmod +x "$TMP/bin/uname"
+    RC=0
+    OUT=$(PATH="$TMP/bin:$PATH" SPINIFEX_OS_RELEASE="$TMP/os-release" SPINIFEX_CPUINFO="$TMP/cpuinfo" \
+        SPINIFEX_SELINUX_CONFIG="$TMP/selinux-config" INSTALL_SPINIFEX_LIB_ONLY=1 bash -c '. "$1"; detect_os' _ "$SETUP" 2>&1) || RC=$?
+    # Some shells keep a prefix assignment to a function call after it returns.
+    unset SELINUX_NOW SELINUX_BOOT
+}
+accepts() {
+    if [ "$RC" = 0 ]; then pass "accepts $1"; else fail "rejects $1: $OUT"; fi
+}
+# rejects <desc> <word> — the refusal must say why, not just exit.
+rejects() {
+    if [ "$RC" = 0 ]; then
+        fail "accepts $1"
+    else
+        case "$OUT" in
+            *"$2"*) pass "rejects $1" ;;
+            *) fail "rejects $1 without naming '$2': $OUT" ;;
+        esac
+    fi
+}
+
+OL='ID="ol"
+NAME="Oracle Linux Server"
+PRETTY_NAME="Oracle Linux Server 10.2"'
+
+gate 'ID=debian
+VERSION_ID="13"
+PRETTY_NAME="Debian GNU/Linux 13 (trixie)"'
+accepts "Debian 13"
+
+# Debian testing and sid carry no VERSION_ID. The gate has always let them
+# through, and Debian behaviour is not this gate's to change.
+gate 'ID=debian
+PRETTY_NAME="Debian GNU/Linux forky/sid"'
+accepts "Debian testing with no VERSION_ID"
+
+gate 'ID=debian
+VERSION_ID="12"'
+rejects "Debian 12" "Minimum: Debian 13"
+
+gate 'ID=ubuntu
+VERSION_ID="26.04"
+PRETTY_NAME="Ubuntu 26.04 LTS"'
+accepts "Ubuntu 26.04"
+
+gate 'ID=ubuntu
+VERSION_ID="22.04"'
+rejects "Ubuntu 22.04" "Minimum: Ubuntu 24.04"
+
+gate "$OL
+VERSION_ID=\"10.2\""
+accepts "Oracle Linux 10.2 on UEK R8, x86_64, x86-64-v3"
+
+gate "$OL
+VERSION_ID=\"10.4\""
+accepts "Oracle Linux 10.4"
+
+gate "$OL
+VERSION_ID=\"10.1\""
+rejects "Oracle Linux 10.1" "Oracle Linux 10.2 or a later 10.x"
+
+gate "$OL
+VERSION_ID=\"10\""
+rejects "Oracle Linux with no minor version" "Oracle Linux 10.2 or a later 10.x"
+
+gate "$OL
+VERSION_ID=\"11.0\""
+rejects "Oracle Linux 11" "Oracle Linux 10.2 or a later 10.x"
+
+gate "$OL
+VERSION_ID=\"9.6\"" x86_64 "5.15.0-310.184.5.2.el9uek.x86_64"
+rejects "Oracle Linux 9" "Oracle Linux 10.2 or a later 10.x"
+
+gate "$OL" x86_64
+rejects "Oracle Linux with no VERSION_ID" "Oracle Linux 10.2 or a later 10.x"
+
+gate "$OL
+VERSION_ID=\"10.2\"" aarch64 "6.12.0-207.111.5.4.el10uek.aarch64"
+rejects "Oracle Linux on aarch64" "x86_64 only"
+
+gate "$OL
+VERSION_ID=\"10.2\"" x86_64 "6.12.0-55.9.1.el10_0.x86_64"
+rejects "Oracle Linux on the Red Hat compatible kernel" "is not UEK R8"
+
+gate "$OL
+VERSION_ID=\"10.2\"" x86_64 "5.15.0-310.184.5.2.el9uek.x86_64"
+rejects "Oracle Linux on a UEK older than R8" "is not UEK R8"
+
+gate "$OL
+VERSION_ID=\"10.2\"" x86_64 "$UEK_R8" "$V2_FLAGS"
+rejects "Oracle Linux on an x86-64-v2 CPU" "missing: avx avx2 bmi1 bmi2 f16c fma abm movbe xsave"
+
+OL102="$OL
+VERSION_ID=\"10.2\""
+
+SELINUX_NOW=Enforcing gate "$OL102"
+rejects "Oracle Linux with SELinux enforcing" "SELinux is enforcing"
+
+SELINUX_BOOT=enforcing gate "$OL102"
+rejects "Oracle Linux with SELinux permissive now but enforcing at next boot" "enforcing at next boot"
+
+SELINUX_BOOT='"enforcing"' gate "$OL102"
+rejects "Oracle Linux with a quoted SELINUX=\"enforcing\"" "enforcing at next boot"
+
+SELINUX_NOW=Disabled SELINUX_BOOT=disabled gate "$OL102"
+accepts "Oracle Linux with SELinux disabled"
+
+SELINUX_NOW=Disabled SELINUX_BOOT=none gate "$OL102"
+accepts "Oracle Linux with no SELinux config file"
+
+# Spinifex has no SELinux policy on any OS, but refusing enforcing on Debian
+# would change Debian behaviour, so only the Oracle Linux gate checks it.
+SELINUX_NOW=Enforcing SELINUX_BOOT=enforcing gate 'ID=debian
+VERSION_ID="13"'
+accepts "Debian 13 regardless of SELinux"
+
+gate 'ID="rocky"
+VERSION_ID="10.0"'
+rejects "Rocky Linux 10" "Unsupported OS: rocky"
+
+gate 'this is not an os-release file'
+rejects "a malformed os-release" "Unsupported OS"
+
+if [ "$fails" -gt 0 ]; then
+    echo "$fails failure(s)"
+    exit 1
+fi

scripts/smoke-test.sh

@@ -60,7 +60,12 @@ export AWS_PROFILE=spinifex
 # + CA bundle) is used when the script is called via sudo.
 INVOKING_USER="${SUDO_USER:-$(id -un)}"
 INVOKING_HOME=$(getent passwd "$INVOKING_USER" | cut -d: -f6)
-aws_as_user() { sudo -u "$INVOKING_USER" env HOME="$INVOKING_HOME" AWS_PROFILE=spinifex aws "$@"; }
+# By absolute path: sudo's secure_path leaves out /usr/local/bin on some hosts,
+# Oracle Linux among them, and setup.sh installs the AWS CLI there.
+AWS_BIN=$(command -v aws || true)
+[ -z "$AWS_BIN" ] && [ -x /usr/local/bin/aws ] && AWS_BIN=/usr/local/bin/aws
+[ -n "$AWS_BIN" ] || { echo "❌ aws CLI not found on PATH or at /usr/local/bin/aws" >&2; exit 1; }
+aws_as_user() { sudo -u "$INVOKING_USER" env HOME="$INVOKING_HOME" AWS_PROFILE=spinifex "$AWS_BIN" "$@"; }
 
 # --- Wait for EC2 daemon to subscribe to NATS ---
 # Port 3000 (UI) becomes ready before the daemon finishes initialising.

scripts/sync-aws-models.sh

@@ -27,6 +27,7 @@ MODELS=(
     elastic-load-balancing-v2/2015-12-01
     iam/2010-05-08
     rds/2014-10-31
+    route-53/2013-04-01
     s3/2006-03-01
     sts/2011-06-15
 )
@@ -34,7 +35,7 @@ MODELS=(
 TMP="$(mktemp -d)"
 trap 'rm -rf "$TMP"' EXIT
 
-# A blobless shallow fetch pulls only the ten files read below.
+# A blobless shallow fetch pulls only the files read below.
 git -C "$TMP" init --quiet repo
 git -C "$TMP/repo" fetch --quiet --depth=1 --filter=blob:none "$REPO" "$COMMIT"
 COMMITTED="$(git -C "$TMP/repo" log -1 --format=%cs "$COMMIT")"

scripts/uninstall-spx.sh

@@ -12,9 +12,10 @@
 # Options:
 #   --purge-data  Also remove /var/lib/spinifex — volumes, S3 objects, JetStream.
 #                 Off by default: uninstalling software should not destroy data.
-#   --purge-deps  Also apt-remove the packages setup.sh installed. Names them and
-#                 confirms first, even under --yes. Off by default because a
-#                 curl|bash install lands on hosts that were already using them.
+#   --purge-deps  Also remove the packages setup.sh installed, with apt. Refused
+#                 on dnf hosts, where it cannot tell those from the host's own.
+#                 Names them and confirms first, even under --yes. Off by default
+#                 because a curl|bash install lands on hosts already using them.
 #   --yes         Skip the confirmation prompt. Does NOT cover --purge-data,
 #                 --purge-deps, or the multi-node refusal.
 #   --dry-run     Print the whole plan, touch nothing, exit 0.
@@ -30,14 +31,22 @@
 # /usr/share/spinifex, every spinifex systemd unit and slice, the eight
 # spinifex-* service users, the sudoers grant, the sysctl/logrotate/chrony/udev
 # drop-ins, the host firewall policy and its loaded nft table, the OVS bridges
-# and OVN databases, and the Spinifex CA from the host trust store.
+# and OVN databases, the Spinifex resolved drop-in, and the Spinifex CA from the
+# host trust store. Where setup.sh turned on systemd-resolved, it restores
+# /etc/resolv.conf and turns resolved off.
 #
 # WHAT THIS LEAVES ALONE
 #
-# /var/lib/spinifex unless --purge-data. Every apt package unless --purge-deps.
+# /var/lib/spinifex unless --purge-data. Every package unless --purge-deps.
 # /swapfile, which setup.sh may have created but cannot be told apart from one
 # the host already had. Anything the operator configured around us: netplan,
-# /etc/iptables/rules.v4, hand-written bridges.
+# /etc/iptables/rules.v4, hand-written bridges. firewalld, which setup.sh
+# masked, is unmasked but not started. Host settings setup.sh changed and other
+# software may now rely on: /etc/modules-load.d/spinifex.conf, the confdir line
+# appended to an EL /etc/chrony.conf, the dnf repositories and AppStream QEMU
+# exclude, OVS_USER_ID and OVN_USER_ID, the OVS and OVN logrotate edits, masked
+# IPsec units, the SELinux mode, and the GPU passthrough config spx admin gpu
+# wrote. The run ends by listing each of these it finds.
 set -euo pipefail
 
 ETC_DIR="${SPX_ETC_DIR:-/etc/spinifex}"
@@ -73,6 +82,36 @@ while [[ $# -gt 0 ]]; do
     shift
 done
 
+# Host-OS capability probes: beside this script in a checkout and in the
+# installed share dir, else where setup.sh installs them.
+DISTRO_LIB=""
+for candidate in "$(dirname "${BASH_SOURCE[0]}")/lib/distro.sh" /usr/local/share/spinifex/lib/distro.sh; do
+    if [ -f "$candidate" ]; then DISTRO_LIB="$candidate"; break; fi
+done
+if [ -z "$DISTRO_LIB" ]; then
+    echo "ERROR: distro.sh not found at $(dirname "${BASH_SOURCE[0]}")/lib/distro.sh or /usr/local/share/spinifex/lib/distro.sh; re-run setup.sh to install it" >&2
+    exit 1
+fi
+# shellcheck source=lib/distro.sh
+. "$DISTRO_LIB"
+
+# Resolved before anything is removed, so --purge-deps on a host with no package
+# manager, or two, stops with nothing changed.
+PKG_MANAGER=""
+if $PURGE_DEPS; then
+    PKG_MANAGER=$(distro_pkg_manager) || exit 1
+    if [ "$PKG_MANAGER" = dnf ]; then
+        echo "ERROR: --purge-deps is not supported on dnf hosts: its package list cannot tell what setup.sh installed from what the host already had, and dnf also removes every package that requires them. Re-run without --purge-deps." >&2
+        exit 1
+    fi
+fi
+# A failed OVS unit is restarted after its drop-in goes; absent when OVS is.
+OVS_UNIT=$(distro_ovs_unit 2>/dev/null) || OVS_UNIT=""
+# Resolved before anything is removed. The CA tools live in sbin, which Debian
+# leaves off a non-root PATH, and failing late would stop the uninstall halfway.
+CA_TRUST=$(PATH="/usr/sbin:/sbin:$PATH" distro_ca_trust) \
+    || { echo "ERROR: cannot tell which trust store to remove the Spinifex CA from" >&2; exit 1; }
+
 log() { echo "[uninstall-spx] $*"; }
 run() {
     if $DRY_RUN; then
@@ -130,6 +169,20 @@ EOF
     exit 1
 fi
 
+# --- BEGIN at-stake counts ---
+# Copied between node-reset.sh and uninstall-spx.sh; node-reset_test.sh keeps them equal.
+# Instances are the daemon's local state records. Only mounted volumes have a
+# local directory under viperblock/; the rest live only in Predastore.
+count_instances() {
+    local state="$1/state/instance-state.json"
+    sudo test -e "$state" || { echo 0; return 0; }
+    sudo jq -r '.vms // {} | length' "$state" 2>/dev/null || echo unknown
+}
+count_mounted_volumes() {
+    sudo find "$1/viperblock" -maxdepth 1 -mindepth 1 -type d -name 'vol-*' ! -name '*-efi' 2>/dev/null | wc -l || true
+}
+# --- END at-stake counts ---
+
 # ---------------------------------------------------------------------------
 # 2. Report what is at stake, then confirm.
 # ---------------------------------------------------------------------------
@@ -139,18 +192,18 @@ if command -v spx >/dev/null 2>&1; then
 fi
 if [ -d "$DATA_DIR" ]; then
     size=$(sudo du -sh "$DATA_DIR" 2>/dev/null | cut -f1 || true)
-    instances=$(sudo find "$DATA_DIR/instances" -maxdepth 1 -mindepth 1 -type d 2>/dev/null | wc -l || true)
-    volumes=$(sudo find "$DATA_DIR/volumes" -maxdepth 1 -mindepth 1 2>/dev/null | wc -l || true)
+    instances=$(count_instances "$DATA_DIR")
+    volumes=$(count_mounted_volumes "$DATA_DIR")
     if $PURGE_DATA; then
-        log "  --purge-data: DESTROYING $instances instance(s), $volumes volume(s), ${size:-unknown} under $DATA_DIR"
+        log "  --purge-data: DESTROYING $instances instance(s), $volumes mounted volume(s), ${size:-unknown} under $DATA_DIR"
     else
-        log "  keeping $DATA_DIR ($instances instance(s), $volumes volume(s), ${size:-unknown}) — pass --purge-data to remove it"
+        log "  keeping $DATA_DIR ($instances instance(s), $volumes mounted volume(s), ${size:-unknown}) — pass --purge-data to remove it"
     fi
 fi
-$PURGE_DEPS && log "  --purge-deps: apt packages will be removed (confirmed separately below)"
+$PURGE_DEPS && log "  --purge-deps: $PKG_MANAGER packages will be removed (confirmed separately below)"
 
 if ! $ASSUME_YES && ! $DRY_RUN; then
-    read -r -p "Remove the Spinifex install from this node? Type 'uninstall' to continue: " reply
+    read -r -p "Remove the Spinifex install from this node? Type 'uninstall' to continue: " reply || reply=""
     [ "$reply" = "uninstall" ] || { log "aborted"; exit 1; }
 fi
 
@@ -166,7 +219,7 @@ if $PURGE_DATA && ! $DRY_RUN; then
     if [ "${SPX_PURGE_CONFIRM:-}" = "destroy" ]; then
         log "SPX_PURGE_CONFIRM=destroy: proceeding without a prompt"
     else
-        read -r -p "Also destroy every volume and object under $DATA_DIR? Type 'destroy' to continue: " reply
+        read -r -p "Also destroy every volume and object under $DATA_DIR? Type 'destroy' to continue: " reply || reply=""
         [ "$reply" = "destroy" ] || { log "aborted"; exit 1; }
     fi
 fi
@@ -184,7 +237,7 @@ fi
 log "stopping services"
 run sudo systemctl stop spinifex.target 2>/dev/null || true
 if ! $DRY_RUN; then
-    for unit in spinifex-predastore spinifex-viperblock spinifex-nats; do
+    for unit in spinifex-predastore spinifex-viperblock spinifex-sqs-store spinifex-nats; do
         if systemctl is-active --quiet "$unit"; then
             sudo systemctl stop "$unit" || true
         fi
@@ -208,7 +261,7 @@ fi
 #
 # It handles the guests, the OVS bridges, the OVN databases and the JetStream
 # store, and it carries correctness that is invisible in its code: QEMU matched
-# as ^qemu-system because comm truncates at 15 characters, a wait for QEMU to
+# by a comm prefix because comm truncates at 15 characters, a wait for QEMU to
 # exit rather than for the signal, and the firewall-mode stash that stops an
 # interrupted run leaving the node unarmed. A second copy of that would drift,
 # and the failure mode of the drift is data loss.
@@ -265,6 +318,16 @@ if command -v iptables-save >/dev/null 2>&1 && ! $DRY_RUN; then
 elif $DRY_RUN; then
     echo "  would remove iptables rules commented spinifex-nat-egress / spinifex-eip-ingress / spinifex-imds / spinifex-imds-remap"
 fi
+# setup.sh masks firewalld. Unmasked but left disabled: its rules were never
+# migrated, so starting it is the operator's decision. A mask keeps the enable
+# links, so without the disable it would start again at the next boot.
+FIREWALLD_UNMASKED=false
+if [ "$(systemctl is-enabled firewalld.service 2>/dev/null)" = masked ]; then
+    run sudo systemctl unmask firewalld.service
+    run sudo systemctl disable firewalld.service 2>/dev/null || true
+    FIREWALLD_UNMASKED=true
+    $DRY_RUN || log "  firewalld unmasked, left disabled"
+fi
 
 # ---------------------------------------------------------------------------
 # 6. systemd units.
@@ -294,12 +357,15 @@ rm_path /etc/systemd/system/spinifex-*.service \
 # exactly that way on the first dev-oci round trip.
 log "removing drop-ins written into other units"
 rm_path /etc/systemd/system/openvswitch-switch.service.d/spinifex-perms.conf \
+        /etc/systemd/system/ovs-vswitchd.service.d/spinifex-perms.conf \
         /etc/systemd/system/ovn-controller.service.d/spinifex-perms.conf \
         /etc/systemd/system/ovn-controller.service.d/log-level.conf \
         /etc/systemd/system/openvswitch-ipsec.service.d/spinifex-perms.conf \
-        /etc/systemd/system/ovn-northd.service.d/no-manage-ovsdb.conf
+        /etc/systemd/system/ovn-northd.service.d/spinifex-perms.conf \
+        /etc/systemd/system/ovn-northd.service.d/no-manage-ovsdb.conf \
+        /etc/systemd/system/ipsec.service.d/spinifex-guard.conf
 # Only if we emptied them — another package may own the same override dir.
-for d in openvswitch-switch ovn-controller openvswitch-ipsec ovn-northd; do
+for d in openvswitch-switch ovs-vswitchd ovn-controller openvswitch-ipsec ovn-northd ipsec; do
     dir="/etc/systemd/system/${d}.service.d"
     if sudo test -d "$dir" && [ -z "$(sudo ls -A "$dir" 2>/dev/null)" ]; then
         rm_path "$dir"
@@ -313,8 +379,11 @@ run sudo systemctl reset-failed 2>/dev/null || true
 # attempt. Restarting them here is not housekeeping: openvswitch-switch failing
 # is how the operator discovers the drop-in problem, and leaving it failed after
 # an uninstall that claims success is the same silent failure one step later.
+# Each unit that carried one of the drop-ins above, under this host's OVS name.
+RESTART_UNITS=(ovs-vswitchd ovn-controller ovn-northd)
+[ -n "$OVS_UNIT" ] && RESTART_UNITS=("$OVS_UNIT" "${RESTART_UNITS[@]}")
 if ! $DRY_RUN; then
-    for u in openvswitch-switch ovn-controller; do
+    for u in "${RESTART_UNITS[@]}"; do
         if systemctl is-failed --quiet "$u" 2>/dev/null; then
             log "  restarting $u, which our drop-in had failed"
             sudo systemctl reset-failed "$u" 2>/dev/null || true
@@ -363,10 +432,19 @@ if [ -z "$PLUGINDIR" ]; then
 fi
 [ -n "$PLUGINDIR" ] && rm_path "$PLUGINDIR/nbdkit-viperblock-plugin.so"
 
+# A leftover guard table drops all Geneve from this host's encap address until
+# reboot, and the helper is the one place that names it.
+if [ -x /usr/local/lib/spinifex/spinifex-set-ipsec-state ]; then
+    run sudo /usr/local/lib/spinifex/spinifex-set-ipsec-state unguard || true
+fi
 rm_path /usr/local/bin/spx \
         /usr/local/share/spinifex \
         /usr/local/lib/spinifex \
         /usr/share/spinifex
+# setup.sh adds this link where sudo's PATH leaves out /usr/local/bin.
+if [ "$(readlink /usr/bin/spx 2>/dev/null)" = /usr/local/bin/spx ]; then
+    rm_path /usr/bin/spx
+fi
 
 # ---------------------------------------------------------------------------
 # 9. Drop-ins. Globbed, not listed: 99-spinifex-vpc.conf is written at runtime
@@ -378,9 +456,15 @@ log "removing configuration drop-ins"
 rm_path /etc/sysctl.d/*spinifex* \
         /etc/sudoers.d/spinifex-network \
         /etc/chrony/conf.d/spinifex.conf \
+        /etc/chrony.d/spinifex.conf \
         /etc/logrotate.d/spinifex \
-        /etc/udev/rules.d/*spinifex*
+        /etc/tmpfiles.d/spinifex.conf \
+        /etc/udev/rules.d/*spinifex* \
+        /etc/NetworkManager/conf.d/*spinifex*
 run sudo sysctl --system >/dev/null 2>&1 || true
+if systemctl is-active --quiet NetworkManager.service 2>/dev/null; then
+    run sudo nmcli general reload conf 2>/dev/null || true
+fi
 
 # ---------------------------------------------------------------------------
 # 10. Spinifex directories.
@@ -415,7 +499,7 @@ fi
 # userdel warns about a missing home. Order here is cosmetic, not correctness.
 # ---------------------------------------------------------------------------
 log "removing service users and groups"
-for svc in nats gw daemon storage northstar viperblock vpcd ui; do
+for svc in nats sqsstore gw daemon storage northstar viperblock vpcd ui; do
     user="spinifex-${svc}"
     if id -u "$user" >/dev/null 2>&1; then
         run sudo userdel "$user" 2>/dev/null || log "  NOTE: could not remove user $user"
@@ -431,14 +515,41 @@ done
 # 12. The CA in the host trust store. Leaving it means the host trusts a CA
 # whose key no longer exists anywhere.
 # ---------------------------------------------------------------------------
-if [ -f /usr/local/share/ca-certificates/spinifex-ca.crt ]; then
+CA_ANCHOR="${CA_TRUST#* }/spinifex-ca.crt"
+if [ -f "$CA_ANCHOR" ]; then
     log "removing the Spinifex CA from the host trust store"
-    rm_path /usr/local/share/ca-certificates/spinifex-ca.crt
-    run sudo update-ca-certificates >/dev/null 2>&1 || true
+    rm_path "$CA_ANCHOR"
+    run sudo "${CA_TRUST%% *}"
 fi
 
 # ---------------------------------------------------------------------------
-# 13. Swap is reported, never removed.
+# 13. Host DNS. setup.sh saved /etc/resolv.conf only where it turned on
+# systemd-resolved, so the saved copy is what says to turn it off again.
+# ---------------------------------------------------------------------------
+RESOLV_CONF_SAVED=/var/lib/spinifex-reset/resolv.conf.orig
+# The daemon's drop-in routes the cluster zones to a northstar that is gone now.
+RESOLVED_DROPIN=/etc/systemd/resolved.conf.d/spinifex-dns.conf
+if exists "$RESOLVED_DROPIN"; then
+    log "removing the systemd-resolved drop-in for the cluster zones"
+    rm_path "$RESOLVED_DROPIN"
+    if systemctl is-active --quiet systemd-resolved.service 2>/dev/null; then
+        run sudo systemctl restart systemd-resolved.service
+    fi
+fi
+if exists "$RESOLV_CONF_SAVED"; then
+    log "restoring /etc/resolv.conf and turning systemd-resolved off, as before the install"
+    run sudo systemctl disable --now systemd-resolved.service
+    run sudo cp -a --remove-destination "$RESOLV_CONF_SAVED" /etc/resolv.conf
+    if systemctl is-active --quiet NetworkManager.service 2>/dev/null; then
+        run sudo systemctl restart NetworkManager.service
+    fi
+    rm_path "$RESOLV_CONF_SAVED"
+fi
+# node-reset keeps its firewall-mode stash here too, so remove it only once empty.
+run sudo rmdir --ignore-fail-on-non-empty "$(dirname "$RESOLV_CONF_SAVED")" 2>/dev/null || true
+
+# ---------------------------------------------------------------------------
+# 14. Swap is reported, never removed.
 #
 # setup_swap creates an 8G /swapfile with the fstab line `/swapfile none swap sw
 # 0 0` — which is byte-identical to the line a host would already have if it
@@ -455,28 +566,34 @@ if [ -f /swapfile ] && grep -q '^/swapfile ' /etc/fstab 2>/dev/null; then
 fi
 
 # ---------------------------------------------------------------------------
-# 14. apt packages, only when asked, and only after naming them.
+# 15. Packages, only when asked, and only after naming them.
 #
 # This is the one irreversible thing in the script. On a dedicated node these
 # are ours; on the host a curl|bash install actually lands on they are someone
 # else's, and openvswitch/libvirt/qemu/chrony going away underneath a running
 # workload is a far worse outcome than a few megabytes left behind. Confirmed
 # separately even under --yes, and never autoremove, which is unbounded.
+#
+# apt only: dnf hosts are refused up front.
 # ---------------------------------------------------------------------------
 if $PURGE_DEPS; then
     PACKAGES="nbdkit qemu-utils gdisk ovmf less libvirt-daemon-system libvirt-clients
 pciutils jq ethtool netcat-openbsd unzip xz-utils
 ovn-central ovn-host openvswitch-switch openvswitch-ipsec strongswan-charon
 chrony nftables"
+    KEPT="curl, wget, file, iproute2, dhcpcd-base, qemu-system-*"
+    REMOVE_CMD="apt-get remove -y"
     log "--purge-deps: these packages would be removed:"
     # shellcheck disable=SC2086
     for p in $PACKAGES; do echo "    $p"; done
-    log "  NOT removed: curl, wget, file, iproute2, dhcpcd-base, qemu-system-* — too"
+    log "  NOT removed: $KEPT — too"
     log "  widely depended on to remove from a host we do not own."
-    if $DRY_RUN; then
-        echo "  would run: apt-get remove -y <the above>"
+    if [ -z "$PACKAGES" ]; then
+        log "  none of them is installed"
+    elif $DRY_RUN; then
+        echo "  would run: $REMOVE_CMD <the above>"
     else
-        read -r -p "Remove these packages? Type 'remove' to continue: " reply
+        read -r -p "Remove these packages? Type 'remove' to continue: " reply || reply=""
         if [ "$reply" = "remove" ]; then
             # shellcheck disable=SC2086
             sudo DEBIAN_FRONTEND=noninteractive apt-get remove -y $PACKAGES || \
@@ -488,7 +605,7 @@ chrony nftables"
 fi
 
 # ---------------------------------------------------------------------------
-# 15. Verify. An uninstaller that reports success over a surviving unit file is
+# 16. Verify. An uninstaller that reports success over a surviving unit file is
 # worse than one that refuses, because the next install silently inherits it.
 # ---------------------------------------------------------------------------
 if ! $DRY_RUN; then
@@ -496,23 +613,29 @@ if ! $DRY_RUN; then
     LEFT=()
     for p in /usr/local/bin/spx /usr/local/share/spinifex /usr/local/lib/spinifex \
              /usr/share/spinifex "$ETC_DIR" "$LOG_DIR" \
-             /etc/sudoers.d/spinifex-network /etc/logrotate.d/spinifex \
-             /etc/chrony/conf.d/spinifex.conf; do
+             /etc/sudoers.d/spinifex-network /etc/logrotate.d/spinifex /etc/tmpfiles.d/spinifex.conf \
+             /etc/chrony/conf.d/spinifex.conf /etc/chrony.d/spinifex.conf "$RESOLV_CONF_SAVED" \
+             "$RESOLVED_DROPIN"; do
         exists "$p" && LEFT+=("$p")
     done
+    [ "$(readlink /usr/bin/spx 2>/dev/null)" = /usr/local/bin/spx ] && LEFT+=(/usr/bin/spx)
     for p in /etc/systemd/system/spinifex-*.service /etc/systemd/system/spinifex*.slice \
              /etc/systemd/system/spinifex.target /etc/sysctl.d/*spinifex* \
              /etc/systemd/network/1[5-8]-spinifex-* \
+             /etc/NetworkManager/conf.d/*spinifex* \
              /etc/systemd/system/*.service.d/spinifex-perms.conf \
              /etc/systemd/system/ovn-northd.service.d/no-manage-ovsdb.conf \
+             /etc/systemd/system/ipsec.service.d/spinifex-guard.conf \
              /etc/systemd/system/ovn-controller.service.d/log-level.conf; do
         exists "$p" && LEFT+=("$p")
     done
+    $FIREWALLD_UNMASKED && systemctl is-enabled --quiet firewalld.service 2>/dev/null &&
+        LEFT+=("firewalld.service is enabled")
     # A unit we broke and could not restart is a failed uninstall, not a wart.
-    for u in openvswitch-switch ovn-controller; do
-        systemctl is-failed --quiet "$u" 2>/dev/null && LEFT+=("$u.service is failed")
+    for u in "${RESTART_UNITS[@]}"; do
+        systemctl is-failed --quiet "$u" 2>/dev/null && LEFT+=("${u%.service}.service is failed")
     done
-    for svc in nats gw daemon storage northstar viperblock vpcd ui; do
+    for svc in nats sqsstore gw daemon storage northstar viperblock vpcd ui; do
         id -u "spinifex-${svc}" >/dev/null 2>&1 && LEFT+=("user spinifex-${svc}")
     done
     if command -v nft >/dev/null 2>&1 && sudo nft list table inet spinifex_filter >/dev/null 2>&1; then
@@ -527,7 +650,51 @@ if ! $DRY_RUN; then
     fi
 fi
 
+# ---------------------------------------------------------------------------
+# 17. Host settings setup.sh changed and this leaves, each listed only where it
+# is found, so the operator can see what to undo by hand.
+# ---------------------------------------------------------------------------
+KEPT_SETTINGS=()
+kept() { KEPT_SETTINGS+=("$1"); }
+$FIREWALLD_UNMASKED && kept "firewalld: unmasked but disabled, with the rules it had before Spinifex"
+exists /etc/modules-load.d/spinifex.conf && kept "/etc/modules-load.d/spinifex.conf: openvswitch geneve vhost_net tun load at boot"
+CHRONY_CONF=$(distro_chrony_conf 2>/dev/null) || CHRONY_CONF=""
+if [ -n "$CHRONY_CONF" ] && grep -q '^# Added by the Spinifex installer' "$CHRONY_CONF" 2>/dev/null; then
+    kept "$CHRONY_CONF: the confdir line the installer appended"
+fi
+if command -v dnf >/dev/null 2>&1; then
+    exists /etc/yum.repos.d/spinifex-nfv-sig.repo &&
+        kept "dnf: the CentOS NFV SIG repository (/etc/yum.repos.d/spinifex-nfv-sig.repo) and its key in /etc/pki/rpm-gpg"
+    for repo in ol10_kvm_utils ol10_UEKR8; do
+        dnf config-manager --dump "$repo" 2>/dev/null | grep -qx 'enabled = 1' && kept "dnf: $repo enabled"
+    done
+    dnf config-manager --dump ol10_appstream 2>/dev/null | grep -q '^excludepkgs = .*qemu-kvm' &&
+        kept "dnf: ol10_appstream excludes qemu-kvm* and qemu-img*, so QEMU stays on KVM Utilities"
+fi
+for setting in /etc/sysconfig/openvswitch:OVS_USER_ID /etc/sysconfig/ovn:OVN_USER_ID; do
+    grep -q "^${setting#*:}=\"root:root\"" "${setting%%:*}" 2>/dev/null &&
+        kept "${setting%%:*}: ${setting#*:}=\"root:root\""
+done
+for lr in "$(distro_ovs_logrotate 2>/dev/null)" "$(distro_ovn_logrotate 2>/dev/null)"; do
+    [ -n "$lr" ] && grep -q 'maxsize 100M' "$lr" 2>/dev/null && kept "$lr: the rotate, maxsize and su edits setup-ovn.sh made"
+done
+IPSEC_UNITS=(openvswitch-ipsec.service strongswan-starter.service)
+distro_has_libreswan && IPSEC_UNITS+=(ipsec.service)
+for u in "${IPSEC_UNITS[@]}"; do
+    [ "$(systemctl is-enabled "$u" 2>/dev/null)" = masked ] && kept "$u: masked, as IPsec was off"
+done
+command -v getenforce >/dev/null 2>&1 && kept "SELinux: $(getenforce 2>/dev/null); setup.sh refuses enforcing, so it was set for the install"
+GPU_FILES=""
+for gpu in /etc/modprobe.d/vfio-pci.conf /etc/dracut.conf.d/spinifex-vfio.conf; do
+    exists "$gpu" && GPU_FILES="${GPU_FILES:+$GPU_FILES, }$gpu"
+done
+[ -n "$GPU_FILES" ] && kept "GPU passthrough: $GPU_FILES and the IOMMU kernel arguments, written by spx admin gpu setup"
+if [ "${#KEPT_SETTINGS[@]}" -gt 0 ]; then
+    log "left in place, as other software on the host may rely on them:"
+    for k in "${KEPT_SETTINGS[@]}"; do log "  - $k"; done
+fi
+
 log "done — Spinifex is removed from this node"
 $PURGE_DATA || { [ -d "$DATA_DIR" ] && log "      $DATA_DIR was kept (--purge-data removes it)"; }
-$PURGE_DEPS || log "      apt dependencies were kept (--purge-deps removes them)"
+$PURGE_DEPS || log "      package dependencies were kept (--purge-deps removes them)"
 exit 0

scripts/uninstall-spx_test.sh

@@ -56,6 +56,17 @@ grep -q "unknown option" "$TMP/err" \
     && pass "unknown option explains itself on stderr" \
     || fail "unknown option wrote nothing useful to stderr"
 
+# --purge-deps on a dnf host stops before anything is touched. PATH holds only
+# the tools the script needs up to that check, with dnf and no apt-get.
+mkdir -p "$TMP/dnfbin"
+for tool in bash dirname; do ln -s "$(command -v "$tool")" "$TMP/dnfbin/$tool"; done
+printf '#!/bin/sh\nexit 0\n' >"$TMP/dnfbin/dnf"
+chmod +x "$TMP/dnfbin/dnf"
+PATH="$TMP/dnfbin" "$TMP/dnfbin/bash" "$UNINSTALL" --purge-deps --yes >"$TMP/out" 2>"$TMP/err" && rc=0 || rc=$?
+[ "$rc" = 1 ] && grep -q "not supported on dnf hosts" "$TMP/err" \
+    && pass "--purge-deps is refused on a dnf host, saying why" \
+    || fail "--purge-deps on a dnf host: rc=$rc: $(cat "$TMP/err")"
+
 # --------------------------------------------------------------------------
 # The multi-node refusal.
 #

scripts/verify-network.sh

@@ -29,7 +29,12 @@ SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 
 INVOKING_USER="${SUDO_USER:-$(id -un)}"
 INVOKING_HOME=$(getent passwd "$INVOKING_USER" | cut -d: -f6)
-aws_as_user() { sudo -u "$INVOKING_USER" env HOME="$INVOKING_HOME" AWS_PROFILE=spinifex aws "$@"; }
+# By absolute path: sudo's secure_path leaves out /usr/local/bin on some hosts,
+# Oracle Linux among them, and setup.sh installs the AWS CLI there.
+AWS_BIN=$(command -v aws || true)
+[ -z "$AWS_BIN" ] && [ -x /usr/local/bin/aws ] && AWS_BIN=/usr/local/bin/aws
+[ -n "$AWS_BIN" ] || { echo "ERROR: aws CLI not found on PATH or at /usr/local/bin/aws" >&2; exit 1; }
+aws_as_user() { sudo -u "$INVOKING_USER" env HOME="$INVOKING_HOME" AWS_PROFILE=spinifex "$AWS_BIN" "$@"; }
 
 FAILURES=0
 pass() { printf '  \033[32m✓\033[0m %s\n' "$*"; }

spinifex/agents/eks/tokenwebhook/readiness.go

@@ -0,0 +1,128 @@
+package tokenwebhook
+
+import (
+	"context"
+	"crypto/tls"
+	"crypto/x509"
+	"fmt"
+	"net"
+	"net/http"
+	"net/url"
+	"os"
+	"sync"
+	"time"
+
+	"github.com/mulgadc/bluebottle/pkg/tlsconfig"
+)
+
+const (
+	readinessTimeout = 2 * time.Second
+	readinessTTL     = 2 * time.Second
+)
+
+// defaultAPIServerAddr is the local k3s apiserver the readiness check dials.
+const defaultAPIServerAddr = "127.0.0.1:6443"
+
+// gatewayProbe reports whether this control plane can serve: the gateway it
+// relays to accepts a TLS handshake and the local apiserver accepts a TCP
+// connection. It sends no credentials and caches the result so frequent load
+// balancer probes never reach either more than once per ttl.
+type gatewayProbe struct {
+	addr    string
+	apiAddr string
+	tlsCfg  *tls.Config
+	timeout time.Duration
+	ttl     time.Duration
+	now     func() time.Time
+
+	mu      sync.Mutex
+	checked time.Time
+	ok      bool
+}
+
+func newGatewayProbe(gatewayURL, caPath, apiAddr string) (*gatewayProbe, error) {
+	u, err := url.Parse(gatewayURL)
+	if err != nil {
+		return nil, fmt.Errorf("parse gateway URL %q: %w", gatewayURL, err)
+	}
+	host, port := u.Hostname(), u.Port()
+	if host == "" {
+		return nil, fmt.Errorf("gateway URL %q has no host", gatewayURL)
+	}
+	if port == "" {
+		port = "443"
+	}
+	cfg := &tls.Config{MinVersion: tls.VersionTLS13, CurvePreferences: tlsconfig.Curves}
+	if caPath != "" {
+		pem, err := os.ReadFile(caPath)
+		if err != nil {
+			return nil, fmt.Errorf("read gateway CA %q: %w", caPath, err)
+		}
+		pool := x509.NewCertPool()
+		if !pool.AppendCertsFromPEM(pem) {
+			return nil, fmt.Errorf("gateway CA %q has no usable certificates", caPath)
+		}
+		cfg.RootCAs = pool
+	}
+	if apiAddr == "" {
+		apiAddr = defaultAPIServerAddr
+	}
+	return &gatewayProbe{
+		addr:    net.JoinHostPort(host, port),
+		apiAddr: apiAddr,
+		tlsCfg:  cfg,
+		timeout: readinessTimeout,
+		ttl:     readinessTTL,
+		now:     time.Now,
+	}, nil
+}
+
+// ready returns the cached result while it is younger than ttl, else probes.
+// Concurrent callers wait for one probe and share its result.
+func (p *gatewayProbe) ready() bool {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	if !p.checked.IsZero() && p.now().Sub(p.checked) < p.ttl {
+		return p.ok
+	}
+	p.ok = p.probe()
+	p.checked = p.now()
+	return p.ok
+}
+
+func (p *gatewayProbe) probe() bool {
+	ctx, cancel := context.WithTimeout(context.Background(), p.timeout)
+	defer cancel()
+	d := tls.Dialer{NetDialer: &net.Dialer{}, Config: p.tlsCfg}
+	conn, err := d.DialContext(ctx, "tcp", p.addr)
+	if err != nil {
+		return false
+	}
+	_ = conn.Close()
+
+	// A control plane whose apiserver is down must leave rotation too.
+	api, err := (&net.Dialer{}).DialContext(ctx, "tcp", p.apiAddr)
+	if err != nil {
+		return false
+	}
+	_ = api.Close()
+	return true
+}
+
+// handle serves 200 when the gateway and apiserver are reachable, else 503.
+func (p *gatewayProbe) handle(w http.ResponseWriter, _ *http.Request) {
+	if p.ready() {
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte("ok"))
+		return
+	}
+	http.Error(w, "not ready", http.StatusServiceUnavailable)
+}
+
+// readinessHandler serves only /readyz; the listener it backs is plain HTTP and
+// reachable beyond loopback, so it must never expose /authenticate.
+func readinessHandler(p *gatewayProbe) http.Handler {
+	mux := http.NewServeMux()
+	mux.HandleFunc("/readyz", p.handle)
+	return mux
+}

spinifex/agents/eks/tokenwebhook/readiness_test.go

@@ -0,0 +1,251 @@
+package tokenwebhook
+
+import (
+	"crypto/tls"
+	"net"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// tlsGateway is a loopback TLS listener standing in for the gateway. It counts
+// accepted connections and can be stopped and restarted on the same address.
+type tlsGateway struct {
+	t       *testing.T
+	api     *fakeAPIServer
+	addr    string
+	cert    tls.Certificate
+	caPath  string
+	ln      net.Listener
+	accepts atomic.Int32
+}
+
+func newTLSGateway(t *testing.T) *tlsGateway {
+	t.Helper()
+	dir := t.TempDir()
+	cert, certPEM, err := ensureServingCert(filepath.Join(dir, "c.pem"), filepath.Join(dir, "k.pem"))
+	require.NoError(t, err)
+	caPath := filepath.Join(dir, "ca.pem")
+	require.NoError(t, os.WriteFile(caPath, certPEM, 0o600))
+
+	g := &tlsGateway{t: t, cert: cert, caPath: caPath, api: newFakeAPIServer(t)}
+	g.start("127.0.0.1:0")
+	t.Cleanup(g.stop)
+	return g
+}
+
+func (g *tlsGateway) start(addr string) {
+	g.t.Helper()
+	ln, err := tls.Listen("tcp", addr, serverTLSConfig(g.cert))
+	require.NoError(g.t, err)
+	g.ln = ln
+	g.addr = ln.Addr().String()
+	go func() {
+		for {
+			c, err := ln.Accept()
+			if err != nil {
+				return
+			}
+			g.accepts.Add(1)
+			go func() {
+				if tc, ok := c.(*tls.Conn); ok {
+					_ = tc.Handshake()
+				}
+				_ = c.Close()
+			}()
+		}
+	}()
+}
+
+func (g *tlsGateway) stop() {
+	if g.ln != nil {
+		_ = g.ln.Close()
+		g.ln = nil
+	}
+}
+
+// fakeAPIServer is a plain TCP listener standing in for the local apiserver.
+type fakeAPIServer struct {
+	addr string
+	ln   net.Listener
+}
+
+func newFakeAPIServer(t *testing.T) *fakeAPIServer {
+	t.Helper()
+	a := &fakeAPIServer{}
+	a.start("127.0.0.1:0", t)
+	t.Cleanup(a.stop)
+	return a
+}
+
+func (a *fakeAPIServer) start(addr string, t *testing.T) {
+	t.Helper()
+	ln, err := net.Listen("tcp", addr)
+	require.NoError(t, err)
+	a.ln = ln
+	a.addr = ln.Addr().String()
+	go func() {
+		for {
+			c, err := ln.Accept()
+			if err != nil {
+				return
+			}
+			_ = c.Close()
+		}
+	}()
+}
+
+func (a *fakeAPIServer) stop() {
+	if a.ln != nil {
+		_ = a.ln.Close()
+		a.ln = nil
+	}
+}
+
+func (g *tlsGateway) probe(t *testing.T) *gatewayProbe {
+	t.Helper()
+	p, err := newGatewayProbe("https://"+g.addr, g.caPath, g.api.addr)
+	require.NoError(t, err)
+	return p
+}
+
+func TestGatewayProbe_UpAndRefused(t *testing.T) {
+	g := newTLSGateway(t)
+	p := g.probe(t)
+	p.ttl = 0
+
+	assert.True(t, p.ready())
+
+	g.stop()
+	assert.False(t, p.ready(), "a closed gateway port is not ready")
+}
+
+func TestGatewayProbe_RecoversAfterDown(t *testing.T) {
+	g := newTLSGateway(t)
+	p := g.probe(t)
+	p.ttl = 0
+	addr := g.addr
+
+	g.stop()
+	require.False(t, p.ready())
+
+	g.start(addr)
+	assert.True(t, p.ready(), "ready again once the gateway is back")
+}
+
+func TestGatewayProbe_TimesOutOnStalledHandshake(t *testing.T) {
+	g := newTLSGateway(t)
+	// A plain listener that accepts and never answers the TLS handshake.
+	ln, err := net.Listen("tcp", "127.0.0.1:0")
+	require.NoError(t, err)
+	t.Cleanup(func() { _ = ln.Close() })
+	go func() {
+		for {
+			c, err := ln.Accept()
+			if err != nil {
+				return
+			}
+			t.Cleanup(func() { _ = c.Close() })
+		}
+	}()
+
+	p, err := newGatewayProbe("https://"+ln.Addr().String(), g.caPath, g.api.addr)
+	require.NoError(t, err)
+	p.timeout = 200 * time.Millisecond
+	p.ttl = 0
+
+	start := time.Now()
+	assert.False(t, p.ready())
+	assert.Less(t, time.Since(start), 2*time.Second, "the probe gives up at its timeout")
+}
+
+func TestGatewayProbe_RejectsUntrustedCertificate(t *testing.T) {
+	g := newTLSGateway(t)
+	other := newTLSGateway(t)
+	p, err := newGatewayProbe("https://"+g.addr, other.caPath, g.api.addr)
+	require.NoError(t, err)
+	p.ttl = 0
+
+	assert.False(t, p.ready(), "a certificate not signed by the configured CA is not ready")
+}
+
+func TestGatewayProbe_CachesWithinTTL(t *testing.T) {
+	g := newTLSGateway(t)
+	p := g.probe(t)
+	clock := time.Unix(1_000_000, 0)
+	p.now = func() time.Time { return clock }
+	p.ttl = 2 * time.Second
+
+	require.True(t, p.ready())
+	require.True(t, p.ready())
+	require.True(t, p.ready())
+	assert.Equal(t, int32(1), g.accepts.Load(), "calls inside the ttl share one probe")
+
+	g.stop()
+	assert.True(t, p.ready(), "a cached answer is served while the ttl holds")
+
+	clock = clock.Add(3 * time.Second)
+	assert.False(t, p.ready(), "after the ttl the gateway is probed again")
+}
+
+func TestGatewayProbe_HandlerStatus(t *testing.T) {
+	g := newTLSGateway(t)
+	p := g.probe(t)
+	p.ttl = 0
+
+	w := httptest.NewRecorder()
+	p.handle(w, httptest.NewRequest(http.MethodGet, "/readyz", nil))
+	assert.Equal(t, http.StatusOK, w.Code)
+
+	g.stop()
+	w = httptest.NewRecorder()
+	p.handle(w, httptest.NewRequest(http.MethodGet, "/readyz", nil))
+	assert.Equal(t, http.StatusServiceUnavailable, w.Code)
+}
+
+func TestNewGatewayProbe_DefaultsPortAndRejectsBadInput(t *testing.T) {
+	p, err := newGatewayProbe("https://gw.example", "", "")
+	require.NoError(t, err)
+	assert.Equal(t, "gw.example:443", p.addr)
+	assert.Equal(t, defaultAPIServerAddr, p.apiAddr)
+
+	_, err = newGatewayProbe("https://", "", "")
+	require.Error(t, err)
+	_, err = newGatewayProbe("https://gw.example:9999", "/no/such/ca.pem", "")
+	require.Error(t, err)
+}
+
+func TestGatewayProbe_NotReadyWhenAPIServerDown(t *testing.T) {
+	g := newTLSGateway(t)
+	p := g.probe(t)
+	p.ttl = 0
+	require.True(t, p.ready())
+
+	g.api.stop()
+	assert.False(t, p.ready(), "a dead apiserver must not look ready while the gateway is up")
+
+	w := httptest.NewRecorder()
+	p.handle(w, httptest.NewRequest(http.MethodGet, "/readyz", nil))
+	assert.Equal(t, http.StatusServiceUnavailable, w.Code)
+}
+
+func TestReadinessHandler_ServesOnlyReadyz(t *testing.T) {
+	g := newTLSGateway(t)
+	h := readinessHandler(g.probe(t))
+
+	for _, path := range []string{"/authenticate", "/healthz", "/"} {
+		w := httptest.NewRecorder()
+		h.ServeHTTP(w, httptest.NewRequest(http.MethodPost, path, nil))
+		assert.Equalf(t, http.StatusNotFound, w.Code, "%s must not be served on the readiness listener", path)
+	}
+	w := httptest.NewRecorder()
+	h.ServeHTTP(w, httptest.NewRequest(http.MethodGet, "/readyz", nil))
+	assert.Equal(t, http.StatusOK, w.Code)
+}

spinifex/agents/eks/tokenwebhook/run_test.go

@@ -128,3 +128,58 @@ func TestRunServesUntilCancelled(t *testing.T) {
 		t.Fatal("Run did not return after cancel")
 	}
 }
+
+func TestRunServesReadinessOnSeparateListener(t *testing.T) {
+	g := newTLSGateway(t)
+	cfg := runConfig(t, freeAddr(t))
+	cfg.GatewayURL = "https://" + g.addr
+	cfg.GatewayCA = g.caPath
+	cfg.APIServerAddr = g.api.addr
+	cfg.ReadyAddr = freeAddr(t)
+
+	ctx, cancel := context.WithCancel(context.Background())
+	done := make(chan error, 1)
+	go func() { done <- Run(ctx, cfg) }()
+
+	client := &http.Client{Timeout: 3 * time.Second}
+	require.Eventually(t, func() bool {
+		resp, err := client.Get("http://" + cfg.ReadyAddr + "/readyz")
+		if err != nil {
+			return false
+		}
+		_ = resp.Body.Close()
+		return resp.StatusCode == http.StatusOK
+	}, 10*time.Second, 50*time.Millisecond, "readiness listener never reported ready")
+
+	resp, err := client.Post("http://"+cfg.ReadyAddr+"/authenticate", "application/json", strings.NewReader("{}"))
+	require.NoError(t, err)
+	_ = resp.Body.Close()
+	assert.Equal(t, http.StatusNotFound, resp.StatusCode, "the readiness listener must not serve /authenticate")
+
+	cancel()
+	select {
+	case err := <-done:
+		require.NoError(t, err)
+	case <-time.After(5 * time.Second):
+		t.Fatal("Run did not return after cancel")
+	}
+}
+
+func TestRunReportsReadinessListenFailure(t *testing.T) {
+	l, err := net.Listen("tcp", "127.0.0.1:0")
+	require.NoError(t, err)
+	defer l.Close()
+
+	cfg := runConfig(t, freeAddr(t))
+	cfg.ReadyAddr = l.Addr().String()
+
+	done := make(chan error, 1)
+	go func() { done <- Run(context.Background(), cfg) }()
+	select {
+	case err := <-done:
+		require.Error(t, err)
+		assert.True(t, strings.HasPrefix(err.Error(), "serve readiness: "), err.Error())
+	case <-time.After(10 * time.Second):
+		t.Fatal("Run kept serving after the readiness listener failed")
+	}
+}

spinifex/agents/eks/tokenwebhook/webhook.go

@@ -43,6 +43,10 @@ type Config struct {
 	CertPath    string
 	KeyPath     string
 	Kubeconfig  string
+	// ReadyAddr, when set, is a plain-HTTP listener serving only /readyz for
+	// load balancer health checks. APIServerAddr defaults to the local k3s.
+	ReadyAddr     string
+	APIServerAddr string
 }
 
 // Run writes the serving certificate and apiserver kubeconfig, then serves
@@ -63,6 +67,11 @@ func Run(ctx context.Context, cfg Config) error {
 		return fmt.Errorf("build gateway client: %w", err)
 	}
 
+	probe, err := newGatewayProbe(cfg.GatewayURL, cfg.GatewayCA, cfg.APIServerAddr)
+	if err != nil {
+		return fmt.Errorf("build readiness probe: %w", err)
+	}
+
 	authr := &authenticator{
 		accountID:   cfg.AccountID,
 		clusterName: cfg.ClusterName,
@@ -74,6 +83,7 @@ func Run(ctx context.Context, cfg Config) error {
 		w.WriteHeader(http.StatusOK)
 		_, _ = w.Write([]byte("ok"))
 	})
+	mux.HandleFunc("/readyz", probe.handle)
 	mux.HandleFunc("/authenticate", authr.handle)
 
 	server := &http.Server{
@@ -82,12 +92,38 @@ func Run(ctx context.Context, cfg Config) error {
 		ReadHeaderTimeout: 5 * time.Second,
 		TLSConfig:         serverTLSConfig(tlsCert),
 	}
-	stop := context.AfterFunc(ctx, func() { _ = server.Close() })
+	readyErr := make(chan error, 1)
+	var readySrv *http.Server
+	if cfg.ReadyAddr != "" {
+		readySrv = &http.Server{
+			Addr:              cfg.ReadyAddr,
+			Handler:           readinessHandler(probe),
+			ReadHeaderTimeout: 5 * time.Second,
+		}
+		go func() {
+			if err := readySrv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
+				readyErr <- fmt.Errorf("serve readiness: %w", err)
+				_ = server.Close()
+			}
+		}()
+		defer func() { _ = readySrv.Close() }()
+	}
+	stop := context.AfterFunc(ctx, func() {
+		_ = server.Close()
+		if readySrv != nil {
+			_ = readySrv.Close()
+		}
+	})
 	defer stop()
-	slog.Info("eks-token-webhook listening", "addr", cfg.Addr, "cluster", cfg.ClusterName)
+	slog.Info("eks-token-webhook listening", "addr", cfg.Addr, "ready_addr", cfg.ReadyAddr, "cluster", cfg.ClusterName)
 	if err := server.ListenAndServeTLS("", ""); err != nil && !errors.Is(err, http.ErrServerClosed) {
 		return fmt.Errorf("serve: %w", err)
 	}
+	select {
+	case err := <-readyErr:
+		return err
+	default:
+	}
 	return nil
 }
 
@@ -101,8 +137,8 @@ type tokenReviewRequest struct {
 
 // gatewayReviewer returns the per-call relay: SigV4-POST {accountId,token} to
 // the gateway and decode the resolved identity. A non-2xx / transport error is
-// surfaced (handle() turns it into a 5xx so the apiserver retries) rather than a
-// silent deny, distinguishing "webhook broker down" from "token rejected".
+// surfaced (handle() turns it into a 5xx) rather than a silent deny, which
+// distinguishes "webhook broker down" from "token rejected" in the logs.
 func gatewayReviewer(client *eksgw.Client, clusterName, accountID string) func(string) (ekscluster.WebhookTokenReviewResult, error) {
 	path := "/clusters/" + clusterName + "/token-review"
 	return func(token string) (ekscluster.WebhookTokenReviewResult, error) {
@@ -142,8 +178,8 @@ func (a *authenticator) handle(w http.ResponseWriter, r *http.Request) {
 
 	res, err := a.review(review.Spec.Token)
 	if err != nil {
-		// Broker fault (gateway down, transport error): a 5xx tells the apiserver
-		// to retry rather than treating a transient outage as a hard deny.
+		// Broker fault (gateway down, transport error). The apiserver treats any
+		// webhook error as an authentication failure and does not retry it.
 		slog.Error("TokenReview relay failed", "cluster", a.clusterName, "err", err)
 		http.Error(w, "token review unavailable", http.StatusServiceUnavailable)
 		return

spinifex/agents/elbv2/health.go

@@ -22,6 +22,11 @@ type HealthReport struct {
 	Servers []ServerStatus `json:"servers"`
 }
 
+// FailoverServerSuffix marks the unchecked backup copy of a server that HAProxy
+// uses only when every checked server is down. '.' cannot appear in a sanitized
+// target name, so a copy never collides with a real server.
+const FailoverServerSuffix = ".fo"
+
 // queryHAProxyStats reads backend server health from the HAProxy stats socket.
 // Parses CSV "show stat" output; cols 0/1/17 = pxname/svname/status.
 func queryHAProxyStats(socketPath string) ([]ServerStatus, error) {
@@ -53,7 +58,7 @@ func queryHAProxyStats(socketPath string) ([]ServerStatus, error) {
 		}
 
 		svname := fields[1]
-		if svname == "FRONTEND" || svname == "BACKEND" {
+		if svname == "FRONTEND" || svname == "BACKEND" || strings.HasSuffix(svname, FailoverServerSuffix) {
 			continue
 		}
 

spinifex/agents/elbv2/health_test.go

@@ -58,3 +58,38 @@ func TestQueryHAProxyStats_SocketNotFound(t *testing.T) {
 		t.Error("expected error for non-existent socket")
 	}
 }
+
+func TestQueryHAProxyStats_IgnoresFailoverCopies(t *testing.T) {
+	dir := t.TempDir()
+	sock := filepath.Join(dir, "haproxy.sock")
+
+	ln, err := net.Listen("unix", sock)
+	if err != nil {
+		t.Fatalf("listen: %v", err)
+	}
+	defer ln.Close()
+
+	go func() {
+		conn, err := ln.Accept()
+		if err != nil {
+			return
+		}
+		defer conn.Close()
+
+		buf := make([]byte, 256)
+		conn.Read(buf)
+
+		csv := "# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight\n"
+		csv += "bk_tg1,srv_i-aaa111,0,0,0,0,0,5,0,0,0,0,0,0,0,0,0,DOWN,1\n"
+		csv += "bk_tg1,srv_i-aaa111" + FailoverServerSuffix + ",0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,UP,1\n"
+		fmt.Fprint(conn, csv)
+	}()
+
+	servers, err := queryHAProxyStats(sock)
+	if err != nil {
+		t.Fatalf("queryHAProxyStats: %v", err)
+	}
+	if len(servers) != 1 || servers[0].Server != "srv_i-aaa111" || servers[0].Status != "DOWN" {
+		t.Fatalf("got %+v, want only srv_i-aaa111/DOWN", servers)
+	}
+}

spinifex/bootstrap/preflight/hostdns.go

@@ -0,0 +1,44 @@
+package preflight
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"slices"
+)
+
+const resolvConf = "/etc/resolv.conf"
+
+// resolvedStubs are the files that send every lookup through systemd-resolved:
+// its runtime stub and the static copy it ships.
+var resolvedStubs = []string{
+	"/run/systemd/resolve/stub-resolv.conf",
+	"/usr/lib/systemd/resolv.conf",
+	"/lib/systemd/resolv.conf",
+}
+
+// checkHostDNS warns when systemd-resolved runs but /etc/resolv.conf bypasses
+// it. The Spinifex zones are routed to northstar inside resolved, so lookups
+// that skip it never see them.
+func checkHostDNS(root string) Result {
+	const unit = "systemd-resolved.service"
+	if !unitActive(unit) {
+		return Result{Path: resolvConf, Kind: "host-dns", Status: OK}
+	}
+	target, err := os.Readlink(filepath.Join(root, resolvConf))
+	if err == nil {
+		if !filepath.IsAbs(target) {
+			target = filepath.Join(filepath.Dir(resolvConf), target)
+		}
+		if slices.Contains(resolvedStubs, filepath.Clean(target)) {
+			return Result{Path: resolvConf, Kind: "host-dns", Status: OK}
+		}
+	}
+	what := "is not a link"
+	if err == nil {
+		what = fmt.Sprintf("links to %s", target)
+	}
+	return Result{Path: resolvConf, Kind: "host-dns", Status: Warning,
+		Detail: fmt.Sprintf("%s is active but %s %s to its stub, so Spinifex names may not resolve on this host; run 'ln -sf %s %s'",
+			unit, resolvConf, what, resolvedStubs[0], resolvConf)}
+}

spinifex/bootstrap/preflight/hostdns_test.go

@@ -0,0 +1,55 @@
+package preflight
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+func TestCheckHostAt_HostDNS(t *testing.T) {
+	for name, tc := range map[string]struct {
+		resolved bool
+		link     string // target of /etc/resolv.conf; "" writes a regular file
+		want     Status
+		detail   string
+	}{
+		"resolved inactive, plain file":       {want: OK},
+		"resolved on the runtime stub":        {resolved: true, link: "/run/systemd/resolve/stub-resolv.conf", want: OK},
+		"resolved on a relative stub link":    {resolved: true, link: "../run/systemd/resolve/stub-resolv.conf", want: OK},
+		"resolved on the static stub":         {resolved: true, link: "/usr/lib/systemd/resolv.conf", want: OK},
+		"resolved behind a plain file":        {resolved: true, want: Warning, detail: "is not a link"},
+		"resolved behind its uplink file":     {resolved: true, link: "/run/systemd/resolve/resolv.conf", want: Warning, detail: "links to /run/systemd/resolve/resolv.conf"},
+		"resolved behind NetworkManager file": {resolved: true, link: "/run/NetworkManager/resolv.conf", want: Warning, detail: "links to"},
+	} {
+		t.Run(name, func(t *testing.T) {
+			orig := unitActive
+			t.Cleanup(func() { unitActive = orig })
+			unitActive = func(unit string) bool { return tc.resolved && unit == "systemd-resolved.service" }
+			root := t.TempDir()
+			path := filepath.Join(root, resolvConf)
+			if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
+				t.Fatal(err)
+			}
+			var err error
+			if tc.link == "" {
+				err = os.WriteFile(path, []byte("nameserver 192.0.2.1\n"), 0o644)
+			} else {
+				err = os.Symlink(tc.link, path)
+			}
+			if err != nil {
+				t.Fatal(err)
+			}
+			r := findResult(t, CheckHostAt(root), resolvConf)
+			if r.Status != tc.want {
+				t.Fatalf("status = %s %q, want %s", r.Status, r.Detail, tc.want)
+			}
+			if !strings.Contains(r.Detail, tc.detail) {
+				t.Errorf("detail %q does not contain %q", r.Detail, tc.detail)
+			}
+			if r.Status == Warning && HasProblem([]Result{r}) {
+				t.Error("a Warning fails preflight; it must only be reported")
+			}
+		})
+	}
+}

spinifex/bootstrap/preflight/hostfirewall.go

@@ -0,0 +1,62 @@
+package preflight
+
+import (
+	"bufio"
+	"bytes"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strings"
+)
+
+// ufwConfig records whether ufw is enabled. ufw.service is a oneshot that stays
+// active after `ufw disable`, so the unit state alone does not say ufw is on.
+const ufwConfig = "/etc/ufw/ufw.conf"
+
+// unitActive is a variable so tests never query a real systemd.
+var unitActive = func(unit string) bool {
+	return exec.Command("systemctl", "is-active", "--quiet", unit).Run() == nil
+}
+
+// checkHostFirewalls fails while firewalld or ufw is active. A forwarded packet
+// must then be accepted by their ruleset as well as Spinifex's, so routed NAT,
+// EIPs and IMDS break with nothing logged.
+func checkHostFirewalls(root string) []Result {
+	return []Result{checkFirewalld(), checkUFW(root)}
+}
+
+func checkFirewalld() Result {
+	const unit = "firewalld.service"
+	if unitActive(unit) {
+		return Result{Path: unit, Kind: "host-firewall", Status: Failed,
+			Detail: "firewalld is active beside the Spinifex nftables policy; run 'systemctl mask --now firewalld'"}
+	}
+	return Result{Path: unit, Kind: "host-firewall", Status: OK}
+}
+
+func checkUFW(root string) Result {
+	const unit = "ufw.service"
+	if unitActive(unit) && ufwEnabled(root) {
+		return Result{Path: unit, Kind: "host-firewall", Status: Failed,
+			Detail: "ufw is enabled beside the Spinifex nftables policy; run 'ufw disable'"}
+	}
+	return Result{Path: unit, Kind: "host-firewall", Status: OK}
+}
+
+// ufwEnabled reads ENABLED= from ufw.conf, as ufw-init does. A missing file
+// means ufw is not installed.
+func ufwEnabled(root string) bool {
+	data, err := os.ReadFile(filepath.Join(root, ufwConfig))
+	if err != nil {
+		return false
+	}
+	sc := bufio.NewScanner(bytes.NewReader(data))
+	for sc.Scan() {
+		key, value, ok := strings.Cut(strings.TrimSpace(sc.Text()), "=")
+		if ok && key == "ENABLED" {
+			value = strings.Trim(value, `"'`)
+			return value == "yes" || value == "YES" // the two spellings ufw-init accepts
+		}
+	}
+	return false
+}

spinifex/bootstrap/preflight/hostfirewall_test.go

@@ -0,0 +1,53 @@
+package preflight
+
+import (
+	"os"
+	"path/filepath"
+	"slices"
+	"strings"
+	"testing"
+)
+
+func TestCheckHostAt_HostFirewalls(t *testing.T) {
+	for name, tc := range map[string]struct {
+		active    []string
+		ufwConf   string
+		firewalld Status
+		ufw       Status
+		detail    string
+	}{
+		"neither installed":                 {firewalld: OK, ufw: OK},
+		"firewalld active":                  {active: []string{"firewalld.service"}, firewalld: Failed, ufw: OK, detail: "systemctl mask --now firewalld"},
+		"ufw unit active after ufw disable": {active: []string{"ufw.service"}, ufwConf: "ENABLED=no\n", firewalld: OK, ufw: OK},
+		"ufw enabled":                       {active: []string{"ufw.service"}, ufwConf: "# /etc/ufw/ufw.conf\nENABLED=yes\nLOGLEVEL=low\n", firewalld: OK, ufw: Failed, detail: "ufw disable"},
+		"ufw enabled, upper case":           {active: []string{"ufw.service"}, ufwConf: "ENABLED=YES\n", firewalld: OK, ufw: Failed, detail: "ufw disable"},
+		"ufw enabled but its unit stopped":  {ufwConf: "ENABLED=yes\n", firewalld: OK, ufw: OK},
+		"both active":                       {active: []string{"firewalld.service", "ufw.service"}, ufwConf: "ENABLED=yes\n", firewalld: Failed, ufw: Failed},
+	} {
+		t.Run(name, func(t *testing.T) {
+			orig := unitActive
+			t.Cleanup(func() { unitActive = orig })
+			unitActive = func(unit string) bool { return slices.Contains(tc.active, unit) }
+			root := t.TempDir()
+			if tc.ufwConf != "" {
+				path := filepath.Join(root, ufwConfig)
+				if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
+					t.Fatal(err)
+				}
+				if err := os.WriteFile(path, []byte(tc.ufwConf), 0o644); err != nil {
+					t.Fatal(err)
+				}
+			}
+			results := CheckHostAt(root)
+			for unit, want := range map[string]Status{"firewalld.service": tc.firewalld, "ufw.service": tc.ufw} {
+				r := findResult(t, results, unit)
+				if r.Status != want {
+					t.Errorf("%s = %s %q, want %s", unit, r.Status, r.Detail, want)
+				}
+				if r.Status == Failed && !strings.Contains(r.Detail, tc.detail) {
+					t.Errorf("%s detail %q does not name the fix %q", unit, r.Detail, tc.detail)
+				}
+			}
+		})
+	}
+}