feat: build an Oracle Linux appliance gold image that sets each unit up on first boot

* feat: add the store-neutral SQS message store contract and conformance suite

Defines message.Store over opaque shard and queue ID namespaces with no
create call, plus a reusable conformance runner exercised against a
test-only in-memory reference store.

* feat: add the SQS message store's JetStream layout, append and state CAS

Provisions the fixed set of shared message streams and state buckets,
publishes each queue on sqs.m.<shard>.<queue>, dedupes idempotent appends
with a queue-scoped Nats-Msg-Id, and maps state revision CAS onto KV.

* fix: give each SQS queue immutable message and state shard allocations

A changed stream or bucket count applies only to new queues: the namespace
names its message shard and state shard directly, retained streams and
buckets stay addressable, and the store refuses cluster sizes other than
one node (R1) or three (R3).

* fix: enumerate complete SQS namespaces and backstop state buckets

Enumeration now returns each namespace with both allocations, so a
state-only namespace can be counted and cleaned without its queue record,
and state buckets expire records after 14 days plus 1 hour.

* feat: add the SQS JetStream store's reads, counts and cleanup

Get and Delete are scoped to the namespace's subject, DeleteThrough and
DeleteNamespace are subject-filtered purges that leave shard neighbours
intact, and Count reads the per-subject message total and live state keys.

* fix: count distinct SQS state keys and validate before namespace purge

The key lister can repeat a key under concurrent writes, so Count tallies
distinct names; DeleteNamespace resolves both the message stream and the
state bucket before purging either, so a wrong state shard deletes nothing.

* feat: add the SQS JetStream store's ordered message-shard feed

Append records the namespace's state shard in an internal header that is
never returned as metadata. Feed reads one shared message stream through
an ephemeral, headers-only ordered consumer with no acknowledgement state,
rebuilds each change's complete namespace, refuses a malformed header with
ErrCorruptRecord, and deletes its consumer on Close.

* feat: enumerate SQS namespaces and run conformance on the JetStream store

Namespaces joins per-subject body counts, whose internal header names the
state shard, with live state keys from every state bucket that exists,
found by listing so a historical bucket counts and none is created. An
allocation conflict is reported as ErrCorruptRecord. The store now
implements message.Store and runs the conformance suite on R1, per case
with crash reopen and on one shared instance.

* fix: check every SQS body's state shard when enumerating namespaces

Sampling the first and last body let a middle body naming another state
shard pass. Namespaces now reads every body's header in one headers-only
ordered pass over the shard's stream, counting each queue's bodies from
that pass, and deletes its consumer when done.

* test: run SQS store conformance on R3 and guard against consumer state

TestR3Conformance runs the conformance suite on a real three-node
JetStream cluster in process, with Reopen restarting every node over the
same data, and a check that every stream and state bucket is R3.
TestS5_NoConsumerDeliveryStateInSQS fails on any acknowledgement,
redelivery, durable-consumer or ack-retention mechanism in SQS source.

* test: bar legacy NATS acknowledgement helpers from SQS source

Adds AckSync, ManualAck, AckAll and AckExplicit to the ADR-0011:S5
consumer-state guard.

* feat: add a bounded SQS store scan and the tombstoned-body sweeper

Store gains Scan: a paginated, headers-only pass over a message shard,
bounded at the shard's last position when it starts, skipping deleted
positions and never returning a later append. The conformance suite and
fake cover continuation, deleted runs and the bound.

The JetStream adapter reads every bounded range (scan, body enumeration
and state keys) through one reader whose completion rests on a direct get
of the next match, not on consumer state; the client's key lister and
pending counts are no longer trusted to end a pass.

message.Sweep walks a shard with Scan and, for each body whose record the
domain reports terminal, deletes the body then the record by revision.

* test: add an opt-in SQS store conformance run on an external R3 cluster

TestR3ExternalConformance runs the conformance suite against the
dedicated three-node SQS JetStream cluster named by SQS_STORE_R3_URLS,
checks every stream is R3 with two peers, and uses SQS_STORE_R3_RESTART
to restart every node for the crash cases. It skips when unset.

* test: run the SQS sweeper against the reference store fake

Covers live records kept, tombstones swept, another shard untouched, a
no-op rerun, and a retry after the body delete succeeds and the state
delete fails.

* test: harden the SQS store R3 harnesses against restart and placement races

The external R3 harness reused one connection across restarts, so a case's
cleanup closed it for every later case; each case and restart now opens its
own connection and waits for stream leaders.

The in-process cluster released its free ports before the servers bound them,
and R3 placement can fail until the meta leader has every peer's config and
stats. Client ports are now server-chosen, route ports are reserved together,
and provisioning retries the idempotent New until placement succeeds.

* fix: stop the SQS store feed idling for a whole pull expiry

Under load an ordered feed on a three-node cluster sat for nats.go's 30s
default pull expiry, once 60s, with stored changes pending and no pull
waiting on its consumer, both on feeds opened after a full restart and on
feeds kept open across one. Feed now sets a 2s pull expiry.

Adds restart tests for a feed opened after a full-cluster restart and one
open across it, each replaying a shared-shard backlog then a live append and
failing on any 5s gap. The in-process cluster keeps client ports across
restarts so open connections reconnect, and can write node logs.

* fix: read SQS message state from the stream leader

GetState used nats.go's KV Get, which switches to a direct get whenever
the bucket allows it. On a replicated bucket a follower can answer that
from state it has not applied yet, so a record the leader had already
deleted or purged could still be returned.

Read the key's last message with a stream message get instead, which
only the stream leader answers, and apply KV Get's delete and purge
marker rules to the result. Bucket configuration is unchanged. Missing,
deleted and purged records still return ErrNotFound; values, revisions,
context errors and the API prefix or domain of the caller's JetStream
context are preserved.

* docs: narrow the SQS GetState leader-read comment to what it avoids

* fix: enumerate SQS state keys from the stream leader

Count and Namespaces found live state keys with a direct-get probe and a
headers-only ordered consumer on the state bucket's stream. On a
replicated bucket a follower can answer either from state it has not
applied, so keys the leader had already deleted could still be counted.

Walk the key prefix with stream message gets by next subject, which
only the stream leader answers, through the stream's last sequence at
the call. Delete and purge markers drop a key, as GetState reads them.
This costs one leader request per record under the prefix. Bucket
configuration, historical state shards, the message-body scans and the
feed are unchanged.

* test: keep SQS API recorder deliveries in arrival order

The recorder fed API subjects and its sentinel to two async
subscriptions, which nats.go delivers on separate goroutines, so drain
could return before the last request was recorded. Both now share one
channel that the connection fills in arrival order.

* fix: read SQS body enumerations from the stream leader

Namespaces and Scan read bodies through an ephemeral ordered consumer,
which the server may place on a follower that has not applied a purge,
so a deleted queue could still be listed or scanned. Each stored match
through the bound is now a leader stream message get. That transfers
every full body, where the consumer read headers only.

* test: stop the SQS API recorder goroutine with its test

The recorder ranged over a caller-owned ChanSubscribe channel, which
nats.go never closes, so its goroutine outlived every test. Cleanup now
unsubscribes both subscriptions, then signals the goroutine and waits
for it to return; the sentinel send also watches that signal.

* fix: recover a stalled SQS feed from its cursor

An ordered feed was seen to idle with stored changes pending after a
cluster restart. Next now waits at most an idle check on the consumer,
then asks the stream leader whether a change follows the cursor; if one
does, the consumer is deleted and reopened just after the cursor, so no
change is skipped or repeated. The root cause of the stall is unknown.

* fix: make closing an SQS feed terminal and safe against Next

Close now marks the feed closed under a lock, cancels any in-flight
Next, stops the iterator and deletes the consumer. A later or in-flight
Next returns ErrFeedClosed and never opens a consumer; one opened by a
recovery racing Close is deleted rather than adopted. Repeated Close
returns nil.

* fix: define SQS feed close in the store contract

message.Feed now documents one Next reader with Close callable at any
time and more than once, and message.ErrFeedClosed replaces the
adapter's own error. Once Close has begun, Next returns ErrFeedClosed
and withholds any change it had received; a feed from the last cursor
still delivers it. Shared conformance cases and the reference fake
enforce this.

The JetStream feed's Close also waits, within a bound, for a consumer
open already in progress, so a consumer created during Close is deleted
before Close returns nil; past the bound, or if that delete fails, Close
returns an error and a repeated Close waits again.

* fix: track SQS feed consumer cleanup until Close succeeds

Every consumer open and release is now tracked work recorded under the
lock that takes the consumer, and a delete that fails is kept for Close
to retry. Close returns nil only once no such work is in progress and no
failed delete remains, waiting within a bound and otherwise returning an
error; concurrent and repeated Close calls wait the same way.

Next claims a change under that lock and advances its cursor there, so a
change is either returned before Close begins or withheld for a feed
resumed from the last cursor.

A consumer create no longer inherits the caller's cancellation, which
was seen to leave most cancelled creates on the server with no owner;
it has its own timeout, and the name of one that fails is deleted and
kept for Close to delete again.

* fix: release SQS feed consumers opened after the caller's context ends

Feed and recovery now check the caller's context before starting a
consumer open and after it completes. A create that finishes after the
context ended is released as tracked work and the context's error is
returned, or ErrFeedClosed when Close won; no change is returned.

The feed contract and adapter docs now describe cleanup as complete for
consumers the feed knows of, with the server's one-minute inactive
threshold removing one an ambiguous create lands after its last delete.

* fix: report cancellation when an SQS feed open fails after it

A first consumer open that fails after the caller's context ended now
returns the context's error joined with the failure, so cancellation
stays detectable. The feed contract's cancellation text is narrowed to
opening and reopening, whose cleanup is best effort.

* docs: describe the internal SQS message store and its limits

DESIGN.md gains a section on the store-neutral SQS message store, its
JetStream adapter and conformance suite: an internal primitive with no
service wired to it, its shared stream and bucket layout, leader-directed
reads and their full-body cost, feed recovery and its best-effort
consumer cleanup, and the absence of end-to-end or multi-host
validation. The Scan contract says metadata only, not headers only.

.github/e2e-cells.json

@@ -223,5 +223,27 @@
     "runner": "ci-single",
     "extra": "nat-single",
     "suite": ""
+  },
+  {
+    "cell": 37,
+    "install": "binary",
+    "topo": "real-multi",
+    "net": "dhcp",
+    "host": "oracle-10",
+    "arch": "x86_64",
+    "runner": "ci-multi",
+    "extra": "",
+    "suite": ""
+  },
+  {
+    "cell": 38,
+    "install": "binary",
+    "topo": "real-multi",
+    "net": "nat",
+    "host": "oracle-10",
+    "arch": "x86_64",
+    "runner": "ci-multi",
+    "extra": "nat-multi",
+    "suite": ""
   }
 ]

.github/workflows/actions.lock

@@ -73,7 +73,6 @@ workflows:
     '.github/workflows/ui.yml':
         - 'actions/checkout@v7.0.1'
         - 'actions/setup-node@v7.1.0'
-        - 'actions/upload-artifact@v7.0.2'
         - 'pnpm/action-setup@v6.1.0'
 dependencies:
     'actions/cache@v6.1.0':

.github/workflows/e2e-nightly.yml

@@ -10,7 +10,7 @@ on:
   workflow_dispatch:
     inputs:
       cells:
-        description: "Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 34-36 Oracle Linux single node (basic, reboot, routed NAT), 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection, 30 routed NAT on three nodes, 31 instance recovery on three nodes, 32 load balancer recovery on three nodes, 33 KV bucket quorum under node loss on three nodes"
+        description: "Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 34-36 Oracle Linux single node (basic, reboot, routed NAT), 37-38 Oracle Linux on three nodes (multi-node subset with IPsec, routed NAT), 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection, 30 routed NAT on three nodes, 31 instance recovery on three nodes, 32 load balancer recovery on three nodes, 33 KV bucket quorum under node loss on three nodes"
         required: false
 
 permissions:

.github/workflows/ui.yml

@@ -69,85 +69,3 @@ jobs:
             echo "$drift"
             exit 1
           fi
-
-  # The pinned image lives in one file that scripts/ui-playwright.sh also reads,
-  # so make ui-baselines and the job below draw screenshots in the same image.
-  ui_e2e_image:
-    name: UI End-to-End Image
-    runs-on: ubuntu-26.04
-    outputs:
-      image: ${{ steps.read.outputs.image }}
-
-    steps:
-      - name: Checkout Code
-        uses: actions/checkout@v7.0.1
-        with:
-          sparse-checkout: spinifex/runtime/roles/spinifexui/frontend/playwright.image
-          sparse-checkout-cone-mode: false
-
-      - name: Read Playwright Image
-        id: read
-        run: echo "image=$(tr -d '[:space:]' < spinifex/runtime/roles/spinifexui/frontend/playwright.image)" >> "$GITHUB_OUTPUT"
-
-  # The browser suite: the console against a faked API on Vite dev, plus the
-  # built dist under the production CSP. It runs in Playwright's own image, which
-  # carries the browsers, so screenshots rasterise as they did for the baselines.
-  ui_e2e:
-    name: UI End-to-End
-    needs: ui_e2e_image
-    runs-on: ubuntu-26.04
-    timeout-minutes: 20
-    container:
-      image: ${{ needs.ui_e2e_image.outputs.image }}
-      options: --ipc=host
-    defaults:
-      run:
-        working-directory: spinifex/runtime/roles/spinifexui/frontend
-
-    steps:
-      - name: Checkout Code
-        uses: actions/checkout@v7.0.1
-        with:
-          sparse-checkout: spinifex/runtime/roles/spinifexui/frontend
-
-      - name: Setup pnpm
-        uses: pnpm/action-setup@v6.1.0
-        with:
-          package_json_file: spinifex/runtime/roles/spinifexui/frontend/package.json
-
-      - name: Setup Node.js
-        uses: actions/setup-node@v7.1.0
-        with:
-          cache: "pnpm"
-          cache-dependency-path: spinifex/runtime/roles/spinifexui/frontend/pnpm-lock.yaml
-          node-version-file: spinifex/runtime/roles/spinifexui/frontend/.nvmrc
-
-      - name: Install Dependencies
-        run: pnpm install --frozen-lockfile
-
-      # The image's browsers belong to one Playwright release, so a lockfile
-      # bump without an image bump would otherwise fail on a missing executable.
-      - name: Check Playwright Matches the Image
-        env:
-          IMAGE: ${{ needs.ui_e2e_image.outputs.image }}
-        run: |
-          version=$(pnpm exec playwright --version | awk '{print $2}')
-          case "$IMAGE" in
-            *":v$version-"*) ;;
-            *) echo "::error::Lockfile has Playwright $version but the image is $IMAGE. Update playwright.image."; exit 1 ;;
-          esac
-
-      - name: Build Frontend
-        run: pnpm build
-
-      - name: Run End-to-End Suite
-        run: pnpm e2e
-
-      - name: Upload Screenshot Diffs
-        if: failure()
-        uses: actions/upload-artifact@v7.0.2
-        with:
-          name: ui-e2e-test-results
-          path: spinifex/runtime/roles/spinifexui/frontend/test-results/
-          if-no-files-found: ignore
-          retention-days: 14

Makefile

@@ -56,19 +56,6 @@ build-ui:
 	@echo -e "\n....Building spinifex-ui frontend...."
 	cd spinifex/runtime/roles/spinifexui/frontend && pnpm build
 
-# Playwright suite for the console, against Vite dev and the built dist under
-# the console's CSP. Not part of preflight; needs `pnpm exec playwright install chromium`.
-test-ui-e2e: build-ui
-	cd spinifex/runtime/roles/spinifexui/frontend && pnpm e2e
-
-# The same suite inside the pinned Playwright image CI uses (podman or docker),
-# so screenshots compare as they will on the runner. ui-baselines redraws them.
-ui-e2e-container:
-	scripts/ui-playwright.sh compare
-
-ui-baselines:
-	scripts/ui-playwright.sh baselines
-
 # GO commands
 VERSION ?= $(shell git describe --tags --always --dirty --exclude '*-dev')
 COMMIT  ?= $(shell git rev-parse --short HEAD)
@@ -199,13 +186,14 @@ test-build-scripts:
 		echo "-- $$t"; \
 		sh "$$t" || exit 1; \
 	done
-	@echo -e "\n....Running scripts/*_test.sh and scripts/lib/*_test.sh...."
-	@for t in $$( (find scripts -maxdepth 1 -name '*_test.sh'; find scripts/lib -name '*_test.sh') | sort); do \
+	@echo -e "\n....Running scripts/*_test.sh, scripts/lib/*_test.sh and scripts/appliance/*_test.sh...."
+	@for t in $$( (find scripts -maxdepth 1 -name '*_test.sh'; find scripts/lib scripts/appliance -name '*_test.sh') | sort); do \
 		echo "-- $$t"; \
 		sh "$$t" || exit 1; \
 	done
 	@echo -e "\n....Running shellcheck over build/scripts/**/*.sh + the node scripts...."
 	shellcheck -S warning $$(find build/scripts -name '*.sh' | sort) $(NODE_SCRIPTS)
+	shellcheck -S warning -x scripts/build-oracle-appliance.sh $$(find scripts/appliance -name '*.sh' | sort)
 	@echo "  test-build-scripts ok"
 
 # E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
@@ -499,7 +487,7 @@ distro-arm64:
 distro-clean:
 	rm -rf dist/
 
-.PHONY: test-package-check build build-ui test-ui-e2e ui-e2e-container ui-baselines build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-rds-postgres-image import-rds-postgres-image build-rds-mariadb-image import-rds-mariadb-image build-microvm-image install-microvm go_build preflight test test-cover test-race diff-coverage bench fuzz test-actions test-images test-build-scripts test-harness test-integration generate-aws-model-coverage aws-model-coverage test-segscan-oracle manifest-check manifest-lint manifest-lint-update \
+.PHONY: test-package-check build build-ui build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-rds-postgres-image import-rds-postgres-image build-rds-mariadb-image import-rds-mariadb-image build-microvm-image install-microvm go_build preflight test test-cover test-race diff-coverage bench fuzz test-actions test-images test-build-scripts test-harness test-integration generate-aws-model-coverage aws-model-coverage test-segscan-oracle manifest-check manifest-lint manifest-lint-update \
 	deploy reinstall clean \
 	install-system install-go install-aws quickinstall \
 	lint fix govulncheck nilaway \

architecture/layering/allowlist_test.go

@@ -29,8 +29,9 @@ var allowlist = []allowance{
 
 	{From: "agents/rds/agent", To: "domains/rds/instance", Debt: "agents/rds/agent (dependency, narrowed)", Reason: "command protocol, wire and record types; needs a guest/controller wire contract"},
 
-	{From: "cmd/aws-model-coverage", To: "domains/acm/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared ECR/ACM inventories; named exception, must not construct a registration"},
-	{From: "cmd/aws-model-coverage", To: "domains/ecr/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared ECR/ACM inventories; named exception, must not construct a registration"},
+	{From: "cmd/aws-model-coverage", To: "domains/acm/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared ECR/ACM/Route 53 inventories; named exception, must not construct a registration"},
+	{From: "cmd/aws-model-coverage", To: "domains/ecr/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared ECR/ACM/Route 53 inventories; named exception, must not construct a registration"},
+	{From: "cmd/aws-model-coverage", To: "domains/route53/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared ECR/ACM/Route 53 inventories; named exception, must not construct a registration"},
 
 	{From: "runtime/daemon", To: "domains/ec2/awsapi/instance", Debt: "Instance adapter consumers outside EC2", Reason: "runtime/daemon/eks_cp_control.go and rds_deps.go call the instance adapter; needs an EC2 capability"},
 

cmd/aws-model-coverage/main.go

@@ -4,11 +4,13 @@ import (
 	"flag"
 	"fmt"
 	"os"
+	"slices"
 
 	"github.com/mulgadc/predastore/s3api"
 	"github.com/mulgadc/spinifex/internal/awsmodel"
 	acmawsapi "github.com/mulgadc/spinifex/spinifex/domains/acm/awsapi"
 	awsapi "github.com/mulgadc/spinifex/spinifex/domains/ecr/awsapi"
+	route53awsapi "github.com/mulgadc/spinifex/spinifex/domains/route53/awsapi"
 	"github.com/mulgadc/spinifex/spinifex/gateway"
 	"github.com/mulgadc/spinifex/spinifex/ingress/aws/dispatch"
 
@@ -49,9 +51,17 @@ func compareAll() ([]awsmodel.OperationCoverage, error) {
 	// Registered services contribute their declared inventory, which needs no
 	// live dependencies; nothing here serves a request.
 	inventories := gateway.AWSOperationInventory(map[string]dispatch.Inventory{
-		awsapi.ServiceName:    awsapi.OperationInventory(),
-		acmawsapi.ServiceName: acmawsapi.OperationInventory(),
+		awsapi.ServiceName:        awsapi.OperationInventory(),
+		acmawsapi.ServiceName:     acmawsapi.OperationInventory(),
+		route53awsapi.ServiceName: route53awsapi.OperationInventory(),
 	})
+	// The loop below walks the embedded models, so a served service with no
+	// model would otherwise drop out of the report without a word.
+	for name := range inventories {
+		if !slices.Contains(awsmodel.Services(), awsmodel.Service(name)) {
+			return nil, fmt.Errorf("no embedded AWS model for served service %s", name)
+		}
+	}
 	coverages := make([]awsmodel.OperationCoverage, 0, len(awsmodel.Services()))
 	for _, service := range awsmodel.Services() {
 		inventory, ok := inventories[string(service)]

cmd/eks-token-webhook/main.go

@@ -29,6 +29,7 @@ func loadConfig() (tokenwebhook.Config, error) {
 		CertPath:    envOr("EKS_WEBHOOK_CERT", "/etc/spinifex-eks/token-webhook.crt"),
 		KeyPath:     envOr("EKS_WEBHOOK_KEY", "/etc/spinifex-eks/token-webhook.key"),
 		Kubeconfig:  envOr("EKS_WEBHOOK_KUBECONFIG", "/etc/spinifex-eks/token-webhook.kubeconfig"),
+		ReadyAddr:   os.Getenv("EKS_WEBHOOK_READY_ADDR"),
 	}
 	// Static SigV4 creds are optional: when absent, eksgw.New signs with the
 	// AWS SDK chain (IMDS instance-role creds), the same path the CP VM's

docs/DESIGN.md

@@ -229,6 +229,20 @@ type EBSRequest struct {
 
 Viperblock responds with an NBD URI that QEMU uses to access the block device.
 
+#### Direct-boot microVM data volumes
+
+A direct-boot microVM keeps the launcher's saved `vm.Config` as a clean base, and `startQEMU` adds data volumes only to a launch-only copy of it (`runtime/compute/vm/lifecycle.go`).
+At each start the daemon takes one snapshot of the instance's already-mounted data requests, probes each NBD or host block export from that snapshot, and builds the command from the same snapshot, so the launched endpoint is the one that was checked and is never written back into the saved base.
+Each data volume becomes a `virtio-blk-device` on the MMIO bus with the same id, node name, I/O thread, serial and `werror=stop,rerror=stop` policy as the PCI device, and takes no PCIe hot-plug port.
+
+Boot and EFI requests are rejected on this path, and a direct-boot instance with no data requests launches its base unchanged.
+Disk-boot (q35) instances keep the existing PCI hot-plug path.
+
+Hot-attaching a volume to a running direct-boot microVM is not supported, because the MMIO bus has no PCIe root port to plug into.
+`AttachVolume` refuses it before mounting the volume or changing any instance or volume state, and the EC2 API returns `UnsupportedOperation`.
+
+This covers only requests that are already mounted when the instance starts; nothing yet supplies data volumes to a system instance, and the path is verified with fake-QEMU unit tests, not with a real EBS volume in a guest.
+
 ### Predastore (S3)
 
 Object storage used for:
@@ -256,6 +270,29 @@ file. It does not connect to Predastore to do so, and no Predastore node
 exposes a status endpoint. This is an operator status interface, not a generic
 storage-provider contract.
 
+### SQS message store (internal)
+
+`domains/sqs/message` defines the store-neutral interface the SQS domain reaches its message store through (ADR-0011:S6), with a JetStream adapter in `domains/sqs/message/jetstream` and a shared conformance suite in `domains/sqs/message/conformance`.
+It is an internal primitive: nothing outside the SQS domain uses it, no daemon or service is wired to it, and it does not itself deliver an SQS API.
+The dedicated JetStream server ADR-0011 describes is not deployed by this code.
+
+The interface covers durable and idempotent append, body and metadata reads, conditional state writes and deletes, per-namespace counts, namespace enumeration, an ordered change feed, a bounded paginated scan, namespace deletion and a sweep of tombstoned bodies.
+A namespace carries an immutable message-shard and state-shard allocation chosen when the queue is created, so a later change in shard counts never moves an existing queue.
+
+The adapter provisions a fixed set of shared message streams (`SQS_M_<shard>`, one subject per queue) and shared state buckets (`sqs_state_<shard>`), keeps historical ones, and applies a 14-day-plus-one-hour retention backstop to both.
+It supports one node (R1) and three nodes (R3) only and refuses any other declared cluster size before provisioning.
+
+Reads that must observe a completed write or delete go to the stream leader: `GetState`, the state-key pass behind `Count` and `Namespaces`, and the body pass behind `Namespaces` and `Scan`.
+This avoids the follower-served direct gets and follower-placed consumers that returned purged data in local three-node fault-injection tests; it is not a general linearizability guarantee.
+Each leader read is one request per record, and the body pass transfers each full body internally even though `Scan` returns metadata only, which is a material cost for a large backlog.
+A state-key pass is bounded but is not a snapshot when a key is overwritten during it.
+
+The feed reads through an in-memory, headers-only ordered consumer.
+If that consumer delivers nothing for about two seconds while the stream holds a change after the feed's cursor, the feed replaces it from the cursor, so a stored change does not wait on a stalled consumer; the cause of those stalls is not established.
+`Close` is terminal, waits for consumer opens and deletes in progress and retries failed deletes, but cleanup is best effort for a create whose outcome is unknown: a consumer that lands after its last delete is removed by the server's one-minute inactive threshold.
+
+Coverage is unit and conformance tests on in-process R1 and R3 servers; there is no end-to-end or multi-host validation of this code, and earlier multi-host evidence predates the final feed.
+
 ## Configuration
 
 Cluster configuration (`spinifex/bootstrap/config/config.go`):

docs/PACKAGE_BOUNDARY_MIGRATION.md

@@ -438,7 +438,7 @@ These residuals were recorded by individual slices and must be resolved once the
 - `runtime/roles/vpcd` (placement): the role still mixes process/role supervision with network-controller responsibilities. `lease_reaper.go` (orphaned external-pool DHCP lease sweep), `ovn_controller_watchdog.go` (OVN controller liveness and restart), `host_eip_loop.go` (host-EIP reconcile loop) and `lease_datapath_reconcile.go` (lease datapath reconcile) are periodic network-control loops, not role wiring; `vpcd.go` itself also carries bridge-mode detection and verification, chassis-list parsing, external-CIDR resolution, DHCP manager startup and lease/IP-change handling, and gateway/host-EIP allocator and neighbour flush/prime construction alongside `Service.Start`. The future placement question is whether these belong in `domains/network`. Removal condition: an accepted decision on network-controller placement, and extraction of those loops and helpers behind the role.
 - ECR control-plane registration: `awsapi.Deps.NATS` is a raw `*nats.Conn`, not a narrow capability. Every NATS-backed `Handler` passes it to `domains/ecr.NewNATSMetaStore`, which calls `natsmsg.NATSRequest`, so narrowing it means changing the exported `Handler` type, every handler and the meta store; it must not be broadened further meanwhile.
 - ACM control-plane registration: `domains/acm/awsapi.Deps.NATS` (`registration.go`) is likewise a raw `*nats.Conn` carried by every `Handler`, for the same reason; it must not be broadened further meanwhile.
-- `cmd/aws-model-coverage` imports `ingress/aws/dispatch`, `domains/ecr/awsapi`, `domains/acm/awsapi` and `gateway` to read ECR's and ACM's declared inventories. It is a non-serving tool and the only named exception to the ADR-0002 composition invariant; it must not construct a registration.
+- `cmd/aws-model-coverage` imports `ingress/aws/dispatch`, `domains/ecr/awsapi`, `domains/acm/awsapi`, `domains/route53/awsapi` and `gateway` to read ECR's, ACM's and Route 53's declared inventories. It is a non-serving tool and the only named exception to the ADR-0002 composition invariant; it must not construct a registration.
 - Ingress pipeline (ADR-0002 phase 4): the request-audit record, the trace action enricher, the throttle key functions and the gateway request-context keys stay in `gateway` until the auth/IAM and request-context boundary is designed. SigV4 authentication writes the audit record and context keys from four places, so moving them now would export exactly the setters ADR-0002 says must not be exported solely to make a directory move possible.
 - NAT projection delivery is inconsistent and is preserved as found: EIP `AssociateAddress` sends `vpc.add-nat` unacknowledged (`AnnounceNATAdd`), skipping the 45s OVN barrier every other add path waits on, and `DeleteNetworkInterface` sends `vpc.delete-nat` unacknowledged (`AnnounceNATDelete`), skipping the ack and the ErrNoResponders retry. The reconciler's orphan sweep is the only backstop for both. Clears when those call sites use `AddNAT`/`AddNATBestEffort` and `RemoveNAT`, a reliability change of its own.
 - `vpc.add-system-egress` has a `vpcd` subscriber and a contract entry but no producer (its former `domains/ec2/eip` publisher is gone); `domains/eks/cluster` still publishes the delete. Either a producer is missing or the add route is dead; it needs an owner decision.

docs/admin/host-firewall/README.md

@@ -158,6 +158,7 @@ Restarting the daemon is what re-arms it: the node rebuilds its peer list from t
 
 > [!WARNING]
 > Do not flush the whole nftables ruleset to clear the policy. The VPC daemon writes SNAT and per-elastic-IP forward rules into the `ip nat` and `ip filter` tables and reinstalls them only when it starts, so `nft flush ruleset` breaks elastic IPs silently until the next `systemctl restart spinifex-vpcd`. Use `disable`, which only ever touches `inet spinifex_filter`.
+> On Oracle Linux with IPsec on, a flush also removes `inet spinifex_ipsec_guard`, so Geneve to a peer without an SA goes out in clear until the daemon puts the table back, within 60 seconds.
 
 ## The Files
 

docs/compute/vpc-networking/README.md

@@ -144,6 +144,27 @@ The guest MTU moves with the setting, but a running guest holds its DHCP lease f
 
 **Expect launches to fail for a minute or two after re-enabling.** `RunInstances` returns `ServerInternal` with `vpc.add-nat` timing out, because the OVN flows-ready barrier cannot confirm while `ovn-controller` reprograms every tunnel for the new SAs. It clears once `ip xfrm state` settles. The error is reported as a capacity problem, which it is not.
 
+### Containers and nested overlays inside a guest
+
+A Docker bridge, Kubernetes pod network or other overlay inside a guest defaults to an MTU of 1500, which is larger than the guest's 1408 or 1442.
+Downloads from the internet still work for TCP on the veth and routed uplinks, because vpcd clamps the TCP MSS on the host hop (see [Performance Tuning → MTU](/docs/performance-tuning#4-mtu)).
+That was measured on a single node (env12) with a Docker container; it was not measured on a physical uplink or on a multi-node cluster.
+
+On a physical uplink there is no host hop to clamp, and UDP is never clamped, so set the inner MTU to the guest MTU.
+For Docker, in `/etc/docker/daemon.json`:
+
+```json
+{
+  "mtu": 1408,
+  "default-network-opts": {
+    "bridge": { "com.docker.network.driver.mtu": "1408" }
+  }
+}
+```
+
+Use 1442 when `ipsec_enabled = false`, and restart the Docker daemon so the default bridge is recreated.
+The gateway MTU follows `ipsec_enabled` and converges on the next reconcile pass, so change the inner MTU together with the guest MTU.
+
 ## Private vs Public Subnets
 
 A subnet's behavior depends on three things: whether the VPC has an Internet Gateway, whether the subnet's route table has a default route to that IGW, and whether the subnet has `MapPublicIpOnLaunch` enabled.

docs/containers/ecr/README.md

@@ -45,6 +45,7 @@ Each repository holds image manifests and their layers. Images are stored in obj
 
 - Repository lifecycle — `CreateRepository`, `DeleteRepository`, `DescribeRepositories`.
 - OCI push and pull — blobs, manifests, and tags, including cross-repository blob mounts.
+- OCI 1.1 referrers — the referrers API, so cosign v3 signatures and other artifacts attached to an image can be listed (see [OCI referrers and image signing](#oci-referrers-and-image-signing)).
 - Token auth — `GetAuthorizationToken` plus bearer/basic auth on `/v2/`.
 - Repository policies — set, get, and delete a repository policy document.
 
@@ -176,6 +177,71 @@ Then authenticate Docker and push using the `repository_url` output, following s
 
 :::
 
+## EKS workers and multiple gateways
+
+Each gateway puts its own address in the `repositoryUri` it returns, so repositories created through different gateways can carry different hosts.
+When an EKS worker launches, the daemon that launches it writes every cluster gateway into the worker's registry mirrors, TLS trust and credential-provider config.
+A pod can then pull from any `repositoryUri` host in the cluster.
+The worker tries the gateway named in the image reference first and falls back to the others in cluster order.
+
+Limits:
+
+- The gateway list comes from the cluster config the launching daemon loaded at start.
+  A gateway added later is not trusted by existing workers; restart the launching daemon and replace the worker.
+- Peer gateways are assumed to listen on port 9999.
+  A hand-edited `awsgw` bind on a peer is not covered.
+- A gateway address that does not answer costs about 30 seconds on each cold pull while the worker's connection attempt gives up.
+  A stopped gateway on a host that is still up refuses the connection, and the worker moves to the next gateway at once.
+
+Measured on a single-node environment: pulls by `repositoryUri` and by the account's ECR DNS name, a pull rejected with `x509: certificate signed by unknown authority` when the gateway host was not in the worker config, and a 31 second pull through an unreachable first endpoint.
+Measured on a 3-node environment: a pod pulling each of the three gateways' `repositoryUri` on a worker launched with this change (two of the three failed with `x509` on a worker launched without it), and a pull of the stopped gateway's own `repositoryUri` that succeeded through another gateway within one retry (the pull itself took 147 ms after the first endpoint refused).
+
+## OCI referrers and image signing
+
+A manifest can name another manifest as its `subject`, which is how tools such as cosign v3 attach signatures and SBOMs to an image.
+The registry records the subject when the manifest is pushed and answers with an `OCI-Subject` header on the `201`.
+The `subject` does not have to exist in the repository when the referrer is pushed.
+
+`GET /v2/<name>/referrers/<digest>` lists the manifests that name `<digest>` as their subject.
+The request is authorised as `ecr:BatchGetImage` on the repository.
+The response is `200` with `Content-Type: application/vnd.oci.image.index.v1+json` and an index whose `manifests` hold each referrer's `mediaType`, `digest`, `size`, `artifactType` and annotations.
+A digest with no referrers, or one that is not stored, returns an empty index.
+A malformed digest returns `400` with `DIGEST_INVALID`.
+
+Adding `?artifactType=<type>` returns only referrers of that type and sets `OCI-Filters-Applied: artifactType`.
+The header is also set when nothing matches.
+An image manifest without an `artifactType` is listed under its config `mediaType`; an index without one has none.
+
+Deleting an image by digest (`DELETE /v2/<name>/manifests/<digest>`, `aws ecr batch-delete-image` with `imageDigest`) also deletes its referrers, and their referrers in turn.
+Deleting by tag removes only the tag and leaves the image and its referrers.
+
+cosign v3 sign and verify work against the registry, including on a child manifest of an image index copied with `cosign copy`.
+cosign v2 keeps writing its `sha256-<digest>.sig` tags and is unaffected.
+`GET` on a manifest still returns `406` when the request's `Accept` header does not include the stored `mediaType`.
+
+Limits:
+
+- `cosign copy` does not copy v3 signatures to the destination repository.
+  This is client behaviour.
+  To keep the original signature, copy with `oras cp -r <source> <mirror>`, which copies the image and its referrers in one command, or `crane copy` the signature manifest by digest after the image.
+  Otherwise, sign the copy again.
+- Referrers pushed before this version are not listed.
+  Push them again to add them to the index.
+- A crash between deleting an image and deleting its referrers can leave referrers behind that nothing lists.
+- The referrers listing is not paginated.
+- `HEAD` on the referrers route is not served.
+
+Measured on a 3-node environment with cosign v3.1.4 and v2.6.5:
+
+- cosign v3.1.4 signed a fresh image index, with one `GET /referrers` answered `200`, no `sha256-<digest>` tag written and no `406`.
+- Verify passed.
+- A copied index's child manifest signed and verified, which returned `406` before this change.
+- The listing returned the signature with its `artifactType`, the filter returned one entry for the signature's type and none for another type.
+- `aws ecr batch-delete-image` of the signed image left an empty listing and `404` for both the signature and the image.
+- cosign v2.6.5 signed and verified a fresh image.
+
+Not measured: `oras`, `notation`, a listing with many referrers, and a crash in the middle of a delete.
+
 ## Troubleshooting
 
 **`docker login` fails with an authorization error.** The token is account-scoped and short-lived. Re-run `aws ecr get-login-password` to mint a fresh token, and confirm the registry host in `docker login` matches your account's endpoint (`aws ecr describe-repositories` returns it).

docs/containers/eks/README.md

@@ -60,6 +60,36 @@ Authentication uses **EKS access entries** (the API authentication mode) — IAM
 - **The worker image is fixed.** Node groups always boot Spinifex's `eks-node` image. `ami_type` is recorded but does not select the image, and the image must be registered on the cluster before you create a node group.
 - **Default Kubernetes version is `1.32`.**
 
+### Control plane readiness and kubectl authentication
+
+Each control-plane VM authenticates `kubectl` tokens through a webhook that calls the AWS gateway on the host the VM runs on.
+When that gateway is stopped, the webhook cannot validate tokens and `kubectl` reports `Unauthorized` for the share of requests that the load balancer sends to that control plane.
+
+The webhook serves `/readyz` on port 8444 of each control-plane VM.
+It returns 200 only when the host gateway answers a TLS handshake and the local API server accepts a TCP connection, and 503 otherwise.
+The API server target group health-checks `HTTP :8444/readyz` every 5 seconds with a healthy and unhealthy threshold of 2.
+The target group still forwards TCP on 6443.
+A control plane that cannot authenticate leaves rotation, and returns when its gateway is back.
+
+Measured on a three-node cluster, one control plane per host, with `kubectl get ns` run continuously after stopping one gateway:
+
+| Gateway stopped | Result |
+| --- | --- |
+| Host without the cluster load balancer VM, first run | 4 of 20 calls failed in the first round, 0 of 20 in the next round 54 s after the stop |
+| Host without the cluster load balancer VM, second run | 2 failures in 100 s, at 3 s and 8 s after the stop |
+| Host running the cluster load balancer VM | Not fixed: 20 of 60 calls failed over 100 s |
+
+Before this change the same test failed 3, 4 and 3 calls of 10 per stopped gateway.
+
+Limits:
+
+- Only clusters created with the current `eks-node` image get the readiness listener, and only clusters created from now on get the health check on the target group.
+  Older clusters keep the TCP check on 6443 and are not changed.
+- A cluster created with an older image gets the health check on its target group but no listener on 8444, so every control plane fails the check and the load balancer fails open to all of them.
+- A stopped gateway on the host that runs the cluster load balancer VM is not covered.
+  The load balancer VM reports target health through its own host's gateway, so while that gateway is down the unhealthy control plane stays in rotation.
+- Not measured: more than one control plane per host, a physical-node failure instead of a stopped gateway, and the target health state as `DescribeTargetHealth` reports it, because the load balancer and control-plane VMs belong to the system account.
+
 ## Prerequisites
 
 > [!IMPORTANT]
@@ -123,7 +153,7 @@ aws eks describe-cluster --name demo --query 'cluster.status'
 aws eks create-nodegroup \
   --cluster-name demo \
   --nodegroup-name default \
-  --node-role-arn arn:aws:iam::000000000000:role/eks-node-role \
+  --node-role arn:aws:iam::000000000000:role/eks-node-role \
   --subnets subnet-aaaa subnet-bbbb \
   --scaling-config minSize=1,maxSize=2,desiredSize=1 \
   --instance-types t3.medium
@@ -256,6 +286,10 @@ The workbooks form a ladder. **eks-quickstart** is a NodePort demo; **eks-https-
 
 **`kubectl` cannot connect.** Re-run `aws eks update-kubeconfig --name <name>` to refresh the endpoint and CA, and confirm the cluster is `ACTIVE`. `aws sts get-caller-identity` verifies your credentials reach the Spinifex endpoint.
 
+**`kubectl` returns `Unauthorized` for some requests and works for others.**
+One control plane cannot reach its host's AWS gateway, so check `systemctl status spinifex-awsgw` on each node.
+Control plane readiness and kubectl authentication above lists which clusters recover on their own.
+
 **Workers stay `NotReady`.** Almost always an egress or IAM problem:
 - Confirm the workers can pull images — a default route to an Internet Gateway (public) or NAT Gateway (private).
 - Confirm the node role has `AmazonEKSWorkerNodePolicy`, `AmazonEKS_CNI_Policy`, and `AmazonEC2ContainerRegistryReadOnly`.

docs/coverage/README.md

@@ -17,7 +17,7 @@ tags:
 
 ## Overview
 
-The platform serves **392 operations** across the AWS APIs below. Every page names the operations its service implements from the pinned model, generated from the dispatch tables on each build rather than written by hand.
+The platform serves **405 operations** across the AWS APIs below. Every page names the operations its service implements from the pinned model, generated from the dispatch tables on each build rather than written by hand.
 
 | Service | Operations |
 |---|---:|
@@ -29,6 +29,7 @@ The platform serves **392 operations** across the AWS APIs below. Every page nam
 | [ELBv2](/coverage/elbv2) | 36 |
 | [IAM](/coverage/iam) | 81 |
 | [RDS](/coverage/rds) | 27 |
+| [Route 53](/coverage/route53) | 13 |
 | [S3](/coverage/s3) | 26 |
 | [STS](/coverage/sts) | 5 |
-| **Total** | **392** |
+| **Total** | **405** |

docs/coverage/coverage.json

@@ -3483,6 +3483,180 @@
         ]
       }
     },
+    {
+      "service": "route53",
+      "api_version": "2013-04-01",
+      "opaque": false,
+      "note": "",
+      "operations": {
+        "modelled": [
+          "ActivateKeySigningKey",
+          "AssociateVPCWithHostedZone",
+          "ChangeCidrCollection",
+          "ChangeResourceRecordSets",
+          "ChangeTagsForResource",
+          "CreateCidrCollection",
+          "CreateHealthCheck",
+          "CreateHostedZone",
+          "CreateKeySigningKey",
+          "CreateQueryLoggingConfig",
+          "CreateReusableDelegationSet",
+          "CreateTrafficPolicy",
+          "CreateTrafficPolicyInstance",
+          "CreateTrafficPolicyVersion",
+          "CreateVPCAssociationAuthorization",
+          "DeactivateKeySigningKey",
+          "DeleteCidrCollection",
+          "DeleteHealthCheck",
+          "DeleteHostedZone",
+          "DeleteKeySigningKey",
+          "DeleteQueryLoggingConfig",
+          "DeleteReusableDelegationSet",
+          "DeleteTrafficPolicy",
+          "DeleteTrafficPolicyInstance",
+          "DeleteVPCAssociationAuthorization",
+          "DisableHostedZoneDNSSEC",
+          "DisassociateVPCFromHostedZone",
+          "EnableHostedZoneDNSSEC",
+          "GetAccountLimit",
+          "GetChange",
+          "GetCheckerIpRanges",
+          "GetDNSSEC",
+          "GetGeoLocation",
+          "GetHealthCheck",
+          "GetHealthCheckCount",
+          "GetHealthCheckLastFailureReason",
+          "GetHealthCheckStatus",
+          "GetHostedZone",
+          "GetHostedZoneCount",
+          "GetHostedZoneLimit",
+          "GetQueryLoggingConfig",
+          "GetReusableDelegationSet",
+          "GetReusableDelegationSetLimit",
+          "GetTrafficPolicy",
+          "GetTrafficPolicyInstance",
+          "GetTrafficPolicyInstanceCount",
+          "ListCidrBlocks",
+          "ListCidrCollections",
+          "ListCidrLocations",
+          "ListGeoLocations",
+          "ListHealthChecks",
+          "ListHostedZones",
+          "ListHostedZonesByName",
+          "ListHostedZonesByVPC",
+          "ListQueryLoggingConfigs",
+          "ListResourceRecordSets",
+          "ListReusableDelegationSets",
+          "ListTagsForResource",
+          "ListTagsForResources",
+          "ListTrafficPolicies",
+          "ListTrafficPolicyInstances",
+          "ListTrafficPolicyInstancesByHostedZone",
+          "ListTrafficPolicyInstancesByPolicy",
+          "ListTrafficPolicyVersions",
+          "ListVPCAssociationAuthorizations",
+          "TestDNSAnswer",
+          "UpdateHealthCheck",
+          "UpdateHostedZoneComment",
+          "UpdateHostedZoneFeatures",
+          "UpdateTrafficPolicyComment",
+          "UpdateTrafficPolicyInstance"
+        ],
+        "registered": [
+          "ChangeResourceRecordSets",
+          "ChangeTagsForResource",
+          "CreateHostedZone",
+          "DeleteHostedZone",
+          "GetChange",
+          "GetHostedZone",
+          "GetHostedZoneCount",
+          "ListHostedZones",
+          "ListHostedZonesByName",
+          "ListResourceRecordSets",
+          "ListTagsForResource",
+          "ListTagsForResources",
+          "UpdateHostedZoneComment"
+        ],
+        "implemented": [
+          "ChangeResourceRecordSets",
+          "ChangeTagsForResource",
+          "CreateHostedZone",
+          "DeleteHostedZone",
+          "GetChange",
+          "GetHostedZone",
+          "GetHostedZoneCount",
+          "ListHostedZones",
+          "ListHostedZonesByName",
+          "ListResourceRecordSets",
+          "ListTagsForResource",
+          "ListTagsForResources",
+          "UpdateHostedZoneComment"
+        ],
+        "stubbed": [],
+        "unsupported": [],
+        "missing": [
+          "ActivateKeySigningKey",
+          "AssociateVPCWithHostedZone",
+          "ChangeCidrCollection",
+          "CreateCidrCollection",
+          "CreateHealthCheck",
+          "CreateKeySigningKey",
+          "CreateQueryLoggingConfig",
+          "CreateReusableDelegationSet",
+          "CreateTrafficPolicy",
+          "CreateTrafficPolicyInstance",
+          "CreateTrafficPolicyVersion",
+          "CreateVPCAssociationAuthorization",
+          "DeactivateKeySigningKey",
+          "DeleteCidrCollection",
+          "DeleteHealthCheck",
+          "DeleteKeySigningKey",
+          "DeleteQueryLoggingConfig",
+          "DeleteReusableDelegationSet",
+          "DeleteTrafficPolicy",
+          "DeleteTrafficPolicyInstance",
+          "DeleteVPCAssociationAuthorization",
+          "DisableHostedZoneDNSSEC",
+          "DisassociateVPCFromHostedZone",
+          "EnableHostedZoneDNSSEC",
+          "GetAccountLimit",
+          "GetCheckerIpRanges",
+          "GetDNSSEC",
+          "GetGeoLocation",
+          "GetHealthCheck",
+          "GetHealthCheckCount",
+          "GetHealthCheckLastFailureReason",
+          "GetHealthCheckStatus",
+          "GetHostedZoneLimit",
+          "GetQueryLoggingConfig",
+          "GetReusableDelegationSet",
+          "GetReusableDelegationSetLimit",
+          "GetTrafficPolicy",
+          "GetTrafficPolicyInstance",
+          "GetTrafficPolicyInstanceCount",
+          "ListCidrBlocks",
+          "ListCidrCollections",
+          "ListCidrLocations",
+          "ListGeoLocations",
+          "ListHealthChecks",
+          "ListHostedZonesByVPC",
+          "ListQueryLoggingConfigs",
+          "ListReusableDelegationSets",
+          "ListTrafficPolicies",
+          "ListTrafficPolicyInstances",
+          "ListTrafficPolicyInstancesByHostedZone",
+          "ListTrafficPolicyInstancesByPolicy",
+          "ListTrafficPolicyVersions",
+          "ListVPCAssociationAuthorizations",
+          "TestDNSAnswer",
+          "UpdateHealthCheck",
+          "UpdateHostedZoneFeatures",
+          "UpdateTrafficPolicyComment",
+          "UpdateTrafficPolicyInstance"
+        ],
+        "extra": []
+      }
+    },
     {
       "service": "s3",
       "api_version": "2006-03-01",

docs/coverage/elbv2/README.md

@@ -24,6 +24,34 @@ Spinifex implements **36 operations** in the ELBv2 `2015-12-01` API model.
 
 The data plane is a system-managed load balancer VM, launched automatically during `CreateLoadBalancer`. Application Load Balancers run HAProxy for the L7 surface — rules, fixed responses and redirects over HTTP and HTTPS. Network Load Balancers run nginx `stream` for the L4 surface — TCP, UDP, TLS and TCP_UDP — because HAProxy cannot load-balance UDP. The agent selects the engine from the configuration the control plane delivers, so the choice follows the load balancer type and is not separately configurable.
 
+### Target health and routing
+
+Traffic goes only to targets that are healthy or still `initial`.
+A target that has failed its health check is removed from the data plane on the next render and put back when it passes again.
+When every target in a group is unhealthy, the load balancer fails open and sends traffic to all of them rather than refusing it.
+A Network Load Balancer renders every registered server in that case, and an Application Load Balancer falls back to unchecked backup servers that HAProxy uses only when every checked server is down.
+`DescribeTargetHealth` keeps reporting the targets as unhealthy while this happens.
+Draining targets and targets without an address never receive traffic.
+
+An `initial` target receives traffic before its first successful check.
+This differs from AWS, where a target takes no traffic until it passes.
+Health state is reset to `initial` for every target when the control plane restarts, so excluding `initial` targets would drop all traffic after a restart.
+
+Changing the health check of a target group with `ModifyTargetGroup` takes effect on the next render of every load balancer that uses it.
+The time from a backend failing to traffic stopping is the health check interval times the unhealthy threshold, plus the agent poll of about 5 seconds and the heartbeat that reports the result.
+Recovery takes the interval times the healthy threshold, plus the same delays.
+
+Measured on a three-node cluster with a 5 second interval and thresholds of 2:
+
+| Event | Time |
+| --- | --- |
+| Backend starts failing to reported `unhealthy` | 21 s |
+| Backend recovers to reported `healthy` | 15 s, serving again 6 s later |
+| Every backend failing to reported `unhealthy` | 12 s, fail open immediate |
+
+A TCP connection held through a Network Load Balancer survived four configuration reloads while the other backend changed state, because the agent applies nginx with a graceful reload and HAProxy with a graceful restart.
+The HAProxy configuration with the backup servers was accepted by a live Application Load Balancer, but its rendered text was not dumped and checked by hand.
+
 ### Operations
 
 | Operation |

docs/coverage/elbv2/intro.md

@@ -1,3 +1,31 @@
 ### Two data planes
 
 The data plane is a system-managed load balancer VM, launched automatically during `CreateLoadBalancer`. Application Load Balancers run HAProxy for the L7 surface — rules, fixed responses and redirects over HTTP and HTTPS. Network Load Balancers run nginx `stream` for the L4 surface — TCP, UDP, TLS and TCP_UDP — because HAProxy cannot load-balance UDP. The agent selects the engine from the configuration the control plane delivers, so the choice follows the load balancer type and is not separately configurable.
+
+### Target health and routing
+
+Traffic goes only to targets that are healthy or still `initial`.
+A target that has failed its health check is removed from the data plane on the next render and put back when it passes again.
+When every target in a group is unhealthy, the load balancer fails open and sends traffic to all of them rather than refusing it.
+A Network Load Balancer renders every registered server in that case, and an Application Load Balancer falls back to unchecked backup servers that HAProxy uses only when every checked server is down.
+`DescribeTargetHealth` keeps reporting the targets as unhealthy while this happens.
+Draining targets and targets without an address never receive traffic.
+
+An `initial` target receives traffic before its first successful check.
+This differs from AWS, where a target takes no traffic until it passes.
+Health state is reset to `initial` for every target when the control plane restarts, so excluding `initial` targets would drop all traffic after a restart.
+
+Changing the health check of a target group with `ModifyTargetGroup` takes effect on the next render of every load balancer that uses it.
+The time from a backend failing to traffic stopping is the health check interval times the unhealthy threshold, plus the agent poll of about 5 seconds and the heartbeat that reports the result.
+Recovery takes the interval times the healthy threshold, plus the same delays.
+
+Measured on a three-node cluster with a 5 second interval and thresholds of 2:
+
+| Event | Time |
+| --- | --- |
+| Backend starts failing to reported `unhealthy` | 21 s |
+| Backend recovers to reported `healthy` | 15 s, serving again 6 s later |
+| Every backend failing to reported `unhealthy` | 12 s, fail open immediate |
+
+A TCP connection held through a Network Load Balancer survived four configuration reloads while the other backend changed state, because the agent applies nginx with a graceful reload and HAProxy with a graceful restart.
+The HAProxy configuration with the backup servers was accepted by a live Application Load Balancer, but its rendered text was not dumped and checked by hand.

docs/coverage/internal-report.md

@@ -1243,6 +1243,71 @@ option-groups: Option groups configure engine add-ons for engines this platform
 - `DescribeOptionGroups`
 - `ModifyOptionGroup`
 
+## route53
+
+`2013-04-01` — 13 of 71 modelled operations implemented (18.3%), 0 stubbed, 0 not applicable, 0 registered outside the pinned model.
+
+### Not implemented
+
+- `ActivateKeySigningKey`
+- `AssociateVPCWithHostedZone`
+- `ChangeCidrCollection`
+- `CreateCidrCollection`
+- `CreateHealthCheck`
+- `CreateKeySigningKey`
+- `CreateQueryLoggingConfig`
+- `CreateReusableDelegationSet`
+- `CreateTrafficPolicy`
+- `CreateTrafficPolicyInstance`
+- `CreateTrafficPolicyVersion`
+- `CreateVPCAssociationAuthorization`
+- `DeactivateKeySigningKey`
+- `DeleteCidrCollection`
+- `DeleteHealthCheck`
+- `DeleteKeySigningKey`
+- `DeleteQueryLoggingConfig`
+- `DeleteReusableDelegationSet`
+- `DeleteTrafficPolicy`
+- `DeleteTrafficPolicyInstance`
+- `DeleteVPCAssociationAuthorization`
+- `DisableHostedZoneDNSSEC`
+- `DisassociateVPCFromHostedZone`
+- `EnableHostedZoneDNSSEC`
+- `GetAccountLimit`
+- `GetCheckerIpRanges`
+- `GetDNSSEC`
+- `GetGeoLocation`
+- `GetHealthCheck`
+- `GetHealthCheckCount`
+- `GetHealthCheckLastFailureReason`
+- `GetHealthCheckStatus`
+- `GetHostedZoneLimit`
+- `GetQueryLoggingConfig`
+- `GetReusableDelegationSet`
+- `GetReusableDelegationSetLimit`
+- `GetTrafficPolicy`
+- `GetTrafficPolicyInstance`
+- `GetTrafficPolicyInstanceCount`
+- `ListCidrBlocks`
+- `ListCidrCollections`
+- `ListCidrLocations`
+- `ListGeoLocations`
+- `ListHealthChecks`
+- `ListHostedZonesByVPC`
+- `ListQueryLoggingConfigs`
+- `ListReusableDelegationSets`
+- `ListTrafficPolicies`
+- `ListTrafficPolicyInstances`
+- `ListTrafficPolicyInstancesByHostedZone`
+- `ListTrafficPolicyInstancesByPolicy`
+- `ListTrafficPolicyVersions`
+- `ListVPCAssociationAuthorizations`
+- `TestDNSAnswer`
+- `UpdateHealthCheck`
+- `UpdateHostedZoneFeatures`
+- `UpdateTrafficPolicyComment`
+- `UpdateTrafficPolicyInstance`
+
 ## s3
 
 `2006-03-01` — 26 of 112 modelled operations implemented (23.2%), 0 stubbed, 6 not applicable, 0 registered outside the pinned model.

docs/coverage/pages.json

@@ -247,6 +247,20 @@
         "option-groups": "Option groups configure engine add-ons for engines this platform does not offer, such as Oracle and SQL Server."
       }
     },
+    "route53": {
+      "slug": "route53",
+      "name": "Route 53",
+      "title": "Route 53 API Coverage",
+      "seoTitle": "Amazon Route 53 API Coverage on Spinifex — Spinifex Docs",
+      "description": "The Amazon Route 53 API operations Spinifex implements, covering the hosted zones it serves, their resource record sets, change status and tags on each zone.",
+      "tags": [
+        "aws",
+        "compatibility",
+        "coverage",
+        "route53",
+        "dns"
+      ]
+    },
     "s3": {
       "slug": "s3",
       "name": "S3",

docs/coverage/route53/README.md

@@ -0,0 +1,38 @@
+---
+title: "Route 53 API Coverage"
+seoTitle: "Amazon Route 53 API Coverage on Spinifex — Spinifex Docs"
+description: "The Amazon Route 53 API operations Spinifex implements, covering the hosted zones it serves, their resource record sets, change status and tags on each zone."
+category: "Coverage"
+sections:
+  - overview
+tags:
+  - aws
+  - compatibility
+  - coverage
+  - route53
+  - dns
+---
+
+# Route 53 API Coverage
+
+## Overview
+
+Spinifex implements **13 operations** in the Route 53 `2013-04-01` API model.
+
+### Operations
+
+| Operation |
+|---|
+| `ChangeResourceRecordSets` |
+| `ChangeTagsForResource` |
+| `CreateHostedZone` |
+| `DeleteHostedZone` |
+| `GetChange` |
+| `GetHostedZone` |
+| `GetHostedZoneCount` |
+| `ListHostedZones` |
+| `ListHostedZonesByName` |
+| `ListResourceRecordSets` |
+| `ListTagsForResource` |
+| `ListTagsForResources` |
+| `UpdateHostedZoneComment` |

docs/install/install-multi-node/README.md

@@ -163,11 +163,14 @@ Open between all hosts:
 | Protocol and port | Used by |
 |---|---|
 | UDP 6081 | Geneve tunnels |
+| UDP 500, 4500 and IP protocol 50 (ESP) | IPsec for the Geneve tunnels, while IPsec is on |
 | TCP 4222, 4248 | NATS |
 | TCP 6641, 6642 | OVN northbound and southbound |
 | TCP 8443 | Predastore S3 gate |
 | UDP 6660, 7660 | Object shards, metadata consensus |
 
+UDP 4500 carries the tunnel traffic only where IKE detects NAT between two nodes; otherwise it travels as ESP, on Oracle Linux as on Debian.
+
 Open between the three OVN database servers (servers 1 to 3) only:
 
 | Protocol and port | Used by |
@@ -207,7 +210,10 @@ Spinifex's own **host** firewall is separate, and already carries this policy
 
 Choose one method and apply it to **every** server in the cluster.
 
-**Option A — existing OS.** On a server already running Ubuntu 26.04 or Debian 13 (Oracle Linux runs a single node only for now, see [Oracle Linux Hosts](/docs/install-oracle-linux)):
+**Option A — existing OS.** Use this on servers already running Ubuntu 26.04, Debian 13 or Oracle Linux 10, and prepare an Oracle Linux server with [Oracle Linux Hosts](/docs/install-oracle-linux) first.
+Every server in a cluster must run the same OS: Oracle Linux never shares a cluster with Debian or Ubuntu, and `spx admin join` does not check this for you.
+
+Run the installer on each server:
 
 ```bash
 curl -fsSL https://install.mulgadc.com | bash

docs/install/install-oracle-linux/README.md

@@ -1,12 +1,13 @@
 ---
 title: "Oracle Linux Hosts"
 seoTitle: "Install Spinifex on Oracle Linux 10 — Spinifex Docs"
-description: "Prepare an Oracle Linux 10 host for Spinifex: UEK R8, SELinux, a NetworkManager br-wan bridge, the CentOS NFV SIG repository, firewalld handover and updates."
+description: "Prepare Oracle Linux 10 hosts for Spinifex: UEK R8, SELinux, a NetworkManager br-wan bridge, the NFV SIG repository, IPsec on libreswan and host updates."
 category: "Install"
 tags:
   - install
   - oracle linux
   - single node
+  - multi node
 resources:
   - title: "Oracle Linux 10 Documentation"
     url: "https://docs.oracle.com/en/operating-systems/oracle-linux/10/"
@@ -35,13 +36,12 @@ Follow it alongside [Single-Node Install](/docs/install).
 
 **Supported on Oracle Linux:**
 
-- A single node.
+- A single node, or a cluster in which every node runs Oracle Linux; servers beyond the third join as compute nodes, as on Debian.
 - Guests, VPCs, routed NAT, Elastic IPs, and Application and Network Load Balancers.
 - Whole-GPU passthrough, on x86_64 GPU hosts (see [GPU Passthrough](/docs/gpu-passthrough)).
 
 **Not supported on Oracle Linux yet:**
 
-- Clusters of more than one node. They wait for IPsec on libreswan, so `spx admin init` turns IPsec off on Oracle Linux and refuses `--ipsec=true`.
 - Clusters that mix Oracle Linux with Debian or Ubuntu nodes, on any release.
 - SELinux enforcing, aarch64, Oracle Linux 9 or earlier, the Red Hat Compatible Kernel, a ZFS root, and NVIDIA MIG.
 
@@ -51,7 +51,7 @@ Spinifex installs `openvswitch3.5` and `ovn25.03`, the versions Debian 13 ships,
 QEMU comes from Oracle's KVM Utilities repository, because only that build has the `microvm` machine type Spinifex load balancers boot on.
 
 An Oracle Linux node must be a dedicated Spinifex host.
-Spinifex claims every `tap*` device and stops libreswan's `ipsec.service`, so VPNs, hand-run QEMU guests and libreswan tunnels of your own cannot share the host.
+Spinifex claims every `tap*` device and owns libreswan's `ipsec.service`, which it keeps stopped on a single node and runs for its own tunnels on a cluster node, so VPNs, hand-run QEMU guests and libreswan tunnels of your own cannot share the host.
 Crypto policies other than `DEFAULT` and `FIPS` are untested, and so is Secure Boot.
 
 ## Prerequisites
@@ -134,8 +134,28 @@ Run the installer as a user with `sudo`, exactly as on Debian:
 curl -fsSL https://install.mulgadc.com | bash
 ```
 
-Then continue with [Single-Node Install](/docs/install) from Step 2: `setup-ovn.sh --management`, `spx admin init` and `systemctl start spinifex.target`.
-`spx admin init` prints that IPsec is off on this host; on a single node there is no peer to encrypt traffic to.
+For a single node, continue with [Single-Node Install](/docs/install) from Step 2: `setup-ovn.sh --management`, `spx admin init` and `systemctl start spinifex.target`.
+IPsec stays off on a single node, because there is no peer to encrypt traffic to.
+
+For a cluster, prepare and install every node this way, then continue with [Multi-Node Install](/docs/install-multi-node) from Step 2.
+
+### Clusters
+
+Every node in an Oracle Linux cluster runs Oracle Linux.
+`install-node.sh` refuses a host list that mixes Oracle Linux with Debian or Ubuntu, but `spx admin join` does not check, so keep to one OS yourself when joining by hand.
+
+IPsec encrypts the Geneve tunnels between nodes and is on by default, as on Debian.
+On Oracle Linux it runs on libreswan, the IKE daemon Oracle ships, rather than strongSwan.
+Once the node's IPsec certificate exists, the daemon unmasks and starts `ipsec.service` and `openvswitch-ipsec.service`.
+`ovs-monitor-ipsec` then loads the node certificate, its key and the cluster CA into libreswan's NSS database at `/var/lib/ipsec/nss`, under nicknames that start with `ovs_cert`.
+
+The tunnels are plain ESP (IP protocol 50), as on Debian, negotiated over UDP 500.
+Open UDP 500, UDP 4500 and ESP between nodes: libreswan moves to UDP 4500 if it detects NAT between two nodes.
+The Spinifex host firewall opens all three to cluster peers; a firewall between the nodes must allow them too.
+While IKE to a peer fails, the node drops its Geneve traffic to that peer rather than send it in clear.
+The nft table `inet spinifex_ipsec_guard` drops outbound UDP 6081 from the encap address whose route carries no SA, which also covers the moments pluto flushes the xfrm policy table as it starts and stops.
+It is not persisted: `ipsec.service` puts it back at boot before ovn-controller builds the tunnels, and the daemon re-syncs it every 60 seconds.
+`spx admin init --ipsec=false` leaves the tunnels unencrypted, as on any OS.
 
 ## Step 2. Know What the Installer Changed
 
@@ -182,7 +202,9 @@ Those advisories do not cover the NFV SIG packages, so follow Open vSwitch and O
 
 ## Step 5. Uninstall
 
-`/usr/local/share/spinifex/uninstall-spx.sh` removes Spinifex as on Debian, and lists what it leaves in place on Oracle Linux: the repository changes and the SIG key, `OVS_USER_ID` and `OVN_USER_ID`, the OVS and OVN logrotate edits, the chrony include, `/etc/modules-load.d/spinifex.conf`, the SELinux mode, a masked `ipsec.service`, firewalld disabled, and any GPU passthrough configuration.
+`/usr/local/share/spinifex/uninstall-spx.sh` removes Spinifex as on Debian.
+On a cluster node it also stops `openvswitch-ipsec.service` and deletes the `ovs_cert*` entries from libreswan's NSS database, so the node can join a different cluster; other NSS identities and your connection files in `/etc/ipsec.d` stay.
+It lists what it leaves in place on Oracle Linux: the repository changes and the SIG key, `OVS_USER_ID` and `OVN_USER_ID`, the OVS and OVN logrotate edits, the chrony include, `/etc/modules-load.d/spinifex.conf`, the SELinux mode, a masked `ipsec.service`, firewalld disabled, and any GPU passthrough configuration.
 `--purge-deps` is refused on Oracle Linux, because dnf cannot tell the packages Spinifex installed from the host's own; remove them with `dnf remove` yourself.
 
 ## Troubleshooting
@@ -224,6 +246,20 @@ If it prints `yes`, run `sudo nmcli connection modify br-wan bridge.stp no` and
 
 Check that `/usr/bin/spx` links to `/usr/local/bin/spx`; run the installer again if it does not, or call `sudo /usr/local/bin/spx` meanwhile.
 
+### Cluster nodes form no IPsec tunnels
+
+Check that both IPsec services run, and that libreswan holds the node's certificate and the cluster CA:
+
+```bash
+systemctl is-active ipsec.service openvswitch-ipsec.service
+sudo certutil -L -d sql:/var/lib/ipsec/nss     # ovs_certkey_<node>, ovs_cert_cacert
+sudo ipsec trafficstatus                       # one line per peer tunnel
+```
+
+If the services are inactive, check the daemon log with `journalctl -u spinifex-daemon` for the IPsec reconcile.
+If the certificates are present but no tunnel is listed, check that UDP 500 and ESP pass between the nodes in both directions, and look for `SEC_ERROR_UNKNOWN_ISSUER` in `journalctl -u ipsec`, which means a peer's certificate is from another CA.
+Until the tunnel forms, guest traffic between those two nodes is dropped, which is the guard in [Clusters](#clusters) working.
+
 ### CA certificate not trusted
 
 `sudo spx admin init` installs the CA automatically. To install it again by hand on Oracle Linux:

docs/install/install/README.md

@@ -39,7 +39,7 @@ This guide installs Spinifex on a single server using the binary installer. For
 
 - Ubuntu 26.04
 - Debian 13
-- Oracle Linux 10.2 or a later 10.x, on UEK R8, x86_64 only, as a single node. Open vSwitch and OVN come from the CentOS NFV SIG, a community repository that Oracle does not support. Prepare the host with [Oracle Linux Hosts](/docs/install-oracle-linux) first.
+- Oracle Linux 10.2 or a later 10.x, on UEK R8, x86_64 only, as a single node or, with [Multi-Node Install](/docs/install-multi-node), a cluster. Open vSwitch and OVN come from the CentOS NFV SIG, a community repository that Oracle does not support. Prepare the host with [Oracle Linux Hosts](/docs/install-oracle-linux) first.
 
 **What Gets Installed:**
 

docs/install/performance-tuning/README.md

@@ -269,6 +269,35 @@ ip link set <dev> mtu 9000        # both ends
 ping -M do -s 8972 -c 3 <peer>    # must succeed
 ```
 
+### The internet path
+
+A guest MTU of 1408 or 1442 sits below the 1500 bytes the internet carries, so a server can send the guest segments it cannot receive.
+The VPC gateway port carries `gateway_mtu = min(guest MTU, 1500)` and answers an oversized packet with ICMP fragmentation-needed (IPv4) or Packet Too Big (IPv6).
+It follows `ipsec_enabled` and `network.underlay_mtu` and converges on the next reconcile pass, with no VPC recreate.
+
+Path MTU discovery depends on that ICMP reaching the server, and an upstream NAT router or a CDN often drops it.
+vpcd therefore also clamps the TCP MSS on the host hop to the gateway MTU minus 40 (IPv4) or minus 60 (IPv6), and re-asserts the rule every 30 seconds so a firewall reload cannot remove it for long.
+
+| Uplink | Clamp | Table |
+| --- | --- | --- |
+| veth (distributed NAT) | yes | `bridge spinifex_mss`, matching the `veth-wan-br` port |
+| routed NAT | yes | `inet spinifex_mss`, matching `spx-nat-host` |
+| physical | no host hop exists | path MTU discovery only |
+
+On a physical uplink the only cover is path MTU discovery, which fails when the upstream drops the ICMP or the peer ignores it.
+
+Measured on one node (env12), IPsec on, Ubuntu 26.04 guest, 16 MB HTTPS download inside a Docker container at the default bridge MTU of 1500:
+
+| Uplink | Without the clamp | With the clamp |
+| --- | --- | --- |
+| veth, instance with a public IP | stalled 3/3 | completed 3/3 |
+| routed NAT | stalled 3/3, no ICMP seen on the host | completed 3/3 |
+
+An instance behind a NAT gateway on the veth uplink also completed 3/3 with the clamp; its unclamped internet run was not measured.
+The guest's own `curl` completed in every case.
+Without the clamp, `gateway_mtu` alone fixed the download only against a server on the same L2 with no NAT in between (completed 3/3), and not against the internet through the lab's upstream NAT router.
+Not measured: a physical uplink, a multi-node cluster, IPv6, and a Debian guest.
+
 ## 5. Measurement notes
 
 Throughput on this stack is **noisy** — repeated identical 20s runs varied between

docs/security/network-connections/README.md

@@ -111,8 +111,8 @@ behavior in other words.
 | 6643 | OVN Northbound DB (RAFT) | OVSDB/TCP | Cluster | NB database RAFT replication between the 3 quorum nodes | Cluster network only; TLS planned |
 | 6644 | OVN Southbound DB (RAFT) | OVSDB/TCP | Cluster | SB database RAFT replication between the 3 quorum nodes | Cluster network only; TLS planned |
 | 6081 | OVN (Geneve) | UDP | Encap | Tenant traffic overlay between chassis. A kernel UDP-tunnel socket, so packets are delivered locally and traverse the host's netfilter input hook before OVS sees them — a host firewall must accept them explicitly. The socket is opened by the kernel tunnel driver and takes no bind address, so it binds the wildcard by design and reach must be restricted by firewall. | None — see 500/4500 |
-| 500, 4500 | strongSwan `charon` | IKEv2 / UDP | Encap | IKE and NAT-T for the IPsec protecting Geneve, managed entirely by `ovs-monitor-ipsec`. Binds the wildcard by design (the upstream strongSwan default, accepted rather than overridden), so reach must be restricted by firewall. | Certificate-based, against the cluster CA |
-| — | ESP | IP proto 50 | Encap | The IPsec payload itself, once IKE has negotiated an SA | Cluster CA |
+| 500, 4500 | strongSwan `charon`, or libreswan `pluto` on Oracle Linux | IKEv2 / UDP | Encap | IKE and NAT-T for the IPsec protecting Geneve, managed entirely by `ovs-monitor-ipsec`; 4500 carries the ESP payload only where IKE detects NAT between two nodes. `charon` binds the wildcard by design (the upstream strongSwan default, accepted rather than overridden); `pluto` binds each host address separately, the WAN address included, which is no narrower. Either way reach must be restricted by firewall. | Certificate-based, against the cluster CA |
+| — | ESP | IP proto 50 | Encap | The IPsec payload itself, once IKE has negotiated an SA, on Oracle Linux as on Debian. | Cluster CA |
 | 8222 | spinifex-nats (monitoring) | HTTP | Localhost | `varz`/`subsz` metrics consumed by the daemon | Loopback only |
 | 323 | chronyd | NTP | Localhost | Time sync client control socket | Loopback only |
 | 169.254.169.254:80 | spinifex-vpcd (IMDS) | HTTP | Guest | Instance metadata service. One socket per instance, bound to that instance's `ime-*` interface. Terminates on the host, so it traverses the netfilter input hook. | Instance identity by interface; IMDSv2 tokens |
@@ -177,7 +177,7 @@ Control-plane and data-plane traffic between Spinifex nodes, for completeness an
 | OVN NB/SB (client) | 6641 / 6642 | Cluster network only (TLS planned) | Network control plane; vpcd and ovn-controller dial the quorum |
 | OVN NB/SB (RAFT) | 6643 / 6644 | Cluster network only (TLS planned) | NB/SB database replication across the 3 quorum nodes |
 | OVN tunnels (Geneve) | UDP 6081 | Encapsulated by IPsec when `network.ipsec_enabled` is true, which is the default on multi-node clusters | Tenant traffic overlay between chassis, on the `vpc` plane |
-| OVN IPsec (IKE / NAT-T / ESP) | UDP 500, UDP 4500, IP proto 50 | Certificate-based against the cluster CA, negotiated by `ovs-monitor-ipsec` | Protects the Geneve tunnels above. OVN-native only — no layer manages strongSwan directly. |
+| OVN IPsec (IKE / NAT-T / ESP) | UDP 500, UDP 4500, IP proto 50 | Certificate-based against the cluster CA, negotiated by `ovs-monitor-ipsec` | Protects the Geneve tunnels above. OVN-native only: no layer configures strongSwan or libreswan directly. On Oracle Linux the daemon also blocks outbound Geneve that no SA covers, so a peer failing IKE gets no plaintext. |
 | Instance DNS forwarding | UDP/TCP 5300 | None | Each node's per-instance DNS shim forwards guest queries to peer nodes' northstar `:5300` |
 
 Nodes **must** sit on a network segment that is not routed to tenant/guest VMs or to the internet. The Predastore blob and meta transports and the OVN DBs are cluster-internal and must not be reachable from anywhere else.
@@ -239,6 +239,7 @@ Notes that make the difference between this working and locking you out:
 
 - **Use a dedicated table and never flush the others.** `vpcd` writes MASQUERADE and per-EIP FORWARD rules into the `ip filter` and `ip nat` tables, and only reinstalls them when the service starts. `iptables -F`, `nft flush ruleset`, `ufw enable` and `firewalld` all destroy them silently, and instance networking stays broken until the next restart.
 - **Do not add a `forward` hook.** nftables evaluates every table registered on a hook and any `drop` is final, so a default-deny forward chain here would override `vpcd`'s accepts in the other table and break every routed-NAT instance and every EIP. Filtering forwarded guest traffic is the security group's job, not the host firewall's.
+- **`vpcd` owns a second table named `spinifex_mss`, separate from `inet spinifex_filter`.** It is a `bridge` table on the veth uplink and an `inet` table on the routed uplink, it has a `forward` hook with policy `accept`, and it only rewrites the MSS option of TCP SYN packets on the host hop to the gateway. It never drops or accepts traffic and does not replace the forward-hook advice above. `vpcd` re-installs it every 30 seconds, so a flush is repaired within that interval, and there is none on a physical uplink.
 - **`output` is deliberately untouched.** The metadata service's reply path egresses through per-instance policy routing; filtering `output` breaks it in ways that are hard to attribute.
 - **On a single-NIC node the cluster and encap sets are the node's public addresses**, because the planes collapse. The rules are still correct, but they are no longer a boundary — an upstream ACL is the only real control there.
 
@@ -250,7 +251,7 @@ Every listener and outbound destination is controlled by one of these files. Cha
 
 | File | Keys | Controls |
 |------|------|----------|
-| `/etc/spinifex/spinifex.toml` | `nodes.<node>.{awsgw,nats,predastore,daemon}.host`, `nodes.<node>.vpcd.ovn_{nb,sb}_addr`, `nodes.<node>.daemon.dev_networking`, `network.ipsec_enabled` | Per-service bind addresses/ports; dev-mode QEMU port forwarding. `network.ipsec_enabled` (default `true`) decides whether OVN-native IPsec protects the Geneve tunnels, and therefore whether `charon` listens on 500/4500 — single-node clusters never enable it. |
+| `/etc/spinifex/spinifex.toml` | `nodes.<node>.{awsgw,nats,predastore,daemon}.host`, `nodes.<node>.vpcd.ovn_{nb,sb}_addr`, `nodes.<node>.daemon.dev_networking`, `network.ipsec_enabled` | Per-service bind addresses/ports; dev-mode QEMU port forwarding. `network.ipsec_enabled` (default `true`) decides whether OVN-native IPsec protects the Geneve tunnels, and therefore whether the IKE daemon (`charon`, or `pluto` on Oracle Linux) listens on 500/4500 — single-node clusters never enable it. |
 | `/etc/spinifex/nats.conf` | `listen`, `cluster.listen`, `cluster.routes`, `http`, `tls`, `cluster.authorization` | NATS client/cluster/monitoring listeners, peer routes, TLS, cluster token. |
 | `/etc/spinifex/predastore/predastore.toml` | `[[host]].bind_addr`, `[[host]].addr`, `[[host]].tls_cert`, `[[host]].tls_key`, `[[host.node]].role`, `[[host.node]].port` | Predastore host and node layout: `bind_addr` is the address the host's sockets bind, `addr` is the address peer hosts dial it on, and both carry no port — the nodes pinned to the host supply their own. The service's `--host`/`--port` override the bind address and the gate's S3 port. |
 | `/etc/spinifex/northstar/northstar.toml` | `listen`, `forwarders` | DNS listen addresses (`:53` on the advertise address, `:5300` wildcard) and upstream resolvers. |

docs/testing/e2e-coverage.md

@@ -40,7 +40,7 @@ Runs **on** a node, shelling out to `ip`/`iptables`/`ovn-nbctl` locally. Skips u
 
 Phases 7 and 8 are gated on a cluster by what the datapath can actually do: the transit segment is a localnet, so host-to-guest Tier 1 ingress only exists where the VPC's gateway router port and the guest are both on this host. The VPC ingress route itself is installed on every node, so phase 8 checks it unconditionally.
 
-Three nightly cells run it: `nat-single` (cell 19), its Oracle Linux twin (cell 36) and `nat-multi` (cell 30). Cell 20 is the bare-metal job and is not in the permutation matrix at all, which is why the multi-node NAT cell is numbered past the end rather than filling a gap.
+Four nightly cells run it: `nat-single` (cell 19) and `nat-multi` (cell 30), and their Oracle Linux twins (cells 36 and 38). Cell 20 is the bare-metal job and is not in the permutation matrix at all, which is why the multi-node NAT cell is numbered past the end rather than filling a gap.
 
 ### `multinode` — behaviour that only exists on more than one node
 
@@ -50,6 +50,17 @@ Three nightly cells run it: `nat-single` (cell 19), its Oracle Linux twin (cell
 
 **`TestMultinodeDNS` convergence budgets are env-tunable.** `SPINIFEX_MULTINODE_DNS_TOPOLOGY_TIMEOUT` (default `90s`) bounds every node agreeing on the NS topology, `SPINIFEX_MULTINODE_DNS_TIMEOUT` (default `2m`) bounds a record becoming answerable from a guest or a node, and `SPINIFEX_MULTINODE_DNS_POLL` (default `3s`) is the poll interval for both. Values are Go durations; a malformed or non-positive value panics rather than falling back to the default.
 
+**`TestMultinodeOVNRaft` restarts each node's OVN central in turn before it kills the NB leader.**
+A `setup-ovn.sh` rerun restarts a node's NB and SB members together, on EL through the one `ovn-northd.service`, so the test does the same on every database node in turn, waits for both members to be back as cluster members under a known leader, and requires an NB write to land after each step and the full quorum with one leader at the end.
+
+**`TestMultinodeIPSec` proves guest traffic is encrypted on the wire, not only that IPsec is configured.**
+It first requires every node to carry the OVS certificate, key and CA pointers, `NB_Global.ipsec=true` never asserted over an unconfigured chassis, and an AES-GCM SA in `ip xfrm state`.
+None of those fails when SAs exist but traffic bypasses them, so it then launches two instances on distinct nodes and sends 200 pings of 1000 bytes between them.
+For each direction of that pair it fails unless the outbound SAs to the peer's `ovn-encap-ip` carried at least the payload sent, an outbound-only capture on the interface routing to the peer saw ESP (raw or UDP/4500-encapsulated), and that capture saw no plaintext Geneve (UDP/6081).
+The capture is outbound-only because inbound ESP is decrypted and seen again as Geneve on the same interface, which would read as plaintext on a healthy node.
+It proves the node pair the guests landed on; the SA check covers every node.
+It skips when the node config sets `network.ipsec_enabled=false`, or when the scheduler places all four launches on one node.
+
 ### `kvquorum` — the cluster's own state survives losing a node (multi-node, needs three)
 
 **The control plane keeps its state in JetStream KV buckets, and this is the suite that proves those buckets are actually replicated.** A bucket on one replica lives on one JetStream-chosen server recorded in no config, so losing that node makes it return `nats: no responders available for request` from *every* node at once — a cluster-wide outage caused by a single-node event. Leader leases are the worst case: acquiring one is a write, so a lease bucket without a quorum cannot be acquired by anyone, and every reconciler sharing it stops cluster-wide at exactly the moment a node has failed and there is repair work to do.
@@ -104,6 +115,9 @@ CA installed and downloadable, fingerprint matches config, hostnames present in
 
 Internal and internet-facing ALB and NLB, HTTPS listeners, listener rules, `ModifyListener`, and NLB UDP.
 
+Unhealthy-target routing: an NLB with an HTTP health check on a TCP target group stops sending traffic to a target that fails it while still accepting TCP, sends traffic again once it recovers, and fails open when every target is unhealthy.
+An ALB whose targets all fail their health check keeps serving from them instead of answering 503.
+
 ### `eks`, `ecs`, `ecr` — container services
 
 `eks`: cluster create/delete, access entries, addon delivery, `GetToken`, kubeconfig artifacts, the EBS CSI volume path, and an IRSA pod wired with a projected SA token. `ecs`: clusters, container instances, the three network modes (`awsvpc`, `bridge`, `host`), task-role credentials fetched from the container credentials endpoint, a service behind an ELB, and ENI release on terminate. `ecr`: control-plane CRUD plus a real OCI data plane — docker login/build/push/pull round trips, multi-layer images, and raw blob HTTP.

docs/ui/design-system.md

@@ -1,254 +0,0 @@
-# Spinifex UI design system
-
-The console's tokens, primitives and the rules that hold pages to them.
-The live catalogue is the `/design` route: every token and primitive below rendered in both themes.
-It is open to admins in production and to anyone under `vite dev`.
-
-Source: `spinifex/runtime/roles/spinifexui/frontend/`.
-Tokens are in `src/styles.css`, primitives in `src/components/ui/`, and the catalogue in `src/routes/_auth/-components/design/`.
-
-## Principles
-
-- **Pages compose primitives; they do not restyle them.** If a page needs something the system lacks, add it to the system first.
-- **Colour is for status.** Surfaces are neutral, and brand orange marks the one primary action on a view.
-- **Borders, not shadows.** Overlays (dialogs, sheets, popovers, menus) are the only elevated surfaces.
-- **AWS semantics, our presentation.** States, field names and actions follow AWS.
-
-## Tokens
-
-All colours are OKLCH and come from mulgadc.com, so product and site match.
-
-### Surfaces
-
-| Token | Use |
-|---|---|
-| `background` | The app page. Warm off-white in light, ink `#14181D` in dark. |
-| `card`, `popover` | Raised surfaces. White in light, about `#1B2027` in dark. |
-| `muted`, `secondary`, `accent` | Hover rows, secondary buttons, quiet fills. |
-| `border` | Every divider and card edge. `#DCD8CE` in light, `oklch(1 0 0 / 8%)` in dark. |
-| `brand` | Brand orange `#EE4B12`, for the logo and decoration only. It fails text contrast, so it never carries text. |
-| `primary` | Darker orange in light, brighter orange in dark: primary button, link, focus ring, active tab. |
-| `sidebar` | The navigation column. |
-| `console`, `console-foreground` | A serial console log. Light grey on the light theme and near-black on the dark one, so the log follows the theme. |
-| `console-stamp`, `console-muted`, `console-ok`, `console-fail` | The log's tones: the kernel timestamp (brand yellow in dark, a darker amber in light), the subsystem or device that logged the line, and systemd's `[ OK ]` and `[FAILED]` markers. |
-
-### Status
-
-Each status has a solid token plus `-foreground` (text on a tinted background), `-subtle` (tinted background) and `-border`.
-
-| Status | Hue | AWS states |
-|---|---|---|
-| `success` | green | running, available, completed, healthy |
-| `warning` | brand yellow `#F5B500` | pending, stopping, creating, modifying, deleting, draining |
-| `danger` | cool red, hue 17 | stopped, error, failed, unhealthy, unavailable |
-| `info` | blue | informational notes |
-| `neutral` | grey `#9AA1A8` | terminated (muted text), and any unmapped state |
-
-`stopped` is danger because EC2 reads it that way, although RDS treats it as a resting state; splitting the map by service is its own change.
-
-`StateBadge` (`src/components/state-badge.tsx`) is the only mapping from an AWS state to a tone, and it exposes the result as `data-tone` for tests.
-Every status badge measures above 4.8:1 against its tinted background in both themes.
-
-**Danger is separated from the primary accent by hue alone in dark mode**: the two have the same lightness.
-So a destructive action is never a solid danger fill, and a danger state always carries a text label or an icon.
-
-### Chart palette
-
-`--chart-1` to `--chart-8` are categorical, with lightness spread so adjacent series separate under protanopia, deuteranopia and tritanopia.
-Measured with a simulated CVD ΔE, the worst pair is 7.4 in light and 7.2 in dark.
-Colour is never the only key: label every series.
-
-### Typography
-
-| Role | Font |
-|---|---|
-| Page titles and section headings (`h1`–`h3`) | Archivo (`font-heading`) |
-| UI text, tables and forms | Geist (`font-sans`) |
-| IDs, ARNs, IPs, CIDRs, code and eyebrow labels | JetBrains Mono (`font-mono`), with `tabular-nums` for figures |
-
-| Step | Size | Use |
-|---|---|---|
-| `2xl` | 24px | page title (`PageHeading`) |
-| `xl` | 20px | section heading |
-| `lg` | 16px | |
-| `base` | 14px | |
-| `sm` | 13px | body and table default |
-| `xs` | 12px | badges, secondary text |
-| `2xs` | 11px | mono eyebrow labels |
-
-### Primary and contrast
-
-The brand orange cannot be the primary: white on `#EE4B12` is 3.72:1, and as link text on a light card it is also 3.72:1, against WCAG AA's 4.5:1.
-So `primary` is its own token, and `brand` keeps the exact site colour for decoration.
-
-| Theme | `primary` | Fill text | Fill contrast | As a link |
-|---|---|---|---|---|
-| light | `oklch(55% 0.19 36.6)` | white | 5.30:1 | 5.30:1 on a card, 4.69:1 on the page |
-| dark | `oklch(68% 0.18 38)` | ink | 5.70:1 | 5.27:1 on a card |
-
-Dark mode puts ink on the fill rather than white, because no orange is both bright enough to read as a link on ink and dark enough to carry white text.
-
-### Radius
-
-`--radius` is 0.5rem; `rounded-md` is that, `rounded-sm` and `rounded-lg` step 2px either side, and status pills are `rounded-full`.
-
-### Themes
-
-The console defaults to dark, and the header switch toggles light and dark (`localStorage["spinifex-ui-theme"]`).
-`public/preboot.js` runs before the bundle. It applies the stored theme and `color-scheme` before first paint, so a dark-mode user never sees a white page while the bundle loads, and it turns off zod's eval probe (below).
-It is an external file because the console's CSP forbids inline scripts.
-
-## Constraints from the console's CSP
-
-The Go console serves `script-src 'self'; style-src 'self'`.
-
-- **No inline `<script>` or `<style>` elements.** A library that injects either breaks in production while working under `vite dev`, which applies no CSP. This is why `sonner` is not used.
-- **No runtime CSS variables set through `style`.** Primitives use fixed class maps instead, as `ToggleGroup` does for `spacing`.
-- **Base UI's `ScrollArea` injects a `<style>` element** to hide native scrollbars. `main.tsx` wraps the app in `CSPProvider disableStyleElements`, and `styles.css` carries the rule instead (`.base-ui-disable-scrollbar`).
-- **zod compiles validators with `new Function`** unless it is told not to, and it probes for that at module load, before any app code runs. `public/preboot.js` sets `globalThis.__zod_globalConfig = { jitless: true }` first.
-- **Chrome reports a violation whenever `DOMParser` meets a body that is not XML**, because the `<parsererror>` element it builds carries an inline style. The AWS SDK parses every query-protocol error body, so a proxy that answers with plain text or HTML produces a console violation as well as the error. Harmless, but it is noise in exactly the place someone debugging a failed call will look.
-- Base UI overlays animate through `data-starting-style` and `data-ending-style` transitions, not keyframe animation libraries, and every transition has `motion-reduce:transition-none`.
-
-## Primitives
-
-`src/components/ui/` holds the shadcn primitives on Base UI.
-Added in Phase 1: breadcrumb, checkbox, command (`cmdk`), dialog, hover-card, popover, resizable (`react-resizable-panels`), scroll-area, slider, switch, table, toggle and toggle-group.
-`button-group` joins buttons edge to edge, as the instance power buttons are.
-
-A disabled control stays visible and shows `cursor: not-allowed`, so it says why it does nothing rather than vanishing.
-An on/off setting is a `Switch`, orange when on; a `Checkbox` only picks items from a list, such as table rows, subnets or security groups.
-
-`InfoTip` (`src/components/info-tip.tsx`) is UniFi's (i): a small button beside a label that opens a short explanation on hover or click, with `InfoList` for the facts inside it.
-`PanelField` takes it through `info`, so the login user names the image's distribution and the image names its AMI details.
-
-Base UI marks orientation with `data-orientation`, so `styles.css` defines the `data-horizontal` and `data-vertical` variants the shadcn sources expect.
-
-Base UI marks the selected tab with `data-active`, not `data-selected`, so `ui/tabs` styles that attribute.
-
-shadcn's `chart` is not used: it pulls `recharts`, whose `reselect@5.1.1` dependency fails the workspace's `trustPolicy: no-downgrade`.
-Charts are uPlot instead (see Charts).
-
-## Enforcement
-
-Four lint layers keep pages on the system.
-
-| Rule | Where | What it catches |
-|---|---|---|
-| `shadcn/no-unknown-classes` | all source, not tests | a class Tailwind does not generate, such as a typo or a removed token |
-| `shadcn/no-restyle` | all source | a page passing layout or colour classes to a primitive |
-| `eslint/no-restricted-imports` | `src/routes/**` | a route importing a chart library directly |
-| `src/routes/route-files.test.ts` | test suite | a new hand-written `<table>` in a route |
-
-The table test is a ratchet: `HAND_WRITTEN_TABLES` lists the routes that still have one, it fails when a new file adds one, and it fails when a listed file no longer has one so the list only shrinks.
-
-## Shell
-
-`src/lib/nav.ts` is the one description of the console's navigation.
-The sidebar, the command palette and the breadcrumbs all read it, and `nav.test.ts` checks every destination against the generated route tree, so a renamed route fails a test rather than leaving a dead link.
-
-- **Sidebar.** Each service group folds, and a folded group stays folded across reloads (`localStorage["spinifex-ui-nav-collapsed"]`). The group holding the current page is always open, so the active item is always visible. On a phone the sidebar is a sheet, and any navigation closes it.
-- **Command palette.** ⌘K or Ctrl K, or the Search button in the header. It lists every page the user can reach, every create flow (`create` on a nav item), and a theme toggle. It is navigation only, with no resource search yet.
-- **Breadcrumbs.** Service, then list page, then the resource or the create flow. A resource ID is set in mono. On a phone only the last crumb shows.
-
-## Layout
-
-The app shell's `<main>` is `min-w-0`, so a wide table scrolls inside its card instead of widening the page.
-The page gutter is 16px on a phone, 24px from `md` and 32px from `xl`.
-`PageHeading` wraps its actions below the title when they do not fit beside it.
-Every page must fit at 390px with `document.documentElement.scrollWidth` equal to the window width.
-
-## Data tables
-
-`DataTable` (`src/components/data-table.tsx`) renders `@tanstack/react-table` over the `ui/table` primitives, so sorting, row selection and shift-click range selection are the library's and not ours to maintain.
-Pages describe columns with the small `DataColumn` shape (`id`, `header`, `cell`, `sortValue`, `hideBelow`, `align`), which the component maps to TanStack column definitions; a column with a `sortValue` sorts, with missing values last and numbers compared as numbers.
-Sort and selection stay controlled by the caller, so a page can keep them in the URL.
-
-A column hides itself through `hideBelow`, which is a **container** query on the table's card (`@max-lg`, `@max-xl`, `@max-4xl`, `@max-5xl`), not a viewport one.
-So columns drop out when a side panel narrows the list as well as on a small screen.
-
-List pages keep their filter, state and sort in the URL, so a filtered view can be bookmarked, shared and survives a reload.
-EC2 Instances is the reference page (`ec2/(instances)/-components/instances-page.tsx`):
-
-- The first column names the resource: the `Name` tag in the primary colour, with the ID in mono beneath it, opening its side panel.
-- Bulk actions appear above the table once a row is selected. Each action counts only the selected instances it applies to, so a mixed selection never sends a call that must fail, and Terminate asks first and lists every ID.
-- An empty account gets a call to action, and an empty filter result says the filters are why.
-
-VPCs and Internet gateways follow the same pattern.
-
-### Selecting rows to delete
-
-Every list with a delete action has row checkboxes. Selecting a row shows `SelectionBar` (`src/components/selection-bar.tsx`) above the table, with the count, the actions and a way to clear, and `useRowSelection` (`src/hooks/use-row-selection.ts`) keeps the selection to rows still in the list.
-`BulkDelete` (`src/components/bulk-delete.tsx`) is the bar's destructive action, and its dialog names every selected resource before anything is sent.
-
-- **One call per resource, one at a time.** Most AWS delete calls take a single ID, so each refusal lands against its own row with AWS's message. A partial failure keeps the dialog open, and its confirm becomes "Retry N" for only the ones that failed.
-- **No pre-filtering.** The bar sends what was selected and lets AWS refuse, rather than guessing which deletes would fail. Instances are the exception, since one call takes many IDs and a mixed selection would fail as a whole.
-- **The bulk delete and the panel's delete are the same function.** Where a single delete is several calls, such as removing dependants first, the sequence lives in one place and both use it.
-- The verb follows the AWS console: Release for an Elastic IP, Deregister for an image.
-
-Key pairs is the smallest example (`ec2/(key)/-components/key-pairs-page.tsx`).
-
-### Instance launch
-
-The image picker shows each distribution's own mark from `src/assets/os/` (Ubuntu, Debian, Rocky, Alpine, CentOS), imported with `?no-inline` so it is served as a file and satisfies `img-src 'self'`. Any other distribution gets a generic server icon, and the "Other images" tile a disc.
-The instance type select states the chosen type's vCPUs and memory beneath it, from `DescribeInstanceTypes`.
-
-## Side panels
-
-Modelled on UniFi's console: a click on a row opens the resource in a panel beside the list, and a create flow opens there too.
-
-- **URL.** The panel is search state on the list route: `selected=<id>`, `tab=overview|monitoring|settings`, and `create=true`. Defaults are stripped, so a closed panel leaves the plain list address. Opening or closing a panel pushes a history entry, so Back closes it; a tab or filter change replaces the entry.
-- **Layout.** From `lg` (1024px) the panel is fixed to the right edge, full height below the header, flush with no outer margin, and it slides in over the list. The list keeps its width and columns underneath, so nothing reflows when a panel opens, and a soft gradient on the panel's left edge reads as the panel lying on top. Below `lg` there is no room to share, so the panel replaces the list as a full page with a back arrow. `useWideLayout` decides, through `matchMedia`.
-- **`DetailPanel`.** A header with the resource's own page (left) and close (right), the title and its ID, a summary of state and quick actions, then icon tabs with tooltips. Escape closes it unless a dialog or menu is open over it. Closing slides the panel out over 150ms, or at once under reduced motion. Each tab, and a create form, sits in its own error boundary, so a failed query shows in the panel and the list keeps working.
-- **Contents.** `PanelSection` is a folding card, `PanelFields` and `PanelField` a label and value list, with `copy` for a one-click copy of IDs and addresses, and `PanelTags` the tag list. Create forms take an `onClose(id?)` prop: embedded, they hide their back link and heading, and a create selects the new resource.
-- **One component is both the panel and the page.** Each resource has a single details component (`InstanceDetails`, `VpcDetails`, `InternetGatewayDetails`) that takes `layout="auto" | "page"`. The list renders it as the panel; `/ec2/describe-instances/$id` and its kind render the same component as a page, as one centred column under a standard left-aligned heading, with state and actions on the right and text tabs. A feature added to the panel is on the page by construction, and the page ignores Escape and has a back arrow in place of close. Expanding the panel to its page keeps the tab, and query results such as fetched console output come along from the cache.
-- **A click on the list's background closes the panel**, as UniFi's does. A row selects another resource instead, and links, buttons and inputs keep their own behaviour. Selecting rows for a bulk action closes it too, so the bulk bar and the panel never compete.
-- **Actions.** State actions sit in the summary as a button group, and the rest go in an Actions menu, Unikraft style. Stop, Reboot and every destructive action confirm in a dialog.
-- **Loading.** A panel or page still loading shows `PanelSpinner`, a centred spinner on the background, not a grey skeleton block.
-- **`DetailCard`, `DetailRow` and `ListCard` are retired.** Every resource page uses the panel sections instead.
-
-## Charts
-
-`MetricChart` (`src/components/charts/metric-chart.tsx`) draws a time series with uPlot, chosen for size and because it draws on a canvas, which the CSP allows where inline style attributes are not; `csp.e2e.ts` renders one under the production header.
-The look follows UniFi's statistics: smoothed lines over a fading fill, a light grid, a legend chip per series that reads the value under the cursor (or the latest), and `-24 hrs … Now` beneath.
-Series colours are the chart palette, and the chart redraws when the theme changes.
-
-Instance monitoring shows EC2's agentless basic metrics live: CPU utilisation, network bytes in and out, and packets in and out, one sample a second over the last five minutes, with no history to range over.
-The chart is built once and fed with `setData`, so a second's tick does not rebuild it, and polling stops while the tab is hidden.
-**The data is simulated** (`src/lib/simulated-metrics.ts`, seeded by instance ID and second) and the panel says so with a badge, until awsgw serves `GetMetricData` at a one-second period.
-
-## Dates
-
-`formatDateTime` writes the AWS console's form in the viewer's zone, `2026/06/04 02:00 GMT-5`, with a 24-hour clock and UTC as plain `GMT`.
-
-## Testing
-
-`make test-ui-e2e` rebuilds `dist` and runs `pnpm e2e`, which drives Playwright against two servers (`playwright.config.ts`):
-
-- **dev** (Vite on 3101) for behaviour: the `/design` catalogue in both themes and four widths, theme bootstrap, primitives' keyboard behaviour, the shell, and the Instances, VPC and Internet gateway list and panel flows.
-- **preview** (the built `dist` on 3102) for `csp.e2e.ts`, which serves the console's own CSP header and fails on any violation. Vite dev applies no CSP, so this is the only place an inline style or `eval` is caught before production.
-
-`e2e/console.ts` is the fixture. It seeds a fake session and answers `/proxy/**` by action, with empty defaults for what the dashboard and shell ask for; a test adds handlers for its own data, and any call nobody answered fails the test through `api.unhandled`.
-Unhandled calls get an error in the protocol's own shape, since a plain-text body would itself trip the CSP test.
-
-### Screenshots
-
-Screenshot baselines live in `e2e/__screenshots__/` and are Chromium renders taken in `en-AU` and UTC, so dates do not move with the host.
-They are drawn inside Playwright's own container image, because font rasterisation differs between hosts and a baseline drawn on a workstation fails on the runner.
-`playwright.image` in the frontend directory pins that image by version and digest, and is the one place it is named.
-Its version must equal the lockfile's `@playwright/test`, since the image carries the browsers for exactly one release, so bump both together.
-
-- `make ui-baselines` redraws every baseline in the pinned image, with `pnpm e2e --update-snapshots=all`, since the default mode keeps any baseline within the diff tolerance and so would keep one drawn on another host. Commit the changed files under `e2e/__screenshots__/` with the change that moved them.
-- `make ui-e2e-container` runs the suite in the same image and compares the screenshots, exactly as CI does.
-
-Both run `scripts/ui-playwright.sh`, which uses podman, or docker where podman is absent, and `CONTAINER_RUNTIME` overrides the choice.
-It mounts the frontend directory, installs with `pnpm install --frozen-lockfile` into a separate `node_modules` under `~/.cache/spinifex-ui-playwright`, so the host's `node_modules` is never touched, then builds `dist` and runs the suite.
-It refuses to run when the image's Node differs from `.nvmrc` or its Playwright differs from the lockfile's.
-`make test-ui-e2e` and `pnpm e2e` still run on the host, where screenshots compare reliably only against baselines drawn on that host, so do not commit baselines from `pnpm e2e:update`.
-
-The `UI End-to-End` job in `.github/workflows/ui.yml` runs the suite inside the pinned image on every pull request touching the frontend, and screenshots gate the merge with every behavioural and CSP test.
-A preceding `UI End-to-End Image` job reads `playwright.image`, so the workflow and the `make` targets cannot disagree about the image.
-On a failure the job uploads `test-results/` as the `ui-e2e-test-results` artifact, holding each failed screenshot's expected, actual and diff images and the trace.
-
-## Open
-
-- **Monitoring data is simulated.** Real samples need a source in the daemon; the plan is `docs/development/feature/ec2-basic-metrics.md` in the mulga repository.

internal/awsmodel/embed.go

@@ -17,6 +17,7 @@ var modelFiles = map[Service]string{
 	ElasticLoadBalancingV2: "models/elastic-load-balancing-v2-2015-12-01.json.gz",
 	IAM:                    "models/iam-2010-05-08.json.gz",
 	RDS:                    "models/rds-2014-10-31.json.gz",
+	Route53:                "models/route-53-2013-04-01.json.gz",
 	S3:                     "models/s3-2006-03-01.json.gz",
 	STS:                    "models/sts-2011-06-15.json.gz",
 }

internal/awsmodel/encode.go

@@ -1,8 +1,10 @@
 package awsmodel
 
 import (
+	"bytes"
 	"encoding/base64"
 	"encoding/json"
+	"encoding/xml"
 	"fmt"
 	"maps"
 	"net/http"
@@ -39,7 +41,7 @@ func (r *EncodedRequest) URL(endpoint string) string {
 }
 
 // EncodeRequest serializes input, as produced by GenerateRequests, for the
-// service's protocol. S3's rest-xml is not supported.
+// service's protocol: query, ec2, json, rest-json or rest-xml.
 func EncodeRequest(service Service, operationName string, input map[string]any) (*EncodedRequest, error) {
 	model, err := Load(service)
 	if err != nil {
@@ -57,6 +59,8 @@ func EncodeRequest(service Service, operationName string, input map[string]any)
 		return encoder.json(operation, input)
 	case "rest-json":
 		return encoder.restJSON(operation, input)
+	case "rest-xml":
+		return encoder.restXML(operation, input)
 	default:
 		return nil, fmt.Errorf("awsmodel: encoding %s requests is not implemented", model.metadata.Protocol)
 	}
@@ -282,18 +286,50 @@ func (e requestEncoder) jsonValue(ref ShapeRef, value any) (any, error) {
 }
 
 func (e requestEncoder) restJSON(operation *Operation, input map[string]any) (*EncodedRequest, error) {
+	request, members, err := e.restBindings(operation, input)
+	if err != nil || len(members) == 0 {
+		return request, err
+	}
+	shape := e.model.shapes[operation.Input.Shape]
+	var document any
+	if payload, ok := members[shape.Payload]; ok && shape.Payload != "" {
+		if document, err = e.jsonValue(shape.Members[shape.Payload], payload); err != nil {
+			return nil, err
+		}
+	} else {
+		body := make(map[string]any, len(members))
+		for member, value := range members {
+			ref := shape.Members[member]
+			encoded, err := e.jsonValue(ref, value)
+			if err != nil {
+				return nil, err
+			}
+			body[elementName(member, ref)] = encoded
+		}
+		document = body
+	}
+	if request.Body, err = json.Marshal(document); err != nil {
+		return nil, err
+	}
+	request.Header.Set("Content-Type", "application/json")
+	return request, nil
+}
+
+// restBindings places the URI, query string and header members of input on a
+// new request, and returns the members left for the body.
+func (e requestEncoder) restBindings(operation *Operation, input map[string]any) (*EncodedRequest, map[string]any, error) {
 	request := &EncodedRequest{Method: operation.HTTP.Method, Query: url.Values{}, Header: http.Header{}}
 	path := operation.HTTP.RequestURI
 	if operation.Input == nil {
 		request.Path = path
-		return request, nil
+		return request, nil, nil
 	}
 	shape := e.model.shapes[operation.Input.Shape]
 	body := map[string]any{}
 	for _, member := range slices.Sorted(maps.Keys(input)) {
 		ref, ok := shape.Members[member]
 		if !ok {
-			return nil, fmt.Errorf("%s has no member %q", operation.Input.Shape, member)
+			return nil, nil, fmt.Errorf("%s has no member %q", operation.Input.Shape, member)
 		}
 		value := input[member]
 		target := e.model.shapes[ref.Shape]
@@ -301,18 +337,18 @@ func (e requestEncoder) restJSON(operation *Operation, input map[string]any) (*E
 		case "uri":
 			text, err := scalarString(value, ref, target, "iso8601")
 			if err != nil {
-				return nil, fmt.Errorf("%s: %w", member, err)
+				return nil, nil, fmt.Errorf("%s: %w", member, err)
 			}
 			path = strings.Replace(path, "{"+member+"+}", escapeGreedyLabel(text), 1)
 			path = strings.Replace(path, "{"+member+"}", url.PathEscape(text), 1)
 		case "querystring":
 			if err := e.restQuery(request.Query, ref, target, value); err != nil {
-				return nil, fmt.Errorf("%s: %w", member, err)
+				return nil, nil, fmt.Errorf("%s: %w", member, err)
 			}
 		case "header":
 			text, err := e.headerValue(ref, target, value)
 			if err != nil {
-				return nil, fmt.Errorf("%s: %w", member, err)
+				return nil, nil, fmt.Errorf("%s: %w", member, err)
 			}
 			request.Header.Set(ref.LocationName, text)
 		case "headers":
@@ -321,41 +357,222 @@ func (e requestEncoder) restJSON(operation *Operation, input map[string]any) (*E
 				request.Header.Set(ref.LocationName+key, fmt.Sprint(entry))
 			}
 		default:
-			encoded, err := e.jsonValue(ref, value)
-			if err != nil {
-				return nil, err
-			}
-			name := member
-			if ref.LocationName != "" {
-				name = ref.LocationName
-			}
-			body[name] = encoded
+			body[member] = value
 		}
 	}
 	request.Path = path
-	if payload := shape.Payload; payload != "" {
-		if document, ok := body[payload]; ok {
-			body = nil
-			encoded, err := json.Marshal(document)
-			if err != nil {
-				return nil, err
-			}
-			request.Body = encoded
-		}
+	return request, body, nil
+}
+
+// restXML encodes the body as AWS SDKs do: a root element named for the input
+// shape, or for the payload member, in the service's namespace. An input with
+// body members sends its root element even when none of them is set.
+func (e requestEncoder) restXML(operation *Operation, input map[string]any) (*EncodedRequest, error) {
+	request, members, err := e.restBindings(operation, input)
+	if err != nil || operation.Input == nil {
+		return request, err
 	}
-	if len(body) > 0 {
-		encoded, err := json.Marshal(body)
-		if err != nil {
+	shape := e.model.shapes[operation.Input.Shape]
+	var root *xmlElement
+	switch {
+	case shape.Payload != "":
+		value, ok := members[shape.Payload]
+		if !ok {
+			return request, nil
+		}
+		ref := shape.Members[shape.Payload]
+		target := e.model.shapes[ref.Shape]
+		if target.Type != "structure" {
+			switch value := value.(type) {
+			case []byte:
+				request.Body = value
+			case string:
+				request.Body = []byte(value)
+			default:
+				return nil, fmt.Errorf("%s: want a blob or string payload, got %T", shape.Payload, value)
+			}
+			return request, nil
+		}
+		name := ref.LocationName
+		if name == "" {
+			name = elementName(ref.Shape, ShapeRef{LocationName: target.LocationName})
+		}
+		if root, err = e.xmlRoot(name, ref, value); err != nil {
 			return nil, err
 		}
-		request.Body = encoded
+	case hasBodyMembers(shape):
+		name := elementName(operation.Input.Shape, ShapeRef{LocationName: shape.LocationName})
+		if root, err = e.xmlRoot(name, *operation.Input, members); err != nil {
+			return nil, err
+		}
+	default:
+		return request, nil
 	}
-	if request.Body != nil {
-		request.Header.Set("Content-Type", "application/json")
+	var body bytes.Buffer
+	encoder := xml.NewEncoder(&body)
+	if err := root.encode(encoder); err != nil {
+		return nil, err
 	}
+	if err := encoder.Flush(); err != nil {
+		return nil, err
+	}
+	request.Body = body.Bytes()
+	request.Header.Set("Content-Type", "application/xml")
 	return request, nil
 }
 
+func hasBodyMembers(shape *Shape) bool {
+	for _, ref := range shape.Members {
+		if ref.Location == "" {
+			return true
+		}
+	}
+	return false
+}
+
+// xmlRoot is the structure value encoded under name, in its shape's namespace
+// or else the service's.
+func (e requestEncoder) xmlRoot(name string, ref ShapeRef, value any) (*xmlElement, error) {
+	holder := &xmlElement{}
+	if err := e.xmlValue(holder, name, ref, value); err != nil {
+		return nil, err
+	}
+	root := holder.children[0]
+	if namespace := e.model.metadata.XMLNamespace; namespace.URI != "" && e.model.shapes[ref.Shape].XMLNamespace.URI == "" {
+		root.attrs = append([]xml.Attr{namespaceAttr(namespace)}, root.attrs...)
+	}
+	return root, nil
+}
+
+func (e requestEncoder) xmlStructure(parent *xmlElement, shapeName string, fields map[string]any) error {
+	shape := e.model.shapes[shapeName]
+	for _, member := range slices.Sorted(maps.Keys(fields)) {
+		ref, ok := shape.Members[member]
+		if !ok {
+			return fmt.Errorf("%s has no member %q", shapeName, member)
+		}
+		name := elementName(member, ref)
+		if ref.XMLAttribute {
+			text, err := scalarString(fields[member], ref, e.model.shapes[ref.Shape], "iso8601")
+			if err != nil {
+				return fmt.Errorf("%s: %w", name, err)
+			}
+			parent.attrs = append(parent.attrs, xml.Attr{Name: xml.Name{Local: name}, Value: text})
+			continue
+		}
+		if err := e.xmlValue(parent, name, ref, fields[member]); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// xmlValue appends the elements value is encoded as to parent. A list or map
+// is wrapped in an element named name unless it is flattened.
+func (e requestEncoder) xmlValue(parent *xmlElement, name string, ref ShapeRef, value any) error {
+	shape := e.model.shapes[ref.Shape]
+	switch shape.Type {
+	case "structure":
+		fields, ok := value.(map[string]any)
+		if !ok {
+			return fmt.Errorf("%s: want a structure, got %T", name, value)
+		}
+		element := parent.add(name)
+		if shape.XMLNamespace.URI != "" {
+			element.attrs = append(element.attrs, namespaceAttr(shape.XMLNamespace))
+		}
+		return e.xmlStructure(element, ref.Shape, fields)
+	case "list":
+		items, ok := value.([]any)
+		if !ok {
+			return fmt.Errorf("%s: want a list, got %T", name, value)
+		}
+		container, itemName := parent, name
+		if !ref.Flattened {
+			container, itemName = parent.add(name), elementName("member", *shape.Member)
+		}
+		for _, item := range items {
+			if err := e.xmlValue(container, itemName, *shape.Member, item); err != nil {
+				return err
+			}
+		}
+		return nil
+	case "map":
+		entries, ok := value.(map[string]any)
+		if !ok {
+			return fmt.Errorf("%s: want a map, got %T", name, value)
+		}
+		container, entryName := parent, name
+		if !ref.Flattened {
+			container, entryName = parent.add(name), "entry"
+		}
+		keyName, valueName := elementName("key", *shape.Key), elementName("value", *shape.Value)
+		for _, key := range slices.Sorted(maps.Keys(entries)) {
+			entry := container.add(entryName)
+			entry.add(keyName).text = key
+			if err := e.xmlValue(entry, valueName, *shape.Value, entries[key]); err != nil {
+				return err
+			}
+		}
+		return nil
+	default:
+		text, err := scalarString(value, ref, shape, "iso8601")
+		if err != nil {
+			return fmt.Errorf("%s: %w", name, err)
+		}
+		parent.add(name).text = text
+		return nil
+	}
+}
+
+// elementName is a member's wire name: its xmlName or jsonName, else its own.
+func elementName(member string, ref ShapeRef) string {
+	if ref.LocationName != "" {
+		return ref.LocationName
+	}
+	return member
+}
+
+func namespaceAttr(namespace XMLNamespace) xml.Attr {
+	name := "xmlns"
+	if namespace.Prefix != "" {
+		name += ":" + namespace.Prefix
+	}
+	return xml.Attr{Name: xml.Name{Local: name}, Value: namespace.URI}
+}
+
+// xmlElement is one element of an encoded rest-xml body.
+type xmlElement struct {
+	name     string
+	attrs    []xml.Attr
+	text     string
+	children []*xmlElement
+}
+
+func (x *xmlElement) add(name string) *xmlElement {
+	child := &xmlElement{name: name}
+	x.children = append(x.children, child)
+	return child
+}
+
+func (x *xmlElement) encode(encoder *xml.Encoder) error {
+	start := xml.StartElement{Name: xml.Name{Local: x.name}, Attr: x.attrs}
+	if err := encoder.EncodeToken(start); err != nil {
+		return err
+	}
+	if x.text != "" {
+		if err := encoder.EncodeToken(xml.CharData(x.text)); err != nil {
+			return err
+		}
+	}
+	for _, child := range x.children {
+		if err := child.encode(encoder); err != nil {
+			return err
+		}
+	}
+	return encoder.EncodeToken(start.End())
+}
+
 func (e requestEncoder) restQuery(query url.Values, ref ShapeRef, shape *Shape, value any) error {
 	switch shape.Type {
 	case "list":

internal/awsmodel/encode_restxml_test.go

@@ -0,0 +1,212 @@
+package awsmodel_test
+
+import (
+	"cmp"
+	"encoding/xml"
+	"io"
+	"net/url"
+	"slices"
+	"strings"
+	"testing"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/aws/credentials"
+	"github.com/aws/aws-sdk-go/aws/request"
+	"github.com/aws/aws-sdk-go/aws/session"
+	"github.com/aws/aws-sdk-go/service/route53"
+	"github.com/aws/aws-sdk-go/service/s3"
+	"github.com/mulgadc/spinifex/internal/awsmodel"
+	"github.com/stretchr/testify/require"
+)
+
+const sdkEndpoint = "https://aws.example"
+
+var sdkSession = session.Must(session.NewSession(&aws.Config{
+	Region:           aws.String("us-east-1"),
+	Endpoint:         aws.String(sdkEndpoint),
+	Credentials:      credentials.AnonymousCredentials,
+	S3ForcePathStyle: aws.Bool(true),
+}))
+
+// TestEncodeRequestRestXMLMatchesSDK checks the encoder against aws-sdk-go's
+// restxml marshaller for the same input, so the expected wire form is the
+// SDK's rather than one written by hand.
+func TestEncodeRequestRestXMLMatchesSDK(t *testing.T) {
+	r53 := route53.New(sdkSession)
+	tests := []struct {
+		name      string
+		service   awsmodel.Service
+		operation string
+		input     map[string]any
+		sdk       *request.Request
+	}{
+		{
+			name:      "CreateHostedZone",
+			service:   awsmodel.Route53,
+			operation: "CreateHostedZone",
+			input: map[string]any{
+				"Name":             "example.com",
+				"CallerReference":  "ref-1",
+				"HostedZoneConfig": map[string]any{"Comment": "a & b", "PrivateZone": true},
+				"VPC":              map[string]any{"VPCRegion": "us-east-1", "VPCId": "vpc-1"},
+			},
+			sdk: requestOf(r53.CreateHostedZoneRequest(&route53.CreateHostedZoneInput{
+				Name:             aws.String("example.com"),
+				CallerReference:  aws.String("ref-1"),
+				HostedZoneConfig: &route53.HostedZoneConfig{Comment: aws.String("a & b"), PrivateZone: aws.Bool(true)},
+				VPC:              &route53.VPC{VPCRegion: aws.String("us-east-1"), VPCId: aws.String("vpc-1")},
+			})),
+		},
+		{
+			name:      "ChangeResourceRecordSets",
+			service:   awsmodel.Route53,
+			operation: "ChangeResourceRecordSets",
+			input: map[string]any{
+				"HostedZoneId": "Z0123456789",
+				"ChangeBatch": map[string]any{
+					"Comment": "two changes",
+					"Changes": []any{
+						map[string]any{"Action": "UPSERT", "ResourceRecordSet": map[string]any{
+							"Name": "www.example.com", "Type": "A", "TTL": int64(300),
+							"ResourceRecords": []any{map[string]any{"Value": "192.0.2.1"}, map[string]any{"Value": "192.0.2.2"}},
+						}},
+						map[string]any{"Action": "DELETE", "ResourceRecordSet": map[string]any{
+							"Name": "old.example.com", "Type": "TXT", "TTL": int64(60),
+							"ResourceRecords": []any{map[string]any{"Value": `"v=1"`}},
+						}},
+					},
+				},
+			},
+			sdk: requestOf(r53.ChangeResourceRecordSetsRequest(&route53.ChangeResourceRecordSetsInput{
+				HostedZoneId: aws.String("Z0123456789"),
+				ChangeBatch: &route53.ChangeBatch{
+					Comment: aws.String("two changes"),
+					Changes: []*route53.Change{
+						{Action: aws.String("UPSERT"), ResourceRecordSet: &route53.ResourceRecordSet{
+							Name: aws.String("www.example.com"), Type: aws.String("A"), TTL: aws.Int64(300),
+							ResourceRecords: []*route53.ResourceRecord{{Value: aws.String("192.0.2.1")}, {Value: aws.String("192.0.2.2")}},
+						}},
+						{Action: aws.String("DELETE"), ResourceRecordSet: &route53.ResourceRecordSet{
+							Name: aws.String("old.example.com"), Type: aws.String("TXT"), TTL: aws.Int64(60),
+							ResourceRecords: []*route53.ResourceRecord{{Value: aws.String(`"v=1"`)}},
+						}},
+					},
+				},
+			})),
+		},
+		{
+			name:      "ChangeTagsForResource",
+			service:   awsmodel.Route53,
+			operation: "ChangeTagsForResource",
+			input: map[string]any{
+				"ResourceType":  "hostedzone",
+				"ResourceId":    "Z0123456789",
+				"AddTags":       []any{map[string]any{"Key": "team", "Value": "dns"}, map[string]any{"Key": "env", "Value": "prod"}},
+				"RemoveTagKeys": []any{"old", "stale"},
+			},
+			sdk: requestOf(r53.ChangeTagsForResourceRequest(&route53.ChangeTagsForResourceInput{
+				ResourceType:  aws.String("hostedzone"),
+				ResourceId:    aws.String("Z0123456789"),
+				AddTags:       []*route53.Tag{{Key: aws.String("team"), Value: aws.String("dns")}, {Key: aws.String("env"), Value: aws.String("prod")}},
+				RemoveTagKeys: []*string{aws.String("old"), aws.String("stale")},
+			})),
+		},
+		{
+			name:      "UpdateHostedZoneComment without a comment",
+			service:   awsmodel.Route53,
+			operation: "UpdateHostedZoneComment",
+			input:     map[string]any{"Id": "Z0123456789"},
+			sdk:       requestOf(r53.UpdateHostedZoneCommentRequest(&route53.UpdateHostedZoneCommentInput{Id: aws.String("Z0123456789")})),
+		},
+		{
+			name:      "ListResourceRecordSets",
+			service:   awsmodel.Route53,
+			operation: "ListResourceRecordSets",
+			input:     map[string]any{"HostedZoneId": "Z0123456789", "StartRecordName": "www.example.com", "StartRecordType": "A", "MaxItems": int64(5)},
+			sdk: requestOf(r53.ListResourceRecordSetsRequest(&route53.ListResourceRecordSetsInput{
+				HostedZoneId: aws.String("Z0123456789"), StartRecordName: aws.String("www.example.com"), StartRecordType: aws.String("A"), MaxItems: aws.String("5"),
+			})),
+		},
+		{
+			name:      "DeleteHostedZone",
+			service:   awsmodel.Route53,
+			operation: "DeleteHostedZone",
+			input:     map[string]any{"Id": "Z0123456789"},
+			sdk:       requestOf(r53.DeleteHostedZoneRequest(&route53.DeleteHostedZoneInput{Id: aws.String("Z0123456789")})),
+		},
+		{
+			name:      "S3 structure payload",
+			service:   awsmodel.S3,
+			operation: "PutBucketTagging",
+			input: map[string]any{
+				"Bucket":  "bucket",
+				"Tagging": map[string]any{"TagSet": []any{map[string]any{"Key": "team", "Value": "storage"}}},
+			},
+			sdk: requestOf(s3.New(sdkSession).PutBucketTaggingRequest(&s3.PutBucketTaggingInput{
+				Bucket:  aws.String("bucket"),
+				Tagging: &s3.Tagging{TagSet: []*s3.Tag{{Key: aws.String("team"), Value: aws.String("storage")}}},
+			})),
+		},
+	}
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			require.NoError(t, test.sdk.Build())
+			want := test.sdk.HTTPRequest
+			wantBody, err := io.ReadAll(test.sdk.GetBody())
+			require.NoError(t, err)
+
+			got, err := awsmodel.EncodeRequest(test.service, test.operation, test.input)
+			require.NoError(t, err)
+			require.Equal(t, want.Method, got.Method)
+			gotURL, err := url.Parse(got.URL(sdkEndpoint))
+			require.NoError(t, err)
+			// The SDK's api-2.json ends ChangeResourceRecordSets' path in a
+			// slash that the Smithy model drops; the gateway accepts both.
+			require.Equal(t, strings.TrimSuffix(want.URL.Path, "/"), gotURL.Path)
+			require.Equal(t, want.URL.Query(), gotURL.Query())
+			if len(wantBody) == 0 {
+				require.Empty(t, got.Body)
+				require.Empty(t, got.Header.Get("Content-Type"))
+				return
+			}
+			require.Equal(t, canonicalXML(t, wantBody), canonicalXML(t, got.Body))
+			require.Equal(t, "application/xml", got.Header.Get("Content-Type"))
+		})
+	}
+}
+
+func requestOf[O any](r *request.Request, _ O) *request.Request { return r }
+
+// xmlTree is a parsed XML element. The SDK emits sibling elements in map
+// order, so canonicalXML sorts them by name, keeping list items in order.
+type xmlTree struct {
+	XMLName  xml.Name   `xml:""`
+	Attrs    []xml.Attr `xml:",any,attr"`
+	Text     string     `xml:",chardata"`
+	Children []xmlTree  `xml:",any"`
+}
+
+func canonicalXML(t *testing.T, body []byte) xmlTree {
+	t.Helper()
+	var tree xmlTree
+	require.NoError(t, xml.Unmarshal(body, &tree), string(body))
+	var canonicalise func(*xmlTree)
+	canonicalise = func(node *xmlTree) {
+		node.Text = strings.TrimSpace(node.Text)
+		slices.SortFunc(node.Attrs, func(a, b xml.Attr) int { return cmp.Compare(a.Name.Space+a.Name.Local, b.Name.Space+b.Name.Local) })
+		slices.SortStableFunc(node.Children, func(a, b xmlTree) int { return cmp.Compare(a.XMLName.Local, b.XMLName.Local) })
+		for i := range node.Children {
+			canonicalise(&node.Children[i])
+		}
+	}
+	canonicalise(&tree)
+	return tree
+}
+
+func TestEncodeRequestRestXMLPutsRootInServiceNamespace(t *testing.T) {
+	request, err := awsmodel.EncodeRequest(awsmodel.Route53, "CreateHostedZone", map[string]any{"Name": "example.com", "CallerReference": "ref"})
+	require.NoError(t, err)
+	tree := canonicalXML(t, request.Body)
+	require.Equal(t, xml.Name{Space: "https://route53.amazonaws.com/doc/2013-04-01/", Local: "CreateHostedZoneRequest"}, tree.XMLName)
+	require.Equal(t, "/2013-04-01/hostedzone", request.Path)
+}

internal/awsmodel/encode_test.go

@@ -117,8 +117,3 @@ func TestEncodeRequestRejectsUnmodelledMember(t *testing.T) {
 	_, err := awsmodel.EncodeRequest(awsmodel.IAM, "GetRole", map[string]any{"RoleName": "app", "Bogus": "x"})
 	require.ErrorContains(t, err, `no member "Bogus"`)
 }
-
-func TestEncodeRequestS3IsNotSupported(t *testing.T) {
-	_, err := awsmodel.EncodeRequest(awsmodel.S3, "ListBuckets", map[string]any{})
-	require.ErrorContains(t, err, "not implemented")
-}

internal/awsmodel/error.go

@@ -119,7 +119,7 @@ func (m *Model) decodeErrorCode(body []byte) (string, error) {
 			return "", fmt.Errorf("JSON error envelope has no code")
 		}
 		return code, nil
-	case "ec2", "query":
+	case "ec2", "query", "rest-xml":
 		root, err := parseXML(body)
 		if err != nil {
 			return "", err

internal/awsmodel/error_test.go

@@ -123,3 +123,16 @@ func TestValidateErrorResponseClassifiesCommonErrorsAsUnmodelled(t *testing.T) {
 		})
 	}
 }
+
+func TestDecodeRequestResultReadsRestXMLErrorEnvelope(t *testing.T) {
+	body := []byte(`<?xml version="1.0"?>
+<ErrorResponse xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><Error><Type>Sender</Type><Code>NoSuchHostedZone</Code><Message>No hosted zone found with ID: Z1</Message></Error><RequestId>r-1</RequestId></ErrorResponse>`)
+	result := DecodeRequestResult(Route53, 404, nil, body)
+	if result.Code != "NoSuchHostedZone" || result.Message != "No hosted zone found with ID: Z1" {
+		t.Fatalf("result = %+v", result)
+	}
+	violations, modelled, err := ValidateErrorResponse(Route53, "GetHostedZone", body)
+	if err != nil || !modelled || len(violations) != 0 {
+		t.Fatalf("ValidateErrorResponse = %v, %v, %v; want a declared code", violations, modelled, err)
+	}
+}

internal/awsmodel/model.go

@@ -34,6 +34,7 @@ const (
 	ElasticLoadBalancingV2 Service = "elasticloadbalancingv2"
 	IAM                    Service = "iam"
 	RDS                    Service = "rds"
+	Route53                Service = "route53"
 	S3                     Service = "s3"
 	STS                    Service = "sts"
 )
@@ -50,6 +51,14 @@ type Metadata struct {
 	ServiceID       string
 	SigningName     string
 	TargetPrefix    string
+	XMLNamespace    XMLNamespace
+}
+
+// XMLNamespace is the namespace an XML element is serialized in. URI is empty
+// when none is declared.
+type XMLNamespace struct {
+	URI    string `json:"uri"`
+	Prefix string `json:"prefix"`
 }
 
 // Operation describes an AWS API operation and the shapes used by its input,
@@ -84,6 +93,7 @@ type Shape struct {
 	Pattern         string
 	LocationName    string
 	TimestampFormat string
+	XMLNamespace    XMLNamespace
 	Payload         string
 	Flattened       bool
 	Sensitive       bool

internal/awsmodel/model_test.go

@@ -8,7 +8,7 @@ import (
 )
 
 func TestServices(t *testing.T) {
-	want := []Service{ACM, EC2, ECR, ECS, EKS, ElasticLoadBalancingV2, IAM, RDS, S3, STS}
+	want := []Service{ACM, EC2, ECR, ECS, EKS, ElasticLoadBalancingV2, IAM, RDS, Route53, S3, STS}
 	if got := Services(); !reflect.DeepEqual(got, want) {
 		t.Fatalf("Services() = %v, want %v", got, want)
 	}
@@ -28,6 +28,7 @@ func TestLoadAllModels(t *testing.T) {
 		ElasticLoadBalancingV2: {"2015-12-01", 51, 418},
 		IAM:                    {"2010-05-08", 180, 624},
 		RDS:                    {"2014-10-31", 164, 791},
+		Route53:                {"2013-04-01", 71, 409},
 		S3:                     {"2006-03-01", 112, 710},
 		STS:                    {"2011-06-15", 11, 101},
 	}

internal/awsmodel/models/README.md

@@ -1,6 +1,6 @@
 # AWS models
 
-The `*.json.gz` files are the Smithy JSON AST service models from [`aws/api-models-aws`](https://github.com/aws/api-models-aws), gzipped and embedded into the binary, so loading them needs no network or Go module cache. All ten come from the one commit recorded in `../model_source.go`.
+The `*.json.gz` files are the Smithy JSON AST service models from [`aws/api-models-aws`](https://github.com/aws/api-models-aws), gzipped and embedded into the binary, so loading them needs no network or Go module cache. All eleven come from the one commit recorded in `../model_source.go`.
 
 Do not edit them by hand. To move to a newer commit, run `scripts/sync-aws-models.sh <commit-sha>`, then `make generate-aws-model-coverage` and review the regenerated `docs/coverage` and the `awsmodel` test expectations: a newer model can add operations, required members and enum values, and each resulting finding is a real gap to triage rather than a reason to stay on an older commit.
 

internal/awsmodel/request.go

@@ -374,6 +374,10 @@ func (g *requestGenerator) structure(name string, path []pathStep, depth int) (m
 	for _, member := range slices.Sorted(maps.Keys(shape.Members)) {
 		ref := shape.Members[member]
 		memberPath := append(slices.Clone(path), member)
+		if !required[member] && slices.Contains(exclusiveMembers[g.model.service][name], member) {
+			g.skip(memberPath, "AWS refuses it alongside the members a valid request carries")
+			continue
+		}
 		memberValue, ok := g.value(ref, memberPath, depth)
 		if !ok {
 			if required[member] {

internal/awsmodel/request_hints.go

@@ -18,7 +18,8 @@ import (
 
 // RequestOptions places generated ARNs in the account and region the
 // requests are sent as. AccessKeyID is the key the requests are signed with.
-// Fixtures maps a member to an existing resource it can name.
+// Fixtures maps a member, or an "Operation.Member" that overrides it for one
+// operation, to an existing resource it can name.
 type RequestOptions struct {
 	AccountID   string
 	Region      string
@@ -73,6 +74,12 @@ var ec2IDMember = regexp.MustCompile(`^(.*?)Ids?$`)
 // with a code the judge cannot tell from a real finding.
 var conditionallyRequired = map[Service]map[string][]string{
 	EC2: {"CreateCapacityReservation": {"AvailabilityZone"}},
+	// A listing that starts at a zone ID needs the name it starts at, and
+	// one that starts at a record type or identifier needs a name and type.
+	Route53: {
+		"ListHostedZonesByName":  {"DNSName"},
+		"ListResourceRecordSets": {"StartRecordName", "StartRecordType"},
+	},
 	ECS: {
 		"CreateCapacityProvider": {"autoScalingGroupProvider"},
 		"CreateService":          {"taskDefinition"},
@@ -123,14 +130,47 @@ var ec2TaggedTypes = map[string]string{
 	"RunInstances":                    "instance",
 }
 
+// exclusiveMembers are optional members a valid request leaves out of the
+// structure that holds them, because AWS refuses them in combination with
+// the rest. Generation skips them, so their constraints are not broken.
+var exclusiveMembers = map[Service]map[string][]string{
+	// A record set takes one routing policy and either records or an alias.
+	Route53: {"ResourceRecordSet": {
+		"AliasTarget", "CidrRoutingConfig", "Failover", "GeoLocation", "GeoProximityLocation", "HealthCheckId",
+		"MultiValueAnswer", "Region", "SetIdentifier", "TrafficPolicyInstanceId", "Weight",
+	}},
+}
+
+// fixture returns the fixture the member at path names, preferring one scoped
+// to the operation.
+func (g *requestGenerator) fixture(member string) (string, bool) {
+	if fixture, ok := g.options.Fixtures[g.operation+"."+member]; ok {
+		return fixture, true
+	}
+	fixture, ok := g.options.Fixtures[member]
+	return fixture, ok
+}
+
 // enumHint returns the enum value a valid request needs at path, where the
 // first value would not do.
 func (g *requestGenerator) enumHint(path []pathStep) (string, bool) {
-	if g.model.service != EC2 || len(path) == 0 || path[0] != "TagSpecifications" || memberName(path) != "ResourceType" {
-		return "", false
+	member := memberName(path)
+	if fixture, ok := g.fixture(member); ok {
+		return fixture, true
 	}
-	value, ok := ec2TaggedTypes[g.operation]
-	return value, ok
+	switch g.model.service {
+	case EC2:
+		if len(path) > 0 && path[0] == "TagSpecifications" && member == "ResourceType" {
+			value, ok := ec2TaggedTypes[g.operation]
+			return value, ok
+		}
+	case Route53:
+		// The first RRType, SOA, is one a zone already has and cannot gain.
+		if member == "Type" {
+			return "A", true
+		}
+	}
+	return "", false
 }
 
 // hint returns a value of the right form for the member at path, if one is
@@ -141,7 +181,7 @@ func (g *requestGenerator) hint(path []pathStep) (string, bool) {
 		return "", false
 	}
 	options := g.options
-	if fixture, ok := options.Fixtures[member]; ok {
+	if fixture, ok := g.fixture(member); ok {
 		return fixture, true
 	}
 	suffix := fmt.Sprintf("%017x", g.saltNumber)
@@ -230,6 +270,17 @@ func (g *requestGenerator) hint(path []pathStep) (string, bool) {
 		if member == "type" && len(path) == 1 {
 			return "STANDARD", true
 		}
+	case Route53:
+		// A generated name is a top-level domain, which cannot be a zone.
+		// Record names fall inside the fixture zone, example.com.
+		switch member {
+		case "Name", "DNSName", "StartRecordName":
+			return name + ".example.com", true
+		case "Value":
+			if slices.Contains(path, pathStep("ResourceRecords")) {
+				return "192.0.2.1", true
+			}
+		}
 	case STS:
 		// GetAccessKeyInfo answers only for a key that exists.
 		if member == "AccessKeyId" && options.AccessKeyID != "" {

internal/awsmodel/request_test.go

@@ -131,6 +131,7 @@ func TestGeneratedRequestsBreakOnlyTheirConstraint(t *testing.T) {
 				for _, request := range plan.Cases {
 					require.False(t, rejecting && request.Acceptance(), "%s acceptance %s follows a rejection", operationName, request.Path)
 					rejecting = !request.Acceptance()
+					// Predastore serves S3, so its requests are never sent.
 					if service != S3 {
 						_, err := EncodeRequest(service, operationName, request.Input)
 						require.NoError(t, err, "%s %s %s encodes", operationName, request.Constraint, request.Path)
@@ -241,6 +242,17 @@ func TestGenerateRequestsTargetsFixturesOutsideCreateAndDelete(t *testing.T) {
 	}
 }
 
+func TestGenerateRequestsPrefersOperationScopedFixtures(t *testing.T) {
+	options := testRequestOptions
+	options.Fixtures = map[string]string{"Id": "Z0FIXTURE", "GetChange.Id": "C0FIXTURE"}
+	for operation, want := range map[string]string{"GetChange": "C0FIXTURE", "GetHostedZone": "Z0FIXTURE"} {
+		plan, err := GenerateRequests(Route53, operation, options)
+		require.NoError(t, err)
+		require.True(t, plan.Cases[0].Acceptance())
+		require.Equal(t, want, plan.Cases[0].Input["Id"], operation)
+	}
+}
+
 func TestGenerateRequestsAttributesRejections(t *testing.T) {
 	plan, err := GenerateRequests(IAM, "CreateRole", testRequestOptions)
 	require.NoError(t, err)

internal/awsmodel/smithy.go

@@ -281,6 +281,9 @@ func (t *smithyTranslator) metadata() (Metadata, error) {
 	if _, err := service.Traits.decode("aws.auth#sigv4", &sigv4); err != nil {
 		return Metadata{}, err
 	}
+	if _, err := service.Traits.decode("smithy.api#xmlNamespace", &metadata.XMLNamespace); err != nil {
+		return Metadata{}, err
+	}
 	metadata.EndpointPrefix = awsService.EndpointPrefix
 	metadata.ServiceID = awsService.SDKID
 	metadata.ServiceFullName = title
@@ -443,6 +446,9 @@ func (t *smithyTranslator) applyShapeTraits(shape *Shape, traits smithyTraits) e
 	if shape.LocationName, err = traits.string("smithy.api#xmlName"); err != nil {
 		return err
 	}
+	if _, err := traits.decode("smithy.api#xmlNamespace", &shape.XMLNamespace); err != nil {
+		return err
+	}
 	if shape.TimestampFormat, err = timestampFormat(traits); err != nil {
 		return err
 	}

scripts/README.md

@@ -7,9 +7,10 @@ Node installation, dev-environment lifecycle, guest image builds, and verificati
 | Script | What it does |
 |---|---|
 | `setup.sh` | The production binary installer, served from `install.mulgadc.com`. Installs deps, users, sudoers, the host firewall policy, files, systemd units, logrotate and udev rules, and loads `openvswitch`, `geneve`, `vhost_net` and `tun`, listing them in `/etc/modules-load.d/spinifex.conf` for later boots. It masks firewalld wherever it is installed, since the Spinifex nftables policy is the only supported node firewall. An unset `--firewall` keeps the mode recorded in `/etc/spinifex/firewall/mode`, and otherwise arms the policy when firewalld was active. Where systemd-resolved is installed but not running, resolvconf is absent and NetworkManager runs, it hands host DNS to resolved: it saves `/etc/resolv.conf` once to `/var/lib/spinifex-reset/resolv.conf.orig`, links `/etc/resolv.conf` to the resolved stub, and stops unless resolved has upstream servers and the package repository host still resolves. It links `/usr/bin/spx` to `/usr/local/bin/spx` where sudo's PATH leaves out `/usr/local/bin`, and takes the nbdkit plugin directory from `nbdkit --dump-config`, stopping when nbdkit cannot say. `SETUP_STAGES` runs a subset; `ISO_BUILD=1` runs it inside the ISO builder's chroot. |
-| `setup-ovn.sh` | Bootstraps a compute node for OVN VPC networking: installs OVN/OVS, creates `br-int`, configures the WAN bridge (bridged, DHCP, or routed-NAT uplink), sets the chassis identity, and applies overlay sysctls. It runs OVS and OVN as root, setting `OVS_USER_ID` and `OVN_USER_ID` where `/etc/sysconfig` holds them, and hooks the socket-permission helper onto `ovs-vswitchd`, `ovn-controller`, `openvswitch-ipsec` and, on EL, `ovn-northd`. `--management` also starts the OVN central services, writes their `ovn-ctl` flags to `/etc/default/ovn-central` (Debian) or `/etc/sysconfig/ovn-northd` (EL), and records the role in `/etc/spinifex/ovn-role`. A later run without `--management` refuses while that file, or the `OVN_CTL_OPTS` line older nodes carry in `/etc/default/ovn-central`, records the role. It adds `maxsize 100M` to the OVS and OVN logrotate files, plus `rotate 5` where the file sets no count, and rotates as root, matching the daemons. On a host whose IPsec stack is libreswan it stops and masks `openvswitch-ipsec.service` and `ipsec.service`, because OVN IPsec does not run on libreswan yet; elsewhere it enables and starts `openvswitch-ipsec.service`, unless the daemon has masked it to keep IPsec off. The links it creates persist through systemd-networkd where that is enabled, and otherwise through `/etc/spinifex/hostnet.json`, which `spinifex-hostnet.service` (`spx admin hostnet apply`) replays at boot and on every OVS start, starting `ovn-controller` again when an OVS stop took it down; the first run records the choice in `/etc/spinifex/hostnet-backend`. Where NetworkManager runs, it leaves the links Spinifex owns unmanaged. |
+| `setup-ovn.sh` | Bootstraps a compute node for OVN VPC networking: installs OVN/OVS, creates `br-int`, configures the WAN bridge (bridged, DHCP, or routed-NAT uplink), sets the chassis identity, and applies overlay sysctls. It runs OVS and OVN as root, setting `OVS_USER_ID` and `OVN_USER_ID` where `/etc/sysconfig` holds them, and hooks the socket-permission helper onto `ovs-vswitchd`, `ovn-controller`, `openvswitch-ipsec` and, on EL, `ovn-northd`. `--management` also starts the OVN central services, writes their `ovn-ctl` flags to `/etc/default/ovn-central` (Debian) or `/etc/sysconfig/ovn-northd` (EL), and records the role in `/etc/spinifex/ovn-role`. With clustered databases it returns only once this node's NB and SB members are back in their RAFT clusters under a known leader, and fails after `OVN_RAFT_REJOIN_WAIT` seconds (default 60) otherwise, so reruns one node at a time never take two members of a cluster down. A later run without `--management` refuses while that file, or the `OVN_CTL_OPTS` line older nodes carry in `/etc/default/ovn-central`, records the role. It adds `maxsize 100M` to the OVS and OVN logrotate files, plus `rotate 5` where the file sets no count, and rotates as root, matching the daemons. On a host whose IPsec stack is libreswan it enables, stops and masks `openvswitch-ipsec.service` and `ipsec.service` for the daemon to unmask when IPsec is on, unless `openvswitch-ipsec.service` is already running; elsewhere it enables and starts `openvswitch-ipsec.service`, unless the daemon has masked it to keep IPsec off. The links it creates persist through systemd-networkd where that is enabled, and otherwise through `/etc/spinifex/hostnet.json`, which `spinifex-hostnet.service` (`spx admin hostnet apply`) replays at boot and on every OVS start, starting `ovn-controller` again when an OVS stop took it down; the first run records the choice in `/etc/spinifex/hostnet-backend`. Where NetworkManager runs, it leaves the links Spinifex owns unmanaged. |
 | `install-node.sh` | Forms a multi-node cluster across hosts that are already running Spinifex: rebuilds the OVN database in clustered form, runs `spx admin init` on the first host and `spx admin join` on the rest, then verifies. Before touching any host it refuses hosts of different OS families (`distro_os_family`) or `spx` versions, and `--ipsec on` when any host uses libreswan. The scripted form of [install-multi-node](../docs/install/install-multi-node/README.md). Destroys the joining nodes' logical network state and crypto — see its `--help`. |
 | `dev-install.sh` | Full local dev setup through the production installer: builds from source, packs a tarball, runs `setup.sh`, initialises the cluster, and starts services under systemd. |
+| `build-oracle-appliance.sh` | Turns an Oracle Linux 10.2 system reachable over SSH with passwordless sudo into a sealed gold image, then powers it off. It updates OL and reboots into the newest UEK R8, sets SELinux permissive, installs Spinifex without initialising it, and leaves `spinifex.target` and the NATS watchdog disabled. It creates the `spinifex` login with a default password that must be changed at first login, installs `appliance/firstboot.sh` as `spinifex-appliance-firstboot.service`, and wipes the system's identity (machine-id, SSH host keys, OVS chassis, NetworkManager profiles, logs). On each unit's first boot that service puts the first physical NIC with carrier on a DHCP `br-wan`, names the host `spinifex-<mac>`, runs `setup-ovn.sh --management` and `spx admin init --nodes 1`, and shows the IP and login on the console and in `/etc/motd`. `--channel`, `--version` and `--tarball` pick what to install. |
 | `clone-deps.sh` | Clones or updates the viperblock and predastore checkouts alongside spinifex for cross-repo development. |
 
 ### `lib/distro.sh`: host-OS capability probes
@@ -58,7 +59,7 @@ Both ship to `/usr/local/share/spinifex/` on every node, so they are available o
 
 | Script | What it does |
 |---|---|
-| `node-reset.sh` | Clears **state** and leaves the node installed: stops services, destroys guests, wipes `/etc/spinifex`, `/var/lib/spinifex`, `/var/log/spinifex` and the OVN databases, and removes the OVS bridges. It stops OVS and OVN central by the unit names the host's packages use. It leaves the NetworkManager unmanaged-device config, which the next `setup-ovn.sh` run rewrites, so NetworkManager cannot take a NIC back in between. The node is then ready for a fresh `spx admin init`. `--keep-data` preserves `/var/lib/spinifex`; `SPX_RESET_CONFIRM=reset` runs it unattended. |
+| `node-reset.sh` | Clears **state** and leaves the node installed: stops services, destroys guests, wipes `/etc/spinifex`, `/var/lib/spinifex`, `/var/log/spinifex` and the OVN databases, and removes the OVS bridges. On a libreswan host it deletes the `ovs_cert_*` and `ovs_certkey_*` entries `ovs-monitor-ipsec` imported into libreswan's NSS database, leaving other identities and `/etc/ipsec.d/*.conf` alone. It stops OVS and OVN central by the unit names the host's packages use. It leaves the NetworkManager unmanaged-device config, which the next `setup-ovn.sh` run rewrites, so NetworkManager cannot take a NIC back in between. The node is then ready for a fresh `spx admin init`. `--keep-data` preserves `/var/lib/spinifex`; `SPX_RESET_CONFIRM=reset` runs it unattended. |
 | `uninstall-spx.sh` | Removes the **software** — the inverse of `setup.sh`. Delegates the state teardown above to `node-reset.sh` rather than reimplementing it, then removes the binary, the nbdkit plugin, the systemd units and slices, the service users, the sudoers grant, the sysctl/logrotate/chrony/udev drop-ins, the `systemd-networkd` units, the host firewall policy and its loaded nft table, the Spinifex systemd-resolved drop-in, and the Spinifex CA from the host trust store. It unmasks firewalld and leaves it disabled. Where `setup.sh` handed DNS to systemd-resolved, it restores the saved `/etc/resolv.conf` and turns resolved off. Verifies afterwards and exits non-zero naming anything that survived, then lists the host settings `setup.sh` changed that it leaves in place, such as the dnf repositories, `OVS_USER_ID`, masked IPsec units and the SELinux mode. |
 
 `node-reset.sh` finds guests by process name, `qemu-system-*` or `qemu-kvm`, owned by the daemon's service user, and waits for them to exit before it deletes any state.
@@ -85,7 +86,6 @@ Both ship to `/usr/local/share/spinifex/` on every node, so they are available o
 | `check-coverage.sh` | Fails the build if total Go coverage falls below the minimum threshold, excluding packages exempt from the requirement. |
 | `diff-coverage.sh` | Coverage check scoped to changed lines only, against an auto-detected base ref (`HEAD~1` on main, `origin/main` on dev, otherwise `origin/dev`). |
 | `sync-aws-models.sh` | Replaces the embedded AWS Smithy models under `internal/awsmodel/models/` with the ten service files from `aws/api-models-aws` at the given commit, and records that commit in `internal/awsmodel/model_source.go`. Takes a full 40-character SHA. |
-| `ui-playwright.sh compare\|baselines [args...]` | Runs the console's Playwright suite inside the image pinned in the frontend's `playwright.image`, the one CI's `UI End-to-End` job uses, with podman or docker. `compare` checks screenshots as CI does and `baselines` redraws them; extra arguments go to `pnpm e2e`. Installs into its own `node_modules` under `~/.cache/spinifex-ui-playwright`, leaving the host's alone. `make ui-e2e-container` and `make ui-baselines` run it. |
 | `run-bench.sh` | Launches a benchmark against running Ubuntu instances, auto-detecting the AMI for the host architecture. Tracks nbdkit and perf alongside the run. |
 | `disk-performance.sh` | Runs a `fio` random 70/30 read-write benchmark swept across 4k/16k/128k/1M. Writes JSON results to `/tmp/spinifex-disk-bench`; override with `BENCH_DIR`, `OUT_DIR`, `SIZE`, `JOBS`, `BLOCK_SIZES`. |
 | `network-performance.sh` | iperf throughput from several clients to one server, driven over SSH, writing a `summary.txt` of Gbit/s per client. `--server-ip` sets the address clients dial, so pointing it at instance private addresses measures the VPC overlay rather than the external pool. |

scripts/appliance/firstboot.sh

@@ -0,0 +1,146 @@
+#!/bin/bash
+# Spinifex appliance first boot, for Oracle Linux units cloned from the gold
+# image that build-oracle-appliance.sh seals. The OL counterpart of the ISO's
+# spinifex-firstboot.sh: it also does the per-unit work the ISO installer does
+# before first boot (hostname, br-wan on the NIC that is present), then sets
+# up a single node. On success it writes $DONE_MARKER and the unit's
+# ConditionPathExists=! stops it running again. A failed run leaves no marker,
+# so the next boot retries from the top; every step below is safe to repeat.
+set -euo pipefail
+
+DONE_MARKER=/var/lib/spinifex/.appliance-firstboot-done
+SHARE=/usr/local/share/spinifex
+
+# shellcheck source=lib.sh
+. "$SHARE/appliance/lib.sh"
+# shellcheck source=../lib/distro.sh
+. "$SHARE/lib/distro.sh"
+
+log() { echo "[appliance-firstboot] $*"; }
+
+if [ -f "$DONE_MARKER" ]; then
+    log "already complete, skipping"
+    exit 0
+fi
+
+# ── Primary NIC ─────────────────────────────────────────────────────────────
+# Links start down when NetworkManager has no profile for them, so bring every
+# physical NIC up to read its carrier, and give a cable up to 60s to come up.
+nics=$(appliance_physical_nics)
+if [ -z "$nics" ]; then
+    log "no physical Ethernet device found"
+    exit 1
+fi
+for nic in $nics; do
+    ip link set "$nic" up || true
+done
+for _i in $(seq 1 60); do
+    for nic in $nics; do
+        [ "$(cat "/sys/class/net/$nic/carrier" 2>/dev/null)" = 1 ] && break 2
+    done
+    sleep 1
+done
+NIC=$(appliance_pick_nic)
+MAC=$(cat "/sys/class/net/$NIC/address")
+log "primary NIC $NIC ($MAC), carrier $(cat "/sys/class/net/$NIC/carrier" 2>/dev/null || echo 0)"
+
+HOSTNAME_NEW=$(appliance_hostname "$MAC")
+hostnamectl set-hostname "$HOSTNAME_NEW"
+log "hostname $HOSTNAME_NEW"
+
+# ── br-wan on the primary NIC, DHCP ─────────────────────────────────────────
+# The bridge takes the NIC's MAC so a DHCP reservation made for the unit's
+# port keeps working. bridge.stp no: with STP on, the port sits in its
+# forwarding delay long enough for DHCP on the bridge to time out.
+for con in $(nmcli -g UUID,DEVICE connection show | awk -F: -v d="$NIC" '$2 == d {print $1}'); do
+    nmcli connection delete "$con" || true
+done
+nmcli connection delete br-wan br-wan-port >/dev/null 2>&1 || true
+nmcli connection add type bridge ifname br-wan con-name br-wan \
+    bridge.stp no bridge.mac-address "$MAC" \
+    ipv4.method auto ipv6.method disabled connection.autoconnect yes
+nmcli connection add type bridge-slave ifname "$NIC" master br-wan con-name br-wan-port \
+    connection.autoconnect yes
+nmcli connection up br-wan || true
+
+log "waiting for a DHCP address on br-wan"
+WAN_IP=""
+for _i in $(seq 1 120); do
+    WAN_IP=$(ip -4 -o addr show br-wan scope global 2>/dev/null | awk '{print $4}' | cut -d/ -f1 | head -1)
+    if [ -n "$WAN_IP" ] && ip -4 route show default dev br-wan | grep -q .; then
+        break
+    fi
+    WAN_IP=""
+    sleep 1
+done
+if [ -z "$WAN_IP" ]; then
+    log "br-wan got no DHCP address and default route in 120s; check the cable on $NIC and the site's DHCP"
+    exit 1
+fi
+log "br-wan $WAN_IP"
+
+# ── Clock ───────────────────────────────────────────────────────────────────
+# Credentials and TLS certificates are minted below, and a certificate issued
+# against a wrong RTC carries a NotBefore that outlives the correction.
+# Bounded and non-fatal, as on the ISO.
+chronyc -a makestep >/dev/null 2>&1 || true
+if chronyc waitsync 12 0.1 0 5 >/dev/null 2>&1; then
+    log "clock synced: $(date -uIs)"
+else
+    log "WARNING: clock not synced, continuing with $(date -uIs)"
+fi
+
+# ── OVS and OVN ─────────────────────────────────────────────────────────────
+# Start them before setup-ovn.sh so their databases exist when it runs: on
+# physical hardware first-boot DB initialisation outlasts its own timeout.
+OVS_UNIT=$(distro_ovs_unit)
+OVN_LAYOUT=$(distro_ovn_central_layout)
+case $OVN_LAYOUT in
+    split) OVN_CENTRAL_UNIT=ovn-central.service ;;
+    lone-northd) OVN_CENTRAL_UNIT=ovn-northd.service ;;
+esac
+systemctl start "$OVS_UNIT" "$OVN_CENTRAL_UNIT"
+for _i in $(seq 1 120); do
+    if ovn-nbctl --timeout=2 get-connection >/dev/null 2>&1; then
+        log "OVN NB DB ready (${_i}s)"
+        break
+    fi
+    sleep 1
+done
+
+"$SHARE/setup-ovn.sh" --management
+
+# ── Single-node cluster ─────────────────────────────────────────────────────
+# Same arguments as the ISO's single-NIC DHCP path. spx writes credentials to
+# disk and never prints them.
+spx admin init --node "$HOSTNAME_NEW" --nodes 1 2>&1
+
+# spx ran as root, so the files it wrote are root:root. setup.sh's fixown
+# stage knows each file's owner and mode.
+SETUP_STAGES=fixown "$SHARE/setup.sh"
+
+# ── spinifex login ──────────────────────────────────────────────────────────
+if [ ! -f /home/spinifex/.ssh/id_ed25519 ]; then
+    install -d -m 700 -o spinifex -g spinifex /home/spinifex/.ssh
+    ssh-keygen -q -t ed25519 -f /home/spinifex/.ssh/id_ed25519 -N "" -C "spinifex@$HOSTNAME_NEW"
+    chown spinifex:spinifex /home/spinifex/.ssh/id_ed25519 /home/spinifex/.ssh/id_ed25519.pub
+fi
+# init wrote the AWS profile under HOME=/root; the operator logs in as spinifex.
+if [ -f /root/.aws/credentials ]; then
+    install -d -m 700 -o spinifex -g spinifex /home/spinifex/.aws
+    install -m 600 -o spinifex -g spinifex /root/.aws/credentials /home/spinifex/.aws/credentials
+    [ -f /root/.aws/config ] && install -m 600 -o spinifex -g spinifex /root/.aws/config /home/spinifex/.aws/config
+fi
+
+# ── Services ────────────────────────────────────────────────────────────────
+# Enabled here, not in the image, so a clone runs nothing of Spinifex before
+# init has written its config.
+systemctl enable spinifex.target spinifex-banner.service spinifex-firewall.service
+systemctl enable --now spinifex-nats-watchdog.timer
+
+mkdir -p "$(dirname "$DONE_MARKER")"
+touch "$DONE_MARKER"
+
+systemctl start spinifex.target
+/usr/local/bin/spx admin banner --boot-check
+log "complete: https://$WAN_IP:3000, login spinifex"

scripts/appliance/getty-wait.conf

@@ -0,0 +1,4 @@
+# Holds the console login prompt until first boot and the banner have written
+# /etc/issue, so the unit's IP shows before anyone logs in.
+[Unit]
+After=spinifex-appliance-firstboot.service spinifex-banner.service

scripts/appliance/lib.sh

@@ -0,0 +1,37 @@
+#!/bin/bash
+# Helpers shared by the appliance first-boot script and its test. Sourced, not run.
+
+# appliance_physical_nics — physical Ethernet devices, one per line, in PCI
+# order. Skips anything without a backing device (bridges, veths, OVS ports,
+# loopback), non-Ethernet link types and wireless. SYSFS overrides /sys.
+appliance_physical_nics() {
+    local sys="${SYSFS:-/sys}/class/net" dev
+    for dev in "$sys"/*; do
+        [ -e "$dev/device" ] || continue
+        [ "$(cat "$dev/type" 2>/dev/null)" = 1 ] || continue
+        [ -d "$dev/wireless" ] && continue
+        printf '%s %s\n' "$(readlink -f "$dev/device")" "$(basename "$dev")"
+    done | sort | awk '{print $2}'
+}
+
+# appliance_pick_nic — the first physical NIC with carrier, else the first
+# physical NIC. Prints nothing when the host has none.
+appliance_pick_nic() {
+    local sys="${SYSFS:-/sys}/class/net" nic first=""
+    for nic in $(appliance_physical_nics); do
+        [ -n "$first" ] || first=$nic
+        if [ "$(cat "$sys/$nic/carrier" 2>/dev/null)" = 1 ]; then
+            echo "$nic"
+            return 0
+        fi
+    done
+    [ -n "$first" ] && echo "$first"
+    return 0
+}
+
+# appliance_hostname <mac> — spinifex-<last 6 hex digits of the MAC>.
+appliance_hostname() {
+    local hex
+    hex=$(echo "$1" | tr -d ':' | tr '[:upper:]' '[:lower:]')
+    echo "spinifex-${hex: -6}"
+}

scripts/appliance/lib_test.sh

@@ -0,0 +1,63 @@
+#!/bin/sh
+# lib_test.sh — tests for the appliance first-boot NIC and hostname helpers,
+# run against a fixture sysfs tree (SYSFS) so the host's own NICs never count.
+set -eu
+
+DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
+TMP=$(mktemp -d)
+trap 'rm -rf "$TMP"' EXIT
+
+fails=0
+pass() { echo "  ok   - $1"; }
+fail() { echo "  FAIL - $1"; fails=$((fails + 1)); }
+
+echo "-- appliance/lib.sh"
+
+# nic <name> <pci-slot> <carrier> [type] — a physical NIC behind a PCI device.
+nic() {
+    mkdir -p "$SYS/devices/pci0000:00/$2" "$SYS/class/net/$1"
+    ln -s "../../../devices/pci0000:00/$2" "$SYS/class/net/$1/device"
+    echo "${4:-1}" > "$SYS/class/net/$1/type"
+    echo "$3" > "$SYS/class/net/$1/carrier"
+}
+# virt <name> — a link with no backing device (bridge, veth, OVS port).
+virt() {
+    mkdir -p "$SYS/class/net/$1"
+    echo 1 > "$SYS/class/net/$1/type"
+    echo 1 > "$SYS/class/net/$1/carrier"
+}
+fixture() { SYS="$TMP/$1"; mkdir -p "$SYS/class/net"; }
+run() { SYSFS="$SYS" bash -c ". '$DIR/lib.sh'; $1"; }
+
+check() {
+    if [ "$2" = "$3" ]; then pass "$1"; else fail "$1: want '$2', got '$3'"; fi
+}
+
+fixture order
+nic enp3s0 0000:00:03.0 0
+nic enp1s0 0000:00:01.0 0
+virt br-wan
+virt lo
+check "physical NICs in PCI order, virtual links skipped" "enp1s0 enp3s0" "$(run appliance_physical_nics | tr '\n' ' ' | sed 's/ $//')"
+check "no carrier anywhere picks the first by PCI order" enp1s0 "$(run appliance_pick_nic)"
+
+fixture carrier
+nic eno1 0000:00:01.0 0
+nic eno2 0000:00:02.0 1
+nic eno3 0000:00:03.0 1
+check "the first NIC with carrier wins over PCI order" eno2 "$(run appliance_pick_nic)"
+
+fixture wireless
+nic wlp1s0 0000:00:01.0 1
+mkdir -p "$SYS/class/net/wlp1s0/wireless"
+nic ib0 0000:00:02.0 1 32
+nic enp4s0 0000:00:04.0 0
+check "wireless and non-Ethernet links are skipped" enp4s0 "$(run appliance_pick_nic)"
+
+fixture none
+virt br-wan
+check "no physical NIC prints nothing" "" "$(run appliance_pick_nic)"
+
+check "hostname from the MAC's last 6 hex digits" spinifex-c0ffee "$(run 'appliance_hostname 52:54:00:C0:FF:EE')"
+
+[ "$fails" -eq 0 ] || { echo "  $fails failed"; exit 1; }

scripts/appliance/prepare.sh

@@ -0,0 +1,178 @@
+#!/bin/bash
+# Runs as root on the Oracle Linux build system, one stage per call, driven by
+# build-oracle-appliance.sh from the payload directory it copied over.
+#   prepare.sh check      refuse anything but OL 10.2+ on UEK R8, x86_64
+#   prepare.sh update     dnf update, SELinux permissive; prints PREPARE_REBOOT=1 if
+#                         the newest kernel is not the running one
+#   prepare.sh install    install Spinifex, uninitialised and not enabled
+#   prepare.sh appliance  spinifex login, first-boot service, console placeholder
+#   prepare.sh seal       wipe this system's identity, then power off
+# INSTALL_SPINIFEX_CHANNEL, INSTALL_SPINIFEX_VERSION and INSTALL_SPINIFEX_TARBALL
+# pass through to the installer. BUILD_USER is the SSH user to lock at seal.
+set -euo pipefail
+
+# Shared by every unit built from this image; first login forces a change.
+DEFAULT_PASSWORD=spinifex
+
+HERE=$(cd "$(dirname "$0")" && pwd)
+SHARE=/usr/local/share/spinifex
+
+log() { echo "[prepare] $*"; }
+die() { echo "[prepare] ERROR: $*" >&2; exit 1; }
+
+[ "$(id -u)" = 0 ] || die "run as root"
+
+stage_check() {
+    # shellcheck source=/dev/null
+    . /etc/os-release
+    [ "${ID:-}" = ol ] || die "not Oracle Linux (ID=${ID:-unset})"
+    local major=${VERSION_ID%%.*} minor=${VERSION_ID#*.}
+    [ "$minor" = "$VERSION_ID" ] && minor=0
+    if [ "$major" -lt 10 ] || { [ "$major" -eq 10 ] && [ "$minor" -lt 2 ]; }; then
+        # A 10.0 or 10.1 system reaches 10.2 through the update stage, which
+        # calls this again with "strict" once it has run.
+        if [ "$major" -ne 10 ] || [ "${1:-}" = strict ]; then
+            die "Oracle Linux $VERSION_ID; 10.2 or later is required"
+        fi
+        log "Oracle Linux $VERSION_ID; the update stage brings it to the current minor"
+    fi
+    [ "$(uname -m)" = x86_64 ] || die "$(uname -m); only x86_64 is supported"
+    case $(uname -r) in
+        6.12.0-*uek*) ;;
+        *) die "running kernel $(uname -r) is not UEK R8; install kernel-uek and boot it first" ;;
+    esac
+    command -v nmcli >/dev/null || die "NetworkManager (nmcli) is required"
+    log "OK: $PRETTY_NAME, kernel $(uname -r)"
+}
+
+stage_update() {
+    setenforce 0 2>/dev/null || true
+    sed -i 's/^SELINUX=.*/SELINUX=permissive/' /etc/selinux/config
+    [ "$(getenforce)" != Enforcing ] || die "SELinux is still enforcing"
+    dnf update -y
+    stage_check strict >/dev/null
+    local newest
+    newest=$(grubby --default-kernel | sed 's|^/boot/vmlinuz-||')
+    case $newest in
+        6.12.0-*uek*) ;;
+        *) die "the default boot kernel is $newest, not UEK R8; set a kernel-uek default with grubby" ;;
+    esac
+    if [ "$newest" != "$(uname -r)" ]; then
+        echo PREPARE_REBOOT=1
+    fi
+}
+
+stage_install() {
+    [ ! -e /etc/spinifex/spinifex.toml ] || die "/etc/spinifex/spinifex.toml exists; this system has been initialised, start from a clean OL install"
+    local args=()
+    [ -n "${INSTALL_SPINIFEX_CHANNEL:-}" ] && args+=(--channel "$INSTALL_SPINIFEX_CHANNEL")
+    [ -n "${INSTALL_SPINIFEX_VERSION:-}" ] && args+=(--version "$INSTALL_SPINIFEX_VERSION")
+    if [ -n "${INSTALL_SPINIFEX_TARBALL:-}" ]; then
+        # A local build: the payload carries the setup.sh of the checkout the
+        # tarball was built from, and setup.sh reads the tarball from the env.
+        [ -f "$HERE/setup.sh" ] || die "no setup.sh in the payload for INSTALL_SPINIFEX_TARBALL"
+        INSTALL_SPINIFEX_SKIP_NEWGRP=1 bash "$HERE/setup.sh" "${args[@]}"
+    else
+        curl -fsSL -o /tmp/spinifex-setup.sh https://install.mulgadc.com
+        INSTALL_SPINIFEX_SKIP_NEWGRP=1 bash /tmp/spinifex-setup.sh "${args[@]}"
+        rm -f /tmp/spinifex-setup.sh
+    fi
+    # The installer enables these on a live host. The ISO image ships them off
+    # and first boot enables them after init; a clone that started them first
+    # would run every service with no config.
+    systemctl disable --now spinifex.target spinifex-nats-watchdog.timer
+    log "Spinifex $(spx version 2>/dev/null | head -1) installed, not initialised"
+}
+
+stage_appliance() {
+    # As on the ISO: the existing spinifex system group as primary, so the
+    # login can reach the sockets setup.sh group-owns, and passwordless sudo.
+    if ! id spinifex >/dev/null 2>&1; then
+        useradd --create-home --shell /bin/bash --gid spinifex --groups wheel spinifex
+    fi
+    echo "spinifex:$DEFAULT_PASSWORD" | chpasswd
+    chage -d 0 spinifex
+    echo 'spinifex ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/spinifex-admin
+    chmod 0440 /etc/sudoers.d/spinifex-admin
+    visudo -cf /etc/sudoers.d/spinifex-admin >/dev/null
+
+    # sshd takes the first value it reads, so this sorts ahead of the
+    # 50-cloud-init.conf that templates ship with passwords off.
+    printf 'PasswordAuthentication yes\n' > /etc/ssh/sshd_config.d/10-spinifex-appliance.conf
+
+    # No DHCP on every port: first boot puts the primary NIC on br-wan and
+    # leaves the rest alone, so no second port competes for the default route.
+    printf '[main]\nno-auto-default=*\n' > /etc/NetworkManager/conf.d/90-spinifex-appliance.conf
+
+    install -D -m 0644 "$HERE/lib.sh" "$SHARE/appliance/lib.sh"
+    install -D -m 0755 "$HERE/firstboot.sh" "$SHARE/appliance/firstboot.sh"
+    install -m 0644 "$HERE/spinifex-appliance-firstboot.service" "$HERE/spinifex-banner.service" /etc/systemd/system/
+    local getty
+    for getty in getty@tty1 serial-getty@ttyS0; do
+        install -D -m 0644 "$HERE/getty-wait.conf" "/etc/systemd/system/$getty.service.d/spinifex-wait.conf"
+    done
+    systemctl daemon-reload
+    systemctl enable spinifex-appliance-firstboot.service
+
+    # Cloud-init would re-run on each unit with no datasource and rewrite the
+    # network and users first boot owns.
+    if [ -d /etc/cloud ]; then
+        touch /etc/cloud/cloud-init.disabled
+    fi
+    log "spinifex login (default password '$DEFAULT_PASSWORD', change forced) and first-boot service installed"
+}
+
+stage_seal() {
+    dnf clean all
+
+    systemctl stop "$(. "$SHARE/lib/distro.sh" && distro_ovs_unit)" ovn-northd.service ovn-controller.service 2>/dev/null || true
+    # OVS writes the chassis UUID on first start; a shared one makes every unit
+    # the same chassis in OVN.
+    rm -f /etc/openvswitch/system-id.conf /etc/openvswitch/conf.db /etc/openvswitch/.conf.db.~lock~
+    rm -f /var/lib/ovn/*.db /etc/ovn/*.db
+    rm -f /etc/spinifex/node.conf
+
+    # Profiles for the build system's NICs, its DHCP leases and NM's
+    # per-machine secret. First boot writes br-wan for the unit's own NIC.
+    rm -f /etc/NetworkManager/system-connections/*
+    rm -f /var/lib/NetworkManager/*.lease /var/lib/NetworkManager/secret_key \
+        /var/lib/NetworkManager/timestamps /var/lib/NetworkManager/seen-bssids
+    if command -v cloud-init >/dev/null; then
+        cloud-init clean --logs || true
+    fi
+
+    # sshd-keygen regenerates host keys before sshd on the next boot; systemd
+    # generates a machine-id when the file is empty.
+    rm -f /etc/ssh/ssh_host_*
+    truncate -s 0 /etc/machine-id
+    rm -f /var/lib/dbus/machine-id /var/lib/systemd/random-seed
+    echo localhost > /etc/hostname
+
+    # The build user's access goes; the spinifex login is the way in.
+    if [ -n "${BUILD_USER:-}" ] && [ "$BUILD_USER" != spinifex ] && [ "$BUILD_USER" != root ]; then
+        local home
+        home=$(getent passwd "$BUILD_USER" | cut -d: -f6)
+        rm -f "$home/.ssh/authorized_keys" "$home/.bash_history"
+        passwd -l "$BUILD_USER" >/dev/null
+        rm -f "/etc/sudoers.d/90-cloud-init-users"
+    fi
+    rm -f /root/.ssh/authorized_keys /root/.bash_history /home/spinifex/.bash_history
+
+    printf '\nSpinifex first boot is in progress. This screen updates when it finishes.\n\n' > /etc/issue
+    sed -i '/^# --- Spinifex ---$/,$d' /etc/motd 2>/dev/null || true
+
+    find /var/log -type f -exec truncate -s 0 {} + 2>/dev/null || true
+    rm -rf /var/log/journal/* /tmp/* /var/tmp/*
+    sync
+    log "sealed; powering off"
+    systemctl poweroff --no-block
+}
+
+case "${1:-}" in
+    check) stage_check ;;
+    update) stage_update ;;
+    install) stage_install ;;
+    appliance) stage_appliance ;;
+    seal) stage_seal ;;
+    *) die "usage: $0 check|update|install|appliance|seal" ;;
+esac

scripts/appliance/spinifex-appliance-firstboot.service

@@ -0,0 +1,22 @@
+[Unit]
+Description=Spinifex appliance first-boot provisioning
+After=NetworkManager.service network-online.target chronyd.service
+Wants=NetworkManager.service network-online.target
+ConditionPathExists=!/var/lib/spinifex/.appliance-firstboot-done
+
+[Service]
+Type=oneshot
+Environment=HOME=/root
+ExecStart=/usr/local/share/spinifex/appliance/firstboot.sh
+RemainAfterExit=yes
+# The banner is never enabled when first boot fails, so without this the
+# console would keep the "first boot in progress" placeholder.
+ExecStopPost=/bin/sh -c 'if [ "$EXIT_STATUS" != "0" ]; then printf "\n*** Spinifex first boot FAILED ***\nLog in as spinifex, then run:\n  sudo journalctl -u spinifex-appliance-firstboot.service\nCheck the network cable and DHCP, then reboot to retry.\n\n" > /etc/issue; fi'
+# setup-ovn.sh and spx admin init must not wedge multi-user.target and keep
+# getty from ever showing a login prompt.
+TimeoutStartSec=600s
+StandardOutput=journal+console
+StandardError=journal+console
+
+[Install]
+WantedBy=multi-user.target

scripts/appliance/spinifex-banner.service

@@ -0,0 +1,15 @@
+[Unit]
+Description=Spinifex console banner and boot health check
+After=network-online.target
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/spx admin banner --boot-check
+RemainAfterExit=yes
+# Cap it so a stuck boot-check cannot hold getty behind the console drop-in.
+TimeoutStartSec=30s
+StandardOutput=journal
+StandardError=journal
+
+[Install]
+WantedBy=multi-user.target

scripts/build-oracle-appliance.sh

@@ -0,0 +1,104 @@
+#!/bin/bash
+# build-oracle-appliance.sh — turn an Oracle Linux 10.2 system reachable over
+# SSH with sudo into a sealed Spinifex gold image: installed, not initialised,
+# with a first-boot service that sets each cloned unit up as a single node.
+# The system powers off at the end; capture its disk as the image.
+#
+# Usage: build-oracle-appliance.sh [options] <user>@<host>
+#   --channel latest|dev   install channel (default: latest)
+#   --version <tag>        install a specific release
+#   --tarball <path>       install a local release tarball instead
+#
+# Ports, keys and jump hosts come from ~/.ssh/config. <user> needs
+# passwordless sudo.
+set -euo pipefail
+
+HERE=$(cd "$(dirname "$0")" && pwd)
+PAYLOAD_SRC="$HERE/appliance"
+REMOTE_DIR=/tmp/spinifex-appliance
+
+CHANNEL="" VERSION="" TARBALL="" TARGET=""
+
+usage() { sed -n '2,13p' "$0" | sed 's/^# \{0,1\}//'; exit "${1:-0}"; }
+log() { echo "==> $*"; }
+die() { echo "ERROR: $*" >&2; exit 1; }
+
+while [ $# -gt 0 ]; do
+    case "$1" in
+        --channel) CHANNEL="${2:-}"; shift ;;
+        --version) VERSION="${2:-}"; shift ;;
+        --tarball) TARBALL="${2:-}"; shift ;;
+        -h | --help) usage ;;
+        -*) die "unknown option: $1" ;;
+        *) [ -z "$TARGET" ] || die "one target only"; TARGET=$1 ;;
+    esac
+    shift
+done
+[ -n "$TARGET" ] || usage 1
+case $TARGET in *@*) ;; *) die "target must be <user>@<host>" ;; esac
+BUILD_USER=${TARGET%@*}
+[ -z "$TARBALL" ] || [ -f "$TARBALL" ] || die "no such tarball: $TARBALL"
+
+# One multiplexed connection; host key checking is relaxed because the build
+# system is usually a fresh install whose key nobody has recorded yet.
+CTL_DIR=$(mktemp -d)
+trap 'ssh -o ControlPath="$CTL_DIR/c" -O exit "$TARGET" >/dev/null 2>&1 || true; rm -rf "$CTL_DIR"' EXIT
+SSH_OPTS=(-o ControlMaster=auto -o ControlPath="$CTL_DIR/c" -o ControlPersist=60
+    -o StrictHostKeyChecking=accept-new -o ServerAliveInterval=15 -o ConnectTimeout=10)
+
+rsh() { ssh "${SSH_OPTS[@]}" "$TARGET" "$@"; }
+
+# prepare <stage> — run one prepare.sh stage as root on the build system.
+prepare() {
+    rsh "sudo env INSTALL_SPINIFEX_CHANNEL='$CHANNEL' INSTALL_SPINIFEX_VERSION='$VERSION' \
+        INSTALL_SPINIFEX_TARBALL='${TARBALL:+$REMOTE_DIR/$(basename "$TARBALL")}' \
+        BUILD_USER='$BUILD_USER' bash $REMOTE_DIR/prepare.sh $1"
+}
+
+push_payload() {
+    rsh "rm -rf $REMOTE_DIR && mkdir -p $REMOTE_DIR"
+    tar -C "$PAYLOAD_SRC" -cf - . | rsh "tar -C $REMOTE_DIR -xf -"
+    if [ -n "$TARBALL" ]; then
+        # setup.sh from this checkout installs the tarball built from it.
+        scp "${SSH_OPTS[@]}" -q "$HERE/setup.sh" "$TARBALL" "$TARGET:$REMOTE_DIR/"
+    fi
+}
+
+wait_for_ssh() {
+    local _i
+    ssh "${SSH_OPTS[@]}" -O exit "$TARGET" >/dev/null 2>&1 || true
+    sleep 10
+    for _i in $(seq 1 60); do
+        if rsh true 2>/dev/null; then return 0; fi
+        sleep 5
+    done
+    die "$TARGET did not come back over SSH after the reboot"
+}
+
+rsh "sudo -n true" || die "$BUILD_USER needs passwordless sudo on the build system"
+
+log "Copying the payload to $TARGET"
+push_payload
+prepare check
+
+log "Updating Oracle Linux"
+prepare update | tee "$CTL_DIR/update.log"
+if grep -qx 'PREPARE_REBOOT=1' "$CTL_DIR/update.log"; then
+    log "Rebooting into the updated kernel"
+    rsh "sudo systemctl reboot" || true
+    wait_for_ssh
+    # /tmp is cleared on boot.
+    push_payload
+    prepare check
+fi
+
+log "Installing Spinifex (not initialising it)"
+prepare install
+
+log "Installing the spinifex login and the first-boot service"
+prepare appliance
+
+log "Sealing; the system powers off when done"
+prepare seal || true
+
+log "Done. Capture $TARGET's disk once it has powered off; each unit set up from it initialises itself on first boot."

scripts/images/eks-node/eks-token-webhook.initd

@@ -25,7 +25,7 @@ error_log="/dev/console"
 if [ -f /etc/spinifex-eks/first-boot.env ]; then
     # shellcheck disable=SC1091
     . /etc/spinifex-eks/first-boot.env
-    export EKS_GATEWAY_URL EKS_GATEWAY_CA EKS_ACCESS_KEY EKS_SECRET_KEY EKS_REGION EKS_ACCOUNT_ID EKS_CLUSTER_NAME
+    export EKS_GATEWAY_URL EKS_GATEWAY_CA EKS_ACCESS_KEY EKS_SECRET_KEY EKS_REGION EKS_ACCOUNT_ID EKS_CLUSTER_NAME EKS_WEBHOOK_READY_ADDR
 fi
 
 depend() {

scripts/install-node.sh

@@ -298,7 +298,6 @@ log "preflight on $N host(s)"
 
 versions=""
 families=""
-libreswan_hosts=""
 for i in $(seq 0 $((N - 1))); do
     host="${HOSTS[$i]}"
 
@@ -322,12 +321,6 @@ for i in $(seq 0 $((N - 1))); do
         fail "$host: cannot tell its OS from /etc/os-release"
     families+="${host}: ${family}"$'\n'
 
-    # libreswan's pluto: Spinifex cannot run OVN IPsec on it yet, and spx admin
-    # init refuses --ipsec=true there, which would stop formation part way.
-    if [ "$(out "$host" "test -x /usr/libexec/ipsec/pluto && echo yes")" = "yes" ]; then
-        libreswan_hosts+=" $host"
-    fi
-
     # A node that is already in a multi-node cluster is one whose keys and
     # logical network state are load-bearing. Refusing here is deliberate:
     # forming a cluster must never be the command that destroys a live one.
@@ -386,12 +379,6 @@ $(sed '/^$/d; s/^/         /' <<<"$families")
        A cluster must run one OS on every host."
 fi
 
-if $IPSEC_ENABLED && [ -n "$libreswan_hosts" ]; then
-    fail "--ipsec on, but these hosts use libreswan for IPsec:$libreswan_hosts
-       Spinifex cannot run OVN IPsec on libreswan yet. Re-run with --ipsec off
-       only if the nodes share a trusted private link."
-fi
-
 # --- Confirm ---------------------------------------------------------------
 
 echo ""

scripts/ipsec_state_helper_test.sh

@@ -0,0 +1,158 @@
+#!/bin/sh
+# ipsec_state_helper_test.sh — tests for the plaintext Geneve guard in the
+# IPsec state helper that setup.sh installs.
+#
+# The helper is cut out of setup.sh's heredoc and run against stub nft,
+# ovs-vsctl and systemctl commands, with pluto's path pointed at a fixture.
+set -eu
+
+SCRIPT_DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
+SETUP="$SCRIPT_DIR/setup.sh"
+TMP=$(mktemp -d)
+trap 'rm -rf "$TMP"' EXIT
+
+fails=0
+pass() { echo "  ok   - $1"; }
+fail() { echo "  FAIL - $1"; fails=$((fails + 1)); }
+
+echo "-- IPsec state helper: plaintext Geneve guard"
+
+HELPER="$TMP/helper"
+awk '/tee "\$\{IPSEC_STATE_HELPER\}"/ { on = 1; next } on && /^HELPER$/ { exit } on { print }' "$SETUP" |
+    sed "s#/usr/libexec/ipsec/pluto#$TMP/pluto#g" > "$HELPER"
+chmod +x "$HELPER"
+grep -q '^    guard)' "$HELPER" || { echo "could not extract the helper from setup.sh" >&2; exit 1; }
+
+mkdir -p "$TMP/bin"
+cat > "$TMP/bin/nft" <<EOF
+#!/bin/sh
+[ "\$*" = "-f -" ] || { echo "nft \$*" >> "$TMP/nft.log"; exit 0; }
+cat >> "$TMP/nft.log"
+echo "--" >> "$TMP/nft.log"
+EOF
+cat > "$TMP/bin/ovs-vsctl" <<EOF
+#!/bin/sh
+[ -e "$TMP/ovs-down" ] && { echo "ovs-vsctl: database connection failed" >&2; exit 1; }
+case "\$*" in
+    *" find Interface type=geneve") cat "$TMP/ifaces" ;;
+    *" get Interface "*)
+        for last; do :; done
+        f="$TMP/opt.\$5.\${last#options:}"
+        if [ -e "\$f" ]; then cat "\$f"; else echo; fi
+        ;;
+    *" get Open_vSwitch . external_ids:ovn-encap-ip")
+        if [ -e "$TMP/encap" ]; then cat "$TMP/encap"; else echo; fi
+        ;;
+    *) exit 1 ;;
+esac
+EOF
+printf '#!/bin/sh\nexit 0\n' > "$TMP/bin/systemctl"
+chmod +x "$TMP/bin/nft" "$TMP/bin/ovs-vsctl" "$TMP/bin/systemctl"
+
+# reset — no tunnels, OVS up, no pluto.
+reset() {
+    rm -f "$TMP"/opt.* "$TMP/ovs-down" "$TMP/nft.log" "$TMP/pluto" "$TMP/encap"
+    : > "$TMP/ifaces"
+    touch "$TMP/nft.log"
+}
+# tunnel <name> <local_ip|-> [remote_name] — a geneve interface as ovn-controller writes it.
+tunnel() {
+    echo "$1" >> "$TMP/ifaces"
+    [ "$2" = - ] || printf '"%s"\n' "$2" > "$TMP/opt.$1.local_ip"
+    [ -z "${3:-}" ] || echo "$3" > "$TMP/opt.$1.remote_name"
+}
+run() {
+    RC=0
+    OUT=$(PATH="$TMP/bin:$PATH" "$HELPER" "$1" 2>&1) || RC=$?
+    NFTLOG=$(cat "$TMP/nft.log")
+}
+LIFTED="table inet spinifex_ipsec_guard
+delete table inet spinifex_ipsec_guard
+--"
+# expect_guarded <desc> <sources> — one transaction that swaps in a table
+# dropping Geneve without an SA from exactly those sources.
+expect_guarded() {
+    if [ "$RC" = 0 ] &&
+        [ "$(grep -c '^--$' "$TMP/nft.log")" = 1 ] &&
+        [ "$(sed -n 2p "$TMP/nft.log")" = "delete table inet spinifex_ipsec_guard" ] &&
+        grep -q "^        elements = { $2 }$" "$TMP/nft.log" &&
+        grep -q '^        ip saddr @srcs udp dport 6081 rt ipsec missing drop$' "$TMP/nft.log"; then
+        pass "$1"
+    else
+        fail "$1 (rc=$RC): got [$NFTLOG] want sources [$2] $OUT"
+    fi
+}
+expect_lifted() {
+    if [ "$RC" = 0 ] && [ "$NFTLOG" = "$LIFTED" ]; then pass "$1"; else fail "$1 (rc=$RC): got [$NFTLOG] $OUT"; fi
+}
+
+reset
+tunnel ovn-node2-0 10.26.7.1 node2
+tunnel ovn-node3-0 10.26.7.1 node3
+run guard
+expect_guarded "guards the encap address once OVN wants the tunnels encrypted" "10.26.7.1/32"
+
+reset
+tunnel ovn-node2-0 10.26.7.1
+tunnel ovn-node3-0 10.26.7.1
+run guard
+expect_lifted "lifts the guard while OVN runs the tunnels in plaintext"
+
+reset
+tunnel ovn-node2-0 10.26.7.1 node2
+tunnel ovn-node3-0 10.26.8.1
+run guard
+expect_guarded "guards only the addresses of tunnels OVN encrypts" "10.26.7.1/32"
+
+reset
+tunnel ovn-node2-0 - node2
+run guard
+expect_guarded "guards a tunnel without a local_ip from every source" "0.0.0.0/0"
+
+reset
+echo '"10.26.7.1"' > "$TMP/encap"
+run guard
+expect_guarded "guards the encap address before ovn-controller has built the tunnels" "10.26.7.1/32"
+
+reset
+echo '"10.26.7.1,10.26.8.1"' > "$TMP/encap"
+run guard
+expect_guarded "guards each encap address of a multi-address host before the tunnels exist" "10.26.7.1/32,10.26.8.1/32"
+
+reset
+run guard
+expect_guarded "guards every source before the tunnels exist when no encap address is set" "0.0.0.0/0"
+
+reset
+tunnel ovn-node2-0 10.26.7.1 node2
+touch "$TMP/ovs-down"
+run guard
+if [ "$RC" != 0 ] && [ -z "$NFTLOG" ]; then
+    pass "keeps the guard and fails when OVS cannot be read"
+else
+    fail "with OVS down: rc=$RC nft calls [$NFTLOG]"
+fi
+
+reset
+touch "$TMP/pluto"
+chmod +x "$TMP/pluto"
+run off
+expect_lifted "off lifts the guard on a libreswan host"
+
+reset
+run off
+if [ "$RC" = 0 ] && [ -z "$NFTLOG" ]; then pass "off leaves nft alone without libreswan"; else fail "off without pluto: rc=$RC [$NFTLOG]"; fi
+
+reset
+run unguard
+expect_lifted "unguard lifts the guard"
+
+reset
+run bogus
+if [ "$RC" = 2 ] && [ -z "$NFTLOG" ]; then pass "refuses an unknown verb"; else fail "unknown verb: rc=$RC [$NFTLOG]"; fi
+
+if [ "$fails" -gt 0 ]; then
+    echo "$fails failure(s)"
+    exit 1
+fi
+echo "all passed"

scripts/lib/distro.sh

@@ -30,10 +30,19 @@ distro_pkg_manager() {
 }
 
 # The OVS umbrella unit: openvswitch-switch.service on Debian, openvswitch.service on EL.
+# Ubuntu's openvswitch-switch declares Alias=openvswitch.service; the symlink that
+# enabling it leaves points back at openvswitch-switch.service and is not the EL unit.
 distro_ovs_unit() {
-    local deb=false el=false
+    local deb=false el=false dir
     distro_unit_exists openvswitch-switch.service && deb=true
-    distro_unit_exists openvswitch.service && el=true
+    for dir in /etc/systemd/system /run/systemd/system /usr/lib/systemd/system /lib/systemd/system; do
+        if [ -e "${DISTRO_ROOT:-}$dir/openvswitch.service" ] || [ -L "${DISTRO_ROOT:-}$dir/openvswitch.service" ]; then
+            case $(readlink "${DISTRO_ROOT:-}$dir/openvswitch.service" || true) in
+                openvswitch-switch.service | */openvswitch-switch.service) ;;
+                *) el=true ;;
+            esac
+        fi
+    done
     if $deb && $el; then
         echo "distro.sh: OVS unit: found both openvswitch-switch.service and openvswitch.service; cannot tell which one owns OVS" >&2
         return 1

scripts/lib/distro_test.sh

@@ -26,6 +26,8 @@ fixture() {
 unit() { mkdir -p "$ROOT/usr/lib/systemd/system" && : > "$ROOT/usr/lib/systemd/system/$1"; }
 file() { mkdir -p "$ROOT$(dirname "$1")" && : > "$ROOT$1"; }
 cmd() { printf '#!/bin/sh\nexit 0\n' > "$BIN/$1" && chmod +x "$BIN/$1"; }
+# realcmd <name> — the host's real command, for probes that read the fixture with it.
+realcmd() { ln -s "$(command -v "$1")" "$BIN/$1"; }
 
 # probe <fn> [arg...] — runs one probe in bash against the current fixture,
 # with only the stub dir on PATH. Sets OUT, ERR and RC.
@@ -73,13 +75,22 @@ expect_ok "OVS unit: Debian name" openvswitch-switch.service
 fixture ovs-el; unit openvswitch.service; probe distro_ovs_unit
 expect_ok "OVS unit: EL name" openvswitch.service
 fixture ovs-masked; mkdir -p "$ROOT/etc/systemd/system"
-ln -s /dev/null "$ROOT/etc/systemd/system/openvswitch.service"; probe distro_ovs_unit
+ln -s /dev/null "$ROOT/etc/systemd/system/openvswitch.service"; realcmd readlink; probe distro_ovs_unit
 expect_ok "OVS unit: a masked unit in /etc counts" openvswitch.service
 fixture ovs-lib; mkdir -p "$ROOT/lib/systemd/system"
 : > "$ROOT/lib/systemd/system/openvswitch-switch.service"; probe distro_ovs_unit
 expect_ok "OVS unit: found under /lib on an unmerged-/usr host" openvswitch-switch.service
 fixture ovs-both; unit openvswitch-switch.service; unit openvswitch.service; probe distro_ovs_unit
 expect_fail "OVS unit: both fails naming both" openvswitch-switch.service openvswitch.service
+fixture ovs-ubuntu-alias; unit openvswitch-switch.service; mkdir -p "$ROOT/etc/systemd/system"
+ln -s /usr/lib/systemd/system/openvswitch-switch.service "$ROOT/etc/systemd/system/openvswitch.service"; realcmd readlink; probe distro_ovs_unit
+expect_ok "OVS unit: Ubuntu's openvswitch.service alias is the Debian unit" openvswitch-switch.service
+fixture ovs-relative-alias; unit openvswitch-switch.service; mkdir -p "$ROOT/etc/systemd/system"
+ln -s openvswitch-switch.service "$ROOT/etc/systemd/system/openvswitch.service"; realcmd readlink; probe distro_ovs_unit
+expect_ok "OVS unit: a relative alias symlink is the Debian unit" openvswitch-switch.service
+fixture ovs-alias-and-el; unit openvswitch-switch.service; unit openvswitch.service; mkdir -p "$ROOT/etc/systemd/system"
+ln -s /usr/lib/systemd/system/openvswitch-switch.service "$ROOT/etc/systemd/system/openvswitch.service"; realcmd readlink; probe distro_ovs_unit
+expect_fail "OVS unit: an alias beside a real EL unit still fails naming both" openvswitch-switch.service openvswitch.service
 fixture ovs-none; probe distro_ovs_unit
 expect_fail "OVS unit: neither fails naming both" openvswitch-switch.service openvswitch.service
 

scripts/node-reset.sh

@@ -71,9 +71,16 @@ distro_unit_exists() {
 }
 
 distro_ovs_unit() {
-    local deb=false el=false
+    local deb=false el=false dir
     distro_unit_exists openvswitch-switch.service && deb=true
-    distro_unit_exists openvswitch.service && el=true
+    for dir in /etc/systemd/system /run/systemd/system /usr/lib/systemd/system /lib/systemd/system; do
+        if [ -e "${DISTRO_ROOT:-}$dir/openvswitch.service" ] || [ -L "${DISTRO_ROOT:-}$dir/openvswitch.service" ]; then
+            case $(readlink "${DISTRO_ROOT:-}$dir/openvswitch.service" || true) in
+                openvswitch-switch.service | */openvswitch-switch.service) ;;
+                *) el=true ;;
+            esac
+        fi
+    done
     if $deb && $el; then
         echo "distro.sh: OVS unit: found both openvswitch-switch.service and openvswitch.service; cannot tell which one owns OVS" >&2
         return 1
@@ -266,6 +273,33 @@ if [ -n "$OVS_UNIT" ]; then
 fi
 run sudo rm -f /etc/openvswitch/system-id.conf
 
+# --- BEGIN ovs nss nicknames ---
+# ovs_nss_nicknames — the entries ovs-monitor-ipsec imported, read from
+# `certutil -L` on stdin: ovs_certkey_* is this node's certificate and key,
+# ovs_cert_* the CA and any peer certificates. Everything else is someone else's.
+ovs_nss_nicknames() {
+    awk '$1 ~ /^ovs_cert(key)?_/ {print $1}' | LC_ALL=C sort -u
+}
+# --- END ovs nss nicknames ---
+
+# A node rejoining another cluster must not keep trusting this one's CA. Only
+# the NSS entries go: other identities and /etc/ipsec.d/*.conf are left alone.
+if [ -x /usr/libexec/ipsec/pluto ] && command -v certutil >/dev/null 2>&1; then
+    log "removing Spinifex IPsec identities from libreswan's NSS database"
+    # A running ovs-monitor-ipsec imports them again from other_config.
+    run sudo systemctl stop openvswitch-ipsec.service 2>/dev/null || true
+    for nss in /var/lib/ipsec/nss /etc/ipsec.d; do
+        sudo test -e "$nss/cert9.db" || continue
+        for nick in $(sudo certutil -L -d "sql:$nss" 2>/dev/null | ovs_nss_nicknames); do
+            log "  deleting $nick from $nss"
+            delete=-D
+            case "$nick" in ovs_certkey_*) delete=-F ;; esac
+            run sudo certutil "$delete" -d "sql:$nss" -n "$nick" ||
+                log "  NOTE: could not delete $nick from $nss; remove it with certutil by hand"
+        done
+    done
+fi
+
 # Delete the OVN DBs outright — a caller re-running setup-ovn.sh gets fresh
 # empty ones. This is what clears stale chassis rows and port bindings, which
 # otherwise accumulate across resets and wedge ovn-controller in a commit loop.

scripts/node-reset_test.sh

@@ -1,7 +1,8 @@
 #!/bin/sh
 # node-reset_test.sh — tests the QEMU process match node-reset.sh uses to stop
 # guests before it deletes Viperblock state, against real processes, and the
-# instance and mounted-volume counts it and uninstall-spx.sh report before deleting.
+# instance and mounted-volume counts it and uninstall-spx.sh report before deleting,
+# and the libreswan NSS entries it deletes.
 #
 # Each fake guest is a copy of sleep named like a QEMU binary, so the kernel
 # gives it the same comm, including the 15-character truncation.
@@ -129,6 +130,32 @@ mkdir -p "$empty"
 expect "counts no instances without a state file" "$(count count_instances "$empty")" 0
 expect "counts no mounted volumes without a viperblock directory" "$(count count_mounted_volumes "$empty")" 0
 
+echo "-- node-reset.sh libreswan NSS nicknames"
+
+sed -n '/^# --- BEGIN ovs nss nicknames ---$/,/^# --- END ovs nss nicknames ---$/p' \
+    "$SCRIPT_DIR/node-reset.sh" > "$TMP/nss.sh"
+# `certutil -L` as it prints a database holding an operator's own identity and
+# CA beside what ovs-monitor-ipsec imported, one peer listed twice.
+cat > "$TMP/certutil-L" <<'EOF'
+
+Certificate Nickname                                         Trust Attributes
+                                                             SSL,S/MIME,JAR/XPI
+
+ovs_certkey_node1                                            u,u,u
+ovs_cert_cacert                                              CT,,
+ovs_cert_node2                                               P,P,P
+ovs_cert_node2                                               P,P,P
+site-vpn                                                     u,u,u
+corp-ca                                                      CT,C,C
+ovs_certificate_store                                        ,,
+EOF
+if [ ! -s "$TMP/nss.sh" ]; then
+    fail "no ovs nss nicknames block in node-reset.sh"
+else
+    got=$(bash -c '. "$1"; ovs_nss_nicknames < "$2"' _ "$TMP/nss.sh" "$TMP/certutil-L" | tr '\n' ' ')
+    expect "selects only ovs-monitor-ipsec's entries, once each" "$got" "ovs_cert_cacert ovs_cert_node2 ovs_certkey_node1 "
+fi
+
 if [ "$fails" -gt 0 ]; then
     echo "$fails failure(s)"
     exit 1

scripts/setup-ovn.sh

@@ -172,6 +172,8 @@ OVN_DBDIR="${OVN_DBDIR:-/var/lib/ovn}"
 # Seconds to wait for the database servers to exit, then for a SIGKILL to land.
 OVN_DB_STOP_WAIT="${OVN_DB_STOP_WAIT:-30}"
 OVN_DB_KILL_WAIT="${OVN_DB_KILL_WAIT:-10}"
+# Seconds to wait for a restarted clustered NB or SB member to rejoin its cluster.
+OVN_RAFT_REJOIN_WAIT="${OVN_RAFT_REJOIN_WAIT:-60}"
 # Records that --management set this node up. Nodes set up before it existed
 # carry only the legacy marker, OVN_CTL_OPTS in /etc/default/ovn-central.
 OVN_ROLE_FILE=/etc/spinifex/ovn-role
@@ -757,6 +759,42 @@ stop_ovn_databases() {
     exit 1
 }
 
+# db_clients — sets NBCTL/SBCTL. A clustered DB has no unix socket to fall back
+# on, and a follower refuses to answer at all unless leader-only is turned off;
+# writes through a follower are forwarded to the leader.
+db_clients() {
+    NBCTL=(ovn-nbctl)
+    SBCTL=(ovn-sbctl)
+    if [ -n "$DB_CLUSTER_LOCAL_ADDR" ]; then
+        NBCTL=(ovn-nbctl --db="tcp:$DB_CLUSTER_LOCAL_ADDR:6641" --no-leader-only)
+        SBCTL=(ovn-sbctl --db="tcp:$DB_CLUSTER_LOCAL_ADDR:6642" --no-leader-only)
+    fi
+}
+
+# set_client_connections — the loopback client listeners. A rerun's restart can
+# hand the leadership away, so this goes through NBCTL/SBCTL, not the socket.
+set_client_connections() {
+    sudo "${NBCTL[@]}" set-connection ptcp:6641:127.0.0.1
+    sudo "${SBCTL[@]}" set-connection ptcp:6642:127.0.0.1
+}
+
+# wait_for_raft_member <ctl> <schema> — waits until this server is a cluster
+# member that knows its leader, which a restarted member is once it rejoins.
+# Returns 1 on timeout.
+wait_for_raft_member() {
+    local ctl=$1 schema=$2 status i
+    for i in $(seq 1 "$OVN_RAFT_REJOIN_WAIT"); do
+        status=$(sudo ovs-appctl -t "$ctl" cluster/status "$schema" 2>/dev/null || true)
+        if grep -qx 'Status: cluster member' <<<"$status" &&
+            grep -q '^Leader: ' <<<"$status" && ! grep -qx 'Leader: unknown' <<<"$status"; then
+            return 0
+        fi
+        echo "  Waiting for $schema to rejoin its RAFT cluster... ($i/$OVN_RAFT_REJOIN_WAIT)"
+        sleep 1
+    done
+    return 1
+}
+
 # --- Step 2: Enable services ---
 # ovn-ctl consults the RAFT flags only when it creates a database, so any
 # existing DB file is served as it stands and the cluster flags are ignored.
@@ -996,16 +1034,9 @@ EOF
         fi
     fi
 
-    # A clustered DB has no unix socket to fall back on, and a follower refuses
-    # to answer at all unless leader-only is turned off. Without both of these
-    # the waits below can never succeed on a clustered node: they burn their
-    # full timeout, gate nothing, and the run still continues.
-    NBCTL=(ovn-nbctl)
-    SBCTL=(ovn-sbctl)
-    if [ -n "$DB_CLUSTER_LOCAL_ADDR" ]; then
-        NBCTL=(ovn-nbctl --db="tcp:$DB_CLUSTER_LOCAL_ADDR:6641" --no-leader-only)
-        SBCTL=(ovn-sbctl --db="tcp:$DB_CLUSTER_LOCAL_ADDR:6642" --no-leader-only)
-    fi
+    # Without these the waits below can never succeed on a clustered node:
+    # they burn their full timeout, gate nothing, and the run still continues.
+    db_clients
 
     # Wait for OVN NB DB socket to become available
     for i in $(seq 1 15); do
@@ -1067,29 +1098,34 @@ EOF
     # with no join step to absorb the window — attaching before the leader is
     # elected (and before northd writes the gateway logical flows) leaves it with
     # a partial datapath that never reconverges. Gate on SB reachable and, when
-    # clustered, an elected leader.
+    # clustered, both members back in their clusters under an elected leader.
     for i in $(seq 1 30); do
-        if sudo "${SBCTL[@]}" --timeout=2 show >/dev/null 2>&1; then
-            if [ -z "$DB_CLUSTER_LOCAL_ADDR" ]; then
-                break
-            fi
-            SB_LEADER=$(sudo ovs-appctl -t /var/run/ovn/ovnsb_db.ctl \
-                cluster/status OVN_Southbound 2>/dev/null \
-                | awk '/^Leader:/{print $2; exit}')
-            if [ -n "$SB_LEADER" ] && [ "$SB_LEADER" != "unknown" ]; then
-                break
-            fi
-        fi
-        echo "  Waiting for OVN SB DB leader... ($i/30)"
+        sudo "${SBCTL[@]}" --timeout=2 show >/dev/null 2>&1 && break
+        echo "  Waiting for OVN SB DB... ($i/30)"
         sleep 1
     done
 
+    # The restart above took both of this node's members down at once. Returning
+    # before they rejoin lets a rerun on the next database node take a second
+    # member of each cluster down, and one of three left is below quorum.
+    if [ -n "$DB_CLUSTER_LOCAL_ADDR" ]; then
+        for member in "/var/run/ovn/ovnnb_db.ctl OVN_Northbound" "/var/run/ovn/ovnsb_db.ctl OVN_Southbound"; do
+            # shellcheck disable=SC2086 # two words: the ctl socket and the schema
+            if ! wait_for_raft_member $member; then
+                echo "ERROR: this node's ${member#* } server is not back in its RAFT cluster after ${OVN_RAFT_REJOIN_WAIT}s." >&2
+                echo "       Restarting OVN central on another database node now would leave the cluster" >&2
+                echo "       without quorum. Check: sudo ovs-appctl -t ${member% *} cluster/status ${member#* }" >&2
+                exit 1
+            fi
+        done
+        echo "  RAFT: NB and SB members rejoined, leader elected"
+    fi
+
     # set-connection replicates through RAFT, so it only runs on the create
     # node. Scoped to loopback, not a wildcard ptcp:6641, because 127.0.0.1
     # is valid on every node; the differing LAN address is set above instead.
     if [ -z "$DB_CLUSTER_REMOTE_ADDR" ]; then
-        sudo ovn-nbctl set-connection ptcp:6641:127.0.0.1
-        sudo ovn-sbctl set-connection ptcp:6642:127.0.0.1
+        set_client_connections
         if [ -n "$LAN_ADDR" ]; then
             echo "  OVN NB DB listening on tcp:127.0.0.1:6641, tcp:$LAN_ADDR:6641"
             echo "  OVN SB DB listening on tcp:127.0.0.1:6642, tcp:$LAN_ADDR:6642"
@@ -1958,16 +1994,21 @@ echo "Step 10: Enabling OVN auto-start on boot..."
 sudo systemctl enable "$OVS_UNIT" 2>/dev/null || true
 sudo systemctl enable ovn-controller 2>/dev/null || true
 printf '  %-21s enabled on boot\n' "${OVS_UNIT%.service}:" ovn-controller:
-if distro_has_libreswan; then
-    # OVN IPsec does not run on libreswan yet, so it stays off here. Starting
-    # openvswitch-ipsec would also start pluto, which listens on UDP 500/4500.
+if distro_has_libreswan && [ "$(systemctl is-active openvswitch-ipsec.service 2>/dev/null)" = active ]; then
+    # The daemon turned IPsec on; stopping it here would drop live tunnels.
+    printf '  %-21s %s\n' openvswitch-ipsec: "active: IPsec is on, left to the daemon" ipsec: "active: IPsec is on, left to the daemon"
+elif distro_has_libreswan; then
+    # Off until the daemon asks: pluto listens on UDP 500/4500 once started.
+    # Enabled under the mask, so the daemon's unmask also brings both back on boot.
     IPSEC_UNITS=(ipsec.service)
     distro_unit_exists openvswitch-ipsec.service && IPSEC_UNITS+=(openvswitch-ipsec.service)
-    if ! sudo systemctl mask --now "${IPSEC_UNITS[@]}" >/dev/null; then
-        echo "ERROR: could not stop and mask ${IPSEC_UNITS[*]}; IPsec must stay off on libreswan" >&2
+    if ! sudo systemctl unmask "${IPSEC_UNITS[@]}" >/dev/null 2>&1 ||
+        ! sudo systemctl enable "${IPSEC_UNITS[@]}" >/dev/null 2>&1 ||
+        ! sudo systemctl mask --now "${IPSEC_UNITS[@]}" >/dev/null; then
+        echo "ERROR: could not enable, stop and mask ${IPSEC_UNITS[*]}; IPsec must stay off until the daemon turns it on" >&2
         exit 1
     fi
-    printf '  %-21s %s\n' openvswitch-ipsec: "masked: IPsec stays off on libreswan" ipsec: "masked: IPsec stays off on libreswan"
+    printf '  %-21s %s\n' openvswitch-ipsec: "masked: IPsec is off, left to the daemon" ipsec: "masked: IPsec is off, left to the daemon"
 elif [ "$(systemctl is-enabled openvswitch-ipsec.service 2>/dev/null)" = masked ]; then
     # The daemon masks it while IPsec is off and unmasks it to turn IPsec on.
     printf '  %-21s %s\n' openvswitch-ipsec: "masked: IPsec is off, left to the daemon"

scripts/setup-ovn_test.sh

@@ -454,6 +454,47 @@ case "$RC:$OUT" in
     *) fail "no recreate-db: standalone DB rc=$RC out=$OUT" ;;
 esac
 
+# --------------------------------------------------------------------------
+# A rerun returns only once both of this node's RAFT members have rejoined.
+# --------------------------------------------------------------------------
+# A rerun restarts the node's NB and SB members together, so returning early
+# lets the next database node's rerun take a second member of each down.
+awk '$0 == "wait_for_raft_member() {" { on = 1 } on { print } on && $0 == "}" { exit }' "$SETUP" >"$TMP/raft_fn.sh"
+mkdir -p "$TMP/raftbin"
+printf '#!/bin/sh\nexit 0\n' >"$TMP/raftbin/sleep"
+printf '#!/bin/sh\n[ -f "%s" ] || exit 1\ncat "%s"\n' "$TMP/cluster-status" "$TMP/cluster-status" >"$TMP/raftbin/ovs-appctl"
+chmod +x "$TMP/raftbin/sleep" "$TMP/raftbin/ovs-appctl"
+
+# raft_member <Status:> <Leader:> — runs wait_for_raft_member against a
+# cluster/status in that state, or none when Status is "absent"; sets RC.
+raft_member() {
+    rm -f "$TMP/cluster-status"
+    if [ "$1" != absent ]; then
+        printf 'a1b2\nName: OVN_Northbound\nStatus: %s\nRole: follower\nTerm: 4\nLeader: %s\nVote: c3d4\n\nServers:\n    a1b2 (a1b2 at tcp:10.0.0.1:6643) (self)\n    c3d4 (c3d4 at tcp:10.0.0.2:6643)\n' \
+            "$1" "$2" >"$TMP/cluster-status"
+    fi
+    RC=0
+    PATH="$TMP/raftbin:$PATH" bash -c 'sudo() { "$@"; }; OVN_RAFT_REJOIN_WAIT=2; . "$1"
+        wait_for_raft_member /var/run/ovn/ovnnb_db.ctl OVN_Northbound' _ "$TMP/raft_fn.sh" >/dev/null 2>&1 || RC=$?
+}
+
+if [ ! -s "$TMP/raft_fn.sh" ]; then
+    fail "raft: no wait_for_raft_member in setup-ovn.sh"
+else
+    raft_member "cluster member" c3d4
+    if [ "$RC" = 0 ]; then pass "raft: a member following a known leader has rejoined"; else fail "raft: follower with leader rc=$RC"; fi
+    raft_member "cluster member" self
+    if [ "$RC" = 0 ]; then pass "raft: a member that is the leader has rejoined"; else fail "raft: leader rc=$RC"; fi
+    raft_member "cluster member" unknown
+    if [ "$RC" != 0 ]; then pass "raft: a member that has not found its leader has not rejoined"; else fail "raft: unknown leader accepted"; fi
+    raft_member "disconnected from the cluster (election timeout)" c3d4
+    if [ "$RC" != 0 ]; then pass "raft: a disconnected member that remembers a leader has not rejoined"; else fail "raft: disconnected member accepted"; fi
+    raft_member "joining cluster" c3d4
+    if [ "$RC" != 0 ]; then pass "raft: a member still joining has not rejoined"; else fail "raft: joining member accepted"; fi
+    raft_member absent ""
+    if [ "$RC" != 0 ]; then pass "raft: a server that does not answer has not rejoined"; else fail "raft: unanswered cluster/status accepted"; fi
+fi
+
 rm -rf "$TMP/dbdir"
 mkdir -p "$TMP/dbdir"
 ensure_db true
@@ -478,6 +519,33 @@ else
 fi
 printf '#!/bin/sh\nexit 1\n' >"$TMP/bin/pgrep"
 
+# A rerun restarts this node's members, which can hand the leadership to a
+# peer, so the create node's set-connection must still land as a follower.
+for fn in db_clients set_client_connections; do
+    awk -v f="$fn() {" '$0 == f { on = 1 } on { print } on && $0 == "}" { exit }' "$SETUP" >>"$TMP/conn_fn.sh"
+done
+mkdir -p "$TMP/followerbin"
+for ctl in ovn-nbctl ovn-sbctl; do
+    cat >"$TMP/followerbin/$ctl" <<EOF
+#!/bin/sh
+case " \$* " in
+    *" --no-leader-only "*) echo "$ctl \$*" >>"$TMP/conn.log" ;;
+    *) echo "$ctl: unix:/var/run/ovn/db.sock: database connection failed ()" >&2; exit 1 ;;
+esac
+EOF
+    chmod +x "$TMP/followerbin/$ctl"
+done
+: >"$TMP/conn.log"
+RC=0
+PATH="$TMP/followerbin:$PATH" bash -c 'set -e; sudo() { "$@"; }; DB_CLUSTER_LOCAL_ADDR=10.0.0.1; . "$1"
+    db_clients; set_client_connections' _ "$TMP/conn_fn.sh" >/dev/null 2>&1 || RC=$?
+if [ "$RC" = 0 ] && [ "$(cat "$TMP/conn.log")" = "ovn-nbctl --db=tcp:10.0.0.1:6641 --no-leader-only set-connection ptcp:6641:127.0.0.1
+ovn-sbctl --db=tcp:10.0.0.1:6642 --no-leader-only set-connection ptcp:6642:127.0.0.1" ]; then
+    pass "connections: the create node sets the client listeners while a follower"
+else
+    fail "connections: as a follower rc=$RC calls [$(cat "$TMP/conn.log")]"
+fi
+
 if [ "$fails" -ne 0 ]; then
     echo "  setup-ovn.sh: $fails failure(s)"
     exit 1

scripts/setup.sh

@@ -390,10 +390,68 @@ install_ipsec_state_helper() {
 set -eu
 
 if [ "$#" -ne 1 ]; then
-    echo "usage: $0 <on|off>" >&2
+    echo "usage: $0 <on|off|guard|unguard>" >&2
     exit 2
 fi
 
+# libreswan drops a tunnel's xfrm policies while IKE fails, and pluto flushes
+# the whole xfrm table when it starts or stops, so Geneve from the encap address
+# is dropped in nft unless the route carries an SA.
+GUARD_TABLE="inet spinifex_ipsec_guard"
+
+unguard() {
+    printf 'table %s\ndelete table %s\n' "$GUARD_TABLE" "$GUARD_TABLE" | nft -f -
+}
+
+want=""
+want_src() {
+    case "$1" in
+        *.*.*.*) case "$1" in *[!0-9.]*) src=0.0.0.0/0 ;; *) src="$1/32" ;; esac ;;
+        *) src=0.0.0.0/0 ;;
+    esac
+    case " $want " in *" $src "*) ;; *) want="$want $src" ;; esac
+}
+
+# One source per local address of a tunnel with a remote_name, which
+# ovn-controller sets only while NB_Global.ipsec is on. A tunnel without a
+# usable local_ip is guarded from every source rather than left open.
+guard() {
+    ifaces=$(ovs-vsctl --timeout=5 --bare --columns=name find Interface type=geneve)
+    for iface in $ifaces; do
+        peer=$(ovs-vsctl --timeout=5 --if-exists get Interface "$iface" options:remote_name)
+        [ -n "$peer" ] || continue
+        want_src "$(ovs-vsctl --timeout=5 --if-exists get Interface "$iface" options:local_ip | tr -d '"')"
+    done
+    # At boot pluto is up before ovn-controller builds the tunnels, and its
+    # callers guard only while encryption is wanted, so guard the encap address
+    # until the tunnels say otherwise.
+    if [ -z "$ifaces" ]; then
+        encap=$(ovs-vsctl --timeout=5 --if-exists get Open_vSwitch . external_ids:ovn-encap-ip | tr -d '"')
+        for ip in $(echo "$encap" | tr ',' ' '); do want_src "$ip"; done
+        [ -n "$want" ] || want_src ""
+    fi
+    if [ -z "$want" ]; then
+        unguard
+        return
+    fi
+    # One transaction, so the old set is never gone before the new one is in.
+    nft -f - << NFT
+table $GUARD_TABLE
+delete table $GUARD_TABLE
+table $GUARD_TABLE {
+    set srcs {
+        type ipv4_addr
+        flags interval
+        elements = { $(echo $want | tr ' ' ',') }
+    }
+    chain out {
+        type filter hook output priority 0; policy accept;
+        ip saddr @srcs udp dport 6081 rt ipsec missing drop
+    }
+}
+NFT
+}
+
 # mask/unmask rather than disable/enable. The daemon's unit sets
 # ProtectSystem=full, so /etc is read-only in the namespace its children inherit,
 # and disabling a unit that also ships a SysV script shells out to update-rc.d
@@ -401,12 +459,24 @@ fi
 
 # ovs-monitor-ipsec execs the strongSwan starter itself rather than going through
 # this unit, so leaving it enabled only contends for UDP 500/4500. Off in both
-# states. Absent on a host without strongswan, which is not an error.
-systemctl mask --now strongswan-starter.service >/dev/null 2>&1 || true
+# states. Masking a unit no package ships leaves a stray /dev/null link, so a
+# host without strongSwan gets none, and loses one an older helper left.
+if [ -e /usr/lib/systemd/system/strongswan-starter.service ] || [ -e /lib/systemd/system/strongswan-starter.service ]; then
+    systemctl mask --now strongswan-starter.service >/dev/null 2>&1 || true
+elif [ "$(systemctl is-enabled strongswan-starter.service 2>/dev/null)" = masked ]; then
+    systemctl unmask strongswan-starter.service >/dev/null 2>&1 || true
+fi
 
 case "$1" in
     on)
-        systemctl unmask openvswitch-ipsec.service
+        if [ -x /usr/libexec/ipsec/pluto ]; then
+            # ovs-monitor-ipsec drives pluto over its control socket but does
+            # not start it, so pluto comes up first under its own unit.
+            systemctl unmask ipsec.service openvswitch-ipsec.service
+            systemctl start ipsec.service
+        else
+            systemctl unmask openvswitch-ipsec.service
+        fi
         exec systemctl start openvswitch-ipsec.service
         ;;
     off)
@@ -415,8 +485,15 @@ case "$1" in
         # On Debian that name is strongswan-starter's alias, masked above.
         if [ -x /usr/libexec/ipsec/pluto ]; then
             systemctl mask --now ipsec.service
+            unguard
         fi
         ;;
+    guard)
+        guard
+        ;;
+    unguard)
+        unguard
+        ;;
     *)
         echo "state not permitted: $1" >&2
         exit 2
@@ -1769,6 +1846,18 @@ CPUWeight=300
 EOF
     info "  /etc/systemd/system/system.slice.d/spinifex-reserve.conf"
 
+    # The plaintext Geneve guard is not persisted, so it goes in when pluto
+    # starts at boot, before ovn-controller builds the tunnels. `-`: the guard
+    # must never fail the unit, and the daemon re-asserts it each pass.
+    if [ -x /usr/libexec/ipsec/pluto ]; then
+        $SUDO mkdir -p /etc/systemd/system/ipsec.service.d
+        $SUDO tee /etc/systemd/system/ipsec.service.d/spinifex-guard.conf > /dev/null << EOF
+[Service]
+ExecStartPost=-${IPSEC_STATE_HELPER} guard
+EOF
+        info "  /etc/systemd/system/ipsec.service.d/spinifex-guard.conf"
+    fi
+
     # daemon-reload / enable require a running systemd — skip inside the ISO
     # chroot. Unit files are still dropped into place; firstboot enables them.
     if [ "${ISO_BUILD:-0}" = "1" ]; then

scripts/sync-aws-models.sh

@@ -27,6 +27,7 @@ MODELS=(
     elastic-load-balancing-v2/2015-12-01
     iam/2010-05-08
     rds/2014-10-31
+    route-53/2013-04-01
     s3/2006-03-01
     sts/2011-06-15
 )
@@ -34,7 +35,7 @@ MODELS=(
 TMP="$(mktemp -d)"
 trap 'rm -rf "$TMP"' EXIT
 
-# A blobless shallow fetch pulls only the ten files read below.
+# A blobless shallow fetch pulls only the files read below.
 git -C "$TMP" init --quiet repo
 git -C "$TMP/repo" fetch --quiet --depth=1 --filter=blob:none "$REPO" "$COMMIT"
 COMMITTED="$(git -C "$TMP/repo" log -1 --format=%cs "$COMMIT")"

scripts/ui-playwright.sh

@@ -1,68 +0,0 @@
-#!/usr/bin/env bash
-# Runs the console's Playwright suite in the pinned image CI uses, so screenshots
-# rasterise identically here and on the runner. `compare` runs it as CI does;
-# `baselines` redraws e2e/__screenshots__/. The host's node_modules is not used.
-set -euo pipefail
-
-repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
-ui_dir="${repo_root}/spinifex/runtime/roles/spinifexui/frontend"
-cache="${UI_PLAYWRIGHT_CACHE:-${XDG_CACHE_HOME:-${HOME}/.cache}/spinifex-ui-playwright}"
-
-die() { echo "ui-playwright: $*" >&2; exit 1; }
-
-mode="${1:-}"
-case "${mode}" in
-    compare)   e2e_args=() ;;
-    baselines) e2e_args=(--update-snapshots=all) ;;
-    *) die "usage: $0 compare|baselines [playwright args...]" ;;
-esac
-shift
-e2e_args+=("$@")
-
-image="$(tr -d '[:space:]' < "${ui_dir}/playwright.image")"
-[ -n "${image}" ] || die "${ui_dir}/playwright.image is empty"
-
-runtime="${CONTAINER_RUNTIME:-}"
-if [ -z "${runtime}" ]; then
-    runtime="$(command -v podman || command -v docker || true)"
-fi
-[ -n "${runtime}" ] || die "neither podman nor docker is installed"
-
-# Files the suite writes (dist, screenshots, test-results) must stay owned by
-# the invoking user, which rootless podman and docker reach differently.
-case "$(basename "${runtime}")" in
-    podman) user_args=(--userns=keep-id) ;;
-    *)      user_args=(--user "$(id -u):$(id -g)") ;;
-esac
-
-mkdir -p "${cache}/node_modules" "${cache}/pnpm-store" "${cache}/corepack"
-
-# Runs in the container: check the image against .nvmrc and the lockfile's
-# Playwright, then install, build dist for the CSP test, and run the suite.
-# shellcheck disable=SC2016
-inner='
-set -eu
-want_node="v$(cat .nvmrc)"
-[ "$(node --version)" = "${want_node}" ] ||
-    { echo "ui-playwright: image has node $(node --version), .nvmrc wants ${want_node}" >&2; exit 1; }
-corepack pnpm install --frozen-lockfile --store-dir /cache/pnpm-store
-pw="$(corepack pnpm exec playwright --version | awk "{print \$2}")"
-case "${UI_PLAYWRIGHT_IMAGE}" in
-    *":v${pw}-"*) ;;
-    *) echo "ui-playwright: lockfile has Playwright ${pw}, image is ${UI_PLAYWRIGHT_IMAGE}" >&2; exit 1 ;;
-esac
-corepack pnpm build
-exec corepack pnpm e2e "$@"
-'
-
-exec "${runtime}" run --rm --init --ipc=host "${user_args[@]}" \
-    -e CI=1 \
-    -e HOME=/tmp/home \
-    -e COREPACK_HOME=/cache/corepack \
-    -e COREPACK_ENABLE_DOWNLOAD_PROMPT=0 \
-    -e UI_PLAYWRIGHT_IMAGE="${image}" \
-    -v "${ui_dir}:/work" \
-    -v "${cache}/node_modules:/work/node_modules" \
-    -v "${cache}:/cache" \
-    -w /work \
-    "${image}" bash -c "${inner}" ui-playwright "${e2e_args[@]}"

scripts/uninstall-spx.sh

@@ -362,9 +362,10 @@ rm_path /etc/systemd/system/openvswitch-switch.service.d/spinifex-perms.conf \
         /etc/systemd/system/ovn-controller.service.d/log-level.conf \
         /etc/systemd/system/openvswitch-ipsec.service.d/spinifex-perms.conf \
         /etc/systemd/system/ovn-northd.service.d/spinifex-perms.conf \
-        /etc/systemd/system/ovn-northd.service.d/no-manage-ovsdb.conf
+        /etc/systemd/system/ovn-northd.service.d/no-manage-ovsdb.conf \
+        /etc/systemd/system/ipsec.service.d/spinifex-guard.conf
 # Only if we emptied them — another package may own the same override dir.
-for d in openvswitch-switch ovs-vswitchd ovn-controller openvswitch-ipsec ovn-northd; do
+for d in openvswitch-switch ovs-vswitchd ovn-controller openvswitch-ipsec ovn-northd ipsec; do
     dir="/etc/systemd/system/${d}.service.d"
     if sudo test -d "$dir" && [ -z "$(sudo ls -A "$dir" 2>/dev/null)" ]; then
         rm_path "$dir"
@@ -431,6 +432,11 @@ if [ -z "$PLUGINDIR" ]; then
 fi
 [ -n "$PLUGINDIR" ] && rm_path "$PLUGINDIR/nbdkit-viperblock-plugin.so"
 
+# A leftover guard table drops all Geneve from this host's encap address until
+# reboot, and the helper is the one place that names it.
+if [ -x /usr/local/lib/spinifex/spinifex-set-ipsec-state ]; then
+    run sudo /usr/local/lib/spinifex/spinifex-set-ipsec-state unguard || true
+fi
 rm_path /usr/local/bin/spx \
         /usr/local/share/spinifex \
         /usr/local/lib/spinifex \
@@ -619,6 +625,7 @@ if ! $DRY_RUN; then
              /etc/NetworkManager/conf.d/*spinifex* \
              /etc/systemd/system/*.service.d/spinifex-perms.conf \
              /etc/systemd/system/ovn-northd.service.d/no-manage-ovsdb.conf \
+             /etc/systemd/system/ipsec.service.d/spinifex-guard.conf \
              /etc/systemd/system/ovn-controller.service.d/log-level.conf; do
         exists "$p" && LEFT+=("$p")
     done

spinifex/agents/eks/tokenwebhook/readiness.go

@@ -0,0 +1,128 @@
+package tokenwebhook
+
+import (
+	"context"
+	"crypto/tls"
+	"crypto/x509"
+	"fmt"
+	"net"
+	"net/http"
+	"net/url"
+	"os"
+	"sync"
+	"time"
+
+	"github.com/mulgadc/bluebottle/pkg/tlsconfig"
+)
+
+const (
+	readinessTimeout = 2 * time.Second
+	readinessTTL     = 2 * time.Second
+)
+
+// defaultAPIServerAddr is the local k3s apiserver the readiness check dials.
+const defaultAPIServerAddr = "127.0.0.1:6443"
+
+// gatewayProbe reports whether this control plane can serve: the gateway it
+// relays to accepts a TLS handshake and the local apiserver accepts a TCP
+// connection. It sends no credentials and caches the result so frequent load
+// balancer probes never reach either more than once per ttl.
+type gatewayProbe struct {
+	addr    string
+	apiAddr string
+	tlsCfg  *tls.Config
+	timeout time.Duration
+	ttl     time.Duration
+	now     func() time.Time
+
+	mu      sync.Mutex
+	checked time.Time
+	ok      bool
+}
+
+func newGatewayProbe(gatewayURL, caPath, apiAddr string) (*gatewayProbe, error) {
+	u, err := url.Parse(gatewayURL)
+	if err != nil {
+		return nil, fmt.Errorf("parse gateway URL %q: %w", gatewayURL, err)
+	}
+	host, port := u.Hostname(), u.Port()
+	if host == "" {
+		return nil, fmt.Errorf("gateway URL %q has no host", gatewayURL)
+	}
+	if port == "" {
+		port = "443"
+	}
+	cfg := &tls.Config{MinVersion: tls.VersionTLS13, CurvePreferences: tlsconfig.Curves}
+	if caPath != "" {
+		pem, err := os.ReadFile(caPath)
+		if err != nil {
+			return nil, fmt.Errorf("read gateway CA %q: %w", caPath, err)
+		}
+		pool := x509.NewCertPool()
+		if !pool.AppendCertsFromPEM(pem) {
+			return nil, fmt.Errorf("gateway CA %q has no usable certificates", caPath)
+		}
+		cfg.RootCAs = pool
+	}
+	if apiAddr == "" {
+		apiAddr = defaultAPIServerAddr
+	}
+	return &gatewayProbe{
+		addr:    net.JoinHostPort(host, port),
+		apiAddr: apiAddr,
+		tlsCfg:  cfg,
+		timeout: readinessTimeout,
+		ttl:     readinessTTL,
+		now:     time.Now,
+	}, nil
+}
+
+// ready returns the cached result while it is younger than ttl, else probes.
+// Concurrent callers wait for one probe and share its result.
+func (p *gatewayProbe) ready() bool {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	if !p.checked.IsZero() && p.now().Sub(p.checked) < p.ttl {
+		return p.ok
+	}
+	p.ok = p.probe()
+	p.checked = p.now()
+	return p.ok
+}
+
+func (p *gatewayProbe) probe() bool {
+	ctx, cancel := context.WithTimeout(context.Background(), p.timeout)
+	defer cancel()
+	d := tls.Dialer{NetDialer: &net.Dialer{}, Config: p.tlsCfg}
+	conn, err := d.DialContext(ctx, "tcp", p.addr)
+	if err != nil {
+		return false
+	}
+	_ = conn.Close()
+
+	// A control plane whose apiserver is down must leave rotation too.
+	api, err := (&net.Dialer{}).DialContext(ctx, "tcp", p.apiAddr)
+	if err != nil {
+		return false
+	}
+	_ = api.Close()
+	return true
+}
+
+// handle serves 200 when the gateway and apiserver are reachable, else 503.
+func (p *gatewayProbe) handle(w http.ResponseWriter, _ *http.Request) {
+	if p.ready() {
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte("ok"))
+		return
+	}
+	http.Error(w, "not ready", http.StatusServiceUnavailable)
+}
+
+// readinessHandler serves only /readyz; the listener it backs is plain HTTP and
+// reachable beyond loopback, so it must never expose /authenticate.
+func readinessHandler(p *gatewayProbe) http.Handler {
+	mux := http.NewServeMux()
+	mux.HandleFunc("/readyz", p.handle)
+	return mux
+}

spinifex/agents/eks/tokenwebhook/readiness_test.go

@@ -0,0 +1,251 @@
+package tokenwebhook
+
+import (
+	"crypto/tls"
+	"net"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// tlsGateway is a loopback TLS listener standing in for the gateway. It counts
+// accepted connections and can be stopped and restarted on the same address.
+type tlsGateway struct {
+	t       *testing.T
+	api     *fakeAPIServer
+	addr    string
+	cert    tls.Certificate
+	caPath  string
+	ln      net.Listener
+	accepts atomic.Int32
+}
+
+func newTLSGateway(t *testing.T) *tlsGateway {
+	t.Helper()
+	dir := t.TempDir()
+	cert, certPEM, err := ensureServingCert(filepath.Join(dir, "c.pem"), filepath.Join(dir, "k.pem"))
+	require.NoError(t, err)
+	caPath := filepath.Join(dir, "ca.pem")
+	require.NoError(t, os.WriteFile(caPath, certPEM, 0o600))
+
+	g := &tlsGateway{t: t, cert: cert, caPath: caPath, api: newFakeAPIServer(t)}
+	g.start("127.0.0.1:0")
+	t.Cleanup(g.stop)
+	return g
+}
+
+func (g *tlsGateway) start(addr string) {
+	g.t.Helper()
+	ln, err := tls.Listen("tcp", addr, serverTLSConfig(g.cert))
+	require.NoError(g.t, err)
+	g.ln = ln
+	g.addr = ln.Addr().String()
+	go func() {
+		for {
+			c, err := ln.Accept()
+			if err != nil {
+				return
+			}
+			g.accepts.Add(1)
+			go func() {
+				if tc, ok := c.(*tls.Conn); ok {
+					_ = tc.Handshake()
+				}
+				_ = c.Close()
+			}()
+		}
+	}()
+}
+
+func (g *tlsGateway) stop() {
+	if g.ln != nil {
+		_ = g.ln.Close()
+		g.ln = nil
+	}
+}
+
+// fakeAPIServer is a plain TCP listener standing in for the local apiserver.
+type fakeAPIServer struct {
+	addr string
+	ln   net.Listener
+}
+
+func newFakeAPIServer(t *testing.T) *fakeAPIServer {
+	t.Helper()
+	a := &fakeAPIServer{}
+	a.start("127.0.0.1:0", t)
+	t.Cleanup(a.stop)
+	return a
+}
+
+func (a *fakeAPIServer) start(addr string, t *testing.T) {
+	t.Helper()
+	ln, err := net.Listen("tcp", addr)
+	require.NoError(t, err)
+	a.ln = ln
+	a.addr = ln.Addr().String()
+	go func() {
+		for {
+			c, err := ln.Accept()
+			if err != nil {
+				return
+			}
+			_ = c.Close()
+		}
+	}()
+}
+
+func (a *fakeAPIServer) stop() {
+	if a.ln != nil {
+		_ = a.ln.Close()
+		a.ln = nil
+	}
+}
+
+func (g *tlsGateway) probe(t *testing.T) *gatewayProbe {
+	t.Helper()
+	p, err := newGatewayProbe("https://"+g.addr, g.caPath, g.api.addr)
+	require.NoError(t, err)
+	return p
+}
+
+func TestGatewayProbe_UpAndRefused(t *testing.T) {
+	g := newTLSGateway(t)
+	p := g.probe(t)
+	p.ttl = 0
+
+	assert.True(t, p.ready())
+
+	g.stop()
+	assert.False(t, p.ready(), "a closed gateway port is not ready")
+}
+
+func TestGatewayProbe_RecoversAfterDown(t *testing.T) {
+	g := newTLSGateway(t)
+	p := g.probe(t)
+	p.ttl = 0
+	addr := g.addr
+
+	g.stop()
+	require.False(t, p.ready())
+
+	g.start(addr)
+	assert.True(t, p.ready(), "ready again once the gateway is back")
+}
+
+func TestGatewayProbe_TimesOutOnStalledHandshake(t *testing.T) {
+	g := newTLSGateway(t)
+	// A plain listener that accepts and never answers the TLS handshake.
+	ln, err := net.Listen("tcp", "127.0.0.1:0")
+	require.NoError(t, err)
+	t.Cleanup(func() { _ = ln.Close() })
+	go func() {
+		for {
+			c, err := ln.Accept()
+			if err != nil {
+				return
+			}
+			t.Cleanup(func() { _ = c.Close() })
+		}
+	}()
+
+	p, err := newGatewayProbe("https://"+ln.Addr().String(), g.caPath, g.api.addr)
+	require.NoError(t, err)
+	p.timeout = 200 * time.Millisecond
+	p.ttl = 0
+
+	start := time.Now()
+	assert.False(t, p.ready())
+	assert.Less(t, time.Since(start), 2*time.Second, "the probe gives up at its timeout")
+}
+
+func TestGatewayProbe_RejectsUntrustedCertificate(t *testing.T) {
+	g := newTLSGateway(t)
+	other := newTLSGateway(t)
+	p, err := newGatewayProbe("https://"+g.addr, other.caPath, g.api.addr)
+	require.NoError(t, err)
+	p.ttl = 0
+
+	assert.False(t, p.ready(), "a certificate not signed by the configured CA is not ready")
+}
+
+func TestGatewayProbe_CachesWithinTTL(t *testing.T) {
+	g := newTLSGateway(t)
+	p := g.probe(t)
+	clock := time.Unix(1_000_000, 0)
+	p.now = func() time.Time { return clock }
+	p.ttl = 2 * time.Second
+
+	require.True(t, p.ready())
+	require.True(t, p.ready())
+	require.True(t, p.ready())
+	assert.Equal(t, int32(1), g.accepts.Load(), "calls inside the ttl share one probe")
+
+	g.stop()
+	assert.True(t, p.ready(), "a cached answer is served while the ttl holds")
+
+	clock = clock.Add(3 * time.Second)
+	assert.False(t, p.ready(), "after the ttl the gateway is probed again")
+}
+
+func TestGatewayProbe_HandlerStatus(t *testing.T) {
+	g := newTLSGateway(t)
+	p := g.probe(t)
+	p.ttl = 0
+
+	w := httptest.NewRecorder()
+	p.handle(w, httptest.NewRequest(http.MethodGet, "/readyz", nil))
+	assert.Equal(t, http.StatusOK, w.Code)
+
+	g.stop()
+	w = httptest.NewRecorder()
+	p.handle(w, httptest.NewRequest(http.MethodGet, "/readyz", nil))
+	assert.Equal(t, http.StatusServiceUnavailable, w.Code)
+}
+
+func TestNewGatewayProbe_DefaultsPortAndRejectsBadInput(t *testing.T) {
+	p, err := newGatewayProbe("https://gw.example", "", "")
+	require.NoError(t, err)
+	assert.Equal(t, "gw.example:443", p.addr)
+	assert.Equal(t, defaultAPIServerAddr, p.apiAddr)
+
+	_, err = newGatewayProbe("https://", "", "")
+	require.Error(t, err)
+	_, err = newGatewayProbe("https://gw.example:9999", "/no/such/ca.pem", "")
+	require.Error(t, err)
+}
+
+func TestGatewayProbe_NotReadyWhenAPIServerDown(t *testing.T) {
+	g := newTLSGateway(t)
+	p := g.probe(t)
+	p.ttl = 0
+	require.True(t, p.ready())
+
+	g.api.stop()
+	assert.False(t, p.ready(), "a dead apiserver must not look ready while the gateway is up")
+
+	w := httptest.NewRecorder()
+	p.handle(w, httptest.NewRequest(http.MethodGet, "/readyz", nil))
+	assert.Equal(t, http.StatusServiceUnavailable, w.Code)
+}
+
+func TestReadinessHandler_ServesOnlyReadyz(t *testing.T) {
+	g := newTLSGateway(t)
+	h := readinessHandler(g.probe(t))
+
+	for _, path := range []string{"/authenticate", "/healthz", "/"} {
+		w := httptest.NewRecorder()
+		h.ServeHTTP(w, httptest.NewRequest(http.MethodPost, path, nil))
+		assert.Equalf(t, http.StatusNotFound, w.Code, "%s must not be served on the readiness listener", path)
+	}
+	w := httptest.NewRecorder()
+	h.ServeHTTP(w, httptest.NewRequest(http.MethodGet, "/readyz", nil))
+	assert.Equal(t, http.StatusOK, w.Code)
+}

spinifex/agents/eks/tokenwebhook/run_test.go

@@ -128,3 +128,58 @@ func TestRunServesUntilCancelled(t *testing.T) {
 		t.Fatal("Run did not return after cancel")
 	}
 }
+
+func TestRunServesReadinessOnSeparateListener(t *testing.T) {
+	g := newTLSGateway(t)
+	cfg := runConfig(t, freeAddr(t))
+	cfg.GatewayURL = "https://" + g.addr
+	cfg.GatewayCA = g.caPath
+	cfg.APIServerAddr = g.api.addr
+	cfg.ReadyAddr = freeAddr(t)
+
+	ctx, cancel := context.WithCancel(context.Background())
+	done := make(chan error, 1)
+	go func() { done <- Run(ctx, cfg) }()
+
+	client := &http.Client{Timeout: 3 * time.Second}
+	require.Eventually(t, func() bool {
+		resp, err := client.Get("http://" + cfg.ReadyAddr + "/readyz")
+		if err != nil {
+			return false
+		}
+		_ = resp.Body.Close()
+		return resp.StatusCode == http.StatusOK
+	}, 10*time.Second, 50*time.Millisecond, "readiness listener never reported ready")
+
+	resp, err := client.Post("http://"+cfg.ReadyAddr+"/authenticate", "application/json", strings.NewReader("{}"))
+	require.NoError(t, err)
+	_ = resp.Body.Close()
+	assert.Equal(t, http.StatusNotFound, resp.StatusCode, "the readiness listener must not serve /authenticate")
+
+	cancel()
+	select {
+	case err := <-done:
+		require.NoError(t, err)
+	case <-time.After(5 * time.Second):
+		t.Fatal("Run did not return after cancel")
+	}
+}
+
+func TestRunReportsReadinessListenFailure(t *testing.T) {
+	l, err := net.Listen("tcp", "127.0.0.1:0")
+	require.NoError(t, err)
+	defer l.Close()
+
+	cfg := runConfig(t, freeAddr(t))
+	cfg.ReadyAddr = l.Addr().String()
+
+	done := make(chan error, 1)
+	go func() { done <- Run(context.Background(), cfg) }()
+	select {
+	case err := <-done:
+		require.Error(t, err)
+		assert.True(t, strings.HasPrefix(err.Error(), "serve readiness: "), err.Error())
+	case <-time.After(10 * time.Second):
+		t.Fatal("Run kept serving after the readiness listener failed")
+	}
+}

spinifex/agents/eks/tokenwebhook/webhook.go

@@ -43,6 +43,10 @@ type Config struct {
 	CertPath    string
 	KeyPath     string
 	Kubeconfig  string
+	// ReadyAddr, when set, is a plain-HTTP listener serving only /readyz for
+	// load balancer health checks. APIServerAddr defaults to the local k3s.
+	ReadyAddr     string
+	APIServerAddr string
 }
 
 // Run writes the serving certificate and apiserver kubeconfig, then serves
@@ -63,6 +67,11 @@ func Run(ctx context.Context, cfg Config) error {
 		return fmt.Errorf("build gateway client: %w", err)
 	}
 
+	probe, err := newGatewayProbe(cfg.GatewayURL, cfg.GatewayCA, cfg.APIServerAddr)
+	if err != nil {
+		return fmt.Errorf("build readiness probe: %w", err)
+	}
+
 	authr := &authenticator{
 		accountID:   cfg.AccountID,
 		clusterName: cfg.ClusterName,
@@ -74,6 +83,7 @@ func Run(ctx context.Context, cfg Config) error {
 		w.WriteHeader(http.StatusOK)
 		_, _ = w.Write([]byte("ok"))
 	})
+	mux.HandleFunc("/readyz", probe.handle)
 	mux.HandleFunc("/authenticate", authr.handle)
 
 	server := &http.Server{
@@ -82,12 +92,38 @@ func Run(ctx context.Context, cfg Config) error {
 		ReadHeaderTimeout: 5 * time.Second,
 		TLSConfig:         serverTLSConfig(tlsCert),
 	}
-	stop := context.AfterFunc(ctx, func() { _ = server.Close() })
+	readyErr := make(chan error, 1)
+	var readySrv *http.Server
+	if cfg.ReadyAddr != "" {
+		readySrv = &http.Server{
+			Addr:              cfg.ReadyAddr,
+			Handler:           readinessHandler(probe),
+			ReadHeaderTimeout: 5 * time.Second,
+		}
+		go func() {
+			if err := readySrv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
+				readyErr <- fmt.Errorf("serve readiness: %w", err)
+				_ = server.Close()
+			}
+		}()
+		defer func() { _ = readySrv.Close() }()
+	}
+	stop := context.AfterFunc(ctx, func() {
+		_ = server.Close()
+		if readySrv != nil {
+			_ = readySrv.Close()
+		}
+	})
 	defer stop()
-	slog.Info("eks-token-webhook listening", "addr", cfg.Addr, "cluster", cfg.ClusterName)
+	slog.Info("eks-token-webhook listening", "addr", cfg.Addr, "ready_addr", cfg.ReadyAddr, "cluster", cfg.ClusterName)
 	if err := server.ListenAndServeTLS("", ""); err != nil && !errors.Is(err, http.ErrServerClosed) {
 		return fmt.Errorf("serve: %w", err)
 	}
+	select {
+	case err := <-readyErr:
+		return err
+	default:
+	}
 	return nil
 }
 
@@ -101,8 +137,8 @@ type tokenReviewRequest struct {
 
 // gatewayReviewer returns the per-call relay: SigV4-POST {accountId,token} to
 // the gateway and decode the resolved identity. A non-2xx / transport error is
-// surfaced (handle() turns it into a 5xx so the apiserver retries) rather than a
-// silent deny, distinguishing "webhook broker down" from "token rejected".
+// surfaced (handle() turns it into a 5xx) rather than a silent deny, which
+// distinguishes "webhook broker down" from "token rejected" in the logs.
 func gatewayReviewer(client *eksgw.Client, clusterName, accountID string) func(string) (ekscluster.WebhookTokenReviewResult, error) {
 	path := "/clusters/" + clusterName + "/token-review"
 	return func(token string) (ekscluster.WebhookTokenReviewResult, error) {
@@ -142,8 +178,8 @@ func (a *authenticator) handle(w http.ResponseWriter, r *http.Request) {
 
 	res, err := a.review(review.Spec.Token)
 	if err != nil {
-		// Broker fault (gateway down, transport error): a 5xx tells the apiserver
-		// to retry rather than treating a transient outage as a hard deny.
+		// Broker fault (gateway down, transport error). The apiserver treats any
+		// webhook error as an authentication failure and does not retry it.
 		slog.Error("TokenReview relay failed", "cluster", a.clusterName, "err", err)
 		http.Error(w, "token review unavailable", http.StatusServiceUnavailable)
 		return

spinifex/agents/elbv2/health.go

@@ -22,6 +22,11 @@ type HealthReport struct {
 	Servers []ServerStatus `json:"servers"`
 }
 
+// FailoverServerSuffix marks the unchecked backup copy of a server that HAProxy
+// uses only when every checked server is down. '.' cannot appear in a sanitized
+// target name, so a copy never collides with a real server.
+const FailoverServerSuffix = ".fo"
+
 // queryHAProxyStats reads backend server health from the HAProxy stats socket.
 // Parses CSV "show stat" output; cols 0/1/17 = pxname/svname/status.
 func queryHAProxyStats(socketPath string) ([]ServerStatus, error) {
@@ -53,7 +58,7 @@ func queryHAProxyStats(socketPath string) ([]ServerStatus, error) {
 		}
 
 		svname := fields[1]
-		if svname == "FRONTEND" || svname == "BACKEND" {
+		if svname == "FRONTEND" || svname == "BACKEND" || strings.HasSuffix(svname, FailoverServerSuffix) {
 			continue
 		}
 

spinifex/agents/elbv2/health_test.go

@@ -58,3 +58,38 @@ func TestQueryHAProxyStats_SocketNotFound(t *testing.T) {
 		t.Error("expected error for non-existent socket")
 	}
 }
+
+func TestQueryHAProxyStats_IgnoresFailoverCopies(t *testing.T) {
+	dir := t.TempDir()
+	sock := filepath.Join(dir, "haproxy.sock")
+
+	ln, err := net.Listen("unix", sock)
+	if err != nil {
+		t.Fatalf("listen: %v", err)
+	}
+	defer ln.Close()
+
+	go func() {
+		conn, err := ln.Accept()
+		if err != nil {
+			return
+		}
+		defer conn.Close()
+
+		buf := make([]byte, 256)
+		conn.Read(buf)
+
+		csv := "# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight\n"
+		csv += "bk_tg1,srv_i-aaa111,0,0,0,0,0,5,0,0,0,0,0,0,0,0,0,DOWN,1\n"
+		csv += "bk_tg1,srv_i-aaa111" + FailoverServerSuffix + ",0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,UP,1\n"
+		fmt.Fprint(conn, csv)
+	}()
+
+	servers, err := queryHAProxyStats(sock)
+	if err != nil {
+		t.Fatalf("queryHAProxyStats: %v", err)
+	}
+	if len(servers) != 1 || servers[0].Server != "srv_i-aaa111" || servers[0].Status != "DOWN" {
+		t.Fatalf("got %+v, want only srv_i-aaa111/DOWN", servers)
+	}
+}

spinifex/domains/dns/publisher_test.go

@@ -12,6 +12,7 @@ import (
 	"strings"
 	"sync"
 	"testing"
+	"time"
 
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
@@ -120,7 +121,9 @@ func TestZonePublisherReloadCountsEveryNorthstarNode(t *testing.T) {
 				return acks
 			})
 
-			served, complete, err := pub.Reload(context.Background(), "example.com")
+			ctx, cancel := context.WithTimeout(t.Context(), 200*time.Millisecond)
+			defer cancel()
+			served, complete, err := pub.Reload(ctx, "example.com")
 			require.NoError(t, err)
 			assert.Equal(t, tc.complete, complete)
 			assert.ElementsMatch(t, tc.served, served)

spinifex/domains/ec2/ebs/metadata/store.go

@@ -8,6 +8,7 @@ import (
 	awsidentifiers "github.com/mulgadc/spinifex/spinifex/foundation/aws/identifiers"
 	"io"
 	"log/slog"
+	"strings"
 	"sync"
 	"time"
 
@@ -99,6 +100,16 @@ func (s *Store) ListAllVolumesStrict(ctx context.Context) ([]Volume, error) {
 	return listDocuments(ctx, s, volumePrefix, "volume", UnmarshalVolume, false)
 }
 
+// ListAllVolumesPartial is ListAllVolumes that hands back what it skipped
+// instead of logging it, for a caller that must decide what an unreadable
+// document means for its own answer. A document deleted mid-listing is absent.
+func (s *Store) ListAllVolumesPartial(ctx context.Context) ([]Volume, []UnreadableDocument, error) {
+	if s == nil || s.objects == nil {
+		return nil, nil, errors.New("metadata store is not configured")
+	}
+	return collectDocuments(ctx, s, volumePrefix, UnmarshalVolume, listPartial)
+}
+
 func (s *Store) listVolumeDocuments(ctx context.Context, accountID string, skipCorrupt bool) ([]Volume, error) {
 	if s == nil || s.objects == nil {
 		return nil, errors.New("metadata store is not configured")
@@ -219,6 +230,14 @@ func (s *Store) ListAMIsStrict(ctx context.Context) ([]AMI, error) {
 	return s.listAMIDocuments(ctx, false)
 }
 
+// ListAMIsPartial is ListAllVolumesPartial for AMI documents.
+func (s *Store) ListAMIsPartial(ctx context.Context) ([]AMI, []UnreadableDocument, error) {
+	if s == nil || s.objects == nil {
+		return nil, nil, errors.New("metadata store is not configured")
+	}
+	return collectDocuments(ctx, s, amiPrefix, UnmarshalAMI, listPartial)
+}
+
 func (s *Store) listAMIDocuments(ctx context.Context, skipCorrupt bool) ([]AMI, error) {
 	return listDocuments(ctx, s, amiPrefix, "AMI", UnmarshalAMI, skipCorrupt)
 }
@@ -285,15 +304,43 @@ func listKeys(ctx context.Context, objects objectstore.ObjectStore, bucket, pref
 	return nil, fmt.Errorf("listing %s did not finish within %d pages", prefix, maxListPages)
 }
 
+// UnreadableDocument is a document a partial listing could not use. AccountID
+// is empty for an unpartitioned kind. Err wraps ErrCorruptDocument when the
+// bytes arrived but would not decode.
+type UnreadableDocument struct {
+	Key       string
+	AccountID string
+	ID        string
+	Err       error
+}
+
+// documentIdentity recovers the account and ID a document key was built from.
+func documentIdentity(prefix, key string) (accountID, id string) {
+	rest := strings.TrimSuffix(strings.TrimPrefix(key, prefix), ".json")
+	if account, docID, found := strings.Cut(rest, "/"); found {
+		return account, docID
+	}
+	return "", rest
+}
+
+// listMode says what a listing does with a document it cannot use.
+type listMode int
+
+const (
+	// listStrict fails the listing.
+	listStrict listMode = iota
+	// listTolerant skips it and logs the skip.
+	listTolerant
+	// listPartial skips it and reports it, treating a deleted one as absent.
+	listPartial
+)
+
 // listDocuments reads and decodes every document under prefix.
 //
 // skipCorrupt tolerates a document that cannot be fetched as well as one that
 // cannot be decoded: an object whose shards no longer join is exactly as
 // unusable as one whose bytes will not parse, and either kind read wholesale
 // as a failure turns a single bad document into a cluster-wide outage.
-//
-// Fetches run concurrently but results are assembled in key order, so a strict
-// listing fails on the same document it would have failed on serially.
 func listDocuments[T any](
 	ctx context.Context,
 	s *Store,
@@ -301,9 +348,35 @@ func listDocuments[T any](
 	unmarshal func([]byte) (T, error),
 	skipCorrupt bool,
 ) ([]T, error) {
+	if !skipCorrupt {
+		documents, _, err := collectDocuments(ctx, s, prefix, unmarshal, listStrict)
+		return documents, err
+	}
+	documents, skipped, err := collectDocuments(ctx, s, prefix, unmarshal, listTolerant)
+	for _, doc := range skipped {
+		if errors.Is(doc.Err, ErrCorruptDocument) {
+			slog.WarnContext(ctx, "skipping undecodable "+kind+" document", "key", doc.Key, "err", doc.Err)
+		} else {
+			slog.WarnContext(ctx, "skipping unreadable "+kind+" document", "key", doc.Key, "err", doc.Err)
+		}
+	}
+	return documents, err
+}
+
+// collectDocuments is listDocuments returning the documents it skipped.
+//
+// Fetches run concurrently but results are assembled in key order, so a strict
+// listing fails on the same document it would have failed on serially.
+func collectDocuments[T any](
+	ctx context.Context,
+	s *Store,
+	prefix string,
+	unmarshal func([]byte) (T, error),
+	mode listMode,
+) ([]T, []UnreadableDocument, error) {
 	keys, err := listKeys(ctx, s.objects, s.bucket, prefix)
 	if err != nil {
-		return nil, err
+		return nil, nil, err
 	}
 
 	// fetchErr and decodeErr are kept apart because they are tolerated on
@@ -341,23 +414,31 @@ func listDocuments[T any](
 	wg.Wait()
 
 	documents := make([]T, 0, len(results))
+	var skipped []UnreadableDocument
+	skip := func(key string, err error) {
+		accountID, id := documentIdentity(prefix, key)
+		skipped = append(skipped, UnreadableDocument{Key: key, AccountID: accountID, ID: id, Err: err})
+	}
 	for _, r := range results {
 		switch {
 		case r.fetchErr != nil:
-			if !skipCorrupt {
-				return nil, r.fetchErr
+			if mode == listStrict {
+				return nil, nil, r.fetchErr
 			}
-			slog.WarnContext(ctx, "skipping unreadable "+kind+" document", "key", r.key, "err", r.fetchErr)
+			if mode == listPartial && objectstore.IsNoSuchKeyError(r.fetchErr) {
+				continue
+			}
+			skip(r.key, r.fetchErr)
 		case r.decodeErr != nil:
-			if !skipCorrupt || !errors.Is(r.decodeErr, ErrCorruptDocument) {
-				return nil, r.decodeErr
+			if mode == listStrict || !errors.Is(r.decodeErr, ErrCorruptDocument) {
+				return nil, nil, r.decodeErr
 			}
-			slog.WarnContext(ctx, "skipping undecodable "+kind+" document", "key", r.key, "err", r.decodeErr)
+			skip(r.key, r.decodeErr)
 		default:
 			documents = append(documents, r.document)
 		}
 	}
-	return documents, nil
+	return documents, skipped, nil
 }
 
 // FetchResult is one document from an explicit-ID fan-out. Err is that ID's

spinifex/domains/ec2/ebs/metadata/store_test.go

@@ -706,3 +706,74 @@ func (s *truncatedNoTokenStore) ListObjectsV2(_ context.Context, _ *s3.ListObjec
 	}
 	return &s3.ListObjectsV2Output{Contents: contents, IsTruncated: aws.Bool(true)}, nil
 }
+
+// A partial listing hands back each document it could not use, identified by
+// the account and ID its key was built from, and leaves out one deleted
+// between listing and read.
+func TestListPartial_ReportsUnreadableAndDropsDeleted(t *testing.T) {
+	objects := objectstore.NewMemoryObjectStore()
+	store := NewStore(objects, "control-plane")
+	ctx := context.Background()
+
+	require.NoError(t, store.PutVolume(ctx, Volume{VolumeID: "vol-good", TenantID: "000000000001", CapacityGiB: 1}))
+	require.NoError(t, store.PutVolume(ctx, Volume{VolumeID: "vol-unreadable", TenantID: "000000000002", CapacityGiB: 1}))
+	unreadableKey, err := VolumeKey("000000000002", "vol-unreadable")
+	require.NoError(t, err)
+	corruptKey, err := VolumeKey("000000000001", "vol-corrupt")
+	require.NoError(t, err)
+	writeRaw(t, objects, corruptKey, []byte("{not json"))
+	badAMIKey, err := AMIKey("ami-bad")
+	require.NoError(t, err)
+	writeRaw(t, objects, badAMIKey, []byte("{not json"))
+	require.NoError(t, store.PutAMI(ctx, AMI{ImageID: "ami-good"}))
+	deletedKey, err := AMIKey("ami-deleted")
+	require.NoError(t, err)
+	writeRaw(t, objects, deletedKey, []byte("{}"))
+
+	broken := NewStore(&deletedAndFailingStore{
+		ObjectStore: objects, failKey: unreadableKey, deletedKey: deletedKey,
+	}, "control-plane")
+
+	volumes, unreadable, err := broken.ListAllVolumesPartial(ctx)
+	require.NoError(t, err)
+	assert.Equal(t, []string{"vol-good"}, volumeIDs(volumes))
+	require.Len(t, unreadable, 2)
+	byID := map[string]UnreadableDocument{}
+	for _, doc := range unreadable {
+		byID[doc.ID] = doc
+	}
+	assert.Equal(t, "000000000001", byID["vol-corrupt"].AccountID)
+	assert.ErrorIs(t, byID["vol-corrupt"].Err, ErrCorruptDocument)
+	assert.Equal(t, "000000000002", byID["vol-unreadable"].AccountID)
+	assert.NotErrorIs(t, byID["vol-unreadable"].Err, ErrCorruptDocument)
+
+	amis, unreadable, err := broken.ListAMIsPartial(ctx)
+	require.NoError(t, err)
+	require.Len(t, amis, 1)
+	assert.Equal(t, "ami-good", amis[0].ImageID)
+	require.Len(t, unreadable, 1, "a deleted document is absent, not unreadable")
+	assert.Equal(t, UnreadableDocument{Key: badAMIKey, ID: "ami-bad", Err: unreadable[0].Err}, unreadable[0])
+
+	_, _, err = (&Store{}).ListAllVolumesPartial(ctx)
+	require.Error(t, err)
+	_, _, err = (&Store{}).ListAMIsPartial(ctx)
+	require.Error(t, err)
+}
+
+// deletedAndFailingStore fails one key's read and reports another as already
+// deleted, while still listing both.
+type deletedAndFailingStore struct {
+	objectstore.ObjectStore
+
+	failKey, deletedKey string
+}
+
+func (s *deletedAndFailingStore) GetObject(ctx context.Context, in *s3.GetObjectInput) (*s3.GetObjectOutput, error) {
+	switch aws.StringValue(in.Key) {
+	case s.failKey:
+		return nil, errors.New("reconstruction failed")
+	case s.deletedKey:
+		return nil, &objectstore.NoSuchKeyError{Key: s.deletedKey}
+	}
+	return s.ObjectStore.GetObject(ctx, in)
+}

spinifex/domains/ec2/eip/service_impl.go

@@ -181,7 +181,7 @@ func (s *EIPServiceImpl) ReleaseAddress(ctx context.Context, input *ec2.ReleaseA
 
 	// Cannot release an EIP that is still associated.
 	if record.State == "associated" {
-		return nil, errors.New(awserrors.ErrorInvalidAddressLocked)
+		return nil, awserrors.Errorf(awserrors.ErrorInvalidIPAddressInUse, "Address %s is in use.", record.PublicIp)
 	}
 
 	// Release IP back to IPAM pool. User-driven release of an already-detached

spinifex/domains/ec2/eip/service_impl_test.go

@@ -169,8 +169,11 @@ func TestEIP_ReleaseWhileAssociated(t *testing.T) {
 	_, err = svc.ReleaseAddress(context.Background(), &ec2.ReleaseAddressInput{
 		AllocationId: aws.String(allocID),
 	}, testAccountID)
-	assert.Error(t, err)
-	assert.Contains(t, err.Error(), "InvalidAddress.Locked")
+	require.Error(t, err)
+	code, message, ok := awserrors.ResolveErrorDetail(err)
+	require.True(t, ok, "error must carry a registered code: %v", err)
+	assert.Equal(t, awserrors.ErrorInvalidIPAddressInUse, code)
+	assert.Equal(t, "Address "+record.PublicIp+" is in use.", message)
 }
 
 func TestEIP_ReleaseByInstanceID_ReclaimsAssociatedEIP(t *testing.T) {

spinifex/domains/ec2/guestmetadata/metadata.go

@@ -246,13 +246,13 @@ func (s *IMDSServiceImpl) dispatch(w http.ResponseWriter, r *http.Request, eni *
 	case prefixMetaData + "security-groups":
 		writeText(w, strings.Join(s.resolver.resolveSGNames(ctx, eni.accountID, eni.securityGroupIDs), "\n"))
 	case prefixMetaData + "hostname", prefixMetaData + "local-hostname":
-		writeText(w, s.privateHostname(eni.privateIP, regionFromAZ(eni.availabilityZone)))
+		writeText(w, s.privateHostname(eni.privateIP, s.region))
 	case prefixMetaData + "placement", prefixMetaData + "placement/":
 		writeText(w, "availability-zone\nregion")
 	case prefixMetaData + "placement/availability-zone":
 		writeText(w, eni.availabilityZone)
 	case prefixMetaData + "placement/region":
-		writeText(w, regionFromAZ(eni.availabilityZone))
+		writeText(w, s.region)
 	case prefixMetaData + "instance-type":
 		s.serveInstanceField(ctx, w, eni, func(i *instanceFacts) string { return i.instanceType })
 	case prefixMetaData + "ami-id":
@@ -419,7 +419,7 @@ func (s *IMDSServiceImpl) serveInstanceIdentityDocument(ctx context.Context, w h
 		InstanceType:     inst.instanceType,
 		PendingTime:      inst.pendingTime.UTC().Format("2006-01-02T15:04:05Z"),
 		PrivateIP:        eni.privateIP,
-		Region:           regionFromAZ(eni.availabilityZone),
+		Region:           s.region,
 		Version:          identityDocSchemaVersion,
 	}
 	writeIdentityDocument(w, doc)
@@ -587,7 +587,7 @@ func (s *IMDSServiceImpl) serveNetworkInterface(ctx context.Context, w http.Resp
 	case "local-ipv4s":
 		writeText(w, eni.privateIP)
 	case "local-hostname":
-		writeText(w, s.privateHostname(eni.privateIP, regionFromAZ(eni.availabilityZone)))
+		writeText(w, s.privateHostname(eni.privateIP, s.region))
 	case "security-group-ids":
 		writeText(w, strings.Join(eni.securityGroupIDs, "\n"))
 	case "security-groups":
@@ -697,7 +697,7 @@ func (s *IMDSServiceImpl) servePublicHostname(w http.ResponseWriter, eni *eniFac
 		w.WriteHeader(http.StatusNotFound) // no public IP → no public hostname
 		return
 	}
-	if h := s.publicHostname(eni.publicIP, regionFromAZ(eni.availabilityZone)); h != "" {
+	if h := s.publicHostname(eni.publicIP, s.region); h != "" {
 		writeText(w, h)
 		return
 	}
@@ -772,18 +772,6 @@ func parseTokenTTL(raw string) (time.Duration, bool) {
 	return time.Duration(n) * time.Second, true
 }
 
-// regionFromAZ strips the trailing AZ letter from an AZ name to get the region.
-func regionFromAZ(az string) string {
-	if az == "" {
-		return ""
-	}
-	last := az[len(az)-1]
-	if last >= 'a' && last <= 'z' {
-		return az[:len(az)-1]
-	}
-	return az
-}
-
 // privateHostname is the AWS-parity local hostname (ip-<dashed>.<region>.<internal
 // domain>) honoring the configured internal_domain, so it matches the record the
 // DNS writer publishes. Empty when the IP or region is missing.

spinifex/domains/ec2/guestmetadata/metadata_test.go

@@ -159,6 +159,7 @@ func newTestService(res eniResolver, fIAM profileLookup, assumer stsAssumer) (*I
 		roleMiss:       newRoleMissLogger(func() time.Time { return now }),
 		pubKeys:        &fakePublicKeys{},
 		now:            func() time.Time { return now },
+		region:         "ap-southeast-2",
 		baseDomain:     "spx3.net",
 		servicesDomain: testServicesDomain,
 	}, now
@@ -513,6 +514,43 @@ func TestHTTP_ServicesDomainEmpty404(t *testing.T) {
 	assert.NotContains(t, rec.Body.String(), "amazonaws.com")
 }
 
+// The region comes from the node's configuration, not the zone name, so a zone
+// named <region>-azN (no trailing letter) still reports the bare region.
+func TestHTTP_RegionFromNodeNotZoneName(t *testing.T) {
+	for _, az := range []string{"us-west-1a", "us-west-1-az2"} {
+		t.Run(az, func(t *testing.T) {
+			eni := testENI()
+			eni.availabilityZone = az
+			res := &fakeResolver{eni: eni, inst: &instanceFacts{instanceType: "t3.micro", imageID: "ami-12345"}}
+			svc, _ := newTestService(res, &fakeIAM{}, &fakeAssumer{})
+			svc.region = "us-west-1"
+			h := withTapENI(svc.httpHandler(), eni)
+			token := issueToken(t, h)
+
+			for path, want := range map[string]string{
+				prefixMetaData + "placement/availability-zone": az,
+				prefixMetaData + "placement/region":            "us-west-1",
+				prefixMetaData + "hostname":                    "ip-10-0-1-5.us-west-1.compute.internal",
+				prefixMetaData + "local-hostname":              "ip-10-0-1-5.us-west-1.compute.internal",
+				prefixMetaData + "public-hostname":             "ec2-203-0-113-7.us-west-1.compute.spx3.net",
+				macPath(eni.mac, "local-hostname"):             "ip-10-0-1-5.us-west-1.compute.internal",
+				macPath(eni.mac, "public-hostname"):            "ec2-203-0-113-7.us-west-1.compute.spx3.net",
+			} {
+				rec := get(t, h, path, token)
+				assert.Equal(t, http.StatusOK, rec.Code, "path=%s", path)
+				assert.Equal(t, want, rec.Body.String(), "path=%s", path)
+			}
+
+			rec := get(t, h, pathIdentityDocument, token)
+			require.Equal(t, http.StatusOK, rec.Code)
+			var doc map[string]any
+			require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &doc))
+			assert.Equal(t, "us-west-1", doc["region"])
+			assert.Equal(t, az, doc["availabilityZone"])
+		})
+	}
+}
+
 // An instance with no public IP has no public hostname or public IPv4: both leaves
 // 404, not an empty 200, matching real EC2 and the meta-data/ listing that omits them.
 func TestHTTP_PublicFieldsAbsent404(t *testing.T) {

spinifex/domains/ec2/guestmetadata/service.go

@@ -75,6 +75,7 @@ type IMDSServiceImpl struct {
 	tapResp        *tapResponderManager
 	listTaps       listTapsFunc
 	now            func() time.Time
+	region         string
 	baseDomain     string
 	internalDomain string
 	servicesDomain string
@@ -82,7 +83,9 @@ type IMDSServiceImpl struct {
 }
 
 // NewIMDSServiceImpl wires the IMDS service. listTaps is injected to avoid a
-// network/host import cycle. baseDomain and internalDomain are the cluster's
+// network/host import cycle. region is this node's configured region, served as
+// placement/region and in the identity document and hostnames; it is not
+// derived from the zone name. baseDomain and internalDomain are the cluster's
 // authoritative public and private (AWS-parity) DNS domains, used for
 // public-hostname and local-hostname so IMDS matches the records the DNS writer
 // publishes. servicesDomain is the cluster's configured AWS.ServicesDomain,
@@ -99,7 +102,7 @@ type IMDSServiceImpl struct {
 // nil, degrading that source to a miss rather than failing IMDS to start —
 // the caller (vpcd) owns their construction and availability.
 // ctx bounds the bucket opens only; each served request carries its own.
-func NewIMDSServiceImpl(ctx context.Context, natsConn *nats.Conn, sts stsAssumer, iamSvc profileLookup, pubKeys publicKeyLookup, localState LocalStateReader, records RecordLoader, listTaps listTapsFunc, baseDomain, internalDomain, servicesDomain, caCertPath string, resolverIPs []string, bind HostBindAddrs) (*IMDSServiceImpl, error) {
+func NewIMDSServiceImpl(ctx context.Context, natsConn *nats.Conn, sts stsAssumer, iamSvc profileLookup, pubKeys publicKeyLookup, localState LocalStateReader, records RecordLoader, listTaps listTapsFunc, region, baseDomain, internalDomain, servicesDomain, caCertPath string, resolverIPs []string, bind HostBindAddrs) (*IMDSServiceImpl, error) {
 	if bind.Meta == "" || bind.DNS == "" {
 		return nil, errors.New("IMDS host bind addresses required (use NewHostBindAddrs)")
 	}
@@ -173,6 +176,7 @@ func NewIMDSServiceImpl(ctx context.Context, natsConn *nats.Conn, sts stsAssumer
 		pubKeys:        pubKeys,
 		listTaps:       listTaps,
 		now:            time.Now,
+		region:         region,
 		baseDomain:     baseDomain,
 		internalDomain: internalDomain,
 		servicesDomain: servicesDomain,

spinifex/domains/ec2/snapshot/delete_snapshot_errors_test.go

@@ -0,0 +1,219 @@
+package snapshot
+
+import (
+	"context"
+	"errors"
+	"strings"
+	"testing"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/service/ec2"
+	"github.com/aws/aws-sdk-go/service/s3"
+	"github.com/mulgadc/spinifex/spinifex/bootstrap/config"
+	"github.com/mulgadc/spinifex/spinifex/domains/ec2/ebs/metadata"
+	"github.com/mulgadc/spinifex/spinifex/foundation/aws/errors"
+	"github.com/mulgadc/spinifex/spinifex/providers/ebs"
+	"github.com/mulgadc/spinifex/spinifex/providers/objectstore"
+	"github.com/nats-io/nats.go"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// putCorruptDocument writes bytes that will not decode under key.
+func putCorruptDocument(t *testing.T, store objectstore.ObjectStore, key string) {
+	t.Helper()
+	_, err := store.PutObject(context.Background(), &s3.PutObjectInput{
+		Bucket: aws.String("test-bucket"),
+		Key:    aws.String(key),
+		Body:   strings.NewReader("not-json"),
+	})
+	require.NoError(t, err)
+}
+
+// unreferencedSnapshot returns a snapshot nothing in the cluster reads from.
+func unreferencedSnapshot(t *testing.T, svc *SnapshotServiceImpl, store *objectstore.MemoryObjectStore) string {
+	t.Helper()
+	createTestVolume(t, svc, store, "vol-source", 50)
+	snap, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{VolumeId: aws.String("vol-source")}, testAccountID)
+	require.NoError(t, err)
+	return aws.StringValue(snap.SnapshotId)
+}
+
+func deleteSnapshot(svc *SnapshotServiceImpl, snapshotID string) error {
+	_, err := svc.DeleteSnapshot(context.Background(), &ec2.DeleteSnapshotInput{SnapshotId: aws.String(snapshotID)}, testAccountID)
+	return err
+}
+
+func requireSnapshotSurvives(t *testing.T, svc *SnapshotServiceImpl, snapshotID string) {
+	t.Helper()
+	out, err := svc.DescribeSnapshots(context.Background(), &ec2.DescribeSnapshotsInput{SnapshotIds: []*string{aws.String(snapshotID)}}, testAccountID)
+	require.NoError(t, err)
+	require.Len(t, out.Snapshots, 1)
+}
+
+// An unreadable volume record cannot be ruled out as a clone of the snapshot,
+// so the delete is refused, but with the record named rather than a bare code.
+func TestDeleteSnapshot_UnreadableOwnVolumeRecordIsNamed(t *testing.T) {
+	svc, store := setupTestSnapshotService(t)
+	snapshotID := unreferencedSnapshot(t, svc, store)
+	key, err := ebsmetadata.VolumeKey(testAccountID, "vol-corrupt")
+	require.NoError(t, err)
+	putCorruptDocument(t, store, key)
+
+	err = deleteSnapshot(svc, snapshotID)
+	code, message := clientErrorDetail(t, err)
+	assert.Equal(t, awserrors.ErrorServerInternal, code)
+	assert.Equal(t, "Cannot confirm that snapshot '"+snapshotID+"' is not in use: the record for volume 'vol-corrupt' could not be read", message)
+	requireSnapshotSurvives(t, svc, snapshotID)
+}
+
+// Another account's unreadable volume blocks the delete without its ID
+// reaching the caller.
+func TestDeleteSnapshot_UnreadableForeignVolumeRecordIsNotNamed(t *testing.T) {
+	svc, store := setupTestSnapshotService(t)
+	snapshotID := unreferencedSnapshot(t, svc, store)
+	key, err := ebsmetadata.VolumeKey(otherAccountID, "vol-foreign-corrupt")
+	require.NoError(t, err)
+	putCorruptDocument(t, store, key)
+
+	err = deleteSnapshot(svc, snapshotID)
+	code, message := clientErrorDetail(t, err)
+	assert.Equal(t, awserrors.ErrorServerInternal, code)
+	assert.Equal(t, "Cannot confirm that snapshot '"+snapshotID+"' is not in use: 1 volume record(s) could not be read", message)
+	assert.NotContains(t, message, "vol-foreign-corrupt")
+	requireSnapshotSurvives(t, svc, snapshotID)
+}
+
+// A readable clone settles the answer whatever else failed to read, so the
+// caller gets the AWS refusal rather than an internal error.
+func TestDeleteSnapshot_ReadableCloneWinsOverUnreadableRecord(t *testing.T) {
+	svc, store := setupTestSnapshotService(t)
+	snapshotID := unreferencedSnapshot(t, svc, store)
+	key, err := ebsmetadata.VolumeKey(testAccountID, "vol-corrupt")
+	require.NoError(t, err)
+	putCorruptDocument(t, store, key)
+	seedVolumeDocument(t, store, ebsmetadata.Volume{VolumeID: "vol-clone", CapacityGiB: 50, SnapshotID: snapshotID})
+
+	code, message := clientErrorDetail(t, deleteSnapshot(svc, snapshotID))
+	assert.Equal(t, awserrors.ErrorInvalidSnapshotInUse, code)
+	assert.Equal(t, "The snapshot '"+snapshotID+"' is in use by volume 'vol-clone'.", message)
+}
+
+// An unreadable AMI record blocks the delete, and says so.
+func TestDeleteSnapshot_UnreadableAMIRecordIsReported(t *testing.T) {
+	svc, store := setupTestSnapshotService(t)
+	snapshotID := unreferencedSnapshot(t, svc, store)
+	key, err := ebsmetadata.AMIKey("ami-corrupt")
+	require.NoError(t, err)
+	putCorruptDocument(t, store, key)
+
+	code, message := clientErrorDetail(t, deleteSnapshot(svc, snapshotID))
+	assert.Equal(t, awserrors.ErrorServerInternal, code)
+	assert.Equal(t, "Cannot confirm that snapshot '"+snapshotID+"' is not in use: 1 AMI record(s) could not be read", message)
+	requireSnapshotSurvives(t, svc, snapshotID)
+}
+
+// A registered AMI naming the snapshot is still the AWS refusal when another
+// AMI record is unreadable.
+func TestDeleteSnapshot_RegisteredAMIWinsOverUnreadableAMI(t *testing.T) {
+	svc, store := setupTestSnapshotService(t)
+	snapshotID := unreferencedSnapshot(t, svc, store)
+	key, err := ebsmetadata.AMIKey("ami-corrupt")
+	require.NoError(t, err)
+	putCorruptDocument(t, store, key)
+	require.NoError(t, svc.metadata.PutAMI(context.Background(), ebsmetadata.AMI{ImageID: "ami-registered", SnapshotID: snapshotID}))
+
+	code, _ := clientErrorDetail(t, deleteSnapshot(svc, snapshotID))
+	assert.Equal(t, awserrors.ErrorInvalidSnapshotInUse, code)
+}
+
+// A registered AMI naming the snapshot is the AWS refusal even when the volume
+// scan, which runs first, could not read a record.
+func TestDeleteSnapshot_RegisteredAMIWinsOverUnreadableVolume(t *testing.T) {
+	svc, store := setupTestSnapshotService(t)
+	snapshotID := unreferencedSnapshot(t, svc, store)
+	key, err := ebsmetadata.VolumeKey(otherAccountID, "vol-foreign-corrupt")
+	require.NoError(t, err)
+	putCorruptDocument(t, store, key)
+	require.NoError(t, svc.metadata.PutAMI(context.Background(), ebsmetadata.AMI{ImageID: "ami-registered", SnapshotID: snapshotID}))
+
+	code, _ := clientErrorDetail(t, deleteSnapshot(svc, snapshotID))
+	assert.Equal(t, awserrors.ErrorInvalidSnapshotInUse, code)
+	requireSnapshotSurvives(t, svc, snapshotID)
+}
+
+// vanishingObjectStore lists keys whose documents have already been deleted,
+// as a listing racing a DeleteVolume or DeregisterImage sees them.
+type vanishingObjectStore struct {
+	objectstore.ObjectStore
+
+	gone map[string]bool
+}
+
+func (s *vanishingObjectStore) GetObject(ctx context.Context, input *s3.GetObjectInput) (*s3.GetObjectOutput, error) {
+	if key := aws.StringValue(input.Key); s.gone[key] {
+		return nil, &objectstore.NoSuchKeyError{Key: key}
+	}
+	return s.ObjectStore.GetObject(ctx, input)
+}
+
+// A record deleted between listing and read is absent, not unreadable, and
+// does not stop an unreferenced snapshot from being deleted.
+func TestDeleteSnapshot_RecordDeletedMidScanIsAbsent(t *testing.T) {
+	memory := objectstore.NewMemoryObjectStore()
+	volumeKey, err := ebsmetadata.VolumeKey(otherAccountID, "vol-deleted")
+	require.NoError(t, err)
+	amiKey, err := ebsmetadata.AMIKey("ami-deregistered")
+	require.NoError(t, err)
+	store := &vanishingObjectStore{ObjectStore: memory, gone: map[string]bool{volumeKey: true, amiKey: true}}
+
+	cfg := &config.Config{Predastore: config.PredastoreConfig{Bucket: "test-bucket"}}
+	svc := NewSnapshotServiceImplWithStore(cfg, store, nil)
+	svc.SetEBSProvider(ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{}))
+	snapshotID := unreferencedSnapshot(t, svc, memory)
+	putCorruptDocument(t, memory, volumeKey)
+	putCorruptDocument(t, memory, amiKey)
+
+	require.NoError(t, deleteSnapshot(svc, snapshotID))
+}
+
+// deleteFailingProvider fails only DeleteSnapshot, with err.
+type deleteFailingProvider struct {
+	ebsprovider.EBSProvider
+
+	err error
+}
+
+func (p *deleteFailingProvider) DeleteSnapshot(context.Context, ebsprovider.DeleteSnapshotRequest) error {
+	return p.err
+}
+
+// A provider failure maps to a code the caller can act on, and leaves the
+// control-plane document so the delete can be retried.
+func TestDeleteSnapshot_ProviderFailureMapsToSpecificError(t *testing.T) {
+	cases := []struct {
+		name     string
+		err      error
+		wantCode string
+	}{
+		{"provider unavailable", &ebsprovider.ProviderError{Code: ebsprovider.ErrorCodeUnavailable, Message: "busy"}, awserrors.ErrorServiceUnavailable},
+		{"no provider responding", errors.Join(errors.New("request ebs.provider.v1.snapshot.delete"), nats.ErrNoResponders), awserrors.ErrorServiceUnavailable},
+		{"request timed out", context.DeadlineExceeded, awserrors.ErrorServiceUnavailable},
+		{"connection closed", nats.ErrConnectionClosed, awserrors.ErrorServiceUnavailable},
+		{"nats timeout", nats.ErrTimeout, awserrors.ErrorServiceUnavailable},
+		{"backend reports in use", &ebsprovider.ProviderError{Code: ebsprovider.ErrorCodeVolumeInUse, Message: "clone"}, awserrors.ErrorInvalidSnapshotInUse},
+		{"internal failure", &ebsprovider.ProviderError{Code: ebsprovider.ErrorCodeInternal, Message: "delete snapshot objects: boom"}, awserrors.ErrorServerInternal},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			svc, store := setupTestSnapshotService(t)
+			snapshotID := unreferencedSnapshot(t, svc, store)
+			svc.SetEBSProvider(&deleteFailingProvider{EBSProvider: svc.provider, err: tc.err})
+
+			code, message := clientErrorDetail(t, deleteSnapshot(svc, snapshotID))
+			assert.Equal(t, tc.wantCode, code)
+			assert.Contains(t, message, snapshotID)
+			requireSnapshotSurvives(t, svc, snapshotID)
+		})
+	}
+}

spinifex/domains/ec2/snapshot/service_impl.go

@@ -670,12 +670,9 @@ func (s *SnapshotServiceImpl) snapshotInUseByVolumes(ctx context.Context, snapsh
 	// point: launching from a system AMI writes a root volume owned by the
 	// launching tenant whose SnapshotID is the system snapshot, so a clone
 	// routinely lives outside the snapshot owner's prefix.
-	//
-	// Strict: a volume that failed to read is not evidence that no clone reads
-	// the snapshot's chunks, and this answer gates deleting them.
-	volumes, err := s.metadata.ListAllVolumesStrict(ctx)
+	volumes, unreadable, err := s.metadata.ListAllVolumesPartial(ctx)
 	if err != nil {
-		return false, "", fmt.Errorf("snapshotInUseByVolumes: failed to list volumes: %w", err)
+		return false, "", awserrors.Errorf(awserrors.ErrorServerInternal, "check whether snapshot %s is in use: list volumes: %v", snapshotID, err)
 	}
 	inUse := false
 	for _, volume := range volumes {
@@ -687,28 +684,71 @@ func (s *SnapshotServiceImpl) snapshotInUseByVolumes(ctx context.Context, snapsh
 		}
 		inUse = true
 	}
-	return inUse, "", nil
+	if inUse {
+		return true, "", nil
+	}
+	// A volume that failed to read is not evidence that no clone reads the
+	// snapshot's chunks, and this answer gates deleting them.
+	if len(unreadable) > 0 {
+		return false, "", unreadableRecordError(ctx, snapshotID, accountID, "volume", unreadable)
+	}
+	return false, "", nil
 }
 
 // amiInUseBySnapshot checks if any registered AMI names the given snapshot as
 // its root device. AWS refuses DeleteSnapshot while a registered AMI
 // references the snapshot, the same way it refuses one backing a volume.
-func (s *SnapshotServiceImpl) amiInUseBySnapshot(ctx context.Context, snapshotID string) (bool, error) {
-	// Whole-cluster and strict for the same reason as the volume check above:
-	// a system AMI can be registered by an account other than the snapshot's
-	// owner, and an AMI this cannot read is not evidence that none exists.
-	amis, err := s.metadata.ListAMIsStrict(ctx)
+func (s *SnapshotServiceImpl) amiInUseBySnapshot(ctx context.Context, snapshotID, accountID string) (bool, error) {
+	// Whole-cluster and fail-closed for the same reason as the volume check
+	// above: a system AMI can be registered by an account other than the
+	// snapshot's owner, and an AMI this cannot read is not evidence none exists.
+	amis, unreadable, err := s.metadata.ListAMIsPartial(ctx)
 	if err != nil {
-		return false, fmt.Errorf("amiInUseBySnapshot: failed to list AMIs: %w", err)
+		return false, awserrors.Errorf(awserrors.ErrorServerInternal, "check whether snapshot %s is in use: list AMIs: %v", snapshotID, err)
 	}
 	for _, ami := range amis {
 		if ami.SnapshotID == snapshotID {
 			return true, nil
 		}
 	}
+	if len(unreadable) > 0 {
+		return false, unreadableRecordError(ctx, snapshotID, accountID, "AMI", unreadable)
+	}
 	return false, nil
 }
 
+// unreadableRecordError refuses a delete the in-use scan could not clear. It
+// names a record only when the caller owns it: the scan is whole-cluster, and
+// an AMI record that will not read cannot say who owns it.
+func unreadableRecordError(ctx context.Context, snapshotID, accountID, kind string, unreadable []ebsmetadata.UnreadableDocument) error {
+	for _, doc := range unreadable {
+		slog.ErrorContext(ctx, "DeleteSnapshot cannot read "+kind+" record", "snapshotId", snapshotID, "key", doc.Key, "err", doc.Err)
+	}
+	for _, doc := range unreadable {
+		if doc.AccountID != "" && doc.AccountID == accountID {
+			return awserrors.Errorf(awserrors.ErrorServerInternal,
+				"Cannot confirm that snapshot '%s' is not in use: the record for %s '%s' could not be read", snapshotID, kind, doc.ID)
+		}
+	}
+	return awserrors.Errorf(awserrors.ErrorServerInternal,
+		"Cannot confirm that snapshot '%s' is not in use: %d %s record(s) could not be read", snapshotID, len(unreadable), kind)
+}
+
+// providerDeleteError maps a provider DeleteSnapshot failure to the code a
+// client can act on. A transport failure is retryable: the delete is idempotent.
+func providerDeleteError(snapshotID string, err error) error {
+	switch {
+	case errors.Is(err, ebsprovider.ErrVolumeInUse):
+		return awserrors.Errorf(awserrors.ErrorInvalidSnapshotInUse, "The snapshot '%s' is in use by a volume.", snapshotID)
+	case errors.Is(err, ebsprovider.ErrUnavailable), errors.Is(err, nats.ErrNoResponders),
+		errors.Is(err, nats.ErrConnectionClosed), errors.Is(err, nats.ErrTimeout),
+		errors.Is(err, context.DeadlineExceeded):
+		return awserrors.Errorf(awserrors.ErrorServiceUnavailable, "delete snapshot %s: storage provider unavailable: %v", snapshotID, err)
+	default:
+		return awserrors.Errorf(awserrors.ErrorServerInternal, "delete snapshot %s: %v", snapshotID, err)
+	}
+}
+
 // DeleteSnapshot deletes a snapshot after verifying the caller owns it.
 func (s *SnapshotServiceImpl) DeleteSnapshot(ctx context.Context, input *ec2.DeleteSnapshotInput, accountID string) (*ec2.DeleteSnapshotOutput, error) {
 	if input == nil || input.SnapshotId == nil {
@@ -739,11 +779,9 @@ func (s *SnapshotServiceImpl) DeleteSnapshot(ctx context.Context, input *ec2.Del
 	}
 
 	// Check if any volumes were created from this snapshot
-	inUse, volumeID, err := s.snapshotInUseByVolumes(ctx, snapshotID, accountID)
-	if err != nil {
-		slog.ErrorContext(ctx, "DeleteSnapshot failed to check snapshot usage", "snapshotId", snapshotID, "err", err)
-		return nil, errors.New(awserrors.ErrorServerInternal)
-	}
+	// A scan's failure is held until both have run, so a readable reference
+	// found by either answers InvalidSnapshot.InUse.
+	inUse, volumeID, volumeErr := s.snapshotInUseByVolumes(ctx, snapshotID, accountID)
 	if inUse {
 		slog.InfoContext(ctx, "DeleteSnapshot blocked: snapshot in use by volume", "snapshotId", snapshotID, "volumeId", volumeID)
 		// The registered message blames AMIs. Name the volume only when the
@@ -755,21 +793,26 @@ func (s *SnapshotServiceImpl) DeleteSnapshot(ctx context.Context, input *ec2.Del
 	}
 
 	// Check if any registered AMI names this snapshot as its root device
-	amiInUse, err := s.amiInUseBySnapshot(ctx, snapshotID)
-	if err != nil {
-		slog.ErrorContext(ctx, "DeleteSnapshot failed to check AMI usage", "snapshotId", snapshotID, "err", err)
-		return nil, errors.New(awserrors.ErrorServerInternal)
-	}
+	amiInUse, amiErr := s.amiInUseBySnapshot(ctx, snapshotID, accountID)
 	if amiInUse {
 		slog.InfoContext(ctx, "DeleteSnapshot blocked: snapshot in use by registered AMI", "snapshotId", snapshotID)
 		return nil, errors.New(awserrors.ErrorInvalidSnapshotInUse)
 	}
+	if volumeErr != nil {
+		slog.ErrorContext(ctx, "DeleteSnapshot failed to check snapshot usage", "snapshotId", snapshotID, "err", volumeErr)
+		return nil, volumeErr
+	}
+	if amiErr != nil {
+		slog.ErrorContext(ctx, "DeleteSnapshot failed to check AMI usage", "snapshotId", snapshotID, "err", amiErr)
+		return nil, amiErr
+	}
 	if s.provider == nil {
 		slog.ErrorContext(ctx, "no EBS provider configured", "op", "DeleteSnapshot")
 		return nil, errors.New(awserrors.ErrorServerInternal)
 	}
 	if err := s.provider.DeleteSnapshot(ctx, ebsprovider.DeleteSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: snapshotID, Handle: cfg.ProviderHandle}); err != nil {
-		return nil, errors.New(awserrors.ErrorServerInternal)
+		slog.ErrorContext(ctx, "DeleteSnapshot provider deletion failed", "snapshotId", snapshotID, "err", err)
+		return nil, providerDeleteError(snapshotID, err)
 	}
 
 	objects, _, err := objectstore.ListAll(ctx, s.store, &s3.ListObjectsV2Input{

spinifex/domains/ec2/snapshot/service_impl_test.go

@@ -602,42 +602,6 @@ func TestDeleteSnapshot_InUseByRegisteredAMI_AnotherAccount(t *testing.T) {
 	assert.Contains(t, err.Error(), awserrors.ErrorInvalidSnapshotInUse)
 }
 
-// TestDeleteSnapshot_AMIReadFailureFailsClosed tests that an AMI document
-// this cannot decode aborts the delete rather than being treated as absent:
-// a partial AMI listing is not evidence that no AMI names the snapshot.
-func TestDeleteSnapshot_AMIReadFailureFailsClosed(t *testing.T) {
-	svc, store := setupTestSnapshotService(t)
-
-	createTestVolume(t, svc, store, "vol-source", 50)
-	snap, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
-		VolumeId: aws.String("vol-source"),
-	}, testAccountID)
-	require.NoError(t, err)
-
-	key, err := ebsmetadata.AMIKey("ami-corrupt")
-	require.NoError(t, err)
-	_, err = store.PutObject(context.Background(), &s3.PutObjectInput{
-		Bucket: aws.String("test-bucket"),
-		Key:    aws.String(key),
-		Body:   strings.NewReader("not-json"),
-	})
-	require.NoError(t, err)
-
-	_, err = svc.DeleteSnapshot(context.Background(), &ec2.DeleteSnapshotInput{
-		SnapshotId: snap.SnapshotId,
-	}, testAccountID)
-	require.Error(t, err)
-	assert.Contains(t, err.Error(), awserrors.ErrorServerInternal)
-
-	// Nothing was deleted: the snapshot survives in both the provider and the
-	// control-plane document.
-	result, err := svc.DescribeSnapshots(context.Background(), &ec2.DescribeSnapshotsInput{
-		SnapshotIds: []*string{snap.SnapshotId},
-	}, testAccountID)
-	require.NoError(t, err)
-	assert.Len(t, result.Snapshots, 1)
-}
-
 // TestDeleteSnapshot_NotFound tests deleting a non-existent snapshot.
 func TestDeleteSnapshot_NotFound(t *testing.T) {
 	svc, _ := setupTestSnapshotService(t)

spinifex/domains/ec2/vpc/service_impl.go

@@ -91,6 +91,7 @@ type VPCServiceImpl struct {
 	defaultsKV jetstream.KeyValue // one default-resource claim per account
 	rtbKV      jetstream.KeyValue // route table bucket for auto-creating main route table
 	igwKV      jetstream.KeyValue // internet gateway bucket for the DeleteVpc dependency check
+	eigwKV     jetstream.KeyValue // egress-only internet gateway bucket for the DeleteVpc dependency check
 	ipam       *IPAM
 
 	// Optional: injected after construction for public IP cleanup in DeleteNetworkInterface.
@@ -224,6 +225,10 @@ func NewVPCServiceImplWithNATS(ctx context.Context, cfg *config.Config, natsConn
 	if err != nil {
 		return nil, fmt.Errorf("failed to create KV bucket spinifex-igw: %w", err)
 	}
+	eigwKV, err := kvutil.GetOrCreateBucket(ctx, js, "spinifex-eigw", 10)
+	if err != nil {
+		return nil, fmt.Errorf("failed to create KV bucket spinifex-eigw: %w", err)
+	}
 
 	ipam, err := NewIPAM(ctx, js)
 	if err != nil {
@@ -248,6 +253,7 @@ func NewVPCServiceImplWithNATS(ctx context.Context, cfg *config.Config, natsConn
 		defaultsKV: defaultsKV,
 		rtbKV:      rtbKV,
 		igwKV:      igwKV,
+		eigwKV:     eigwKV,
 		ipam:       ipam,
 	}, nil
 }
@@ -492,10 +498,13 @@ func (s *VPCServiceImpl) DeleteVpc(ctx context.Context, input *ec2.DeleteVpcInpu
 		}
 	}
 
-	// Reject if an internet gateway still names this VPC, as AWS does. Without
-	// this the gateway outlives the only VPC id that can detach it, and it can
-	// then never be detached or deleted.
-	if err := s.rejectAttachedIGW(ctx, accountID, vpcID); err != nil {
+	// Reject if an internet gateway or egress-only internet gateway still names
+	// this VPC, as AWS does. Without this the gateway outlives the only VPC id
+	// that can detach it, and it can then never be detached or deleted.
+	if err := rejectAttachedGateway(ctx, s.igwKV, accountID, vpcID); err != nil {
+		return nil, err
+	}
+	if err := rejectAttachedGateway(ctx, s.eigwKV, accountID, vpcID); err != nil {
 		return nil, err
 	}
 
@@ -538,22 +547,22 @@ func (s *VPCServiceImpl) DeleteVpc(ctx context.Context, input *ec2.DeleteVpcInpu
 	return &ec2.DeleteVpcOutput{}, nil
 }
 
-// rejectAttachedIGW returns DependencyViolation when an internet gateway record
+// rejectAttachedGateway returns DependencyViolation when a gateway record in kv
 // still names vpcID. It reads the raw record rather than a describe projection,
 // because an attachment awaiting confirmation is hidden from the AWS surface but
 // still blocks the delete.
-func (s *VPCServiceImpl) rejectAttachedIGW(ctx context.Context, accountID, vpcID string) error {
-	if s.igwKV == nil {
+func rejectAttachedGateway(ctx context.Context, kv jetstream.KeyValue, accountID, vpcID string) error {
+	if kv == nil {
 		return nil
 	}
 
 	prefix := accountID + "."
-	keys, err := s.igwKV.Keys(ctx)
+	keys, err := kv.Keys(ctx)
 	if err != nil {
 		if errors.Is(err, jetstream.ErrNoKeysFound) {
 			return nil
 		}
-		slog.WarnContext(ctx, "DeleteVpc: IGW key listing failed", "vpcId", vpcID, "err", err)
+		slog.WarnContext(ctx, "DeleteVpc: gateway key listing failed", "bucket", kv.Bucket(), "vpcId", vpcID, "err", err)
 		return errors.New(awserrors.ErrorServerInternal)
 	}
 
@@ -561,7 +570,7 @@ func (s *VPCServiceImpl) rejectAttachedIGW(ctx context.Context, accountID, vpcID
 		if k == kvutil.VersionKey || !strings.HasPrefix(k, prefix) {
 			continue
 		}
-		entry, err := s.igwKV.Get(ctx, k)
+		entry, err := kv.Get(ctx, k)
 		if err != nil {
 			// Deleted between Keys() and Get() is fine to skip. Any other read
 			// error is fail-closed: an attachment this cannot see must not let
@@ -569,19 +578,18 @@ func (s *VPCServiceImpl) rejectAttachedIGW(ctx context.Context, accountID, vpcID
 			if errors.Is(err, jetstream.ErrKeyNotFound) {
 				continue
 			}
-			slog.WarnContext(ctx, "DeleteVpc: IGW read failed", "key", k, "err", err)
+			slog.WarnContext(ctx, "DeleteVpc: gateway read failed", "bucket", kv.Bucket(), "key", k, "err", err)
 			return errors.New(awserrors.ErrorServerInternal)
 		}
-		var igw struct {
-			InternetGatewayId string `json:"internet_gateway_id"`
-			VpcId             string `json:"vpc_id"`
+		var gw struct {
+			VpcId string `json:"vpc_id"`
 		}
-		if err := json.Unmarshal(entry.Value(), &igw); err != nil {
-			slog.WarnContext(ctx, "DeleteVpc: IGW unmarshal failed", "key", k, "err", err)
+		if err := json.Unmarshal(entry.Value(), &gw); err != nil {
+			slog.WarnContext(ctx, "DeleteVpc: gateway unmarshal failed", "bucket", kv.Bucket(), "key", k, "err", err)
 			return errors.New(awserrors.ErrorServerInternal)
 		}
-		if igw.VpcId == vpcID {
-			slog.WarnContext(ctx, "DeleteVpc: VPC has an attached internet gateway", "vpcId", vpcID, "internetGatewayId", igw.InternetGatewayId)
+		if gw.VpcId == vpcID {
+			slog.WarnContext(ctx, "DeleteVpc: VPC has an attached gateway", "bucket", kv.Bucket(), "vpcId", vpcID, "key", k)
 			return awserrors.HasDependencies("vpc", vpcID)
 		}
 	}
@@ -896,7 +904,8 @@ func (s *VPCServiceImpl) checkSubnetResidents(ctx context.Context, accountID, su
 		}
 		var record ENIRecord
 		if err := json.Unmarshal(entry.Value(), &record); err != nil {
-			continue
+			slog.ErrorContext(ctx, "DeleteSubnet: ENI unmarshal failed, blocking delete to avoid orphaning ports", "subnetId", subnetID, "key", key, "err", err)
+			return errors.New(awserrors.ErrorServerInternal)
 		}
 		if record.SubnetId == subnetID && eniIsLiveAttachment(&record) {
 			return awserrors.HasDependencies("subnet", subnetID)

spinifex/domains/ec2/vpc/service_impl_test.go

@@ -864,6 +864,26 @@ func TestDeleteVpc_RejectsAttachedInternetGateway(t *testing.T) {
 	require.Len(t, desc.Vpcs, 1, "VPC must persist so the caller can detach the gateway and retry")
 }
 
+// AWS refuses DeleteVpc while an egress-only internet gateway names the VPC;
+// deleting it anyway leaves the gateway pointing at a VPC that no longer exists.
+func TestDeleteVpc_RejectsEgressOnlyInternetGateway(t *testing.T) {
+	t.Parallel()
+	svc := setupTestVPCService(t)
+	vpcID := createTestVPC(t, svc, "10.66.0.0/16")
+
+	_, err := svc.eigwKV.Put(t.Context(), testAccountID+".eigw-attached",
+		[]byte(`{"egress_only_internet_gateway_id":"eigw-attached","vpc_id":"`+vpcID+`","state":"attached"}`))
+	require.NoError(t, err)
+
+	_, err = svc.DeleteVpc(context.Background(), &ec2.DeleteVpcInput{VpcId: aws.String(vpcID)}, testAccountID)
+	require.Error(t, err)
+	requireAWSError(t, err, awserrors.ErrorDependencyViolation, "The vpc '"+vpcID+"' has dependencies and cannot be deleted.")
+
+	desc, err := svc.DescribeVpcs(context.Background(), &ec2.DescribeVpcsInput{VpcIds: []*string{aws.String(vpcID)}}, testAccountID)
+	require.NoError(t, err)
+	require.Len(t, desc.Vpcs, 1, "VPC must persist so the caller can delete the gateway and retry")
+}
+
 // A gateway attached to some other VPC must not block this one.
 func TestDeleteVpc_IgnoresInternetGatewayOnAnotherVPC(t *testing.T) {
 	t.Parallel()
@@ -994,6 +1014,28 @@ func TestDeleteSubnet_RouteTableCheckFailsClosedOnCorruptRTB(t *testing.T) {
 	require.Len(t, desc.Subnets, 1, "subnet must persist so a KV read failure never strands an association unseen")
 }
 
+// TestDeleteSubnet_ResidentCheckFailsClosedOnCorruptENI asserts an ENI
+// record that cannot be decoded blocks DeleteSubnet, since it may be a live
+// port in this subnet.
+func TestDeleteSubnet_ResidentCheckFailsClosedOnCorruptENI(t *testing.T) {
+	t.Parallel()
+	svc := setupTestVPCService(t)
+	vpcID := createTestVPC(t, svc, "10.66.0.0/16")
+	subnetID := createTestSubnet(t, svc, vpcID, "10.66.1.0/24")
+	eniID := createTestENI(t, svc, subnetID)
+
+	_, err := svc.eniKV.Put(t.Context(), testAccountID+"."+eniID, []byte("{not json"))
+	require.NoError(t, err)
+
+	_, err = svc.DeleteSubnet(context.Background(), &ec2.DeleteSubnetInput{SubnetId: aws.String(subnetID)}, testAccountID)
+	require.Error(t, err)
+	assert.ErrorContains(t, err, "ServerInternal")
+
+	desc, err := svc.DescribeSubnets(context.Background(), &ec2.DescribeSubnetsInput{SubnetIds: []*string{aws.String(subnetID)}}, testAccountID)
+	require.NoError(t, err)
+	require.Len(t, desc.Subnets, 1)
+}
+
 // TestEnsureDefaultVPC_NoVpcdResponder simulates the daemon-startup race where
 // EnsureDefaultVPC runs before vpcd has subscribed. The SG step is best-effort,
 // so subnet and RTB must still land in KV when vpcd is absent.

spinifex/domains/ecr/awsapi/repository.go

@@ -29,10 +29,7 @@ func (e RepositoryEndpoint) RegistryURIHost(accountID string) string {
 	if host == "" {
 		host = accountID + ".dkr.ecr." + e.Region + "." + e.ServicesDomain
 	}
-	if e.RegistryPort != "" && e.RegistryPort != "443" {
-		return host + ":" + e.RegistryPort
-	}
-	return host
+	return ecr.RegistryURIHost(host, e.RegistryPort)
 }
 
 // RepositoryURI returns the pull/push URI for an account-scoped repository.

spinifex/domains/ecr/meta.go

@@ -78,6 +78,31 @@ type ManifestMeta struct {
 	Size         int64     `json:"size"`
 	PushedAt     time.Time `json:"pushedAt"`
 	ChildDigests []string  `json:"childDigests,omitempty"`
+	// SubjectDigest and ArtifactType are set only for manifests that declare a
+	// subject, and feed the OCI referrers listing for that subject.
+	SubjectDigest string            `json:"subjectDigest,omitempty"`
+	ArtifactType  string            `json:"artifactType,omitempty"`
+	Annotations   map[string]string `json:"annotations,omitempty"`
+}
+
+// ReferrerDescriptor is one entry of a subject's referrers listing: the OCI
+// descriptor of a manifest that names the subject.
+type ReferrerDescriptor struct {
+	MediaType    string            `json:"mediaType"`
+	Digest       string            `json:"digest"`
+	Size         int64             `json:"size"`
+	ArtifactType string            `json:"artifactType,omitempty"`
+	Annotations  map[string]string `json:"annotations,omitempty"`
+}
+
+func (m ManifestMeta) referrerDescriptor() ReferrerDescriptor {
+	return ReferrerDescriptor{
+		MediaType:    m.MediaType,
+		Digest:       m.Digest,
+		Size:         m.Size,
+		ArtifactType: m.ArtifactType,
+		Annotations:  m.Annotations,
+	}
 }
 
 // UploadState tracks an in-progress chunked blob upload. The sha256 hash state
@@ -124,6 +149,10 @@ type MetaStore interface {
 	GetManifestMeta(ctx context.Context, accountID, repo, digest string) (ManifestMeta, error)
 	DeleteManifestMeta(ctx context.Context, accountID, repo, digest string) error
 
+	// ListReferrers returns the manifests in repo that declare subjectDigest as
+	// their subject, ordered by digest. A subject with none yields an empty slice.
+	ListReferrers(ctx context.Context, accountID, repo, subjectDigest string) ([]ReferrerDescriptor, error)
+
 	PutUpload(ctx context.Context, accountID, uploadID string, state UploadState) (uint64, error)
 	GetUpload(ctx context.Context, accountID, uploadID string) (UploadState, uint64, error)
 	UpdateUpload(ctx context.Context, accountID, uploadID string, state UploadState, rev uint64) (uint64, error)
@@ -234,11 +263,11 @@ func (s *KVMetaStore) DeleteRepo(ctx context.Context, accountID, repo string) er
 		}
 		return err
 	}
-	tagsPrefix, manifestsPrefix := KVTagsPrefix(repo), KVManifestsPrefix(repo)
+	tagsPrefix, manifestsPrefix, referrersPrefix := KVTagsPrefix(repo), KVManifestsPrefix(repo), KVReferrersPrefix(repo)
 	metaKey, policyKey, lifecycleKey := KVRepoMetaKey(repo), KVRepoPolicyKey(repo), KVLifecyclePolicyKey(repo)
 	for _, k := range keys {
 		if k == metaKey || k == policyKey || k == lifecycleKey ||
-			strings.HasPrefix(k, tagsPrefix) || strings.HasPrefix(k, manifestsPrefix) {
+			strings.HasPrefix(k, tagsPrefix) || strings.HasPrefix(k, manifestsPrefix) || strings.HasPrefix(k, referrersPrefix) {
 			if err := kv.Delete(ctx, k); err != nil {
 				return mapKVErr(err)
 			}
@@ -415,10 +444,55 @@ func (s *KVMetaStore) PutManifestMeta(ctx context.Context, accountID, repo strin
 	if err != nil {
 		return err
 	}
-	_, err = kv.Put(ctx, KVManifestKey(repo, meta.Digest), data)
+	if _, err = kv.Put(ctx, KVManifestKey(repo, meta.Digest), data); err != nil {
+		return err
+	}
+	if meta.SubjectDigest == "" {
+		return nil
+	}
+	entry, err := json.Marshal(meta.referrerDescriptor())
+	if err != nil {
+		return err
+	}
+	_, err = kv.Put(ctx, KVReferrerKey(repo, meta.SubjectDigest, meta.Digest), entry)
 	return err
 }
 
+func (s *KVMetaStore) ListReferrers(ctx context.Context, accountID, repo, subjectDigest string) ([]ReferrerDescriptor, error) {
+	kv, err := s.bucket(ctx, accountID)
+	if err != nil {
+		return nil, err
+	}
+	keys, err := kv.Keys(ctx)
+	if err != nil {
+		if errors.Is(err, jetstream.ErrNoKeysFound) {
+			return []ReferrerDescriptor{}, nil
+		}
+		return nil, err
+	}
+	prefix := KVSubjectReferrersPrefix(repo, subjectDigest)
+	out := []ReferrerDescriptor{}
+	for _, k := range keys {
+		if !strings.HasPrefix(k, prefix) {
+			continue
+		}
+		entry, err := kv.Get(ctx, k)
+		if err != nil {
+			if errors.Is(err, jetstream.ErrKeyNotFound) {
+				continue
+			}
+			return nil, err
+		}
+		var d ReferrerDescriptor
+		if err := json.Unmarshal(entry.Value(), &d); err != nil {
+			return nil, err
+		}
+		out = append(out, d)
+	}
+	sort.Slice(out, func(i, j int) bool { return out[i].Digest < out[j].Digest })
+	return out, nil
+}
+
 func (s *KVMetaStore) GetManifestMeta(ctx context.Context, accountID, repo, digest string) (ManifestMeta, error) {
 	kv, err := s.bucket(ctx, accountID)
 	if err != nil {
@@ -440,9 +514,19 @@ func (s *KVMetaStore) DeleteManifestMeta(ctx context.Context, accountID, repo, d
 	if err != nil {
 		return err
 	}
-	if _, err := kv.Get(ctx, KVManifestKey(repo, digest)); err != nil {
+	entry, err := kv.Get(ctx, KVManifestKey(repo, digest))
+	if err != nil {
 		return mapKVErr(err)
 	}
+	var m ManifestMeta
+	if err := json.Unmarshal(entry.Value(), &m); err != nil {
+		return err
+	}
+	if m.SubjectDigest != "" {
+		if err := kv.Delete(ctx, KVReferrerKey(repo, m.SubjectDigest, digest)); err != nil {
+			return mapKVErr(err)
+		}
+	}
 	return kv.Delete(ctx, KVManifestKey(repo, digest))
 }
 
@@ -738,6 +822,19 @@ func (m *MemoryMetaStore) DeleteManifestMeta(_ context.Context, accountID, repo,
 	return nil
 }
 
+func (m *MemoryMetaStore) ListReferrers(_ context.Context, accountID, repo, subjectDigest string) ([]ReferrerDescriptor, error) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	out := []ReferrerDescriptor{}
+	for k, meta := range m.manifests[accountID] {
+		if r, _, ok := strings.Cut(k, "|"); ok && r == repo && meta.SubjectDigest == subjectDigest {
+			out = append(out, meta.referrerDescriptor())
+		}
+	}
+	sort.Slice(out, func(i, j int) bool { return out[i].Digest < out[j].Digest })
+	return out, nil
+}
+
 func (m *MemoryMetaStore) PutUpload(_ context.Context, accountID, uploadID string, state UploadState) (uint64, error) {
 	m.mu.Lock()
 	defer m.mu.Unlock()

spinifex/domains/ecr/meta_nats.go

@@ -184,6 +184,14 @@ func (s *NATSMetaStore) DeleteManifestMeta(ctx context.Context, accountID, repo,
 	return nil
 }
 
+func (s *NATSMetaStore) ListReferrers(ctx context.Context, accountID, repo, subjectDigest string) ([]ReferrerDescriptor, error) {
+	resp, err := natsmsg.NATSRequest[ReferrerListResponse](ctx, s.conn, SubjectReferrerList, &ReferrerListRequest{Repo: repo, Subject: subjectDigest}, metaRequestTimeout, accountID)
+	if err != nil {
+		return nil, err
+	}
+	return resp.Referrers, nil
+}
+
 func (s *NATSMetaStore) PutUpload(ctx context.Context, accountID, uploadID string, state UploadState) (uint64, error) {
 	resp, err := natsmsg.NATSRequest[UploadCreateResponse](ctx, s.conn, SubjectUploadCreate, &UploadCreateRequest{UploadID: uploadID, State: state}, metaRequestTimeout, accountID)
 	if err != nil {

spinifex/domains/ecr/meta_test.go

@@ -223,3 +223,12 @@ func TestMemoryMetaStore_LifecyclePolicy(t *testing.T) {
 	_, err = m.GetLifecyclePolicy(context.Background(), acct, "team/app")
 	assert.ErrorIs(t, err, ErrNotFound)
 }
+
+func TestManifestMeta_DecodesRecordWithoutReferrerFields(t *testing.T) {
+	var m ManifestMeta
+	require.NoError(t, json.Unmarshal([]byte(`{"digest":"sha256:ccc","mediaType":"application/vnd.oci.image.manifest.v1+json","size":10,"pushedAt":"2026-01-01T00:00:00Z","childDigests":["sha256:ddd"]}`), &m))
+	assert.Equal(t, "sha256:ccc", m.Digest)
+	assert.Equal(t, []string{"sha256:ddd"}, m.ChildDigests)
+	assert.Empty(t, m.SubjectDigest)
+	assert.Empty(t, m.ArtifactType)
+}

spinifex/domains/ecr/referrers_test.go

@@ -0,0 +1,94 @@
+package ecr
+
+import (
+	"context"
+	"testing"
+
+	"github.com/mulgadc/spinifex/internal/testkit"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+const (
+	refSubject = "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+	refSigA    = "sha256:1111111111111111111111111111111111111111111111111111111111111111"
+	refSigB    = "sha256:2222222222222222222222222222222222222222222222222222222222222222"
+)
+
+func referrerMeta(digest, subject string) ManifestMeta {
+	return ManifestMeta{
+		Digest:        digest,
+		MediaType:     "application/vnd.oci.image.manifest.v1+json",
+		Size:          42,
+		SubjectDigest: subject,
+		ArtifactType:  "application/vnd.example.sig.v1",
+		Annotations:   map[string]string{"org.example.key": "v"},
+	}
+}
+
+// referrerStores runs a test against every MetaStore implementation that keeps
+// its own referrers state.
+func referrerStores(t *testing.T) map[string]MetaStore {
+	t.Helper()
+	_, nc, _ := testutil.StartTestJetStream(t)
+	return map[string]MetaStore{
+		"memory": NewMemoryMetaStore(),
+		"kv":     NewKVMetaStore(testutil.NewJetStream(t, nc)),
+	}
+}
+
+func TestListReferrers(t *testing.T) {
+	ctx := context.Background()
+	for name, store := range referrerStores(t) {
+		t.Run(name, func(t *testing.T) {
+			acct, repo := "111111111111", "team/app"
+
+			got, err := store.ListReferrers(ctx, acct, repo, refSubject)
+			require.NoError(t, err)
+			assert.Empty(t, got, "a subject with no referrers lists empty")
+
+			require.NoError(t, store.PutManifestMeta(ctx, acct, repo, referrerMeta(refSigB, refSubject)))
+			require.NoError(t, store.PutManifestMeta(ctx, acct, repo, referrerMeta(refSigA, refSubject)))
+			// Re-putting the same digest must not duplicate the entry.
+			require.NoError(t, store.PutManifestMeta(ctx, acct, repo, referrerMeta(refSigA, refSubject)))
+
+			got, err = store.ListReferrers(ctx, acct, repo, refSubject)
+			require.NoError(t, err)
+			require.Len(t, got, 2)
+			assert.Equal(t, refSigA, got[0].Digest, "entries are ordered by digest")
+			assert.Equal(t, refSigB, got[1].Digest)
+			assert.Equal(t, ReferrerDescriptor{
+				MediaType:    "application/vnd.oci.image.manifest.v1+json",
+				Digest:       refSigA,
+				Size:         42,
+				ArtifactType: "application/vnd.example.sig.v1",
+				Annotations:  map[string]string{"org.example.key": "v"},
+			}, got[0])
+
+			otherRepo, err := store.ListReferrers(ctx, acct, "team/other", refSubject)
+			require.NoError(t, err)
+			assert.Empty(t, otherRepo, "referrers do not cross repositories")
+			otherAcct, err := store.ListReferrers(ctx, "222222222222", repo, refSubject)
+			require.NoError(t, err)
+			assert.Empty(t, otherAcct, "referrers do not cross accounts")
+
+			require.NoError(t, store.DeleteManifestMeta(ctx, acct, repo, refSigA))
+			got, err = store.ListReferrers(ctx, acct, repo, refSubject)
+			require.NoError(t, err)
+			require.Len(t, got, 1)
+			assert.Equal(t, refSigB, got[0].Digest)
+		})
+	}
+}
+
+func TestListReferrers_ManifestWithoutSubjectIsNotListed(t *testing.T) {
+	ctx := context.Background()
+	for name, store := range referrerStores(t) {
+		t.Run(name, func(t *testing.T) {
+			require.NoError(t, store.PutManifestMeta(ctx, "111111111111", "r", referrerMeta(refSigA, "")))
+			got, err := store.ListReferrers(ctx, "111111111111", "r", refSubject)
+			require.NoError(t, err)
+			assert.Empty(t, got)
+		})
+	}
+}

spinifex/domains/ecr/registry/authorization.go

@@ -99,6 +99,8 @@ func ClassifyOperation(method, urlPath string, query url.Values) (ClassifiedOper
 		return classifyBlobOperation(method, name, ref, query)
 	case "manifests":
 		return classifyManifestOperation(method, name, ref)
+	case "referrers":
+		return classifyReferrersOperation(method, name, ref)
 	default:
 		return ClassifiedOperation{}, false
 	}
@@ -165,6 +167,15 @@ func classifyBlobOperation(method, name, ref string, query url.Values) (Classifi
 	}
 }
 
+// classifyReferrersOperation authorizes the referrers listing as a read of the
+// repository's images.
+func classifyReferrersOperation(method, name, subject string) (ClassifiedOperation, bool) {
+	if method != http.MethodGet || subject == "" {
+		return ClassifiedOperation{}, false
+	}
+	return ClassifiedOperation{Repo: name, Requirements: []ActionRequirement{{ActionBatchGetImage, ScopeDestination}}}, true
+}
+
 func classifyManifestOperation(method, name, reference string) (ClassifiedOperation, bool) {
 	if reference == "" {
 		return ClassifiedOperation{}, false

spinifex/domains/ecr/registry/authorization_test.go

@@ -132,6 +132,15 @@ func TestClassifyOperation_RouteMatrix(t *testing.T) {
 				Requirements: []ActionRequirement{{ActionBatchDeleteImage, ScopeDestination}},
 			},
 		},
+		{
+			name:   "GET referrers is a read of the repository images",
+			method: http.MethodGet,
+			path:   "/v2/team/app/referrers/sha256:abc",
+			want: ClassifiedOperation{
+				Repo:         "team/app",
+				Requirements: []ActionRequirement{{ActionBatchGetImage, ScopeDestination}},
+			},
+		},
 		{
 			name:   "cross-repo mount requires destination initiate + source read",
 			method: http.MethodPost,
@@ -205,6 +214,9 @@ func TestClassifyOperation_UnsupportedNeverClassifies(t *testing.T) {
 		{"manifest POST unsupported", http.MethodPost, "/v2/team/app/manifests/latest"},
 		{"manifest empty reference", http.MethodGet, "/v2/team/app/manifests/"},
 		{"catalog wrong method", http.MethodPost, "/v2/_catalog"},
+		{"referrers PUT unsupported", http.MethodPut, "/v2/team/app/referrers/sha256:abc"},
+		{"referrers DELETE unsupported", http.MethodDelete, "/v2/team/app/referrers/sha256:abc"},
+		{"referrers empty subject", http.MethodGet, "/v2/team/app/referrers/"},
 	}
 	for _, c := range cases {
 		t.Run(c.name, func(t *testing.T) {

spinifex/domains/ecr/registry/registry.go

@@ -142,6 +142,8 @@ func (reg *Registry) serve(w http.ResponseWriter, r *http.Request) {
 		reg.routeManifests(w, r, name, ref)
 	case "tags":
 		reg.routeTags(w, r, name, ref)
+	case "referrers":
+		reg.getReferrers(w, r, name, ref)
 	default:
 		WriteError(w, http.StatusNotFound, "NAME_UNKNOWN", "unrecognized registry path")
 	}
@@ -151,7 +153,7 @@ func (reg *Registry) serve(w http.ResponseWriter, r *http.Request) {
 // kind is one of blobs/manifests/tags. ref is the remainder (digest, reference,
 // "uploads", "uploads/{uuid}", or "list"). ok is false if no marker is found.
 func splitV2Path(path string) (name, kind, ref string, ok bool) {
-	for _, marker := range []string{"/blobs/", "/manifests/", "/tags/"} {
+	for _, marker := range []string{"/blobs/", "/manifests/", "/tags/", "/referrers/"} {
 		if before, after, found := strings.Cut(path, marker); found {
 			kind = strings.Trim(marker, "/")
 			return before, kind, after, before != ""
@@ -590,7 +592,7 @@ func (reg *Registry) putManifest(w http.ResponseWriter, r *http.Request, name, r
 		return
 	}
 
-	digest, err := reg.StoreManifest(ctx, reg.AccountID, name, reference, r.Header.Get("Content-Type"), body)
+	digest, subject, err := reg.storeManifest(ctx, reg.AccountID, name, reference, r.Header.Get("Content-Type"), body)
 	if err != nil {
 		if mErr, ok := errors.AsType[*ManifestStoreError](err); ok {
 			WriteError(w, mErr.Status, mErr.Code, mErr.Msg)
@@ -602,6 +604,9 @@ func (reg *Registry) putManifest(w http.ResponseWriter, r *http.Request, name, r
 
 	w.Header().Set("Docker-Content-Digest", digest)
 	w.Header().Set("Location", fmt.Sprintf("/v2/%s/manifests/%s", name, digest))
+	if subject != "" {
+		w.Header().Set("Oci-Subject", subject)
+	}
 	w.WriteHeader(http.StatusCreated)
 }
 
@@ -635,9 +640,15 @@ func (reg *Registry) deleteManifest(ctx context.Context, w http.ResponseWriter,
 
 // manifestDoc is the subset of an OCI image manifest the validator inspects.
 type manifestDoc struct {
-	MediaType string `json:"mediaType"`
-	Config    struct {
+	MediaType    string            `json:"mediaType"`
+	ArtifactType string            `json:"artifactType"`
+	Annotations  map[string]string `json:"annotations"`
+	Subject      *struct {
 		Digest string `json:"digest"`
+	} `json:"subject"`
+	Config struct {
+		MediaType string `json:"mediaType"`
+		Digest    string `json:"digest"`
 	} `json:"config"`
 	Layers []struct {
 		Digest string `json:"digest"`
@@ -648,6 +659,30 @@ type manifestDoc struct {
 	} `json:"manifests"`
 }
 
+// referrerFields returns the subject digest, artifactType and annotations of a
+// manifest that declares a subject. mediaType is the type it is stored under;
+// an image manifest without an artifactType uses its config mediaType.
+func referrerFields(mediaType string, body []byte) (subject, artifactType string, annotations map[string]string, err error) {
+	var doc manifestDoc
+	if err := json.Unmarshal(body, &doc); err != nil {
+		return "", "", nil, fmt.Errorf("manifest is not valid JSON: %w", err)
+	}
+	if doc.Subject == nil {
+		return "", "", nil, nil
+	}
+	if !ecr.ValidateDigest(doc.Subject.Digest) {
+		return "", "", nil, errors.New("subject references malformed digest")
+	}
+	if mediaType == "" {
+		mediaType = doc.MediaType
+	}
+	artifactType = doc.ArtifactType
+	if artifactType == "" && mediaType != mediaTypeOCIIndex && mediaType != mediaTypeDockerManifestList {
+		artifactType = doc.Config.MediaType
+	}
+	return doc.Subject.Digest, artifactType, doc.Annotations, nil
+}
+
 // validateManifest performs type-aware shallow validation, returning the child
 // digests recorded in the manifest metadata. For an image manifest every config
 // and layer blob must exist; for an index every child manifest must exist with a
@@ -697,6 +732,57 @@ func (reg *Registry) validateManifest(ctx context.Context, name, contentType str
 	}
 }
 
+// ---- referrers ----
+
+// referrersIndex is the OCI image index the referrers API answers with.
+type referrersIndex struct {
+	SchemaVersion int                      `json:"schemaVersion"`
+	MediaType     string                   `json:"mediaType"`
+	Manifests     []ecr.ReferrerDescriptor `json:"manifests"`
+}
+
+// getReferrers answers GET /v2/<name>/referrers/<digest> with an index of the
+// manifests that name the digest as their subject. A subject with none, or one
+// that is not stored, yields an empty index.
+func (reg *Registry) getReferrers(w http.ResponseWriter, r *http.Request, name, subject string) {
+	ctx := r.Context()
+	if !ecr.ValidateDigest(subject) {
+		WriteError(w, http.StatusBadRequest, "DIGEST_INVALID", "malformed subject digest")
+		return
+	}
+	if _, err := reg.Meta.GetRepo(ctx, reg.AccountID, name); err != nil {
+		if errors.Is(err, ecr.ErrNotFound) {
+			WriteError(w, http.StatusNotFound, "NAME_UNKNOWN", "repository unknown")
+			return
+		}
+		reg.internal(ctx, w, "get repository", err)
+		return
+	}
+	referrers, err := reg.Meta.ListReferrers(ctx, reg.AccountID, name, subject)
+	if err != nil {
+		reg.internal(ctx, w, "list referrers", err)
+		return
+	}
+	if want := r.URL.Query().Get("artifactType"); want != "" {
+		referrers = slices.DeleteFunc(referrers, func(d ecr.ReferrerDescriptor) bool { return d.ArtifactType != want })
+		w.Header().Set("Oci-Filters-Applied", "artifactType")
+	}
+	if referrers == nil {
+		referrers = []ecr.ReferrerDescriptor{}
+	}
+	out, err := json.Marshal(referrersIndex{SchemaVersion: 2, MediaType: mediaTypeOCIIndex, Manifests: referrers})
+	if err != nil {
+		reg.internal(ctx, w, "encode referrers", err)
+		return
+	}
+	w.Header().Set("Content-Type", mediaTypeOCIIndex)
+	w.Header().Set("Content-Length", strconv.Itoa(len(out)))
+	w.WriteHeader(http.StatusOK)
+	if _, err := w.Write(out); err != nil {
+		slog.ErrorContext(ctx, "ECR/OCI: referrers write failed", "err", err)
+	}
+}
+
 // ---- tags ----
 
 func (reg *Registry) routeTags(w http.ResponseWriter, r *http.Request, name, ref string) {

spinifex/domains/ecr/registry/registry_image.go

@@ -6,6 +6,7 @@ import (
 	"errors"
 	"io"
 	"log/slog"
+	"mime"
 	"net/http"
 	"slices"
 	"strings"
@@ -52,20 +53,29 @@ type ImageRecord struct {
 // case it is detected. On a validation/size failure it returns a
 // *ManifestStoreError carrying the OCI code.
 func (reg *Registry) StoreManifest(ctx context.Context, account, repo, ref, contentType string, body []byte) (string, error) {
+	digest, _, err := reg.storeManifest(ctx, account, repo, ref, contentType, body)
+	return digest, err
+}
+
+// storeManifest is StoreManifest that also returns the subject digest the
+// manifest declares ("" when it has none).
+func (reg *Registry) storeManifest(ctx context.Context, account, repo, ref, contentType string, body []byte) (string, string, error) {
 	scoped := reg.forAccount(account)
 	if err := scoped.requireRepo(ctx, repo); err != nil {
-		return "", err
+		return "", "", err
 	}
 	if len(body) > maxManifestBytes {
-		return "", &ManifestStoreError{http.StatusRequestEntityTooLarge, "MANIFEST_INVALID", "manifest too large"}
+		return "", "", &ManifestStoreError{http.StatusRequestEntityTooLarge, "MANIFEST_INVALID", "manifest too large"}
 	}
 	if contentType == "" {
 		contentType = detectManifestType(body)
+	} else if mt, _, err := mime.ParseMediaType(contentType); err == nil {
+		contentType = mt
 	}
 
 	digest := "sha256:" + hex.EncodeToString(sha256Sum(body))
 	if ecr.ValidateDigest(ref) && ref != digest {
-		return "", &ManifestStoreError{http.StatusBadRequest, "DIGEST_INVALID", "reference digest does not match content"}
+		return "", "", &ManifestStoreError{http.StatusBadRequest, "DIGEST_INVALID", "reference digest does not match content"}
 	}
 
 	// On an IMMUTABLE repo, re-tagging an existing tag onto a different digest is
@@ -73,22 +83,27 @@ func (reg *Registry) StoreManifest(ctx context.Context, account, repo, ref, cont
 	if ref != "" && !ecr.ValidateDigest(ref) {
 		meta, err := scoped.Meta.GetRepo(ctx, account, repo)
 		if err != nil {
-			return "", err
+			return "", "", err
 		}
 		if meta.TagMutability() == ecr.TagMutabilityImmutable {
 			existing, err := scoped.Meta.GetTag(ctx, account, repo, ref)
 			switch {
 			case err == nil && existing != digest:
-				return "", &ManifestStoreError{http.StatusConflict, "TAG_IMMUTABLE", "tag is immutable and already resolves to a different digest"}
+				return "", "", &ManifestStoreError{http.StatusConflict, "TAG_IMMUTABLE", "tag is immutable and already resolves to a different digest"}
 			case err != nil && !errors.Is(err, ecr.ErrNotFound):
-				return "", err
+				return "", "", err
 			}
 		}
 	}
 
 	children, code, vErr := scoped.validateManifest(ctx, repo, contentType, body)
 	if vErr != nil {
-		return "", &ManifestStoreError{http.StatusBadRequest, code, vErr.Error()}
+		return "", "", &ManifestStoreError{http.StatusBadRequest, code, vErr.Error()}
+	}
+
+	subject, artifactType, annotations, rErr := referrerFields(contentType, body)
+	if rErr != nil {
+		return "", "", &ManifestStoreError{http.StatusBadRequest, "MANIFEST_INVALID", rErr.Error()}
 	}
 
 	if _, err := scoped.Store.PutObject(ctx, &s3.PutObjectInput{
@@ -97,7 +112,7 @@ func (reg *Registry) StoreManifest(ctx context.Context, account, repo, ref, cont
 		Body:        aws.ReadSeekCloser(strings.NewReader(string(body))),
 		ContentType: aws.String(contentType),
 	}); err != nil {
-		return "", err
+		return "", "", err
 	}
 
 	if err := scoped.Meta.PutManifestMeta(ctx, account, repo, ecr.ManifestMeta{
@@ -106,16 +121,20 @@ func (reg *Registry) StoreManifest(ctx context.Context, account, repo, ref, cont
 		Size:         int64(len(body)),
 		PushedAt:     time.Now().UTC(),
 		ChildDigests: children,
+
+		SubjectDigest: subject,
+		ArtifactType:  artifactType,
+		Annotations:   annotations,
 	}); err != nil {
-		return "", err
+		return "", "", err
 	}
 
 	if ref != "" && !ecr.ValidateDigest(ref) {
 		if err := scoped.Meta.PutTag(ctx, account, repo, ref, digest); err != nil {
-			return "", err
+			return "", "", err
 		}
 	}
-	return digest, nil
+	return digest, subject, nil
 }
 
 // GetManifest returns the verbatim manifest bytes, media type and digest for a
@@ -230,40 +249,66 @@ func (reg *Registry) DeleteImage(ctx context.Context, account, repo, tag, digest
 		return resolved, nil
 	}
 
-	meta, err := scoped.Meta.GetManifestMeta(ctx, account, repo, digest)
+	if err := scoped.deleteManifestCascade(ctx, account, repo, digest, map[string]bool{}); err != nil {
+		return "", err
+	}
+	return digest, nil
+}
+
+// deleteManifestCascade deletes one manifest by digest with its tags, then the
+// referrers that name it as subject, each through this same path. Referrer
+// failures are logged, not returned: the subject is already deleted and ECR
+// removes referrers asynchronously. visited guards against subject cycles.
+func (reg *Registry) deleteManifestCascade(ctx context.Context, account, repo, digest string, visited map[string]bool) error {
+	visited[digest] = true
+	meta, err := reg.Meta.GetManifestMeta(ctx, account, repo, digest)
 	if err != nil {
 		if errors.Is(err, ecr.ErrNotFound) {
-			return "", ErrImageNotFound
+			return ErrImageNotFound
 		}
-		return "", err
+		return err
 	}
-	tags, err := scoped.Meta.ListTags(ctx, account, repo)
+	tags, err := reg.Meta.ListTags(ctx, account, repo)
 	if err != nil {
-		return "", err
+		return err
 	}
 	for _, t := range tags {
-		d, err := scoped.Meta.GetTag(ctx, account, repo, t)
+		d, err := reg.Meta.GetTag(ctx, account, repo, t)
 		if err != nil {
 			if errors.Is(err, ecr.ErrNotFound) {
 				continue
 			}
-			return "", err
+			return err
 		}
 		if d == digest {
-			if err := scoped.Meta.DeleteTag(ctx, account, repo, t); err != nil {
-				return "", err
+			if err := reg.Meta.DeleteTag(ctx, account, repo, t); err != nil {
+				return err
 			}
 		}
 	}
-	if err := scoped.Meta.DeleteManifestMeta(ctx, account, repo, digest); err != nil {
-		return "", err
+	if err := reg.Meta.DeleteManifestMeta(ctx, account, repo, digest); err != nil {
+		return err
 	}
 
 	// The KV record is gone, so the image is logically deleted. Predastore
 	// reclaim is best-effort: a failure leaks bytes (a capacity nuisance) but
 	// does not undo the delete.
-	scoped.reclaimManifest(ctx, account, repo, digest, meta.ChildDigests)
-	return digest, nil
+	reg.reclaimManifest(ctx, account, repo, digest, meta.ChildDigests)
+
+	referrers, err := reg.Meta.ListReferrers(ctx, account, repo, digest)
+	if err != nil {
+		slog.WarnContext(ctx, "ECR/referrers: listing referrers for cascade failed", "repo", repo, "subject", digest, "err", err)
+		return nil
+	}
+	for _, ref := range referrers {
+		if visited[ref.Digest] {
+			continue
+		}
+		if err := reg.deleteManifestCascade(ctx, account, repo, ref.Digest, visited); err != nil && !errors.Is(err, ErrImageNotFound) {
+			slog.WarnContext(ctx, "ECR/referrers: referrer delete failed", "repo", repo, "subject", digest, "referrer", ref.Digest, "err", err)
+		}
+	}
+	return nil
 }
 
 // reclaimManifest deletes a manifest's predastore object and any child blob no

spinifex/domains/ecr/registry/registry_referrer_test.go

@@ -0,0 +1,409 @@
+package ecrregistry
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/mulgadc/spinifex/spinifex/domains/ecr"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+const (
+	referrerRepo     = "team/app"
+	sigConfigType    = "application/vnd.example.sig.config.v1+json"
+	sigArtifactType  = "application/vnd.example.sig.v1"
+	absentSubjectDgt = "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+)
+
+func imageWithSubject(cfgDigest, subject, artifactType string) string {
+	extra := ""
+	if artifactType != "" {
+		extra = fmt.Sprintf(`"artifactType":%q,`, artifactType)
+	}
+	return fmt.Sprintf(`{"schemaVersion":2,"mediaType":%q,%s"config":{"mediaType":%q,"digest":%q,"size":3},"layers":[],"subject":{"mediaType":%q,"digest":%q,"size":10}}`,
+		mediaTypeOCIManifest, extra, sigConfigType, cfgDigest, mediaTypeOCIManifest, subject)
+}
+
+func storedMeta(t *testing.T, reg *Registry, body string) (subject, artifactType string) {
+	t.Helper()
+	m, err := reg.Meta.GetManifestMeta(context.Background(), testAccount, referrerRepo, digestOf([]byte(body)))
+	require.NoError(t, err)
+	return m.SubjectDigest, m.ArtifactType
+}
+
+func TestPutManifest_SubjectRecordedAndAdvertised(t *testing.T) {
+	reg := newTestRegistry()
+	cfg := pushBlob(t, reg, referrerRepo, []byte("cfg"))
+
+	tests := []struct {
+		name         string
+		artifactType string
+		wantType     string
+	}{
+		{"artifactType falls back to the config mediaType", "", sigConfigType},
+		{"explicit artifactType wins over the config mediaType", sigArtifactType, sigArtifactType},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			body := imageWithSubject(cfg, absentSubjectDgt, tt.artifactType)
+			w := do(reg, http.MethodPut, "/v2/"+referrerRepo+"/manifests/"+digestOf([]byte(body)), []byte(body),
+				map[string]string{"Content-Type": mediaTypeOCIManifest})
+			require.Equal(t, http.StatusCreated, w.Code, "body: %s", w.Body.String())
+			assert.Equal(t, absentSubjectDgt, w.Header().Get("Oci-Subject"))
+
+			subject, artifactType := storedMeta(t, reg, body)
+			assert.Equal(t, absentSubjectDgt, subject)
+			assert.Equal(t, tt.wantType, artifactType)
+		})
+	}
+}
+
+func TestPutManifest_IndexSubject(t *testing.T) {
+	reg := newTestRegistry()
+	seedRepo(t, reg, referrerRepo)
+
+	body := fmt.Sprintf(`{"schemaVersion":2,"mediaType":%q,"manifests":[],"subject":{"mediaType":%q,"digest":%q,"size":10}}`,
+		mediaTypeOCIIndex, mediaTypeOCIManifest, absentSubjectDgt)
+	w := do(reg, http.MethodPut, "/v2/"+referrerRepo+"/manifests/"+digestOf([]byte(body)), []byte(body),
+		map[string]string{"Content-Type": mediaTypeOCIIndex})
+	require.Equal(t, http.StatusCreated, w.Code, "body: %s", w.Body.String())
+	assert.Equal(t, absentSubjectDgt, w.Header().Get("Oci-Subject"))
+
+	subject, artifactType := storedMeta(t, reg, body)
+	assert.Equal(t, absentSubjectDgt, subject)
+	assert.Empty(t, artifactType, "an index has no config mediaType to fall back on")
+}
+
+func TestPutManifest_NoSubjectLeavesFieldsEmpty(t *testing.T) {
+	reg := newTestRegistry()
+	cfg := pushBlob(t, reg, referrerRepo, []byte("cfg"))
+	body := fmt.Sprintf(`{"schemaVersion":2,"mediaType":%q,"artifactType":%q,"config":{"mediaType":%q,"digest":%q,"size":3},"layers":[]}`,
+		mediaTypeOCIManifest, sigArtifactType, sigConfigType, cfg)
+
+	w := do(reg, http.MethodPut, "/v2/"+referrerRepo+"/manifests/v1", []byte(body),
+		map[string]string{"Content-Type": mediaTypeOCIManifest})
+	require.Equal(t, http.StatusCreated, w.Code, "body: %s", w.Body.String())
+	assert.Empty(t, w.Header().Get("Oci-Subject"))
+
+	subject, artifactType := storedMeta(t, reg, body)
+	assert.Empty(t, subject)
+	assert.Empty(t, artifactType, "artifactType is recorded only for manifests that declare a subject")
+}
+
+func TestPutManifest_MalformedSubjectDigestRejected(t *testing.T) {
+	reg := newTestRegistry()
+	cfg := pushBlob(t, reg, referrerRepo, []byte("cfg"))
+	body := imageWithSubject(cfg, "sha256:nothex", "")
+
+	w := do(reg, http.MethodPut, "/v2/"+referrerRepo+"/manifests/v1", []byte(body),
+		map[string]string{"Content-Type": mediaTypeOCIManifest})
+	assert.Equal(t, http.StatusBadRequest, w.Code)
+	assertCode(t, w, "MANIFEST_INVALID")
+
+	_, err := reg.Meta.GetManifestMeta(context.Background(), testAccount, referrerRepo, digestOf([]byte(body)))
+	assert.Error(t, err, "a rejected manifest must not leave a record behind")
+}
+
+func TestPutManifest_IndexWithContentTypeParameterGetsNoConfigArtifactType(t *testing.T) {
+	reg := newTestRegistry()
+	seedRepo(t, reg, referrerRepo)
+
+	body := fmt.Sprintf(`{"schemaVersion":2,"mediaType":%q,"manifests":[],"subject":{"mediaType":%q,"digest":%q,"size":10}}`,
+		mediaTypeOCIIndex, mediaTypeOCIManifest, absentSubjectDgt)
+	w := do(reg, http.MethodPut, "/v2/"+referrerRepo+"/manifests/v1", []byte(body),
+		map[string]string{"Content-Type": mediaTypeOCIIndex + "; charset=utf-8"})
+	require.Equal(t, http.StatusCreated, w.Code, "body: %s", w.Body.String())
+	assert.Equal(t, absentSubjectDgt, w.Header().Get("Oci-Subject"))
+
+	m, err := reg.Meta.GetManifestMeta(context.Background(), testAccount, referrerRepo, digestOf([]byte(body)))
+	require.NoError(t, err)
+	assert.Equal(t, mediaTypeOCIIndex, m.MediaType, "the stored type must not carry the parameter")
+	assert.Empty(t, m.ArtifactType)
+}
+
+func TestDeleteReferrerManifest_RemovesItFromTheSubjectListing(t *testing.T) {
+	reg := newTestRegistry()
+	cfg := pushBlob(t, reg, referrerRepo, []byte("cfg"))
+	body := imageWithSubject(cfg, absentSubjectDgt, sigArtifactType)
+	dg := digestOf([]byte(body))
+	w := do(reg, http.MethodPut, "/v2/"+referrerRepo+"/manifests/"+dg, []byte(body),
+		map[string]string{"Content-Type": mediaTypeOCIManifest})
+	require.Equal(t, http.StatusCreated, w.Code, "body: %s", w.Body.String())
+
+	listed, err := reg.Meta.ListReferrers(context.Background(), testAccount, referrerRepo, absentSubjectDgt)
+	require.NoError(t, err)
+	require.Len(t, listed, 1)
+	assert.Equal(t, dg, listed[0].Digest)
+	assert.Equal(t, sigArtifactType, listed[0].ArtifactType)
+
+	w = do(reg, http.MethodDelete, "/v2/"+referrerRepo+"/manifests/"+dg, nil, nil)
+	require.Equal(t, http.StatusAccepted, w.Code, "body: %s", w.Body.String())
+
+	listed, err = reg.Meta.ListReferrers(context.Background(), testAccount, referrerRepo, absentSubjectDgt)
+	require.NoError(t, err)
+	assert.Empty(t, listed)
+}
+
+// pushReferrer stores a signature-shaped manifest that names subject and returns its digest.
+func pushReferrer(t *testing.T, reg *Registry, repo, subject, artifactType, blobData string) string {
+	t.Helper()
+	cfg := pushBlob(t, reg, repo, []byte(blobData))
+	body := imageWithSubject(cfg, subject, artifactType)
+	dg := digestOf([]byte(body))
+	w := do(reg, http.MethodPut, "/v2/"+repo+"/manifests/"+dg, []byte(body),
+		map[string]string{"Content-Type": mediaTypeOCIManifest})
+	require.Equal(t, http.StatusCreated, w.Code, "body: %s", w.Body.String())
+	return dg
+}
+
+type referrersBody struct {
+	SchemaVersion int    `json:"schemaVersion"`
+	MediaType     string `json:"mediaType"`
+	Manifests     []struct {
+		MediaType    string `json:"mediaType"`
+		Digest       string `json:"digest"`
+		ArtifactType string `json:"artifactType"`
+	} `json:"manifests"`
+}
+
+func getReferrersBody(t *testing.T, reg *Registry, path string) (*httptest.ResponseRecorder, referrersBody) {
+	t.Helper()
+	w := do(reg, http.MethodGet, path, nil, nil)
+	var out referrersBody
+	if w.Code == http.StatusOK {
+		require.NoError(t, json.Unmarshal(w.Body.Bytes(), &out))
+	}
+	return w, out
+}
+
+func TestGetReferrers_ListsReferrersAsAnOCIIndex(t *testing.T) {
+	reg := newTestRegistry()
+	dgA := pushReferrer(t, reg, referrerRepo, absentSubjectDgt, sigArtifactType, "cfgA")
+	dgB := pushReferrer(t, reg, referrerRepo, absentSubjectDgt, "application/vnd.example.sbom.v1", "cfgB")
+
+	w, out := getReferrersBody(t, reg, "/v2/"+referrerRepo+"/referrers/"+absentSubjectDgt)
+	require.Equal(t, http.StatusOK, w.Code, "body: %s", w.Body.String())
+	assert.Equal(t, "application/vnd.oci.image.index.v1+json", w.Header().Get("Content-Type"))
+	assert.Empty(t, w.Header().Get("Oci-Filters-Applied"))
+	assert.Equal(t, 2, out.SchemaVersion)
+	assert.Equal(t, mediaTypeOCIIndex, out.MediaType)
+	got := map[string]string{}
+	for _, m := range out.Manifests {
+		assert.Equal(t, mediaTypeOCIManifest, m.MediaType)
+		got[m.Digest] = m.ArtifactType
+	}
+	assert.Equal(t, map[string]string{dgA: sigArtifactType, dgB: "application/vnd.example.sbom.v1"}, got)
+}
+
+func TestGetReferrers_NoReferrersIsAnEmptyIndexNotAnError(t *testing.T) {
+	reg := newTestRegistry()
+	seedRepo(t, reg, referrerRepo)
+
+	w := do(reg, http.MethodGet, "/v2/"+referrerRepo+"/referrers/"+absentSubjectDgt, nil, nil)
+	require.Equal(t, http.StatusOK, w.Code, "body: %s", w.Body.String())
+	assert.Equal(t, "application/vnd.oci.image.index.v1+json", w.Header().Get("Content-Type"))
+	assert.JSONEq(t, fmt.Sprintf(`{"schemaVersion":2,"mediaType":%q,"manifests":[]}`, mediaTypeOCIIndex), w.Body.String())
+}
+
+func TestGetReferrers_ArtifactTypeFilter(t *testing.T) {
+	reg := newTestRegistry()
+	dgSig := pushReferrer(t, reg, referrerRepo, absentSubjectDgt, sigArtifactType, "cfgA")
+	pushReferrer(t, reg, referrerRepo, absentSubjectDgt, "application/vnd.example.sbom.v1", "cfgB")
+
+	w, out := getReferrersBody(t, reg, "/v2/"+referrerRepo+"/referrers/"+absentSubjectDgt+"?artifactType="+sigArtifactType)
+	require.Equal(t, http.StatusOK, w.Code)
+	assert.Equal(t, "artifactType", w.Header().Get("Oci-Filters-Applied"))
+	require.Len(t, out.Manifests, 1)
+	assert.Equal(t, dgSig, out.Manifests[0].Digest)
+
+	w, out = getReferrersBody(t, reg, "/v2/"+referrerRepo+"/referrers/"+absentSubjectDgt+"?artifactType=application/none")
+	require.Equal(t, http.StatusOK, w.Code)
+	assert.Equal(t, "artifactType", w.Header().Get("Oci-Filters-Applied"), "the header reports the filter even when nothing matches")
+	assert.Empty(t, out.Manifests)
+	assert.Contains(t, w.Body.String(), `"manifests":[]`)
+}
+
+func TestGetReferrers_MalformedDigest(t *testing.T) {
+	reg := newTestRegistry()
+	seedRepo(t, reg, referrerRepo)
+
+	w := do(reg, http.MethodGet, "/v2/"+referrerRepo+"/referrers/not-a-digest", nil, nil)
+	assert.Equal(t, http.StatusBadRequest, w.Code)
+	assertCode(t, w, "DIGEST_INVALID")
+}
+
+func TestGetReferrers_UnknownRepository(t *testing.T) {
+	reg := newTestRegistry()
+	w := do(reg, http.MethodGet, "/v2/team/ghost/referrers/"+absentSubjectDgt, nil, nil)
+	assert.Equal(t, http.StatusNotFound, w.Code)
+	assertCode(t, w, "NAME_UNKNOWN")
+}
+
+func TestGetReferrers_DoesNotCrossRepositories(t *testing.T) {
+	reg := newTestRegistry()
+	pushReferrer(t, reg, referrerRepo, absentSubjectDgt, sigArtifactType, "cfgA")
+	seedRepo(t, reg, "team/other")
+
+	w, out := getReferrersBody(t, reg, "/v2/team/other/referrers/"+absentSubjectDgt)
+	require.Equal(t, http.StatusOK, w.Code)
+	assert.Empty(t, out.Manifests)
+}
+
+func TestReferrersRoute_OnlyGET(t *testing.T) {
+	reg := newTestRegistry()
+	seedRepo(t, reg, referrerRepo)
+	for _, method := range []string{http.MethodPut, http.MethodDelete, http.MethodPost, http.MethodPatch} {
+		w := do(reg, method, "/v2/"+referrerRepo+"/referrers/"+absentSubjectDgt, []byte("{}"), nil)
+		assert.Equal(t, http.StatusNotFound, w.Code, method)
+		assertCode(t, w, "NAME_UNKNOWN")
+	}
+}
+
+type failingReferrerStore struct{ ecr.MetaStore }
+
+func (failingReferrerStore) ListReferrers(context.Context, string, string, string) ([]ecr.ReferrerDescriptor, error) {
+	return nil, errors.New("no responders available for ecr.referrer.list")
+}
+
+func TestGetReferrers_MetadataFailureIsA5xxNotA404(t *testing.T) {
+	reg := newTestRegistry()
+	seedRepo(t, reg, referrerRepo)
+	reg.Meta = failingReferrerStore{reg.Meta}
+
+	w := do(reg, http.MethodGet, "/v2/"+referrerRepo+"/referrers/"+absentSubjectDgt, nil, nil)
+	assert.Equal(t, http.StatusInternalServerError, w.Code)
+}
+
+// pushSubjectImage stores a plain image (no subject) and returns its digest.
+func pushSubjectImage(t *testing.T, reg *Registry, repo, blobData string) string {
+	t.Helper()
+	cfg := pushBlob(t, reg, repo, []byte(blobData))
+	body := fmt.Sprintf(`{"schemaVersion":2,"mediaType":%q,"config":{"mediaType":%q,"digest":%q,"size":3},"layers":[]}`,
+		mediaTypeOCIManifest, sigConfigType, cfg)
+	dg := digestOf([]byte(body))
+	w := do(reg, http.MethodPut, "/v2/"+repo+"/manifests/"+dg, []byte(body),
+		map[string]string{"Content-Type": mediaTypeOCIManifest})
+	require.Equal(t, http.StatusCreated, w.Code, "body: %s", w.Body.String())
+	return dg
+}
+
+func manifestExists(reg *Registry, repo, dg string) bool {
+	_, err := reg.Meta.GetManifestMeta(context.Background(), testAccount, repo, dg)
+	return err == nil
+}
+
+func referrerDigests(t *testing.T, reg *Registry, repo, subject string) []string {
+	t.Helper()
+	refs, err := reg.Meta.ListReferrers(context.Background(), testAccount, repo, subject)
+	require.NoError(t, err)
+	out := make([]string, 0, len(refs))
+	for _, r := range refs {
+		out = append(out, r.Digest)
+	}
+	return out
+}
+
+func TestDeleteManifest_SubjectDeleteRemovesItsReferrers(t *testing.T) {
+	reg := newTestRegistry()
+	subject := pushSubjectImage(t, reg, referrerRepo, "subject")
+	sig := pushReferrer(t, reg, referrerRepo, subject, sigArtifactType, "sig")
+	sbom := pushReferrer(t, reg, referrerRepo, subject, "application/vnd.example.sbom.v1", "sbom")
+	require.ElementsMatch(t, []string{sig, sbom}, referrerDigests(t, reg, referrerRepo, subject))
+
+	w := do(reg, http.MethodDelete, "/v2/"+referrerRepo+"/manifests/"+subject, nil, nil)
+	require.Equal(t, http.StatusAccepted, w.Code, "body: %s", w.Body.String())
+
+	assert.False(t, manifestExists(reg, referrerRepo, subject))
+	assert.False(t, manifestExists(reg, referrerRepo, sig))
+	assert.False(t, manifestExists(reg, referrerRepo, sbom))
+	assert.Empty(t, referrerDigests(t, reg, referrerRepo, subject), "index keys go with the referrers")
+}
+
+func TestDeleteManifest_NestedReferrersAreRemoved(t *testing.T) {
+	reg := newTestRegistry()
+	subject := pushSubjectImage(t, reg, referrerRepo, "subject")
+	sig := pushReferrer(t, reg, referrerRepo, subject, sigArtifactType, "sig")
+	sigOfSig := pushReferrer(t, reg, referrerRepo, sig, sigArtifactType, "sig-of-sig")
+
+	_, err := reg.DeleteImage(context.Background(), testAccount, referrerRepo, "", subject)
+	require.NoError(t, err)
+
+	for _, dg := range []string{subject, sig, sigOfSig} {
+		assert.False(t, manifestExists(reg, referrerRepo, dg), dg)
+	}
+	assert.Empty(t, referrerDigests(t, reg, referrerRepo, sig))
+}
+
+func TestDeleteManifest_ReferrerDeleteLeavesTheSubject(t *testing.T) {
+	reg := newTestRegistry()
+	subject := pushSubjectImage(t, reg, referrerRepo, "subject")
+	sig := pushReferrer(t, reg, referrerRepo, subject, sigArtifactType, "sig")
+	other := pushReferrer(t, reg, referrerRepo, subject, "application/vnd.example.sbom.v1", "sbom")
+
+	_, err := reg.DeleteImage(context.Background(), testAccount, referrerRepo, "", sig)
+	require.NoError(t, err)
+
+	assert.True(t, manifestExists(reg, referrerRepo, subject))
+	assert.True(t, manifestExists(reg, referrerRepo, other))
+	assert.Equal(t, []string{other}, referrerDigests(t, reg, referrerRepo, subject))
+}
+
+func TestDeleteManifest_TagDeleteDoesNotCascade(t *testing.T) {
+	reg := newTestRegistry()
+	subject := pushSubjectImage(t, reg, referrerRepo, "subject")
+	sig := pushReferrer(t, reg, referrerRepo, subject, sigArtifactType, "sig")
+	require.NoError(t, reg.Meta.PutTag(context.Background(), testAccount, referrerRepo, "v1", subject))
+
+	w := do(reg, http.MethodDelete, "/v2/"+referrerRepo+"/manifests/v1", nil, nil)
+	require.Equal(t, http.StatusAccepted, w.Code)
+
+	assert.True(t, manifestExists(reg, referrerRepo, subject), "a tag delete only removes the pointer")
+	assert.True(t, manifestExists(reg, referrerRepo, sig))
+}
+
+func TestDeleteManifest_CascadeStaysInsideTheRepository(t *testing.T) {
+	reg := newTestRegistry()
+	subject := pushSubjectImage(t, reg, referrerRepo, "subject")
+	pushReferrer(t, reg, referrerRepo, subject, sigArtifactType, "sig")
+	elsewhere := pushReferrer(t, reg, "team/other", subject, sigArtifactType, "sig-other")
+
+	_, err := reg.DeleteImage(context.Background(), testAccount, referrerRepo, "", subject)
+	require.NoError(t, err)
+
+	assert.True(t, manifestExists(reg, "team/other", elsewhere))
+	assert.Equal(t, []string{elsewhere}, referrerDigests(t, reg, "team/other", subject))
+}
+
+type failingDeleteStore struct {
+	ecr.MetaStore
+
+	failDigest string
+}
+
+func (s failingDeleteStore) DeleteManifestMeta(ctx context.Context, account, repo, digest string) error {
+	if digest == s.failDigest {
+		return errors.New("kv unavailable")
+	}
+	return s.MetaStore.DeleteManifestMeta(ctx, account, repo, digest)
+}
+
+func TestDeleteManifest_ReferrerFailureDoesNotFailTheSubjectDelete(t *testing.T) {
+	reg := newTestRegistry()
+	subject := pushSubjectImage(t, reg, referrerRepo, "subject")
+	stuck := pushReferrer(t, reg, referrerRepo, subject, sigArtifactType, "stuck")
+	fine := pushReferrer(t, reg, referrerRepo, subject, "application/vnd.example.sbom.v1", "fine")
+	reg.Meta = failingDeleteStore{MetaStore: reg.Meta, failDigest: stuck}
+
+	got, err := reg.DeleteImage(context.Background(), testAccount, referrerRepo, "", subject)
+	require.NoError(t, err)
+	assert.Equal(t, subject, got)
+	assert.False(t, manifestExists(reg, referrerRepo, subject))
+	assert.False(t, manifestExists(reg, referrerRepo, fine), "one stuck referrer does not stop the others")
+	assert.True(t, manifestExists(reg, referrerRepo, stuck))
+}

spinifex/domains/ecr/registry_address.go

@@ -0,0 +1,34 @@
+package ecr
+
+import "net"
+
+// RegistryAddress returns the registry host and port a gateway node advertises.
+// A concrete AWSGW bind host wins, then a concrete AdvertiseIP; an empty host
+// means the per-account parity name applies. The port comes from the bind.
+func RegistryAddress(awsgwHost, advertiseIP string) (host, port string) {
+	if h, p, err := net.SplitHostPort(awsgwHost); err == nil {
+		port = p
+		if isConcreteRegistryHost(h) {
+			host = h
+		}
+	}
+	if host == "" && isConcreteRegistryHost(advertiseIP) {
+		host = advertiseIP
+	}
+	return host, port
+}
+
+// RegistryURIHost joins a registry host and port the way a repositoryUri
+// carries them: the standard HTTPS port 443 is omitted.
+func RegistryURIHost(host, port string) string {
+	if port != "" && port != "443" {
+		return host + ":" + port
+	}
+	return host
+}
+
+// isConcreteRegistryHost reports whether host is a non-empty, non-wildcard
+// literal, so a registry URI never carries 0.0.0.0 or ::.
+func isConcreteRegistryHost(host string) bool {
+	return host != "" && host != "0.0.0.0" && host != "::"
+}

spinifex/domains/ecr/registry_address_test.go

@@ -0,0 +1,48 @@
+package ecr
+
+import "testing"
+
+func TestRegistryAddress(t *testing.T) {
+	tests := []struct {
+		name      string
+		bind      string
+		advertise string
+		wantHost  string
+		wantPort  string
+	}{
+		{"wildcard bind falls back to advertise", "0.0.0.0:9999", "192.168.0.12", "192.168.0.12", "9999"},
+		{"concrete bind equal to advertise", "10.0.0.5:9999", "10.0.0.5", "10.0.0.5", "9999"},
+		{"concrete bind wins over a different advertise", "10.0.0.5:9999", "192.168.0.12", "10.0.0.5", "9999"},
+		{"ipv6 wildcard falls back to advertise", "[::]:9999", "192.168.0.12", "192.168.0.12", "9999"},
+		{"wildcard bind and empty advertise leaves the parity name", "0.0.0.0:9999", "", "", "9999"},
+		{"wildcard bind and wildcard advertise leaves the parity name", "0.0.0.0:9999", "0.0.0.0", "", "9999"},
+		{"unparseable bind falls back to advertise with no port", "not-a-hostport", "192.168.0.12", "192.168.0.12", ""},
+		{"empty bind and empty advertise", "", "", "", ""},
+		{"port is taken from the bind", "10.0.0.5:443", "", "10.0.0.5", "443"},
+		{"loopback bind counts as concrete", "127.0.0.1:9999", "192.168.0.12", "127.0.0.1", "9999"},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			host, port := RegistryAddress(tt.bind, tt.advertise)
+			if host != tt.wantHost || port != tt.wantPort {
+				t.Errorf("RegistryAddress(%q, %q) = (%q, %q), want (%q, %q)",
+					tt.bind, tt.advertise, host, port, tt.wantHost, tt.wantPort)
+			}
+		})
+	}
+}
+
+func TestRegistryURIHost(t *testing.T) {
+	tests := []struct {
+		host, port, want string
+	}{
+		{"10.0.0.1", "9999", "10.0.0.1:9999"},
+		{"10.0.0.1", "443", "10.0.0.1"},
+		{"10.0.0.1", "", "10.0.0.1"},
+	}
+	for _, tt := range tests {
+		if got := RegistryURIHost(tt.host, tt.port); got != tt.want {
+			t.Errorf("RegistryURIHost(%q, %q) = %q, want %q", tt.host, tt.port, got, tt.want)
+		}
+	}
+}

spinifex/domains/ecr/service.go

@@ -28,6 +28,7 @@ const (
 	SubjectManifestDescribe = "ecr.manifest.describe"
 	SubjectManifestList     = "ecr.manifest.list"
 	SubjectManifestDelete   = "ecr.manifest.delete"
+	SubjectReferrerList     = "ecr.referrer.list"
 
 	SubjectUploadCreate = "ecr.upload.create"
 	SubjectUploadGet    = "ecr.upload.get"
@@ -223,6 +224,17 @@ type ManifestListResponse struct {
 	Digests []string `json:"digests"`
 }
 
+// ReferrerListRequest is the payload for SubjectReferrerList, naming the subject whose referrers to list.
+type ReferrerListRequest struct {
+	Repo    string `json:"repo"`
+	Subject string `json:"subject"`
+}
+
+// ReferrerListResponse is the reply to SubjectReferrerList, holding one descriptor per referrer.
+type ReferrerListResponse struct {
+	Referrers []ReferrerDescriptor `json:"referrers"`
+}
+
 // ManifestDeleteRequest is the payload for SubjectManifestDelete, naming the manifest metadata record to remove.
 type ManifestDeleteRequest struct {
 	Repo   string `json:"repo"`
@@ -313,6 +325,7 @@ type MetaService interface {
 	ManifestDescribe(ctx context.Context, req *ManifestDescribeRequest, accountID string) (*ManifestDescribeResponse, error)
 	ManifestList(ctx context.Context, req *ManifestListRequest, accountID string) (*ManifestListResponse, error)
 	ManifestDelete(ctx context.Context, req *ManifestDeleteRequest, accountID string) (*ManifestDeleteResponse, error)
+	ReferrerList(ctx context.Context, req *ReferrerListRequest, accountID string) (*ReferrerListResponse, error)
 
 	UploadCreate(ctx context.Context, req *UploadCreateRequest, accountID string) (*UploadCreateResponse, error)
 	UploadGet(ctx context.Context, req *UploadGetRequest, accountID string) (*UploadGetResponse, error)

spinifex/domains/ecr/service_impl.go

@@ -185,6 +185,14 @@ func (s *MetaServiceImpl) ManifestDelete(ctx context.Context, req *ManifestDelet
 	return &ManifestDeleteResponse{Found: true}, nil
 }
 
+func (s *MetaServiceImpl) ReferrerList(ctx context.Context, req *ReferrerListRequest, accountID string) (*ReferrerListResponse, error) {
+	referrers, err := s.store.ListReferrers(ctx, accountID, req.Repo, req.Subject)
+	if err != nil {
+		return nil, err
+	}
+	return &ReferrerListResponse{Referrers: referrers}, nil
+}
+
 func (s *MetaServiceImpl) ManifestDescribe(ctx context.Context, req *ManifestDescribeRequest, accountID string) (*ManifestDescribeResponse, error) {
 	meta, err := s.store.GetManifestMeta(ctx, accountID, req.Repo, req.Digest)
 	if errors.Is(err, ErrNotFound) {

spinifex/domains/ecr/storage.go

@@ -89,6 +89,7 @@ func KVAccountBucket(accountID string) string {
 //	repos/{name}/meta
 //	repos/{name}/tags/{tag}
 //	repos/{name}/manifests/{digest}
+//	repos/{name}/referrers/{subject}/{referrer}
 //	uploads/{uuid}
 const (
 	kvRepoMetaKeyFmt     = "repos/%s/meta"
@@ -98,6 +99,8 @@ const (
 	kvTagKeyFmt          = "repos/%s/tags/%s"
 	kvManifestKeyFmt     = "repos/%s/manifests/%s"
 	kvManifestsPrefixFmt = "repos/%s/manifests/"
+	kvReferrersPrefixFmt = "repos/%s/referrers/"
+	kvReferrerKeyFmt     = "repos/%s/referrers/%s/%s"
 	kvReposPrefix        = "repos/"
 	kvUploadKeyFmt       = "uploads/%s"
 )
@@ -128,6 +131,21 @@ func KVManifestKey(repo, digest string) string {
 // manifest metadata records.
 func KVManifestsPrefix(repo string) string { return fmt.Sprintf(kvManifestsPrefixFmt, repo) }
 
+// KVReferrersPrefix returns the KV key prefix enumerating every referrer entry
+// of a repository.
+func KVReferrersPrefix(repo string) string { return fmt.Sprintf(kvReferrersPrefixFmt, repo) }
+
+// KVSubjectReferrersPrefix returns the KV key prefix enumerating the referrers
+// of one subject manifest.
+func KVSubjectReferrersPrefix(repo, subjectDigest string) string {
+	return KVReferrersPrefix(repo) + DigestToken(subjectDigest) + "/"
+}
+
+// KVReferrerKey returns the KV key of one referrer entry under its subject.
+func KVReferrerKey(repo, subjectDigest, referrerDigest string) string {
+	return fmt.Sprintf(kvReferrerKeyFmt, repo, DigestToken(subjectDigest), DigestToken(referrerDigest))
+}
+
 // KVReposPrefix is the prefix under which every repo meta key lives. Used by
 // the catalog listing to enumerate account repositories.
 const KVReposPrefix = kvReposPrefix

spinifex/domains/eks/cluster/k3s_server_vm.go

@@ -566,6 +566,7 @@ func buildK3sUserData(in K3sServerInput) string {
 		"EKS_KONNECTIVITY_HOST="+konnHost,
 		"EKS_KONNECTIVITY_SANS="+strings.Join(konnSANs, ","),
 		"EKS_KONNECTIVITY_SERVER_COUNT="+strconv.Itoa(konnCount),
+		"EKS_WEBHOOK_READY_ADDR=0.0.0.0:"+strconv.FormatInt(webhookReadyPort, 10),
 	)
 	// A replacement CP prunes the terminated member it supersedes from etcd once
 	// it has joined (k3s-first-boot deletes the dead Node → embedded-etcd drops

spinifex/domains/eks/cluster/k3s_server_vm_test.go

@@ -4,6 +4,8 @@ import (
 	"context"
 	"errors"
 	"fmt"
+	"os"
+	"slices"
 	"strings"
 	"sync"
 	"testing"
@@ -489,6 +491,29 @@ func TestBuildK3sUserData_KonnectivityEnv(t *testing.T) {
 	assert.Contains(t, ud, "EKS_KONNECTIVITY_SANS=10.32.100.4,203.0.113.9,"+in.NLBDNS)
 }
 
+func TestBuildK3sUserData_WebhookReadinessListener(t *testing.T) {
+	t.Parallel()
+	ud := buildK3sUserData(validK3sInput())
+	assert.Contains(t, ud, "EKS_WEBHOOK_READY_ADDR=0.0.0.0:8444",
+		"the webhook must serve /readyz beyond loopback for the NLB health check")
+}
+
+// The env line alone does nothing: the webhook service only passes through
+// the variables its init script exports.
+func TestWebhookInitExportsReadyAddr(t *testing.T) {
+	t.Parallel()
+	b, err := os.ReadFile("../../../../scripts/images/eks-node/eks-token-webhook.initd")
+	require.NoError(t, err)
+	var exported bool
+	for line := range strings.SplitSeq(string(b), "\n") {
+		f := strings.Fields(line)
+		if len(f) > 1 && f[0] == "export" && slices.Contains(f[1:], "EKS_WEBHOOK_READY_ADDR") {
+			exported = true
+		}
+	}
+	assert.True(t, exported, "eks-token-webhook.initd must export EKS_WEBHOOK_READY_ADDR or the readiness listener never starts")
+}
+
 func TestBuildK3sUserData_KonnServerCountDefaultsToOne(t *testing.T) {
 	t.Parallel()
 	in := validK3sInput()