Merge remote-tracking branch 'origin/dev' into feat/oracle-linux-multinode

* feat: add the store-neutral SQS message store contract and conformance suite

Defines message.Store over opaque shard and queue ID namespaces with no
create call, plus a reusable conformance runner exercised against a
test-only in-memory reference store.

* feat: add the SQS message store's JetStream layout, append and state CAS

Provisions the fixed set of shared message streams and state buckets,
publishes each queue on sqs.m.<shard>.<queue>, dedupes idempotent appends
with a queue-scoped Nats-Msg-Id, and maps state revision CAS onto KV.

* fix: give each SQS queue immutable message and state shard allocations

A changed stream or bucket count applies only to new queues: the namespace
names its message shard and state shard directly, retained streams and
buckets stay addressable, and the store refuses cluster sizes other than
one node (R1) or three (R3).

* fix: enumerate complete SQS namespaces and backstop state buckets

Enumeration now returns each namespace with both allocations, so a
state-only namespace can be counted and cleaned without its queue record,
and state buckets expire records after 14 days plus 1 hour.

* feat: add the SQS JetStream store's reads, counts and cleanup

Get and Delete are scoped to the namespace's subject, DeleteThrough and
DeleteNamespace are subject-filtered purges that leave shard neighbours
intact, and Count reads the per-subject message total and live state keys.

* fix: count distinct SQS state keys and validate before namespace purge

The key lister can repeat a key under concurrent writes, so Count tallies
distinct names; DeleteNamespace resolves both the message stream and the
state bucket before purging either, so a wrong state shard deletes nothing.

* feat: add the SQS JetStream store's ordered message-shard feed

Append records the namespace's state shard in an internal header that is
never returned as metadata. Feed reads one shared message stream through
an ephemeral, headers-only ordered consumer with no acknowledgement state,
rebuilds each change's complete namespace, refuses a malformed header with
ErrCorruptRecord, and deletes its consumer on Close.

* feat: enumerate SQS namespaces and run conformance on the JetStream store

Namespaces joins per-subject body counts, whose internal header names the
state shard, with live state keys from every state bucket that exists,
found by listing so a historical bucket counts and none is created. An
allocation conflict is reported as ErrCorruptRecord. The store now
implements message.Store and runs the conformance suite on R1, per case
with crash reopen and on one shared instance.

* fix: check every SQS body's state shard when enumerating namespaces

Sampling the first and last body let a middle body naming another state
shard pass. Namespaces now reads every body's header in one headers-only
ordered pass over the shard's stream, counting each queue's bodies from
that pass, and deletes its consumer when done.

* test: run SQS store conformance on R3 and guard against consumer state

TestR3Conformance runs the conformance suite on a real three-node
JetStream cluster in process, with Reopen restarting every node over the
same data, and a check that every stream and state bucket is R3.
TestS5_NoConsumerDeliveryStateInSQS fails on any acknowledgement,
redelivery, durable-consumer or ack-retention mechanism in SQS source.

* test: bar legacy NATS acknowledgement helpers from SQS source

Adds AckSync, ManualAck, AckAll and AckExplicit to the ADR-0011:S5
consumer-state guard.

* feat: add a bounded SQS store scan and the tombstoned-body sweeper

Store gains Scan: a paginated, headers-only pass over a message shard,
bounded at the shard's last position when it starts, skipping deleted
positions and never returning a later append. The conformance suite and
fake cover continuation, deleted runs and the bound.

The JetStream adapter reads every bounded range (scan, body enumeration
and state keys) through one reader whose completion rests on a direct get
of the next match, not on consumer state; the client's key lister and
pending counts are no longer trusted to end a pass.

message.Sweep walks a shard with Scan and, for each body whose record the
domain reports terminal, deletes the body then the record by revision.

* test: add an opt-in SQS store conformance run on an external R3 cluster

TestR3ExternalConformance runs the conformance suite against the
dedicated three-node SQS JetStream cluster named by SQS_STORE_R3_URLS,
checks every stream is R3 with two peers, and uses SQS_STORE_R3_RESTART
to restart every node for the crash cases. It skips when unset.

* test: run the SQS sweeper against the reference store fake

Covers live records kept, tombstones swept, another shard untouched, a
no-op rerun, and a retry after the body delete succeeds and the state
delete fails.

* test: harden the SQS store R3 harnesses against restart and placement races

The external R3 harness reused one connection across restarts, so a case's
cleanup closed it for every later case; each case and restart now opens its
own connection and waits for stream leaders.

The in-process cluster released its free ports before the servers bound them,
and R3 placement can fail until the meta leader has every peer's config and
stats. Client ports are now server-chosen, route ports are reserved together,
and provisioning retries the idempotent New until placement succeeds.

* fix: stop the SQS store feed idling for a whole pull expiry

Under load an ordered feed on a three-node cluster sat for nats.go's 30s
default pull expiry, once 60s, with stored changes pending and no pull
waiting on its consumer, both on feeds opened after a full restart and on
feeds kept open across one. Feed now sets a 2s pull expiry.

Adds restart tests for a feed opened after a full-cluster restart and one
open across it, each replaying a shared-shard backlog then a live append and
failing on any 5s gap. The in-process cluster keeps client ports across
restarts so open connections reconnect, and can write node logs.

* fix: read SQS message state from the stream leader

GetState used nats.go's KV Get, which switches to a direct get whenever
the bucket allows it. On a replicated bucket a follower can answer that
from state it has not applied yet, so a record the leader had already
deleted or purged could still be returned.

Read the key's last message with a stream message get instead, which
only the stream leader answers, and apply KV Get's delete and purge
marker rules to the result. Bucket configuration is unchanged. Missing,
deleted and purged records still return ErrNotFound; values, revisions,
context errors and the API prefix or domain of the caller's JetStream
context are preserved.

* docs: narrow the SQS GetState leader-read comment to what it avoids

* fix: enumerate SQS state keys from the stream leader

Count and Namespaces found live state keys with a direct-get probe and a
headers-only ordered consumer on the state bucket's stream. On a
replicated bucket a follower can answer either from state it has not
applied, so keys the leader had already deleted could still be counted.

Walk the key prefix with stream message gets by next subject, which
only the stream leader answers, through the stream's last sequence at
the call. Delete and purge markers drop a key, as GetState reads them.
This costs one leader request per record under the prefix. Bucket
configuration, historical state shards, the message-body scans and the
feed are unchanged.

* test: keep SQS API recorder deliveries in arrival order

The recorder fed API subjects and its sentinel to two async
subscriptions, which nats.go delivers on separate goroutines, so drain
could return before the last request was recorded. Both now share one
channel that the connection fills in arrival order.

* fix: read SQS body enumerations from the stream leader

Namespaces and Scan read bodies through an ephemeral ordered consumer,
which the server may place on a follower that has not applied a purge,
so a deleted queue could still be listed or scanned. Each stored match
through the bound is now a leader stream message get. That transfers
every full body, where the consumer read headers only.

* test: stop the SQS API recorder goroutine with its test

The recorder ranged over a caller-owned ChanSubscribe channel, which
nats.go never closes, so its goroutine outlived every test. Cleanup now
unsubscribes both subscriptions, then signals the goroutine and waits
for it to return; the sentinel send also watches that signal.

* fix: recover a stalled SQS feed from its cursor

An ordered feed was seen to idle with stored changes pending after a
cluster restart. Next now waits at most an idle check on the consumer,
then asks the stream leader whether a change follows the cursor; if one
does, the consumer is deleted and reopened just after the cursor, so no
change is skipped or repeated. The root cause of the stall is unknown.

* fix: make closing an SQS feed terminal and safe against Next

Close now marks the feed closed under a lock, cancels any in-flight
Next, stops the iterator and deletes the consumer. A later or in-flight
Next returns ErrFeedClosed and never opens a consumer; one opened by a
recovery racing Close is deleted rather than adopted. Repeated Close
returns nil.

* fix: define SQS feed close in the store contract

message.Feed now documents one Next reader with Close callable at any
time and more than once, and message.ErrFeedClosed replaces the
adapter's own error. Once Close has begun, Next returns ErrFeedClosed
and withholds any change it had received; a feed from the last cursor
still delivers it. Shared conformance cases and the reference fake
enforce this.

The JetStream feed's Close also waits, within a bound, for a consumer
open already in progress, so a consumer created during Close is deleted
before Close returns nil; past the bound, or if that delete fails, Close
returns an error and a repeated Close waits again.

* fix: track SQS feed consumer cleanup until Close succeeds

Every consumer open and release is now tracked work recorded under the
lock that takes the consumer, and a delete that fails is kept for Close
to retry. Close returns nil only once no such work is in progress and no
failed delete remains, waiting within a bound and otherwise returning an
error; concurrent and repeated Close calls wait the same way.

Next claims a change under that lock and advances its cursor there, so a
change is either returned before Close begins or withheld for a feed
resumed from the last cursor.

A consumer create no longer inherits the caller's cancellation, which
was seen to leave most cancelled creates on the server with no owner;
it has its own timeout, and the name of one that fails is deleted and
kept for Close to delete again.

* fix: release SQS feed consumers opened after the caller's context ends

Feed and recovery now check the caller's context before starting a
consumer open and after it completes. A create that finishes after the
context ended is released as tracked work and the context's error is
returned, or ErrFeedClosed when Close won; no change is returned.

The feed contract and adapter docs now describe cleanup as complete for
consumers the feed knows of, with the server's one-minute inactive
threshold removing one an ambiguous create lands after its last delete.

* fix: report cancellation when an SQS feed open fails after it

A first consumer open that fails after the caller's context ended now
returns the context's error joined with the failure, so cancellation
stays detectable. The feed contract's cancellation text is narrowed to
opening and reopening, whose cleanup is best effort.

* docs: describe the internal SQS message store and its limits

DESIGN.md gains a section on the store-neutral SQS message store, its
JetStream adapter and conformance suite: an internal primitive with no
service wired to it, its shared stream and bucket layout, leader-directed
reads and their full-body cost, feed recovery and its best-effort
consumer cleanup, and the absence of end-to-end or multi-host
validation. The Scan contract says metadata only, not headers only.

.github/e2e-cells.json

@@ -223,5 +223,27 @@
     "runner": "ci-single",
     "extra": "nat-single",
     "suite": ""
+  },
+  {
+    "cell": 37,
+    "install": "binary",
+    "topo": "real-multi",
+    "net": "dhcp",
+    "host": "oracle-10",
+    "arch": "x86_64",
+    "runner": "ci-multi",
+    "extra": "",
+    "suite": ""
+  },
+  {
+    "cell": 38,
+    "install": "binary",
+    "topo": "real-multi",
+    "net": "nat",
+    "host": "oracle-10",
+    "arch": "x86_64",
+    "runner": "ci-multi",
+    "extra": "nat-multi",
+    "suite": ""
   }
 ]

.github/workflows/e2e-nightly.yml

@@ -10,7 +10,7 @@ on:
   workflow_dispatch:
     inputs:
       cells:
-        description: "Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 34-36 Oracle Linux single node (basic, reboot, routed NAT), 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection, 30 routed NAT on three nodes, 31 instance recovery on three nodes, 32 load balancer recovery on three nodes, 33 KV bucket quorum under node loss on three nodes"
+        description: "Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 34-36 Oracle Linux single node (basic, reboot, routed NAT), 37-38 Oracle Linux on three nodes (multi-node subset with IPsec, routed NAT), 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection, 30 routed NAT on three nodes, 31 instance recovery on three nodes, 32 load balancer recovery on three nodes, 33 KV bucket quorum under node loss on three nodes"
         required: false
 
 permissions:

cmd/eks-token-webhook/main.go

@@ -29,6 +29,7 @@ func loadConfig() (tokenwebhook.Config, error) {
 		CertPath:    envOr("EKS_WEBHOOK_CERT", "/etc/spinifex-eks/token-webhook.crt"),
 		KeyPath:     envOr("EKS_WEBHOOK_KEY", "/etc/spinifex-eks/token-webhook.key"),
 		Kubeconfig:  envOr("EKS_WEBHOOK_KUBECONFIG", "/etc/spinifex-eks/token-webhook.kubeconfig"),
+		ReadyAddr:   os.Getenv("EKS_WEBHOOK_READY_ADDR"),
 	}
 	// Static SigV4 creds are optional: when absent, eksgw.New signs with the
 	// AWS SDK chain (IMDS instance-role creds), the same path the CP VM's

docs/DESIGN.md

@@ -229,6 +229,20 @@ type EBSRequest struct {
 
 Viperblock responds with an NBD URI that QEMU uses to access the block device.
 
+#### Direct-boot microVM data volumes
+
+A direct-boot microVM keeps the launcher's saved `vm.Config` as a clean base, and `startQEMU` adds data volumes only to a launch-only copy of it (`runtime/compute/vm/lifecycle.go`).
+At each start the daemon takes one snapshot of the instance's already-mounted data requests, probes each NBD or host block export from that snapshot, and builds the command from the same snapshot, so the launched endpoint is the one that was checked and is never written back into the saved base.
+Each data volume becomes a `virtio-blk-device` on the MMIO bus with the same id, node name, I/O thread, serial and `werror=stop,rerror=stop` policy as the PCI device, and takes no PCIe hot-plug port.
+
+Boot and EFI requests are rejected on this path, and a direct-boot instance with no data requests launches its base unchanged.
+Disk-boot (q35) instances keep the existing PCI hot-plug path.
+
+Hot-attaching a volume to a running direct-boot microVM is not supported, because the MMIO bus has no PCIe root port to plug into.
+`AttachVolume` refuses it before mounting the volume or changing any instance or volume state, and the EC2 API returns `UnsupportedOperation`.
+
+This covers only requests that are already mounted when the instance starts; nothing yet supplies data volumes to a system instance, and the path is verified with fake-QEMU unit tests, not with a real EBS volume in a guest.
+
 ### Predastore (S3)
 
 Object storage used for:
@@ -256,6 +270,29 @@ file. It does not connect to Predastore to do so, and no Predastore node
 exposes a status endpoint. This is an operator status interface, not a generic
 storage-provider contract.
 
+### SQS message store (internal)
+
+`domains/sqs/message` defines the store-neutral interface the SQS domain reaches its message store through (ADR-0011:S6), with a JetStream adapter in `domains/sqs/message/jetstream` and a shared conformance suite in `domains/sqs/message/conformance`.
+It is an internal primitive: nothing outside the SQS domain uses it, no daemon or service is wired to it, and it does not itself deliver an SQS API.
+The dedicated JetStream server ADR-0011 describes is not deployed by this code.
+
+The interface covers durable and idempotent append, body and metadata reads, conditional state writes and deletes, per-namespace counts, namespace enumeration, an ordered change feed, a bounded paginated scan, namespace deletion and a sweep of tombstoned bodies.
+A namespace carries an immutable message-shard and state-shard allocation chosen when the queue is created, so a later change in shard counts never moves an existing queue.
+
+The adapter provisions a fixed set of shared message streams (`SQS_M_<shard>`, one subject per queue) and shared state buckets (`sqs_state_<shard>`), keeps historical ones, and applies a 14-day-plus-one-hour retention backstop to both.
+It supports one node (R1) and three nodes (R3) only and refuses any other declared cluster size before provisioning.
+
+Reads that must observe a completed write or delete go to the stream leader: `GetState`, the state-key pass behind `Count` and `Namespaces`, and the body pass behind `Namespaces` and `Scan`.
+This avoids the follower-served direct gets and follower-placed consumers that returned purged data in local three-node fault-injection tests; it is not a general linearizability guarantee.
+Each leader read is one request per record, and the body pass transfers each full body internally even though `Scan` returns metadata only, which is a material cost for a large backlog.
+A state-key pass is bounded but is not a snapshot when a key is overwritten during it.
+
+The feed reads through an in-memory, headers-only ordered consumer.
+If that consumer delivers nothing for about two seconds while the stream holds a change after the feed's cursor, the feed replaces it from the cursor, so a stored change does not wait on a stalled consumer; the cause of those stalls is not established.
+`Close` is terminal, waits for consumer opens and deletes in progress and retries failed deletes, but cleanup is best effort for a create whose outcome is unknown: a consumer that lands after its last delete is removed by the server's one-minute inactive threshold.
+
+Coverage is unit and conformance tests on in-process R1 and R3 servers; there is no end-to-end or multi-host validation of this code, and earlier multi-host evidence predates the final feed.
+
 ## Configuration
 
 Cluster configuration (`spinifex/bootstrap/config/config.go`):

docs/admin/host-firewall/README.md

@@ -158,6 +158,7 @@ Restarting the daemon is what re-arms it: the node rebuilds its peer list from t
 
 > [!WARNING]
 > Do not flush the whole nftables ruleset to clear the policy. The VPC daemon writes SNAT and per-elastic-IP forward rules into the `ip nat` and `ip filter` tables and reinstalls them only when it starts, so `nft flush ruleset` breaks elastic IPs silently until the next `systemctl restart spinifex-vpcd`. Use `disable`, which only ever touches `inet spinifex_filter`.
+> On Oracle Linux with IPsec on, a flush also removes `inet spinifex_ipsec_guard`, so Geneve to a peer without an SA goes out in clear until the daemon puts the table back, within 60 seconds.
 
 ## The Files
 

docs/compute/vpc-networking/README.md

@@ -144,6 +144,27 @@ The guest MTU moves with the setting, but a running guest holds its DHCP lease f
 
 **Expect launches to fail for a minute or two after re-enabling.** `RunInstances` returns `ServerInternal` with `vpc.add-nat` timing out, because the OVN flows-ready barrier cannot confirm while `ovn-controller` reprograms every tunnel for the new SAs. It clears once `ip xfrm state` settles. The error is reported as a capacity problem, which it is not.
 
+### Containers and nested overlays inside a guest
+
+A Docker bridge, Kubernetes pod network or other overlay inside a guest defaults to an MTU of 1500, which is larger than the guest's 1408 or 1442.
+Downloads from the internet still work for TCP on the veth and routed uplinks, because vpcd clamps the TCP MSS on the host hop (see [Performance Tuning → MTU](/docs/performance-tuning#4-mtu)).
+That was measured on a single node (env12) with a Docker container; it was not measured on a physical uplink or on a multi-node cluster.
+
+On a physical uplink there is no host hop to clamp, and UDP is never clamped, so set the inner MTU to the guest MTU.
+For Docker, in `/etc/docker/daemon.json`:
+
+```json
+{
+  "mtu": 1408,
+  "default-network-opts": {
+    "bridge": { "com.docker.network.driver.mtu": "1408" }
+  }
+}
+```
+
+Use 1442 when `ipsec_enabled = false`, and restart the Docker daemon so the default bridge is recreated.
+The gateway MTU follows `ipsec_enabled` and converges on the next reconcile pass, so change the inner MTU together with the guest MTU.
+
 ## Private vs Public Subnets
 
 A subnet's behavior depends on three things: whether the VPC has an Internet Gateway, whether the subnet's route table has a default route to that IGW, and whether the subnet has `MapPublicIpOnLaunch` enabled.

docs/containers/ecr/README.md

@@ -45,6 +45,7 @@ Each repository holds image manifests and their layers. Images are stored in obj
 
 - Repository lifecycle — `CreateRepository`, `DeleteRepository`, `DescribeRepositories`.
 - OCI push and pull — blobs, manifests, and tags, including cross-repository blob mounts.
+- OCI 1.1 referrers — the referrers API, so cosign v3 signatures and other artifacts attached to an image can be listed (see [OCI referrers and image signing](#oci-referrers-and-image-signing)).
 - Token auth — `GetAuthorizationToken` plus bearer/basic auth on `/v2/`.
 - Repository policies — set, get, and delete a repository policy document.
 
@@ -176,6 +177,71 @@ Then authenticate Docker and push using the `repository_url` output, following s
 
 :::
 
+## EKS workers and multiple gateways
+
+Each gateway puts its own address in the `repositoryUri` it returns, so repositories created through different gateways can carry different hosts.
+When an EKS worker launches, the daemon that launches it writes every cluster gateway into the worker's registry mirrors, TLS trust and credential-provider config.
+A pod can then pull from any `repositoryUri` host in the cluster.
+The worker tries the gateway named in the image reference first and falls back to the others in cluster order.
+
+Limits:
+
+- The gateway list comes from the cluster config the launching daemon loaded at start.
+  A gateway added later is not trusted by existing workers; restart the launching daemon and replace the worker.
+- Peer gateways are assumed to listen on port 9999.
+  A hand-edited `awsgw` bind on a peer is not covered.
+- A gateway address that does not answer costs about 30 seconds on each cold pull while the worker's connection attempt gives up.
+  A stopped gateway on a host that is still up refuses the connection, and the worker moves to the next gateway at once.
+
+Measured on a single-node environment: pulls by `repositoryUri` and by the account's ECR DNS name, a pull rejected with `x509: certificate signed by unknown authority` when the gateway host was not in the worker config, and a 31 second pull through an unreachable first endpoint.
+Measured on a 3-node environment: a pod pulling each of the three gateways' `repositoryUri` on a worker launched with this change (two of the three failed with `x509` on a worker launched without it), and a pull of the stopped gateway's own `repositoryUri` that succeeded through another gateway within one retry (the pull itself took 147 ms after the first endpoint refused).
+
+## OCI referrers and image signing
+
+A manifest can name another manifest as its `subject`, which is how tools such as cosign v3 attach signatures and SBOMs to an image.
+The registry records the subject when the manifest is pushed and answers with an `OCI-Subject` header on the `201`.
+The `subject` does not have to exist in the repository when the referrer is pushed.
+
+`GET /v2/<name>/referrers/<digest>` lists the manifests that name `<digest>` as their subject.
+The request is authorised as `ecr:BatchGetImage` on the repository.
+The response is `200` with `Content-Type: application/vnd.oci.image.index.v1+json` and an index whose `manifests` hold each referrer's `mediaType`, `digest`, `size`, `artifactType` and annotations.
+A digest with no referrers, or one that is not stored, returns an empty index.
+A malformed digest returns `400` with `DIGEST_INVALID`.
+
+Adding `?artifactType=<type>` returns only referrers of that type and sets `OCI-Filters-Applied: artifactType`.
+The header is also set when nothing matches.
+An image manifest without an `artifactType` is listed under its config `mediaType`; an index without one has none.
+
+Deleting an image by digest (`DELETE /v2/<name>/manifests/<digest>`, `aws ecr batch-delete-image` with `imageDigest`) also deletes its referrers, and their referrers in turn.
+Deleting by tag removes only the tag and leaves the image and its referrers.
+
+cosign v3 sign and verify work against the registry, including on a child manifest of an image index copied with `cosign copy`.
+cosign v2 keeps writing its `sha256-<digest>.sig` tags and is unaffected.
+`GET` on a manifest still returns `406` when the request's `Accept` header does not include the stored `mediaType`.
+
+Limits:
+
+- `cosign copy` does not copy v3 signatures to the destination repository.
+  This is client behaviour.
+  To keep the original signature, copy with `oras cp -r <source> <mirror>`, which copies the image and its referrers in one command, or `crane copy` the signature manifest by digest after the image.
+  Otherwise, sign the copy again.
+- Referrers pushed before this version are not listed.
+  Push them again to add them to the index.
+- A crash between deleting an image and deleting its referrers can leave referrers behind that nothing lists.
+- The referrers listing is not paginated.
+- `HEAD` on the referrers route is not served.
+
+Measured on a 3-node environment with cosign v3.1.4 and v2.6.5:
+
+- cosign v3.1.4 signed a fresh image index, with one `GET /referrers` answered `200`, no `sha256-<digest>` tag written and no `406`.
+- Verify passed.
+- A copied index's child manifest signed and verified, which returned `406` before this change.
+- The listing returned the signature with its `artifactType`, the filter returned one entry for the signature's type and none for another type.
+- `aws ecr batch-delete-image` of the signed image left an empty listing and `404` for both the signature and the image.
+- cosign v2.6.5 signed and verified a fresh image.
+
+Not measured: `oras`, `notation`, a listing with many referrers, and a crash in the middle of a delete.
+
 ## Troubleshooting
 
 **`docker login` fails with an authorization error.** The token is account-scoped and short-lived. Re-run `aws ecr get-login-password` to mint a fresh token, and confirm the registry host in `docker login` matches your account's endpoint (`aws ecr describe-repositories` returns it).

docs/containers/eks/README.md

@@ -60,6 +60,36 @@ Authentication uses **EKS access entries** (the API authentication mode) — IAM
 - **The worker image is fixed.** Node groups always boot Spinifex's `eks-node` image. `ami_type` is recorded but does not select the image, and the image must be registered on the cluster before you create a node group.
 - **Default Kubernetes version is `1.32`.**
 
+### Control plane readiness and kubectl authentication
+
+Each control-plane VM authenticates `kubectl` tokens through a webhook that calls the AWS gateway on the host the VM runs on.
+When that gateway is stopped, the webhook cannot validate tokens and `kubectl` reports `Unauthorized` for the share of requests that the load balancer sends to that control plane.
+
+The webhook serves `/readyz` on port 8444 of each control-plane VM.
+It returns 200 only when the host gateway answers a TLS handshake and the local API server accepts a TCP connection, and 503 otherwise.
+The API server target group health-checks `HTTP :8444/readyz` every 5 seconds with a healthy and unhealthy threshold of 2.
+The target group still forwards TCP on 6443.
+A control plane that cannot authenticate leaves rotation, and returns when its gateway is back.
+
+Measured on a three-node cluster, one control plane per host, with `kubectl get ns` run continuously after stopping one gateway:
+
+| Gateway stopped | Result |
+| --- | --- |
+| Host without the cluster load balancer VM, first run | 4 of 20 calls failed in the first round, 0 of 20 in the next round 54 s after the stop |
+| Host without the cluster load balancer VM, second run | 2 failures in 100 s, at 3 s and 8 s after the stop |
+| Host running the cluster load balancer VM | Not fixed: 20 of 60 calls failed over 100 s |
+
+Before this change the same test failed 3, 4 and 3 calls of 10 per stopped gateway.
+
+Limits:
+
+- Only clusters created with the current `eks-node` image get the readiness listener, and only clusters created from now on get the health check on the target group.
+  Older clusters keep the TCP check on 6443 and are not changed.
+- A cluster created with an older image gets the health check on its target group but no listener on 8444, so every control plane fails the check and the load balancer fails open to all of them.
+- A stopped gateway on the host that runs the cluster load balancer VM is not covered.
+  The load balancer VM reports target health through its own host's gateway, so while that gateway is down the unhealthy control plane stays in rotation.
+- Not measured: more than one control plane per host, a physical-node failure instead of a stopped gateway, and the target health state as `DescribeTargetHealth` reports it, because the load balancer and control-plane VMs belong to the system account.
+
 ## Prerequisites
 
 > [!IMPORTANT]
@@ -256,6 +286,10 @@ The workbooks form a ladder. **eks-quickstart** is a NodePort demo; **eks-https-
 
 **`kubectl` cannot connect.** Re-run `aws eks update-kubeconfig --name <name>` to refresh the endpoint and CA, and confirm the cluster is `ACTIVE`. `aws sts get-caller-identity` verifies your credentials reach the Spinifex endpoint.
 
+**`kubectl` returns `Unauthorized` for some requests and works for others.**
+One control plane cannot reach its host's AWS gateway, so check `systemctl status spinifex-awsgw` on each node.
+Control plane readiness and kubectl authentication above lists which clusters recover on their own.
+
 **Workers stay `NotReady`.** Almost always an egress or IAM problem:
 - Confirm the workers can pull images — a default route to an Internet Gateway (public) or NAT Gateway (private).
 - Confirm the node role has `AmazonEKSWorkerNodePolicy`, `AmazonEKS_CNI_Policy`, and `AmazonEC2ContainerRegistryReadOnly`.

docs/coverage/elbv2/README.md

@@ -24,6 +24,34 @@ Spinifex implements **36 operations** in the ELBv2 `2015-12-01` API model.
 
 The data plane is a system-managed load balancer VM, launched automatically during `CreateLoadBalancer`. Application Load Balancers run HAProxy for the L7 surface — rules, fixed responses and redirects over HTTP and HTTPS. Network Load Balancers run nginx `stream` for the L4 surface — TCP, UDP, TLS and TCP_UDP — because HAProxy cannot load-balance UDP. The agent selects the engine from the configuration the control plane delivers, so the choice follows the load balancer type and is not separately configurable.
 
+### Target health and routing
+
+Traffic goes only to targets that are healthy or still `initial`.
+A target that has failed its health check is removed from the data plane on the next render and put back when it passes again.
+When every target in a group is unhealthy, the load balancer fails open and sends traffic to all of them rather than refusing it.
+A Network Load Balancer renders every registered server in that case, and an Application Load Balancer falls back to unchecked backup servers that HAProxy uses only when every checked server is down.
+`DescribeTargetHealth` keeps reporting the targets as unhealthy while this happens.
+Draining targets and targets without an address never receive traffic.
+
+An `initial` target receives traffic before its first successful check.
+This differs from AWS, where a target takes no traffic until it passes.
+Health state is reset to `initial` for every target when the control plane restarts, so excluding `initial` targets would drop all traffic after a restart.
+
+Changing the health check of a target group with `ModifyTargetGroup` takes effect on the next render of every load balancer that uses it.
+The time from a backend failing to traffic stopping is the health check interval times the unhealthy threshold, plus the agent poll of about 5 seconds and the heartbeat that reports the result.
+Recovery takes the interval times the healthy threshold, plus the same delays.
+
+Measured on a three-node cluster with a 5 second interval and thresholds of 2:
+
+| Event | Time |
+| --- | --- |
+| Backend starts failing to reported `unhealthy` | 21 s |
+| Backend recovers to reported `healthy` | 15 s, serving again 6 s later |
+| Every backend failing to reported `unhealthy` | 12 s, fail open immediate |
+
+A TCP connection held through a Network Load Balancer survived four configuration reloads while the other backend changed state, because the agent applies nginx with a graceful reload and HAProxy with a graceful restart.
+The HAProxy configuration with the backup servers was accepted by a live Application Load Balancer, but its rendered text was not dumped and checked by hand.
+
 ### Operations
 
 | Operation |

docs/coverage/elbv2/intro.md

@@ -1,3 +1,31 @@
 ### Two data planes
 
 The data plane is a system-managed load balancer VM, launched automatically during `CreateLoadBalancer`. Application Load Balancers run HAProxy for the L7 surface — rules, fixed responses and redirects over HTTP and HTTPS. Network Load Balancers run nginx `stream` for the L4 surface — TCP, UDP, TLS and TCP_UDP — because HAProxy cannot load-balance UDP. The agent selects the engine from the configuration the control plane delivers, so the choice follows the load balancer type and is not separately configurable.
+
+### Target health and routing
+
+Traffic goes only to targets that are healthy or still `initial`.
+A target that has failed its health check is removed from the data plane on the next render and put back when it passes again.
+When every target in a group is unhealthy, the load balancer fails open and sends traffic to all of them rather than refusing it.
+A Network Load Balancer renders every registered server in that case, and an Application Load Balancer falls back to unchecked backup servers that HAProxy uses only when every checked server is down.
+`DescribeTargetHealth` keeps reporting the targets as unhealthy while this happens.
+Draining targets and targets without an address never receive traffic.
+
+An `initial` target receives traffic before its first successful check.
+This differs from AWS, where a target takes no traffic until it passes.
+Health state is reset to `initial` for every target when the control plane restarts, so excluding `initial` targets would drop all traffic after a restart.
+
+Changing the health check of a target group with `ModifyTargetGroup` takes effect on the next render of every load balancer that uses it.
+The time from a backend failing to traffic stopping is the health check interval times the unhealthy threshold, plus the agent poll of about 5 seconds and the heartbeat that reports the result.
+Recovery takes the interval times the healthy threshold, plus the same delays.
+
+Measured on a three-node cluster with a 5 second interval and thresholds of 2:
+
+| Event | Time |
+| --- | --- |
+| Backend starts failing to reported `unhealthy` | 21 s |
+| Backend recovers to reported `healthy` | 15 s, serving again 6 s later |
+| Every backend failing to reported `unhealthy` | 12 s, fail open immediate |
+
+A TCP connection held through a Network Load Balancer survived four configuration reloads while the other backend changed state, because the agent applies nginx with a graceful reload and HAProxy with a graceful restart.
+The HAProxy configuration with the backup servers was accepted by a live Application Load Balancer, but its rendered text was not dumped and checked by hand.

docs/install/install-multi-node/README.md

@@ -163,11 +163,14 @@ Open between all hosts:
 | Protocol and port | Used by |
 |---|---|
 | UDP 6081 | Geneve tunnels |
+| UDP 500, 4500 and IP protocol 50 (ESP) | IPsec for the Geneve tunnels, while IPsec is on |
 | TCP 4222, 4248 | NATS |
 | TCP 6641, 6642 | OVN northbound and southbound |
 | TCP 8443 | Predastore S3 gate |
 | UDP 6660, 7660 | Object shards, metadata consensus |
 
+UDP 4500 carries the tunnel traffic only where IKE detects NAT between two nodes; otherwise it travels as ESP, on Oracle Linux as on Debian.
+
 Open between the three OVN database servers (servers 1 to 3) only:
 
 | Protocol and port | Used by |
@@ -207,7 +210,10 @@ Spinifex's own **host** firewall is separate, and already carries this policy
 
 Choose one method and apply it to **every** server in the cluster.
 
-**Option A — existing OS.** On a server already running Ubuntu 26.04 or Debian 13 (Oracle Linux runs a single node only for now, see [Oracle Linux Hosts](/docs/install-oracle-linux)):
+**Option A — existing OS.** Use this on servers already running Ubuntu 26.04, Debian 13 or Oracle Linux 10, and prepare an Oracle Linux server with [Oracle Linux Hosts](/docs/install-oracle-linux) first.
+Every server in a cluster must run the same OS: Oracle Linux never shares a cluster with Debian or Ubuntu, and `spx admin join` does not check this for you.
+
+Run the installer on each server:
 
 ```bash
 curl -fsSL https://install.mulgadc.com | bash

docs/install/install-oracle-linux/README.md

@@ -1,12 +1,13 @@
 ---
 title: "Oracle Linux Hosts"
 seoTitle: "Install Spinifex on Oracle Linux 10 — Spinifex Docs"
-description: "Prepare an Oracle Linux 10 host for Spinifex: UEK R8, SELinux, a NetworkManager br-wan bridge, the CentOS NFV SIG repository, firewalld handover and updates."
+description: "Prepare Oracle Linux 10 hosts for Spinifex: UEK R8, SELinux, a NetworkManager br-wan bridge, the NFV SIG repository, IPsec on libreswan and host updates."
 category: "Install"
 tags:
   - install
   - oracle linux
   - single node
+  - multi node
 resources:
   - title: "Oracle Linux 10 Documentation"
     url: "https://docs.oracle.com/en/operating-systems/oracle-linux/10/"
@@ -35,13 +36,12 @@ Follow it alongside [Single-Node Install](/docs/install).
 
 **Supported on Oracle Linux:**
 
-- A single node.
+- A single node, or a cluster in which every node runs Oracle Linux; servers beyond the third join as compute nodes, as on Debian.
 - Guests, VPCs, routed NAT, Elastic IPs, and Application and Network Load Balancers.
 - Whole-GPU passthrough, on x86_64 GPU hosts (see [GPU Passthrough](/docs/gpu-passthrough)).
 
 **Not supported on Oracle Linux yet:**
 
-- Clusters of more than one node. They wait for IPsec on libreswan, so `spx admin init` turns IPsec off on Oracle Linux and refuses `--ipsec=true`.
 - Clusters that mix Oracle Linux with Debian or Ubuntu nodes, on any release.
 - SELinux enforcing, aarch64, Oracle Linux 9 or earlier, the Red Hat Compatible Kernel, a ZFS root, and NVIDIA MIG.
 
@@ -51,7 +51,7 @@ Spinifex installs `openvswitch3.5` and `ovn25.03`, the versions Debian 13 ships,
 QEMU comes from Oracle's KVM Utilities repository, because only that build has the `microvm` machine type Spinifex load balancers boot on.
 
 An Oracle Linux node must be a dedicated Spinifex host.
-Spinifex claims every `tap*` device and stops libreswan's `ipsec.service`, so VPNs, hand-run QEMU guests and libreswan tunnels of your own cannot share the host.
+Spinifex claims every `tap*` device and owns libreswan's `ipsec.service`, which it keeps stopped on a single node and runs for its own tunnels on a cluster node, so VPNs, hand-run QEMU guests and libreswan tunnels of your own cannot share the host.
 Crypto policies other than `DEFAULT` and `FIPS` are untested, and so is Secure Boot.
 
 ## Prerequisites
@@ -134,8 +134,28 @@ Run the installer as a user with `sudo`, exactly as on Debian:
 curl -fsSL https://install.mulgadc.com | bash
 ```
 
-Then continue with [Single-Node Install](/docs/install) from Step 2: `setup-ovn.sh --management`, `spx admin init` and `systemctl start spinifex.target`.
-`spx admin init` prints that IPsec is off on this host; on a single node there is no peer to encrypt traffic to.
+For a single node, continue with [Single-Node Install](/docs/install) from Step 2: `setup-ovn.sh --management`, `spx admin init` and `systemctl start spinifex.target`.
+IPsec stays off on a single node, because there is no peer to encrypt traffic to.
+
+For a cluster, prepare and install every node this way, then continue with [Multi-Node Install](/docs/install-multi-node) from Step 2.
+
+### Clusters
+
+Every node in an Oracle Linux cluster runs Oracle Linux.
+`install-node.sh` refuses a host list that mixes Oracle Linux with Debian or Ubuntu, but `spx admin join` does not check, so keep to one OS yourself when joining by hand.
+
+IPsec encrypts the Geneve tunnels between nodes and is on by default, as on Debian.
+On Oracle Linux it runs on libreswan, the IKE daemon Oracle ships, rather than strongSwan.
+Once the node's IPsec certificate exists, the daemon unmasks and starts `ipsec.service` and `openvswitch-ipsec.service`.
+`ovs-monitor-ipsec` then loads the node certificate, its key and the cluster CA into libreswan's NSS database at `/var/lib/ipsec/nss`, under nicknames that start with `ovs_cert`.
+
+The tunnels are plain ESP (IP protocol 50), as on Debian, negotiated over UDP 500.
+Open UDP 500, UDP 4500 and ESP between nodes: libreswan moves to UDP 4500 if it detects NAT between two nodes.
+The Spinifex host firewall opens all three to cluster peers; a firewall between the nodes must allow them too.
+While IKE to a peer fails, the node drops its Geneve traffic to that peer rather than send it in clear.
+The nft table `inet spinifex_ipsec_guard` drops outbound UDP 6081 from the encap address whose route carries no SA, which also covers the moments pluto flushes the xfrm policy table as it starts and stops.
+It is not persisted: `ipsec.service` puts it back at boot before ovn-controller builds the tunnels, and the daemon re-syncs it every 60 seconds.
+`spx admin init --ipsec=false` leaves the tunnels unencrypted, as on any OS.
 
 ## Step 2. Know What the Installer Changed
 
@@ -182,7 +202,9 @@ Those advisories do not cover the NFV SIG packages, so follow Open vSwitch and O
 
 ## Step 5. Uninstall
 
-`/usr/local/share/spinifex/uninstall-spx.sh` removes Spinifex as on Debian, and lists what it leaves in place on Oracle Linux: the repository changes and the SIG key, `OVS_USER_ID` and `OVN_USER_ID`, the OVS and OVN logrotate edits, the chrony include, `/etc/modules-load.d/spinifex.conf`, the SELinux mode, a masked `ipsec.service`, firewalld disabled, and any GPU passthrough configuration.
+`/usr/local/share/spinifex/uninstall-spx.sh` removes Spinifex as on Debian.
+On a cluster node it also stops `openvswitch-ipsec.service` and deletes the `ovs_cert*` entries from libreswan's NSS database, so the node can join a different cluster; other NSS identities and your connection files in `/etc/ipsec.d` stay.
+It lists what it leaves in place on Oracle Linux: the repository changes and the SIG key, `OVS_USER_ID` and `OVN_USER_ID`, the OVS and OVN logrotate edits, the chrony include, `/etc/modules-load.d/spinifex.conf`, the SELinux mode, a masked `ipsec.service`, firewalld disabled, and any GPU passthrough configuration.
 `--purge-deps` is refused on Oracle Linux, because dnf cannot tell the packages Spinifex installed from the host's own; remove them with `dnf remove` yourself.
 
 ## Troubleshooting
@@ -224,6 +246,20 @@ If it prints `yes`, run `sudo nmcli connection modify br-wan bridge.stp no` and
 
 Check that `/usr/bin/spx` links to `/usr/local/bin/spx`; run the installer again if it does not, or call `sudo /usr/local/bin/spx` meanwhile.
 
+### Cluster nodes form no IPsec tunnels
+
+Check that both IPsec services run, and that libreswan holds the node's certificate and the cluster CA:
+
+```bash
+systemctl is-active ipsec.service openvswitch-ipsec.service
+sudo certutil -L -d sql:/var/lib/ipsec/nss     # ovs_certkey_<node>, ovs_cert_cacert
+sudo ipsec trafficstatus                       # one line per peer tunnel
+```
+
+If the services are inactive, check the daemon log with `journalctl -u spinifex-daemon` for the IPsec reconcile.
+If the certificates are present but no tunnel is listed, check that UDP 500 and ESP pass between the nodes in both directions, and look for `SEC_ERROR_UNKNOWN_ISSUER` in `journalctl -u ipsec`, which means a peer's certificate is from another CA.
+Until the tunnel forms, guest traffic between those two nodes is dropped, which is the guard in [Clusters](#clusters) working.
+
 ### CA certificate not trusted
 
 `sudo spx admin init` installs the CA automatically. To install it again by hand on Oracle Linux:

docs/install/install/README.md

@@ -39,7 +39,7 @@ This guide installs Spinifex on a single server using the binary installer. For
 
 - Ubuntu 26.04
 - Debian 13
-- Oracle Linux 10.2 or a later 10.x, on UEK R8, x86_64 only, as a single node. Open vSwitch and OVN come from the CentOS NFV SIG, a community repository that Oracle does not support. Prepare the host with [Oracle Linux Hosts](/docs/install-oracle-linux) first.
+- Oracle Linux 10.2 or a later 10.x, on UEK R8, x86_64 only, as a single node or, with [Multi-Node Install](/docs/install-multi-node), a cluster. Open vSwitch and OVN come from the CentOS NFV SIG, a community repository that Oracle does not support. Prepare the host with [Oracle Linux Hosts](/docs/install-oracle-linux) first.
 
 **What Gets Installed:**
 

docs/install/performance-tuning/README.md

@@ -269,6 +269,35 @@ ip link set <dev> mtu 9000        # both ends
 ping -M do -s 8972 -c 3 <peer>    # must succeed
 ```
 
+### The internet path
+
+A guest MTU of 1408 or 1442 sits below the 1500 bytes the internet carries, so a server can send the guest segments it cannot receive.
+The VPC gateway port carries `gateway_mtu = min(guest MTU, 1500)` and answers an oversized packet with ICMP fragmentation-needed (IPv4) or Packet Too Big (IPv6).
+It follows `ipsec_enabled` and `network.underlay_mtu` and converges on the next reconcile pass, with no VPC recreate.
+
+Path MTU discovery depends on that ICMP reaching the server, and an upstream NAT router or a CDN often drops it.
+vpcd therefore also clamps the TCP MSS on the host hop to the gateway MTU minus 40 (IPv4) or minus 60 (IPv6), and re-asserts the rule every 30 seconds so a firewall reload cannot remove it for long.
+
+| Uplink | Clamp | Table |
+| --- | --- | --- |
+| veth (distributed NAT) | yes | `bridge spinifex_mss`, matching the `veth-wan-br` port |
+| routed NAT | yes | `inet spinifex_mss`, matching `spx-nat-host` |
+| physical | no host hop exists | path MTU discovery only |
+
+On a physical uplink the only cover is path MTU discovery, which fails when the upstream drops the ICMP or the peer ignores it.
+
+Measured on one node (env12), IPsec on, Ubuntu 26.04 guest, 16 MB HTTPS download inside a Docker container at the default bridge MTU of 1500:
+
+| Uplink | Without the clamp | With the clamp |
+| --- | --- | --- |
+| veth, instance with a public IP | stalled 3/3 | completed 3/3 |
+| routed NAT | stalled 3/3, no ICMP seen on the host | completed 3/3 |
+
+An instance behind a NAT gateway on the veth uplink also completed 3/3 with the clamp; its unclamped internet run was not measured.
+The guest's own `curl` completed in every case.
+Without the clamp, `gateway_mtu` alone fixed the download only against a server on the same L2 with no NAT in between (completed 3/3), and not against the internet through the lab's upstream NAT router.
+Not measured: a physical uplink, a multi-node cluster, IPv6, and a Debian guest.
+
 ## 5. Measurement notes
 
 Throughput on this stack is **noisy** — repeated identical 20s runs varied between

docs/security/network-connections/README.md

@@ -111,8 +111,8 @@ behavior in other words.
 | 6643 | OVN Northbound DB (RAFT) | OVSDB/TCP | Cluster | NB database RAFT replication between the 3 quorum nodes | Cluster network only; TLS planned |
 | 6644 | OVN Southbound DB (RAFT) | OVSDB/TCP | Cluster | SB database RAFT replication between the 3 quorum nodes | Cluster network only; TLS planned |
 | 6081 | OVN (Geneve) | UDP | Encap | Tenant traffic overlay between chassis. A kernel UDP-tunnel socket, so packets are delivered locally and traverse the host's netfilter input hook before OVS sees them — a host firewall must accept them explicitly. The socket is opened by the kernel tunnel driver and takes no bind address, so it binds the wildcard by design and reach must be restricted by firewall. | None — see 500/4500 |
-| 500, 4500 | strongSwan `charon` | IKEv2 / UDP | Encap | IKE and NAT-T for the IPsec protecting Geneve, managed entirely by `ovs-monitor-ipsec`. Binds the wildcard by design (the upstream strongSwan default, accepted rather than overridden), so reach must be restricted by firewall. | Certificate-based, against the cluster CA |
-| — | ESP | IP proto 50 | Encap | The IPsec payload itself, once IKE has negotiated an SA | Cluster CA |
+| 500, 4500 | strongSwan `charon`, or libreswan `pluto` on Oracle Linux | IKEv2 / UDP | Encap | IKE and NAT-T for the IPsec protecting Geneve, managed entirely by `ovs-monitor-ipsec`; 4500 carries the ESP payload only where IKE detects NAT between two nodes. `charon` binds the wildcard by design (the upstream strongSwan default, accepted rather than overridden); `pluto` binds each host address separately, the WAN address included, which is no narrower. Either way reach must be restricted by firewall. | Certificate-based, against the cluster CA |
+| — | ESP | IP proto 50 | Encap | The IPsec payload itself, once IKE has negotiated an SA, on Oracle Linux as on Debian. | Cluster CA |
 | 8222 | spinifex-nats (monitoring) | HTTP | Localhost | `varz`/`subsz` metrics consumed by the daemon | Loopback only |
 | 323 | chronyd | NTP | Localhost | Time sync client control socket | Loopback only |
 | 169.254.169.254:80 | spinifex-vpcd (IMDS) | HTTP | Guest | Instance metadata service. One socket per instance, bound to that instance's `ime-*` interface. Terminates on the host, so it traverses the netfilter input hook. | Instance identity by interface; IMDSv2 tokens |
@@ -177,7 +177,7 @@ Control-plane and data-plane traffic between Spinifex nodes, for completeness an
 | OVN NB/SB (client) | 6641 / 6642 | Cluster network only (TLS planned) | Network control plane; vpcd and ovn-controller dial the quorum |
 | OVN NB/SB (RAFT) | 6643 / 6644 | Cluster network only (TLS planned) | NB/SB database replication across the 3 quorum nodes |
 | OVN tunnels (Geneve) | UDP 6081 | Encapsulated by IPsec when `network.ipsec_enabled` is true, which is the default on multi-node clusters | Tenant traffic overlay between chassis, on the `vpc` plane |
-| OVN IPsec (IKE / NAT-T / ESP) | UDP 500, UDP 4500, IP proto 50 | Certificate-based against the cluster CA, negotiated by `ovs-monitor-ipsec` | Protects the Geneve tunnels above. OVN-native only — no layer manages strongSwan directly. |
+| OVN IPsec (IKE / NAT-T / ESP) | UDP 500, UDP 4500, IP proto 50 | Certificate-based against the cluster CA, negotiated by `ovs-monitor-ipsec` | Protects the Geneve tunnels above. OVN-native only: no layer configures strongSwan or libreswan directly. On Oracle Linux the daemon also blocks outbound Geneve that no SA covers, so a peer failing IKE gets no plaintext. |
 | Instance DNS forwarding | UDP/TCP 5300 | None | Each node's per-instance DNS shim forwards guest queries to peer nodes' northstar `:5300` |
 
 Nodes **must** sit on a network segment that is not routed to tenant/guest VMs or to the internet. The Predastore blob and meta transports and the OVN DBs are cluster-internal and must not be reachable from anywhere else.
@@ -239,6 +239,7 @@ Notes that make the difference between this working and locking you out:
 
 - **Use a dedicated table and never flush the others.** `vpcd` writes MASQUERADE and per-EIP FORWARD rules into the `ip filter` and `ip nat` tables, and only reinstalls them when the service starts. `iptables -F`, `nft flush ruleset`, `ufw enable` and `firewalld` all destroy them silently, and instance networking stays broken until the next restart.
 - **Do not add a `forward` hook.** nftables evaluates every table registered on a hook and any `drop` is final, so a default-deny forward chain here would override `vpcd`'s accepts in the other table and break every routed-NAT instance and every EIP. Filtering forwarded guest traffic is the security group's job, not the host firewall's.
+- **`vpcd` owns a second table named `spinifex_mss`, separate from `inet spinifex_filter`.** It is a `bridge` table on the veth uplink and an `inet` table on the routed uplink, it has a `forward` hook with policy `accept`, and it only rewrites the MSS option of TCP SYN packets on the host hop to the gateway. It never drops or accepts traffic and does not replace the forward-hook advice above. `vpcd` re-installs it every 30 seconds, so a flush is repaired within that interval, and there is none on a physical uplink.
 - **`output` is deliberately untouched.** The metadata service's reply path egresses through per-instance policy routing; filtering `output` breaks it in ways that are hard to attribute.
 - **On a single-NIC node the cluster and encap sets are the node's public addresses**, because the planes collapse. The rules are still correct, but they are no longer a boundary — an upstream ACL is the only real control there.
 
@@ -250,7 +251,7 @@ Every listener and outbound destination is controlled by one of these files. Cha
 
 | File | Keys | Controls |
 |------|------|----------|
-| `/etc/spinifex/spinifex.toml` | `nodes.<node>.{awsgw,nats,predastore,daemon}.host`, `nodes.<node>.vpcd.ovn_{nb,sb}_addr`, `nodes.<node>.daemon.dev_networking`, `network.ipsec_enabled` | Per-service bind addresses/ports; dev-mode QEMU port forwarding. `network.ipsec_enabled` (default `true`) decides whether OVN-native IPsec protects the Geneve tunnels, and therefore whether `charon` listens on 500/4500 — single-node clusters never enable it. |
+| `/etc/spinifex/spinifex.toml` | `nodes.<node>.{awsgw,nats,predastore,daemon}.host`, `nodes.<node>.vpcd.ovn_{nb,sb}_addr`, `nodes.<node>.daemon.dev_networking`, `network.ipsec_enabled` | Per-service bind addresses/ports; dev-mode QEMU port forwarding. `network.ipsec_enabled` (default `true`) decides whether OVN-native IPsec protects the Geneve tunnels, and therefore whether the IKE daemon (`charon`, or `pluto` on Oracle Linux) listens on 500/4500 — single-node clusters never enable it. |
 | `/etc/spinifex/nats.conf` | `listen`, `cluster.listen`, `cluster.routes`, `http`, `tls`, `cluster.authorization` | NATS client/cluster/monitoring listeners, peer routes, TLS, cluster token. |
 | `/etc/spinifex/predastore/predastore.toml` | `[[host]].bind_addr`, `[[host]].addr`, `[[host]].tls_cert`, `[[host]].tls_key`, `[[host.node]].role`, `[[host.node]].port` | Predastore host and node layout: `bind_addr` is the address the host's sockets bind, `addr` is the address peer hosts dial it on, and both carry no port — the nodes pinned to the host supply their own. The service's `--host`/`--port` override the bind address and the gate's S3 port. |
 | `/etc/spinifex/northstar/northstar.toml` | `listen`, `forwarders` | DNS listen addresses (`:53` on the advertise address, `:5300` wildcard) and upstream resolvers. |

docs/testing/e2e-coverage.md

@@ -40,7 +40,7 @@ Runs **on** a node, shelling out to `ip`/`iptables`/`ovn-nbctl` locally. Skips u
 
 Phases 7 and 8 are gated on a cluster by what the datapath can actually do: the transit segment is a localnet, so host-to-guest Tier 1 ingress only exists where the VPC's gateway router port and the guest are both on this host. The VPC ingress route itself is installed on every node, so phase 8 checks it unconditionally.
 
-Three nightly cells run it: `nat-single` (cell 19), its Oracle Linux twin (cell 36) and `nat-multi` (cell 30). Cell 20 is the bare-metal job and is not in the permutation matrix at all, which is why the multi-node NAT cell is numbered past the end rather than filling a gap.
+Four nightly cells run it: `nat-single` (cell 19) and `nat-multi` (cell 30), and their Oracle Linux twins (cells 36 and 38). Cell 20 is the bare-metal job and is not in the permutation matrix at all, which is why the multi-node NAT cell is numbered past the end rather than filling a gap.
 
 ### `multinode` — behaviour that only exists on more than one node
 
@@ -50,6 +50,9 @@ Three nightly cells run it: `nat-single` (cell 19), its Oracle Linux twin (cell
 
 **`TestMultinodeDNS` convergence budgets are env-tunable.** `SPINIFEX_MULTINODE_DNS_TOPOLOGY_TIMEOUT` (default `90s`) bounds every node agreeing on the NS topology, `SPINIFEX_MULTINODE_DNS_TIMEOUT` (default `2m`) bounds a record becoming answerable from a guest or a node, and `SPINIFEX_MULTINODE_DNS_POLL` (default `3s`) is the poll interval for both. Values are Go durations; a malformed or non-positive value panics rather than falling back to the default.
 
+**`TestMultinodeOVNRaft` restarts each node's OVN central in turn before it kills the NB leader.**
+A `setup-ovn.sh` rerun restarts a node's NB and SB members together, on EL through the one `ovn-northd.service`, so the test does the same on every database node in turn, waits for both members to be back as cluster members under a known leader, and requires an NB write to land after each step and the full quorum with one leader at the end.
+
 **`TestMultinodeIPSec` proves guest traffic is encrypted on the wire, not only that IPsec is configured.**
 It first requires every node to carry the OVS certificate, key and CA pointers, `NB_Global.ipsec=true` never asserted over an unconfigured chassis, and an AES-GCM SA in `ip xfrm state`.
 None of those fails when SAs exist but traffic bypasses them, so it then launches two instances on distinct nodes and sends 200 pings of 1000 bytes between them.
@@ -112,6 +115,9 @@ CA installed and downloadable, fingerprint matches config, hostnames present in
 
 Internal and internet-facing ALB and NLB, HTTPS listeners, listener rules, `ModifyListener`, and NLB UDP.
 
+Unhealthy-target routing: an NLB with an HTTP health check on a TCP target group stops sending traffic to a target that fails it while still accepting TCP, sends traffic again once it recovers, and fails open when every target is unhealthy.
+An ALB whose targets all fail their health check keeps serving from them instead of answering 503.
+
 ### `eks`, `ecs`, `ecr` — container services
 
 `eks`: cluster create/delete, access entries, addon delivery, `GetToken`, kubeconfig artifacts, the EBS CSI volume path, and an IRSA pod wired with a projected SA token. `ecs`: clusters, container instances, the three network modes (`awsvpc`, `bridge`, `host`), task-role credentials fetched from the container credentials endpoint, a service behind an ELB, and ENI release on terminate. `ecr`: control-plane CRUD plus a real OCI data plane — docker login/build/push/pull round trips, multi-layer images, and raw blob HTTP.

scripts/README.md

@@ -7,7 +7,7 @@ Node installation, dev-environment lifecycle, guest image builds, and verificati
 | Script | What it does |
 |---|---|
 | `setup.sh` | The production binary installer, served from `install.mulgadc.com`. Installs deps, users, sudoers, the host firewall policy, files, systemd units, logrotate and udev rules, and loads `openvswitch`, `geneve`, `vhost_net` and `tun`, listing them in `/etc/modules-load.d/spinifex.conf` for later boots. It masks firewalld wherever it is installed, since the Spinifex nftables policy is the only supported node firewall. An unset `--firewall` keeps the mode recorded in `/etc/spinifex/firewall/mode`, and otherwise arms the policy when firewalld was active. Where systemd-resolved is installed but not running, resolvconf is absent and NetworkManager runs, it hands host DNS to resolved: it saves `/etc/resolv.conf` once to `/var/lib/spinifex-reset/resolv.conf.orig`, links `/etc/resolv.conf` to the resolved stub, and stops unless resolved has upstream servers and the package repository host still resolves. It links `/usr/bin/spx` to `/usr/local/bin/spx` where sudo's PATH leaves out `/usr/local/bin`, and takes the nbdkit plugin directory from `nbdkit --dump-config`, stopping when nbdkit cannot say. `SETUP_STAGES` runs a subset; `ISO_BUILD=1` runs it inside the ISO builder's chroot. |
-| `setup-ovn.sh` | Bootstraps a compute node for OVN VPC networking: installs OVN/OVS, creates `br-int`, configures the WAN bridge (bridged, DHCP, or routed-NAT uplink), sets the chassis identity, and applies overlay sysctls. It runs OVS and OVN as root, setting `OVS_USER_ID` and `OVN_USER_ID` where `/etc/sysconfig` holds them, and hooks the socket-permission helper onto `ovs-vswitchd`, `ovn-controller`, `openvswitch-ipsec` and, on EL, `ovn-northd`. `--management` also starts the OVN central services, writes their `ovn-ctl` flags to `/etc/default/ovn-central` (Debian) or `/etc/sysconfig/ovn-northd` (EL), and records the role in `/etc/spinifex/ovn-role`. A later run without `--management` refuses while that file, or the `OVN_CTL_OPTS` line older nodes carry in `/etc/default/ovn-central`, records the role. It adds `maxsize 100M` to the OVS and OVN logrotate files, plus `rotate 5` where the file sets no count, and rotates as root, matching the daemons. On a host whose IPsec stack is libreswan it stops and masks `openvswitch-ipsec.service` and `ipsec.service`, because OVN IPsec does not run on libreswan yet; elsewhere it enables and starts `openvswitch-ipsec.service`, unless the daemon has masked it to keep IPsec off. The links it creates persist through systemd-networkd where that is enabled, and otherwise through `/etc/spinifex/hostnet.json`, which `spinifex-hostnet.service` (`spx admin hostnet apply`) replays at boot and on every OVS start, starting `ovn-controller` again when an OVS stop took it down; the first run records the choice in `/etc/spinifex/hostnet-backend`. Where NetworkManager runs, it leaves the links Spinifex owns unmanaged. |
+| `setup-ovn.sh` | Bootstraps a compute node for OVN VPC networking: installs OVN/OVS, creates `br-int`, configures the WAN bridge (bridged, DHCP, or routed-NAT uplink), sets the chassis identity, and applies overlay sysctls. It runs OVS and OVN as root, setting `OVS_USER_ID` and `OVN_USER_ID` where `/etc/sysconfig` holds them, and hooks the socket-permission helper onto `ovs-vswitchd`, `ovn-controller`, `openvswitch-ipsec` and, on EL, `ovn-northd`. `--management` also starts the OVN central services, writes their `ovn-ctl` flags to `/etc/default/ovn-central` (Debian) or `/etc/sysconfig/ovn-northd` (EL), and records the role in `/etc/spinifex/ovn-role`. With clustered databases it returns only once this node's NB and SB members are back in their RAFT clusters under a known leader, and fails after `OVN_RAFT_REJOIN_WAIT` seconds (default 60) otherwise, so reruns one node at a time never take two members of a cluster down. A later run without `--management` refuses while that file, or the `OVN_CTL_OPTS` line older nodes carry in `/etc/default/ovn-central`, records the role. It adds `maxsize 100M` to the OVS and OVN logrotate files, plus `rotate 5` where the file sets no count, and rotates as root, matching the daemons. On a host whose IPsec stack is libreswan it enables, stops and masks `openvswitch-ipsec.service` and `ipsec.service` for the daemon to unmask when IPsec is on, unless `openvswitch-ipsec.service` is already running; elsewhere it enables and starts `openvswitch-ipsec.service`, unless the daemon has masked it to keep IPsec off. The links it creates persist through systemd-networkd where that is enabled, and otherwise through `/etc/spinifex/hostnet.json`, which `spinifex-hostnet.service` (`spx admin hostnet apply`) replays at boot and on every OVS start, starting `ovn-controller` again when an OVS stop took it down; the first run records the choice in `/etc/spinifex/hostnet-backend`. Where NetworkManager runs, it leaves the links Spinifex owns unmanaged. |
 | `install-node.sh` | Forms a multi-node cluster across hosts that are already running Spinifex: rebuilds the OVN database in clustered form, runs `spx admin init` on the first host and `spx admin join` on the rest, then verifies. Before touching any host it refuses hosts of different OS families (`distro_os_family`) or `spx` versions, and `--ipsec on` when any host uses libreswan. The scripted form of [install-multi-node](../docs/install/install-multi-node/README.md). Destroys the joining nodes' logical network state and crypto — see its `--help`. |
 | `dev-install.sh` | Full local dev setup through the production installer: builds from source, packs a tarball, runs `setup.sh`, initialises the cluster, and starts services under systemd. |
 | `clone-deps.sh` | Clones or updates the viperblock and predastore checkouts alongside spinifex for cross-repo development. |
@@ -58,7 +58,7 @@ Both ship to `/usr/local/share/spinifex/` on every node, so they are available o
 
 | Script | What it does |
 |---|---|
-| `node-reset.sh` | Clears **state** and leaves the node installed: stops services, destroys guests, wipes `/etc/spinifex`, `/var/lib/spinifex`, `/var/log/spinifex` and the OVN databases, and removes the OVS bridges. It stops OVS and OVN central by the unit names the host's packages use. It leaves the NetworkManager unmanaged-device config, which the next `setup-ovn.sh` run rewrites, so NetworkManager cannot take a NIC back in between. The node is then ready for a fresh `spx admin init`. `--keep-data` preserves `/var/lib/spinifex`; `SPX_RESET_CONFIRM=reset` runs it unattended. |
+| `node-reset.sh` | Clears **state** and leaves the node installed: stops services, destroys guests, wipes `/etc/spinifex`, `/var/lib/spinifex`, `/var/log/spinifex` and the OVN databases, and removes the OVS bridges. On a libreswan host it deletes the `ovs_cert_*` and `ovs_certkey_*` entries `ovs-monitor-ipsec` imported into libreswan's NSS database, leaving other identities and `/etc/ipsec.d/*.conf` alone. It stops OVS and OVN central by the unit names the host's packages use. It leaves the NetworkManager unmanaged-device config, which the next `setup-ovn.sh` run rewrites, so NetworkManager cannot take a NIC back in between. The node is then ready for a fresh `spx admin init`. `--keep-data` preserves `/var/lib/spinifex`; `SPX_RESET_CONFIRM=reset` runs it unattended. |
 | `uninstall-spx.sh` | Removes the **software** — the inverse of `setup.sh`. Delegates the state teardown above to `node-reset.sh` rather than reimplementing it, then removes the binary, the nbdkit plugin, the systemd units and slices, the service users, the sudoers grant, the sysctl/logrotate/chrony/udev drop-ins, the `systemd-networkd` units, the host firewall policy and its loaded nft table, the Spinifex systemd-resolved drop-in, and the Spinifex CA from the host trust store. It unmasks firewalld and leaves it disabled. Where `setup.sh` handed DNS to systemd-resolved, it restores the saved `/etc/resolv.conf` and turns resolved off. Verifies afterwards and exits non-zero naming anything that survived, then lists the host settings `setup.sh` changed that it leaves in place, such as the dnf repositories, `OVS_USER_ID`, masked IPsec units and the SELinux mode. |
 
 `node-reset.sh` finds guests by process name, `qemu-system-*` or `qemu-kvm`, owned by the daemon's service user, and waits for them to exit before it deletes any state.

scripts/images/eks-node/eks-token-webhook.initd

@@ -25,7 +25,7 @@ error_log="/dev/console"
 if [ -f /etc/spinifex-eks/first-boot.env ]; then
     # shellcheck disable=SC1091
     . /etc/spinifex-eks/first-boot.env
-    export EKS_GATEWAY_URL EKS_GATEWAY_CA EKS_ACCESS_KEY EKS_SECRET_KEY EKS_REGION EKS_ACCOUNT_ID EKS_CLUSTER_NAME
+    export EKS_GATEWAY_URL EKS_GATEWAY_CA EKS_ACCESS_KEY EKS_SECRET_KEY EKS_REGION EKS_ACCOUNT_ID EKS_CLUSTER_NAME EKS_WEBHOOK_READY_ADDR
 fi
 
 depend() {

scripts/install-node.sh

@@ -298,7 +298,6 @@ log "preflight on $N host(s)"
 
 versions=""
 families=""
-libreswan_hosts=""
 for i in $(seq 0 $((N - 1))); do
     host="${HOSTS[$i]}"
 
@@ -322,12 +321,6 @@ for i in $(seq 0 $((N - 1))); do
         fail "$host: cannot tell its OS from /etc/os-release"
     families+="${host}: ${family}"$'\n'
 
-    # libreswan's pluto: Spinifex cannot run OVN IPsec on it yet, and spx admin
-    # init refuses --ipsec=true there, which would stop formation part way.
-    if [ "$(out "$host" "test -x /usr/libexec/ipsec/pluto && echo yes")" = "yes" ]; then
-        libreswan_hosts+=" $host"
-    fi
-
     # A node that is already in a multi-node cluster is one whose keys and
     # logical network state are load-bearing. Refusing here is deliberate:
     # forming a cluster must never be the command that destroys a live one.
@@ -386,12 +379,6 @@ $(sed '/^$/d; s/^/         /' <<<"$families")
        A cluster must run one OS on every host."
 fi
 
-if $IPSEC_ENABLED && [ -n "$libreswan_hosts" ]; then
-    fail "--ipsec on, but these hosts use libreswan for IPsec:$libreswan_hosts
-       Spinifex cannot run OVN IPsec on libreswan yet. Re-run with --ipsec off
-       only if the nodes share a trusted private link."
-fi
-
 # --- Confirm ---------------------------------------------------------------
 
 echo ""

scripts/ipsec_state_helper_test.sh

@@ -0,0 +1,158 @@
+#!/bin/sh
+# ipsec_state_helper_test.sh — tests for the plaintext Geneve guard in the
+# IPsec state helper that setup.sh installs.
+#
+# The helper is cut out of setup.sh's heredoc and run against stub nft,
+# ovs-vsctl and systemctl commands, with pluto's path pointed at a fixture.
+set -eu
+
+SCRIPT_DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
+SETUP="$SCRIPT_DIR/setup.sh"
+TMP=$(mktemp -d)
+trap 'rm -rf "$TMP"' EXIT
+
+fails=0
+pass() { echo "  ok   - $1"; }
+fail() { echo "  FAIL - $1"; fails=$((fails + 1)); }
+
+echo "-- IPsec state helper: plaintext Geneve guard"
+
+HELPER="$TMP/helper"
+awk '/tee "\$\{IPSEC_STATE_HELPER\}"/ { on = 1; next } on && /^HELPER$/ { exit } on { print }' "$SETUP" |
+    sed "s#/usr/libexec/ipsec/pluto#$TMP/pluto#g" > "$HELPER"
+chmod +x "$HELPER"
+grep -q '^    guard)' "$HELPER" || { echo "could not extract the helper from setup.sh" >&2; exit 1; }
+
+mkdir -p "$TMP/bin"
+cat > "$TMP/bin/nft" <<EOF
+#!/bin/sh
+[ "\$*" = "-f -" ] || { echo "nft \$*" >> "$TMP/nft.log"; exit 0; }
+cat >> "$TMP/nft.log"
+echo "--" >> "$TMP/nft.log"
+EOF
+cat > "$TMP/bin/ovs-vsctl" <<EOF
+#!/bin/sh
+[ -e "$TMP/ovs-down" ] && { echo "ovs-vsctl: database connection failed" >&2; exit 1; }
+case "\$*" in
+    *" find Interface type=geneve") cat "$TMP/ifaces" ;;
+    *" get Interface "*)
+        for last; do :; done
+        f="$TMP/opt.\$5.\${last#options:}"
+        if [ -e "\$f" ]; then cat "\$f"; else echo; fi
+        ;;
+    *" get Open_vSwitch . external_ids:ovn-encap-ip")
+        if [ -e "$TMP/encap" ]; then cat "$TMP/encap"; else echo; fi
+        ;;
+    *) exit 1 ;;
+esac
+EOF
+printf '#!/bin/sh\nexit 0\n' > "$TMP/bin/systemctl"
+chmod +x "$TMP/bin/nft" "$TMP/bin/ovs-vsctl" "$TMP/bin/systemctl"
+
+# reset — no tunnels, OVS up, no pluto.
+reset() {
+    rm -f "$TMP"/opt.* "$TMP/ovs-down" "$TMP/nft.log" "$TMP/pluto" "$TMP/encap"
+    : > "$TMP/ifaces"
+    touch "$TMP/nft.log"
+}
+# tunnel <name> <local_ip|-> [remote_name] — a geneve interface as ovn-controller writes it.
+tunnel() {
+    echo "$1" >> "$TMP/ifaces"
+    [ "$2" = - ] || printf '"%s"\n' "$2" > "$TMP/opt.$1.local_ip"
+    [ -z "${3:-}" ] || echo "$3" > "$TMP/opt.$1.remote_name"
+}
+run() {
+    RC=0
+    OUT=$(PATH="$TMP/bin:$PATH" "$HELPER" "$1" 2>&1) || RC=$?
+    NFTLOG=$(cat "$TMP/nft.log")
+}
+LIFTED="table inet spinifex_ipsec_guard
+delete table inet spinifex_ipsec_guard
+--"
+# expect_guarded <desc> <sources> — one transaction that swaps in a table
+# dropping Geneve without an SA from exactly those sources.
+expect_guarded() {
+    if [ "$RC" = 0 ] &&
+        [ "$(grep -c '^--$' "$TMP/nft.log")" = 1 ] &&
+        [ "$(sed -n 2p "$TMP/nft.log")" = "delete table inet spinifex_ipsec_guard" ] &&
+        grep -q "^        elements = { $2 }$" "$TMP/nft.log" &&
+        grep -q '^        ip saddr @srcs udp dport 6081 rt ipsec missing drop$' "$TMP/nft.log"; then
+        pass "$1"
+    else
+        fail "$1 (rc=$RC): got [$NFTLOG] want sources [$2] $OUT"
+    fi
+}
+expect_lifted() {
+    if [ "$RC" = 0 ] && [ "$NFTLOG" = "$LIFTED" ]; then pass "$1"; else fail "$1 (rc=$RC): got [$NFTLOG] $OUT"; fi
+}
+
+reset
+tunnel ovn-node2-0 10.26.7.1 node2
+tunnel ovn-node3-0 10.26.7.1 node3
+run guard
+expect_guarded "guards the encap address once OVN wants the tunnels encrypted" "10.26.7.1/32"
+
+reset
+tunnel ovn-node2-0 10.26.7.1
+tunnel ovn-node3-0 10.26.7.1
+run guard
+expect_lifted "lifts the guard while OVN runs the tunnels in plaintext"
+
+reset
+tunnel ovn-node2-0 10.26.7.1 node2
+tunnel ovn-node3-0 10.26.8.1
+run guard
+expect_guarded "guards only the addresses of tunnels OVN encrypts" "10.26.7.1/32"
+
+reset
+tunnel ovn-node2-0 - node2
+run guard
+expect_guarded "guards a tunnel without a local_ip from every source" "0.0.0.0/0"
+
+reset
+echo '"10.26.7.1"' > "$TMP/encap"
+run guard
+expect_guarded "guards the encap address before ovn-controller has built the tunnels" "10.26.7.1/32"
+
+reset
+echo '"10.26.7.1,10.26.8.1"' > "$TMP/encap"
+run guard
+expect_guarded "guards each encap address of a multi-address host before the tunnels exist" "10.26.7.1/32,10.26.8.1/32"
+
+reset
+run guard
+expect_guarded "guards every source before the tunnels exist when no encap address is set" "0.0.0.0/0"
+
+reset
+tunnel ovn-node2-0 10.26.7.1 node2
+touch "$TMP/ovs-down"
+run guard
+if [ "$RC" != 0 ] && [ -z "$NFTLOG" ]; then
+    pass "keeps the guard and fails when OVS cannot be read"
+else
+    fail "with OVS down: rc=$RC nft calls [$NFTLOG]"
+fi
+
+reset
+touch "$TMP/pluto"
+chmod +x "$TMP/pluto"
+run off
+expect_lifted "off lifts the guard on a libreswan host"
+
+reset
+run off
+if [ "$RC" = 0 ] && [ -z "$NFTLOG" ]; then pass "off leaves nft alone without libreswan"; else fail "off without pluto: rc=$RC [$NFTLOG]"; fi
+
+reset
+run unguard
+expect_lifted "unguard lifts the guard"
+
+reset
+run bogus
+if [ "$RC" = 2 ] && [ -z "$NFTLOG" ]; then pass "refuses an unknown verb"; else fail "unknown verb: rc=$RC [$NFTLOG]"; fi
+
+if [ "$fails" -gt 0 ]; then
+    echo "$fails failure(s)"
+    exit 1
+fi
+echo "all passed"

scripts/node-reset.sh

@@ -273,6 +273,33 @@ if [ -n "$OVS_UNIT" ]; then
 fi
 run sudo rm -f /etc/openvswitch/system-id.conf
 
+# --- BEGIN ovs nss nicknames ---
+# ovs_nss_nicknames — the entries ovs-monitor-ipsec imported, read from
+# `certutil -L` on stdin: ovs_certkey_* is this node's certificate and key,
+# ovs_cert_* the CA and any peer certificates. Everything else is someone else's.
+ovs_nss_nicknames() {
+    awk '$1 ~ /^ovs_cert(key)?_/ {print $1}' | LC_ALL=C sort -u
+}
+# --- END ovs nss nicknames ---
+
+# A node rejoining another cluster must not keep trusting this one's CA. Only
+# the NSS entries go: other identities and /etc/ipsec.d/*.conf are left alone.
+if [ -x /usr/libexec/ipsec/pluto ] && command -v certutil >/dev/null 2>&1; then
+    log "removing Spinifex IPsec identities from libreswan's NSS database"
+    # A running ovs-monitor-ipsec imports them again from other_config.
+    run sudo systemctl stop openvswitch-ipsec.service 2>/dev/null || true
+    for nss in /var/lib/ipsec/nss /etc/ipsec.d; do
+        sudo test -e "$nss/cert9.db" || continue
+        for nick in $(sudo certutil -L -d "sql:$nss" 2>/dev/null | ovs_nss_nicknames); do
+            log "  deleting $nick from $nss"
+            delete=-D
+            case "$nick" in ovs_certkey_*) delete=-F ;; esac
+            run sudo certutil "$delete" -d "sql:$nss" -n "$nick" ||
+                log "  NOTE: could not delete $nick from $nss; remove it with certutil by hand"
+        done
+    done
+fi
+
 # Delete the OVN DBs outright — a caller re-running setup-ovn.sh gets fresh
 # empty ones. This is what clears stale chassis rows and port bindings, which
 # otherwise accumulate across resets and wedge ovn-controller in a commit loop.

scripts/node-reset_test.sh

@@ -1,7 +1,8 @@
 #!/bin/sh
 # node-reset_test.sh — tests the QEMU process match node-reset.sh uses to stop
 # guests before it deletes Viperblock state, against real processes, and the
-# instance and mounted-volume counts it and uninstall-spx.sh report before deleting.
+# instance and mounted-volume counts it and uninstall-spx.sh report before deleting,
+# and the libreswan NSS entries it deletes.
 #
 # Each fake guest is a copy of sleep named like a QEMU binary, so the kernel
 # gives it the same comm, including the 15-character truncation.
@@ -129,6 +130,32 @@ mkdir -p "$empty"
 expect "counts no instances without a state file" "$(count count_instances "$empty")" 0
 expect "counts no mounted volumes without a viperblock directory" "$(count count_mounted_volumes "$empty")" 0
 
+echo "-- node-reset.sh libreswan NSS nicknames"
+
+sed -n '/^# --- BEGIN ovs nss nicknames ---$/,/^# --- END ovs nss nicknames ---$/p' \
+    "$SCRIPT_DIR/node-reset.sh" > "$TMP/nss.sh"
+# `certutil -L` as it prints a database holding an operator's own identity and
+# CA beside what ovs-monitor-ipsec imported, one peer listed twice.
+cat > "$TMP/certutil-L" <<'EOF'
+
+Certificate Nickname                                         Trust Attributes
+                                                             SSL,S/MIME,JAR/XPI
+
+ovs_certkey_node1                                            u,u,u
+ovs_cert_cacert                                              CT,,
+ovs_cert_node2                                               P,P,P
+ovs_cert_node2                                               P,P,P
+site-vpn                                                     u,u,u
+corp-ca                                                      CT,C,C
+ovs_certificate_store                                        ,,
+EOF
+if [ ! -s "$TMP/nss.sh" ]; then
+    fail "no ovs nss nicknames block in node-reset.sh"
+else
+    got=$(bash -c '. "$1"; ovs_nss_nicknames < "$2"' _ "$TMP/nss.sh" "$TMP/certutil-L" | tr '\n' ' ')
+    expect "selects only ovs-monitor-ipsec's entries, once each" "$got" "ovs_cert_cacert ovs_cert_node2 ovs_certkey_node1 "
+fi
+
 if [ "$fails" -gt 0 ]; then
     echo "$fails failure(s)"
     exit 1

scripts/setup-ovn.sh

@@ -172,6 +172,8 @@ OVN_DBDIR="${OVN_DBDIR:-/var/lib/ovn}"
 # Seconds to wait for the database servers to exit, then for a SIGKILL to land.
 OVN_DB_STOP_WAIT="${OVN_DB_STOP_WAIT:-30}"
 OVN_DB_KILL_WAIT="${OVN_DB_KILL_WAIT:-10}"
+# Seconds to wait for a restarted clustered NB or SB member to rejoin its cluster.
+OVN_RAFT_REJOIN_WAIT="${OVN_RAFT_REJOIN_WAIT:-60}"
 # Records that --management set this node up. Nodes set up before it existed
 # carry only the legacy marker, OVN_CTL_OPTS in /etc/default/ovn-central.
 OVN_ROLE_FILE=/etc/spinifex/ovn-role
@@ -757,6 +759,42 @@ stop_ovn_databases() {
     exit 1
 }
 
+# db_clients — sets NBCTL/SBCTL. A clustered DB has no unix socket to fall back
+# on, and a follower refuses to answer at all unless leader-only is turned off;
+# writes through a follower are forwarded to the leader.
+db_clients() {
+    NBCTL=(ovn-nbctl)
+    SBCTL=(ovn-sbctl)
+    if [ -n "$DB_CLUSTER_LOCAL_ADDR" ]; then
+        NBCTL=(ovn-nbctl --db="tcp:$DB_CLUSTER_LOCAL_ADDR:6641" --no-leader-only)
+        SBCTL=(ovn-sbctl --db="tcp:$DB_CLUSTER_LOCAL_ADDR:6642" --no-leader-only)
+    fi
+}
+
+# set_client_connections — the loopback client listeners. A rerun's restart can
+# hand the leadership away, so this goes through NBCTL/SBCTL, not the socket.
+set_client_connections() {
+    sudo "${NBCTL[@]}" set-connection ptcp:6641:127.0.0.1
+    sudo "${SBCTL[@]}" set-connection ptcp:6642:127.0.0.1
+}
+
+# wait_for_raft_member <ctl> <schema> — waits until this server is a cluster
+# member that knows its leader, which a restarted member is once it rejoins.
+# Returns 1 on timeout.
+wait_for_raft_member() {
+    local ctl=$1 schema=$2 status i
+    for i in $(seq 1 "$OVN_RAFT_REJOIN_WAIT"); do
+        status=$(sudo ovs-appctl -t "$ctl" cluster/status "$schema" 2>/dev/null || true)
+        if grep -qx 'Status: cluster member' <<<"$status" &&
+            grep -q '^Leader: ' <<<"$status" && ! grep -qx 'Leader: unknown' <<<"$status"; then
+            return 0
+        fi
+        echo "  Waiting for $schema to rejoin its RAFT cluster... ($i/$OVN_RAFT_REJOIN_WAIT)"
+        sleep 1
+    done
+    return 1
+}
+
 # --- Step 2: Enable services ---
 # ovn-ctl consults the RAFT flags only when it creates a database, so any
 # existing DB file is served as it stands and the cluster flags are ignored.
@@ -996,16 +1034,9 @@ EOF
         fi
     fi
 
-    # A clustered DB has no unix socket to fall back on, and a follower refuses
-    # to answer at all unless leader-only is turned off. Without both of these
-    # the waits below can never succeed on a clustered node: they burn their
-    # full timeout, gate nothing, and the run still continues.
-    NBCTL=(ovn-nbctl)
-    SBCTL=(ovn-sbctl)
-    if [ -n "$DB_CLUSTER_LOCAL_ADDR" ]; then
-        NBCTL=(ovn-nbctl --db="tcp:$DB_CLUSTER_LOCAL_ADDR:6641" --no-leader-only)
-        SBCTL=(ovn-sbctl --db="tcp:$DB_CLUSTER_LOCAL_ADDR:6642" --no-leader-only)
-    fi
+    # Without these the waits below can never succeed on a clustered node:
+    # they burn their full timeout, gate nothing, and the run still continues.
+    db_clients
 
     # Wait for OVN NB DB socket to become available
     for i in $(seq 1 15); do
@@ -1067,29 +1098,34 @@ EOF
     # with no join step to absorb the window — attaching before the leader is
     # elected (and before northd writes the gateway logical flows) leaves it with
     # a partial datapath that never reconverges. Gate on SB reachable and, when
-    # clustered, an elected leader.
+    # clustered, both members back in their clusters under an elected leader.
     for i in $(seq 1 30); do
-        if sudo "${SBCTL[@]}" --timeout=2 show >/dev/null 2>&1; then
-            if [ -z "$DB_CLUSTER_LOCAL_ADDR" ]; then
-                break
-            fi
-            SB_LEADER=$(sudo ovs-appctl -t /var/run/ovn/ovnsb_db.ctl \
-                cluster/status OVN_Southbound 2>/dev/null \
-                | awk '/^Leader:/{print $2; exit}')
-            if [ -n "$SB_LEADER" ] && [ "$SB_LEADER" != "unknown" ]; then
-                break
-            fi
-        fi
-        echo "  Waiting for OVN SB DB leader... ($i/30)"
+        sudo "${SBCTL[@]}" --timeout=2 show >/dev/null 2>&1 && break
+        echo "  Waiting for OVN SB DB... ($i/30)"
         sleep 1
     done
 
+    # The restart above took both of this node's members down at once. Returning
+    # before they rejoin lets a rerun on the next database node take a second
+    # member of each cluster down, and one of three left is below quorum.
+    if [ -n "$DB_CLUSTER_LOCAL_ADDR" ]; then
+        for member in "/var/run/ovn/ovnnb_db.ctl OVN_Northbound" "/var/run/ovn/ovnsb_db.ctl OVN_Southbound"; do
+            # shellcheck disable=SC2086 # two words: the ctl socket and the schema
+            if ! wait_for_raft_member $member; then
+                echo "ERROR: this node's ${member#* } server is not back in its RAFT cluster after ${OVN_RAFT_REJOIN_WAIT}s." >&2
+                echo "       Restarting OVN central on another database node now would leave the cluster" >&2
+                echo "       without quorum. Check: sudo ovs-appctl -t ${member% *} cluster/status ${member#* }" >&2
+                exit 1
+            fi
+        done
+        echo "  RAFT: NB and SB members rejoined, leader elected"
+    fi
+
     # set-connection replicates through RAFT, so it only runs on the create
     # node. Scoped to loopback, not a wildcard ptcp:6641, because 127.0.0.1
     # is valid on every node; the differing LAN address is set above instead.
     if [ -z "$DB_CLUSTER_REMOTE_ADDR" ]; then
-        sudo ovn-nbctl set-connection ptcp:6641:127.0.0.1
-        sudo ovn-sbctl set-connection ptcp:6642:127.0.0.1
+        set_client_connections
         if [ -n "$LAN_ADDR" ]; then
             echo "  OVN NB DB listening on tcp:127.0.0.1:6641, tcp:$LAN_ADDR:6641"
             echo "  OVN SB DB listening on tcp:127.0.0.1:6642, tcp:$LAN_ADDR:6642"
@@ -1958,16 +1994,21 @@ echo "Step 10: Enabling OVN auto-start on boot..."
 sudo systemctl enable "$OVS_UNIT" 2>/dev/null || true
 sudo systemctl enable ovn-controller 2>/dev/null || true
 printf '  %-21s enabled on boot\n' "${OVS_UNIT%.service}:" ovn-controller:
-if distro_has_libreswan; then
-    # OVN IPsec does not run on libreswan yet, so it stays off here. Starting
-    # openvswitch-ipsec would also start pluto, which listens on UDP 500/4500.
+if distro_has_libreswan && [ "$(systemctl is-active openvswitch-ipsec.service 2>/dev/null)" = active ]; then
+    # The daemon turned IPsec on; stopping it here would drop live tunnels.
+    printf '  %-21s %s\n' openvswitch-ipsec: "active: IPsec is on, left to the daemon" ipsec: "active: IPsec is on, left to the daemon"
+elif distro_has_libreswan; then
+    # Off until the daemon asks: pluto listens on UDP 500/4500 once started.
+    # Enabled under the mask, so the daemon's unmask also brings both back on boot.
     IPSEC_UNITS=(ipsec.service)
     distro_unit_exists openvswitch-ipsec.service && IPSEC_UNITS+=(openvswitch-ipsec.service)
-    if ! sudo systemctl mask --now "${IPSEC_UNITS[@]}" >/dev/null; then
-        echo "ERROR: could not stop and mask ${IPSEC_UNITS[*]}; IPsec must stay off on libreswan" >&2
+    if ! sudo systemctl unmask "${IPSEC_UNITS[@]}" >/dev/null 2>&1 ||
+        ! sudo systemctl enable "${IPSEC_UNITS[@]}" >/dev/null 2>&1 ||
+        ! sudo systemctl mask --now "${IPSEC_UNITS[@]}" >/dev/null; then
+        echo "ERROR: could not enable, stop and mask ${IPSEC_UNITS[*]}; IPsec must stay off until the daemon turns it on" >&2
         exit 1
     fi
-    printf '  %-21s %s\n' openvswitch-ipsec: "masked: IPsec stays off on libreswan" ipsec: "masked: IPsec stays off on libreswan"
+    printf '  %-21s %s\n' openvswitch-ipsec: "masked: IPsec is off, left to the daemon" ipsec: "masked: IPsec is off, left to the daemon"
 elif [ "$(systemctl is-enabled openvswitch-ipsec.service 2>/dev/null)" = masked ]; then
     # The daemon masks it while IPsec is off and unmasks it to turn IPsec on.
     printf '  %-21s %s\n' openvswitch-ipsec: "masked: IPsec is off, left to the daemon"

scripts/setup-ovn_test.sh

@@ -454,6 +454,47 @@ case "$RC:$OUT" in
     *) fail "no recreate-db: standalone DB rc=$RC out=$OUT" ;;
 esac
 
+# --------------------------------------------------------------------------
+# A rerun returns only once both of this node's RAFT members have rejoined.
+# --------------------------------------------------------------------------
+# A rerun restarts the node's NB and SB members together, so returning early
+# lets the next database node's rerun take a second member of each down.
+awk '$0 == "wait_for_raft_member() {" { on = 1 } on { print } on && $0 == "}" { exit }' "$SETUP" >"$TMP/raft_fn.sh"
+mkdir -p "$TMP/raftbin"
+printf '#!/bin/sh\nexit 0\n' >"$TMP/raftbin/sleep"
+printf '#!/bin/sh\n[ -f "%s" ] || exit 1\ncat "%s"\n' "$TMP/cluster-status" "$TMP/cluster-status" >"$TMP/raftbin/ovs-appctl"
+chmod +x "$TMP/raftbin/sleep" "$TMP/raftbin/ovs-appctl"
+
+# raft_member <Status:> <Leader:> — runs wait_for_raft_member against a
+# cluster/status in that state, or none when Status is "absent"; sets RC.
+raft_member() {
+    rm -f "$TMP/cluster-status"
+    if [ "$1" != absent ]; then
+        printf 'a1b2\nName: OVN_Northbound\nStatus: %s\nRole: follower\nTerm: 4\nLeader: %s\nVote: c3d4\n\nServers:\n    a1b2 (a1b2 at tcp:10.0.0.1:6643) (self)\n    c3d4 (c3d4 at tcp:10.0.0.2:6643)\n' \
+            "$1" "$2" >"$TMP/cluster-status"
+    fi
+    RC=0
+    PATH="$TMP/raftbin:$PATH" bash -c 'sudo() { "$@"; }; OVN_RAFT_REJOIN_WAIT=2; . "$1"
+        wait_for_raft_member /var/run/ovn/ovnnb_db.ctl OVN_Northbound' _ "$TMP/raft_fn.sh" >/dev/null 2>&1 || RC=$?
+}
+
+if [ ! -s "$TMP/raft_fn.sh" ]; then
+    fail "raft: no wait_for_raft_member in setup-ovn.sh"
+else
+    raft_member "cluster member" c3d4
+    if [ "$RC" = 0 ]; then pass "raft: a member following a known leader has rejoined"; else fail "raft: follower with leader rc=$RC"; fi
+    raft_member "cluster member" self
+    if [ "$RC" = 0 ]; then pass "raft: a member that is the leader has rejoined"; else fail "raft: leader rc=$RC"; fi
+    raft_member "cluster member" unknown
+    if [ "$RC" != 0 ]; then pass "raft: a member that has not found its leader has not rejoined"; else fail "raft: unknown leader accepted"; fi
+    raft_member "disconnected from the cluster (election timeout)" c3d4
+    if [ "$RC" != 0 ]; then pass "raft: a disconnected member that remembers a leader has not rejoined"; else fail "raft: disconnected member accepted"; fi
+    raft_member "joining cluster" c3d4
+    if [ "$RC" != 0 ]; then pass "raft: a member still joining has not rejoined"; else fail "raft: joining member accepted"; fi
+    raft_member absent ""
+    if [ "$RC" != 0 ]; then pass "raft: a server that does not answer has not rejoined"; else fail "raft: unanswered cluster/status accepted"; fi
+fi
+
 rm -rf "$TMP/dbdir"
 mkdir -p "$TMP/dbdir"
 ensure_db true
@@ -478,6 +519,33 @@ else
 fi
 printf '#!/bin/sh\nexit 1\n' >"$TMP/bin/pgrep"
 
+# A rerun restarts this node's members, which can hand the leadership to a
+# peer, so the create node's set-connection must still land as a follower.
+for fn in db_clients set_client_connections; do
+    awk -v f="$fn() {" '$0 == f { on = 1 } on { print } on && $0 == "}" { exit }' "$SETUP" >>"$TMP/conn_fn.sh"
+done
+mkdir -p "$TMP/followerbin"
+for ctl in ovn-nbctl ovn-sbctl; do
+    cat >"$TMP/followerbin/$ctl" <<EOF
+#!/bin/sh
+case " \$* " in
+    *" --no-leader-only "*) echo "$ctl \$*" >>"$TMP/conn.log" ;;
+    *) echo "$ctl: unix:/var/run/ovn/db.sock: database connection failed ()" >&2; exit 1 ;;
+esac
+EOF
+    chmod +x "$TMP/followerbin/$ctl"
+done
+: >"$TMP/conn.log"
+RC=0
+PATH="$TMP/followerbin:$PATH" bash -c 'set -e; sudo() { "$@"; }; DB_CLUSTER_LOCAL_ADDR=10.0.0.1; . "$1"
+    db_clients; set_client_connections' _ "$TMP/conn_fn.sh" >/dev/null 2>&1 || RC=$?
+if [ "$RC" = 0 ] && [ "$(cat "$TMP/conn.log")" = "ovn-nbctl --db=tcp:10.0.0.1:6641 --no-leader-only set-connection ptcp:6641:127.0.0.1
+ovn-sbctl --db=tcp:10.0.0.1:6642 --no-leader-only set-connection ptcp:6642:127.0.0.1" ]; then
+    pass "connections: the create node sets the client listeners while a follower"
+else
+    fail "connections: as a follower rc=$RC calls [$(cat "$TMP/conn.log")]"
+fi
+
 if [ "$fails" -ne 0 ]; then
     echo "  setup-ovn.sh: $fails failure(s)"
     exit 1

scripts/setup.sh

@@ -390,10 +390,68 @@ install_ipsec_state_helper() {
 set -eu
 
 if [ "$#" -ne 1 ]; then
-    echo "usage: $0 <on|off>" >&2
+    echo "usage: $0 <on|off|guard|unguard>" >&2
     exit 2
 fi
 
+# libreswan drops a tunnel's xfrm policies while IKE fails, and pluto flushes
+# the whole xfrm table when it starts or stops, so Geneve from the encap address
+# is dropped in nft unless the route carries an SA.
+GUARD_TABLE="inet spinifex_ipsec_guard"
+
+unguard() {
+    printf 'table %s\ndelete table %s\n' "$GUARD_TABLE" "$GUARD_TABLE" | nft -f -
+}
+
+want=""
+want_src() {
+    case "$1" in
+        *.*.*.*) case "$1" in *[!0-9.]*) src=0.0.0.0/0 ;; *) src="$1/32" ;; esac ;;
+        *) src=0.0.0.0/0 ;;
+    esac
+    case " $want " in *" $src "*) ;; *) want="$want $src" ;; esac
+}
+
+# One source per local address of a tunnel with a remote_name, which
+# ovn-controller sets only while NB_Global.ipsec is on. A tunnel without a
+# usable local_ip is guarded from every source rather than left open.
+guard() {
+    ifaces=$(ovs-vsctl --timeout=5 --bare --columns=name find Interface type=geneve)
+    for iface in $ifaces; do
+        peer=$(ovs-vsctl --timeout=5 --if-exists get Interface "$iface" options:remote_name)
+        [ -n "$peer" ] || continue
+        want_src "$(ovs-vsctl --timeout=5 --if-exists get Interface "$iface" options:local_ip | tr -d '"')"
+    done
+    # At boot pluto is up before ovn-controller builds the tunnels, and its
+    # callers guard only while encryption is wanted, so guard the encap address
+    # until the tunnels say otherwise.
+    if [ -z "$ifaces" ]; then
+        encap=$(ovs-vsctl --timeout=5 --if-exists get Open_vSwitch . external_ids:ovn-encap-ip | tr -d '"')
+        for ip in $(echo "$encap" | tr ',' ' '); do want_src "$ip"; done
+        [ -n "$want" ] || want_src ""
+    fi
+    if [ -z "$want" ]; then
+        unguard
+        return
+    fi
+    # One transaction, so the old set is never gone before the new one is in.
+    nft -f - << NFT
+table $GUARD_TABLE
+delete table $GUARD_TABLE
+table $GUARD_TABLE {
+    set srcs {
+        type ipv4_addr
+        flags interval
+        elements = { $(echo $want | tr ' ' ',') }
+    }
+    chain out {
+        type filter hook output priority 0; policy accept;
+        ip saddr @srcs udp dport 6081 rt ipsec missing drop
+    }
+}
+NFT
+}
+
 # mask/unmask rather than disable/enable. The daemon's unit sets
 # ProtectSystem=full, so /etc is read-only in the namespace its children inherit,
 # and disabling a unit that also ships a SysV script shells out to update-rc.d
@@ -401,12 +459,24 @@ fi
 
 # ovs-monitor-ipsec execs the strongSwan starter itself rather than going through
 # this unit, so leaving it enabled only contends for UDP 500/4500. Off in both
-# states. Absent on a host without strongswan, which is not an error.
-systemctl mask --now strongswan-starter.service >/dev/null 2>&1 || true
+# states. Masking a unit no package ships leaves a stray /dev/null link, so a
+# host without strongSwan gets none, and loses one an older helper left.
+if [ -e /usr/lib/systemd/system/strongswan-starter.service ] || [ -e /lib/systemd/system/strongswan-starter.service ]; then
+    systemctl mask --now strongswan-starter.service >/dev/null 2>&1 || true
+elif [ "$(systemctl is-enabled strongswan-starter.service 2>/dev/null)" = masked ]; then
+    systemctl unmask strongswan-starter.service >/dev/null 2>&1 || true
+fi
 
 case "$1" in
     on)
-        systemctl unmask openvswitch-ipsec.service
+        if [ -x /usr/libexec/ipsec/pluto ]; then
+            # ovs-monitor-ipsec drives pluto over its control socket but does
+            # not start it, so pluto comes up first under its own unit.
+            systemctl unmask ipsec.service openvswitch-ipsec.service
+            systemctl start ipsec.service
+        else
+            systemctl unmask openvswitch-ipsec.service
+        fi
         exec systemctl start openvswitch-ipsec.service
         ;;
     off)
@@ -415,8 +485,15 @@ case "$1" in
         # On Debian that name is strongswan-starter's alias, masked above.
         if [ -x /usr/libexec/ipsec/pluto ]; then
             systemctl mask --now ipsec.service
+            unguard
         fi
         ;;
+    guard)
+        guard
+        ;;
+    unguard)
+        unguard
+        ;;
     *)
         echo "state not permitted: $1" >&2
         exit 2
@@ -1769,6 +1846,18 @@ CPUWeight=300
 EOF
     info "  /etc/systemd/system/system.slice.d/spinifex-reserve.conf"
 
+    # The plaintext Geneve guard is not persisted, so it goes in when pluto
+    # starts at boot, before ovn-controller builds the tunnels. `-`: the guard
+    # must never fail the unit, and the daemon re-asserts it each pass.
+    if [ -x /usr/libexec/ipsec/pluto ]; then
+        $SUDO mkdir -p /etc/systemd/system/ipsec.service.d
+        $SUDO tee /etc/systemd/system/ipsec.service.d/spinifex-guard.conf > /dev/null << EOF
+[Service]
+ExecStartPost=-${IPSEC_STATE_HELPER} guard
+EOF
+        info "  /etc/systemd/system/ipsec.service.d/spinifex-guard.conf"
+    fi
+
     # daemon-reload / enable require a running systemd — skip inside the ISO
     # chroot. Unit files are still dropped into place; firstboot enables them.
     if [ "${ISO_BUILD:-0}" = "1" ]; then

scripts/uninstall-spx.sh

@@ -362,9 +362,10 @@ rm_path /etc/systemd/system/openvswitch-switch.service.d/spinifex-perms.conf \
         /etc/systemd/system/ovn-controller.service.d/log-level.conf \
         /etc/systemd/system/openvswitch-ipsec.service.d/spinifex-perms.conf \
         /etc/systemd/system/ovn-northd.service.d/spinifex-perms.conf \
-        /etc/systemd/system/ovn-northd.service.d/no-manage-ovsdb.conf
+        /etc/systemd/system/ovn-northd.service.d/no-manage-ovsdb.conf \
+        /etc/systemd/system/ipsec.service.d/spinifex-guard.conf
 # Only if we emptied them — another package may own the same override dir.
-for d in openvswitch-switch ovs-vswitchd ovn-controller openvswitch-ipsec ovn-northd; do
+for d in openvswitch-switch ovs-vswitchd ovn-controller openvswitch-ipsec ovn-northd ipsec; do
     dir="/etc/systemd/system/${d}.service.d"
     if sudo test -d "$dir" && [ -z "$(sudo ls -A "$dir" 2>/dev/null)" ]; then
         rm_path "$dir"
@@ -431,6 +432,11 @@ if [ -z "$PLUGINDIR" ]; then
 fi
 [ -n "$PLUGINDIR" ] && rm_path "$PLUGINDIR/nbdkit-viperblock-plugin.so"
 
+# A leftover guard table drops all Geneve from this host's encap address until
+# reboot, and the helper is the one place that names it.
+if [ -x /usr/local/lib/spinifex/spinifex-set-ipsec-state ]; then
+    run sudo /usr/local/lib/spinifex/spinifex-set-ipsec-state unguard || true
+fi
 rm_path /usr/local/bin/spx \
         /usr/local/share/spinifex \
         /usr/local/lib/spinifex \
@@ -619,6 +625,7 @@ if ! $DRY_RUN; then
              /etc/NetworkManager/conf.d/*spinifex* \
              /etc/systemd/system/*.service.d/spinifex-perms.conf \
              /etc/systemd/system/ovn-northd.service.d/no-manage-ovsdb.conf \
+             /etc/systemd/system/ipsec.service.d/spinifex-guard.conf \
              /etc/systemd/system/ovn-controller.service.d/log-level.conf; do
         exists "$p" && LEFT+=("$p")
     done

spinifex/agents/eks/tokenwebhook/readiness.go

@@ -0,0 +1,128 @@
+package tokenwebhook
+
+import (
+	"context"
+	"crypto/tls"
+	"crypto/x509"
+	"fmt"
+	"net"
+	"net/http"
+	"net/url"
+	"os"
+	"sync"
+	"time"
+
+	"github.com/mulgadc/bluebottle/pkg/tlsconfig"
+)
+
+const (
+	readinessTimeout = 2 * time.Second
+	readinessTTL     = 2 * time.Second
+)
+
+// defaultAPIServerAddr is the local k3s apiserver the readiness check dials.
+const defaultAPIServerAddr = "127.0.0.1:6443"
+
+// gatewayProbe reports whether this control plane can serve: the gateway it
+// relays to accepts a TLS handshake and the local apiserver accepts a TCP
+// connection. It sends no credentials and caches the result so frequent load
+// balancer probes never reach either more than once per ttl.
+type gatewayProbe struct {
+	addr    string
+	apiAddr string
+	tlsCfg  *tls.Config
+	timeout time.Duration
+	ttl     time.Duration
+	now     func() time.Time
+
+	mu      sync.Mutex
+	checked time.Time
+	ok      bool
+}
+
+func newGatewayProbe(gatewayURL, caPath, apiAddr string) (*gatewayProbe, error) {
+	u, err := url.Parse(gatewayURL)
+	if err != nil {
+		return nil, fmt.Errorf("parse gateway URL %q: %w", gatewayURL, err)
+	}
+	host, port := u.Hostname(), u.Port()
+	if host == "" {
+		return nil, fmt.Errorf("gateway URL %q has no host", gatewayURL)
+	}
+	if port == "" {
+		port = "443"
+	}
+	cfg := &tls.Config{MinVersion: tls.VersionTLS13, CurvePreferences: tlsconfig.Curves}
+	if caPath != "" {
+		pem, err := os.ReadFile(caPath)
+		if err != nil {
+			return nil, fmt.Errorf("read gateway CA %q: %w", caPath, err)
+		}
+		pool := x509.NewCertPool()
+		if !pool.AppendCertsFromPEM(pem) {
+			return nil, fmt.Errorf("gateway CA %q has no usable certificates", caPath)
+		}
+		cfg.RootCAs = pool
+	}
+	if apiAddr == "" {
+		apiAddr = defaultAPIServerAddr
+	}
+	return &gatewayProbe{
+		addr:    net.JoinHostPort(host, port),
+		apiAddr: apiAddr,
+		tlsCfg:  cfg,
+		timeout: readinessTimeout,
+		ttl:     readinessTTL,
+		now:     time.Now,
+	}, nil
+}
+
+// ready returns the cached result while it is younger than ttl, else probes.
+// Concurrent callers wait for one probe and share its result.
+func (p *gatewayProbe) ready() bool {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	if !p.checked.IsZero() && p.now().Sub(p.checked) < p.ttl {
+		return p.ok
+	}
+	p.ok = p.probe()
+	p.checked = p.now()
+	return p.ok
+}
+
+func (p *gatewayProbe) probe() bool {
+	ctx, cancel := context.WithTimeout(context.Background(), p.timeout)
+	defer cancel()
+	d := tls.Dialer{NetDialer: &net.Dialer{}, Config: p.tlsCfg}
+	conn, err := d.DialContext(ctx, "tcp", p.addr)
+	if err != nil {
+		return false
+	}
+	_ = conn.Close()
+
+	// A control plane whose apiserver is down must leave rotation too.
+	api, err := (&net.Dialer{}).DialContext(ctx, "tcp", p.apiAddr)
+	if err != nil {
+		return false
+	}
+	_ = api.Close()
+	return true
+}
+
+// handle serves 200 when the gateway and apiserver are reachable, else 503.
+func (p *gatewayProbe) handle(w http.ResponseWriter, _ *http.Request) {
+	if p.ready() {
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte("ok"))
+		return
+	}
+	http.Error(w, "not ready", http.StatusServiceUnavailable)
+}
+
+// readinessHandler serves only /readyz; the listener it backs is plain HTTP and
+// reachable beyond loopback, so it must never expose /authenticate.
+func readinessHandler(p *gatewayProbe) http.Handler {
+	mux := http.NewServeMux()
+	mux.HandleFunc("/readyz", p.handle)
+	return mux
+}

spinifex/agents/eks/tokenwebhook/readiness_test.go

@@ -0,0 +1,251 @@
+package tokenwebhook
+
+import (
+	"crypto/tls"
+	"net"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// tlsGateway is a loopback TLS listener standing in for the gateway. It counts
+// accepted connections and can be stopped and restarted on the same address.
+type tlsGateway struct {
+	t       *testing.T
+	api     *fakeAPIServer
+	addr    string
+	cert    tls.Certificate
+	caPath  string
+	ln      net.Listener
+	accepts atomic.Int32
+}
+
+func newTLSGateway(t *testing.T) *tlsGateway {
+	t.Helper()
+	dir := t.TempDir()
+	cert, certPEM, err := ensureServingCert(filepath.Join(dir, "c.pem"), filepath.Join(dir, "k.pem"))
+	require.NoError(t, err)
+	caPath := filepath.Join(dir, "ca.pem")
+	require.NoError(t, os.WriteFile(caPath, certPEM, 0o600))
+
+	g := &tlsGateway{t: t, cert: cert, caPath: caPath, api: newFakeAPIServer(t)}
+	g.start("127.0.0.1:0")
+	t.Cleanup(g.stop)
+	return g
+}
+
+func (g *tlsGateway) start(addr string) {
+	g.t.Helper()
+	ln, err := tls.Listen("tcp", addr, serverTLSConfig(g.cert))
+	require.NoError(g.t, err)
+	g.ln = ln
+	g.addr = ln.Addr().String()
+	go func() {
+		for {
+			c, err := ln.Accept()
+			if err != nil {
+				return
+			}
+			g.accepts.Add(1)
+			go func() {
+				if tc, ok := c.(*tls.Conn); ok {
+					_ = tc.Handshake()
+				}
+				_ = c.Close()
+			}()
+		}
+	}()
+}
+
+func (g *tlsGateway) stop() {
+	if g.ln != nil {
+		_ = g.ln.Close()
+		g.ln = nil
+	}
+}
+
+// fakeAPIServer is a plain TCP listener standing in for the local apiserver.
+type fakeAPIServer struct {
+	addr string
+	ln   net.Listener
+}
+
+func newFakeAPIServer(t *testing.T) *fakeAPIServer {
+	t.Helper()
+	a := &fakeAPIServer{}
+	a.start("127.0.0.1:0", t)
+	t.Cleanup(a.stop)
+	return a
+}
+
+func (a *fakeAPIServer) start(addr string, t *testing.T) {
+	t.Helper()
+	ln, err := net.Listen("tcp", addr)
+	require.NoError(t, err)
+	a.ln = ln
+	a.addr = ln.Addr().String()
+	go func() {
+		for {
+			c, err := ln.Accept()
+			if err != nil {
+				return
+			}
+			_ = c.Close()
+		}
+	}()
+}
+
+func (a *fakeAPIServer) stop() {
+	if a.ln != nil {
+		_ = a.ln.Close()
+		a.ln = nil
+	}
+}
+
+func (g *tlsGateway) probe(t *testing.T) *gatewayProbe {
+	t.Helper()
+	p, err := newGatewayProbe("https://"+g.addr, g.caPath, g.api.addr)
+	require.NoError(t, err)
+	return p
+}
+
+func TestGatewayProbe_UpAndRefused(t *testing.T) {
+	g := newTLSGateway(t)
+	p := g.probe(t)
+	p.ttl = 0
+
+	assert.True(t, p.ready())
+
+	g.stop()
+	assert.False(t, p.ready(), "a closed gateway port is not ready")
+}
+
+func TestGatewayProbe_RecoversAfterDown(t *testing.T) {
+	g := newTLSGateway(t)
+	p := g.probe(t)
+	p.ttl = 0
+	addr := g.addr
+
+	g.stop()
+	require.False(t, p.ready())
+
+	g.start(addr)
+	assert.True(t, p.ready(), "ready again once the gateway is back")
+}
+
+func TestGatewayProbe_TimesOutOnStalledHandshake(t *testing.T) {
+	g := newTLSGateway(t)
+	// A plain listener that accepts and never answers the TLS handshake.
+	ln, err := net.Listen("tcp", "127.0.0.1:0")
+	require.NoError(t, err)
+	t.Cleanup(func() { _ = ln.Close() })
+	go func() {
+		for {
+			c, err := ln.Accept()
+			if err != nil {
+				return
+			}
+			t.Cleanup(func() { _ = c.Close() })
+		}
+	}()
+
+	p, err := newGatewayProbe("https://"+ln.Addr().String(), g.caPath, g.api.addr)
+	require.NoError(t, err)
+	p.timeout = 200 * time.Millisecond
+	p.ttl = 0
+
+	start := time.Now()
+	assert.False(t, p.ready())
+	assert.Less(t, time.Since(start), 2*time.Second, "the probe gives up at its timeout")
+}
+
+func TestGatewayProbe_RejectsUntrustedCertificate(t *testing.T) {
+	g := newTLSGateway(t)
+	other := newTLSGateway(t)
+	p, err := newGatewayProbe("https://"+g.addr, other.caPath, g.api.addr)
+	require.NoError(t, err)
+	p.ttl = 0
+
+	assert.False(t, p.ready(), "a certificate not signed by the configured CA is not ready")
+}
+
+func TestGatewayProbe_CachesWithinTTL(t *testing.T) {
+	g := newTLSGateway(t)
+	p := g.probe(t)
+	clock := time.Unix(1_000_000, 0)
+	p.now = func() time.Time { return clock }
+	p.ttl = 2 * time.Second
+
+	require.True(t, p.ready())
+	require.True(t, p.ready())
+	require.True(t, p.ready())
+	assert.Equal(t, int32(1), g.accepts.Load(), "calls inside the ttl share one probe")
+
+	g.stop()
+	assert.True(t, p.ready(), "a cached answer is served while the ttl holds")
+
+	clock = clock.Add(3 * time.Second)
+	assert.False(t, p.ready(), "after the ttl the gateway is probed again")
+}
+
+func TestGatewayProbe_HandlerStatus(t *testing.T) {
+	g := newTLSGateway(t)
+	p := g.probe(t)
+	p.ttl = 0
+
+	w := httptest.NewRecorder()
+	p.handle(w, httptest.NewRequest(http.MethodGet, "/readyz", nil))
+	assert.Equal(t, http.StatusOK, w.Code)
+
+	g.stop()
+	w = httptest.NewRecorder()
+	p.handle(w, httptest.NewRequest(http.MethodGet, "/readyz", nil))
+	assert.Equal(t, http.StatusServiceUnavailable, w.Code)
+}
+
+func TestNewGatewayProbe_DefaultsPortAndRejectsBadInput(t *testing.T) {
+	p, err := newGatewayProbe("https://gw.example", "", "")
+	require.NoError(t, err)
+	assert.Equal(t, "gw.example:443", p.addr)
+	assert.Equal(t, defaultAPIServerAddr, p.apiAddr)
+
+	_, err = newGatewayProbe("https://", "", "")
+	require.Error(t, err)
+	_, err = newGatewayProbe("https://gw.example:9999", "/no/such/ca.pem", "")
+	require.Error(t, err)
+}
+
+func TestGatewayProbe_NotReadyWhenAPIServerDown(t *testing.T) {
+	g := newTLSGateway(t)
+	p := g.probe(t)
+	p.ttl = 0
+	require.True(t, p.ready())
+
+	g.api.stop()
+	assert.False(t, p.ready(), "a dead apiserver must not look ready while the gateway is up")
+
+	w := httptest.NewRecorder()
+	p.handle(w, httptest.NewRequest(http.MethodGet, "/readyz", nil))
+	assert.Equal(t, http.StatusServiceUnavailable, w.Code)
+}
+
+func TestReadinessHandler_ServesOnlyReadyz(t *testing.T) {
+	g := newTLSGateway(t)
+	h := readinessHandler(g.probe(t))
+
+	for _, path := range []string{"/authenticate", "/healthz", "/"} {
+		w := httptest.NewRecorder()
+		h.ServeHTTP(w, httptest.NewRequest(http.MethodPost, path, nil))
+		assert.Equalf(t, http.StatusNotFound, w.Code, "%s must not be served on the readiness listener", path)
+	}
+	w := httptest.NewRecorder()
+	h.ServeHTTP(w, httptest.NewRequest(http.MethodGet, "/readyz", nil))
+	assert.Equal(t, http.StatusOK, w.Code)
+}

spinifex/agents/eks/tokenwebhook/run_test.go

@@ -128,3 +128,58 @@ func TestRunServesUntilCancelled(t *testing.T) {
 		t.Fatal("Run did not return after cancel")
 	}
 }
+
+func TestRunServesReadinessOnSeparateListener(t *testing.T) {
+	g := newTLSGateway(t)
+	cfg := runConfig(t, freeAddr(t))
+	cfg.GatewayURL = "https://" + g.addr
+	cfg.GatewayCA = g.caPath
+	cfg.APIServerAddr = g.api.addr
+	cfg.ReadyAddr = freeAddr(t)
+
+	ctx, cancel := context.WithCancel(context.Background())
+	done := make(chan error, 1)
+	go func() { done <- Run(ctx, cfg) }()
+
+	client := &http.Client{Timeout: 3 * time.Second}
+	require.Eventually(t, func() bool {
+		resp, err := client.Get("http://" + cfg.ReadyAddr + "/readyz")
+		if err != nil {
+			return false
+		}
+		_ = resp.Body.Close()
+		return resp.StatusCode == http.StatusOK
+	}, 10*time.Second, 50*time.Millisecond, "readiness listener never reported ready")
+
+	resp, err := client.Post("http://"+cfg.ReadyAddr+"/authenticate", "application/json", strings.NewReader("{}"))
+	require.NoError(t, err)
+	_ = resp.Body.Close()
+	assert.Equal(t, http.StatusNotFound, resp.StatusCode, "the readiness listener must not serve /authenticate")
+
+	cancel()
+	select {
+	case err := <-done:
+		require.NoError(t, err)
+	case <-time.After(5 * time.Second):
+		t.Fatal("Run did not return after cancel")
+	}
+}
+
+func TestRunReportsReadinessListenFailure(t *testing.T) {
+	l, err := net.Listen("tcp", "127.0.0.1:0")
+	require.NoError(t, err)
+	defer l.Close()
+
+	cfg := runConfig(t, freeAddr(t))
+	cfg.ReadyAddr = l.Addr().String()
+
+	done := make(chan error, 1)
+	go func() { done <- Run(context.Background(), cfg) }()
+	select {
+	case err := <-done:
+		require.Error(t, err)
+		assert.True(t, strings.HasPrefix(err.Error(), "serve readiness: "), err.Error())
+	case <-time.After(10 * time.Second):
+		t.Fatal("Run kept serving after the readiness listener failed")
+	}
+}

spinifex/agents/eks/tokenwebhook/webhook.go

@@ -43,6 +43,10 @@ type Config struct {
 	CertPath    string
 	KeyPath     string
 	Kubeconfig  string
+	// ReadyAddr, when set, is a plain-HTTP listener serving only /readyz for
+	// load balancer health checks. APIServerAddr defaults to the local k3s.
+	ReadyAddr     string
+	APIServerAddr string
 }
 
 // Run writes the serving certificate and apiserver kubeconfig, then serves
@@ -63,6 +67,11 @@ func Run(ctx context.Context, cfg Config) error {
 		return fmt.Errorf("build gateway client: %w", err)
 	}
 
+	probe, err := newGatewayProbe(cfg.GatewayURL, cfg.GatewayCA, cfg.APIServerAddr)
+	if err != nil {
+		return fmt.Errorf("build readiness probe: %w", err)
+	}
+
 	authr := &authenticator{
 		accountID:   cfg.AccountID,
 		clusterName: cfg.ClusterName,
@@ -74,6 +83,7 @@ func Run(ctx context.Context, cfg Config) error {
 		w.WriteHeader(http.StatusOK)
 		_, _ = w.Write([]byte("ok"))
 	})
+	mux.HandleFunc("/readyz", probe.handle)
 	mux.HandleFunc("/authenticate", authr.handle)
 
 	server := &http.Server{
@@ -82,12 +92,38 @@ func Run(ctx context.Context, cfg Config) error {
 		ReadHeaderTimeout: 5 * time.Second,
 		TLSConfig:         serverTLSConfig(tlsCert),
 	}
-	stop := context.AfterFunc(ctx, func() { _ = server.Close() })
+	readyErr := make(chan error, 1)
+	var readySrv *http.Server
+	if cfg.ReadyAddr != "" {
+		readySrv = &http.Server{
+			Addr:              cfg.ReadyAddr,
+			Handler:           readinessHandler(probe),
+			ReadHeaderTimeout: 5 * time.Second,
+		}
+		go func() {
+			if err := readySrv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
+				readyErr <- fmt.Errorf("serve readiness: %w", err)
+				_ = server.Close()
+			}
+		}()
+		defer func() { _ = readySrv.Close() }()
+	}
+	stop := context.AfterFunc(ctx, func() {
+		_ = server.Close()
+		if readySrv != nil {
+			_ = readySrv.Close()
+		}
+	})
 	defer stop()
-	slog.Info("eks-token-webhook listening", "addr", cfg.Addr, "cluster", cfg.ClusterName)
+	slog.Info("eks-token-webhook listening", "addr", cfg.Addr, "ready_addr", cfg.ReadyAddr, "cluster", cfg.ClusterName)
 	if err := server.ListenAndServeTLS("", ""); err != nil && !errors.Is(err, http.ErrServerClosed) {
 		return fmt.Errorf("serve: %w", err)
 	}
+	select {
+	case err := <-readyErr:
+		return err
+	default:
+	}
 	return nil
 }
 
@@ -101,8 +137,8 @@ type tokenReviewRequest struct {
 
 // gatewayReviewer returns the per-call relay: SigV4-POST {accountId,token} to
 // the gateway and decode the resolved identity. A non-2xx / transport error is
-// surfaced (handle() turns it into a 5xx so the apiserver retries) rather than a
-// silent deny, distinguishing "webhook broker down" from "token rejected".
+// surfaced (handle() turns it into a 5xx) rather than a silent deny, which
+// distinguishes "webhook broker down" from "token rejected" in the logs.
 func gatewayReviewer(client *eksgw.Client, clusterName, accountID string) func(string) (ekscluster.WebhookTokenReviewResult, error) {
 	path := "/clusters/" + clusterName + "/token-review"
 	return func(token string) (ekscluster.WebhookTokenReviewResult, error) {
@@ -142,8 +178,8 @@ func (a *authenticator) handle(w http.ResponseWriter, r *http.Request) {
 
 	res, err := a.review(review.Spec.Token)
 	if err != nil {
-		// Broker fault (gateway down, transport error): a 5xx tells the apiserver
-		// to retry rather than treating a transient outage as a hard deny.
+		// Broker fault (gateway down, transport error). The apiserver treats any
+		// webhook error as an authentication failure and does not retry it.
 		slog.Error("TokenReview relay failed", "cluster", a.clusterName, "err", err)
 		http.Error(w, "token review unavailable", http.StatusServiceUnavailable)
 		return

spinifex/agents/elbv2/health.go

@@ -22,6 +22,11 @@ type HealthReport struct {
 	Servers []ServerStatus `json:"servers"`
 }
 
+// FailoverServerSuffix marks the unchecked backup copy of a server that HAProxy
+// uses only when every checked server is down. '.' cannot appear in a sanitized
+// target name, so a copy never collides with a real server.
+const FailoverServerSuffix = ".fo"
+
 // queryHAProxyStats reads backend server health from the HAProxy stats socket.
 // Parses CSV "show stat" output; cols 0/1/17 = pxname/svname/status.
 func queryHAProxyStats(socketPath string) ([]ServerStatus, error) {
@@ -53,7 +58,7 @@ func queryHAProxyStats(socketPath string) ([]ServerStatus, error) {
 		}
 
 		svname := fields[1]
-		if svname == "FRONTEND" || svname == "BACKEND" {
+		if svname == "FRONTEND" || svname == "BACKEND" || strings.HasSuffix(svname, FailoverServerSuffix) {
 			continue
 		}
 

spinifex/agents/elbv2/health_test.go

@@ -58,3 +58,38 @@ func TestQueryHAProxyStats_SocketNotFound(t *testing.T) {
 		t.Error("expected error for non-existent socket")
 	}
 }
+
+func TestQueryHAProxyStats_IgnoresFailoverCopies(t *testing.T) {
+	dir := t.TempDir()
+	sock := filepath.Join(dir, "haproxy.sock")
+
+	ln, err := net.Listen("unix", sock)
+	if err != nil {
+		t.Fatalf("listen: %v", err)
+	}
+	defer ln.Close()
+
+	go func() {
+		conn, err := ln.Accept()
+		if err != nil {
+			return
+		}
+		defer conn.Close()
+
+		buf := make([]byte, 256)
+		conn.Read(buf)
+
+		csv := "# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight\n"
+		csv += "bk_tg1,srv_i-aaa111,0,0,0,0,0,5,0,0,0,0,0,0,0,0,0,DOWN,1\n"
+		csv += "bk_tg1,srv_i-aaa111" + FailoverServerSuffix + ",0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,UP,1\n"
+		fmt.Fprint(conn, csv)
+	}()
+
+	servers, err := queryHAProxyStats(sock)
+	if err != nil {
+		t.Fatalf("queryHAProxyStats: %v", err)
+	}
+	if len(servers) != 1 || servers[0].Server != "srv_i-aaa111" || servers[0].Status != "DOWN" {
+		t.Fatalf("got %+v, want only srv_i-aaa111/DOWN", servers)
+	}
+}

spinifex/domains/ec2/eip/service_impl.go

@@ -181,7 +181,7 @@ func (s *EIPServiceImpl) ReleaseAddress(ctx context.Context, input *ec2.ReleaseA
 
 	// Cannot release an EIP that is still associated.
 	if record.State == "associated" {
-		return nil, errors.New(awserrors.ErrorInvalidAddressLocked)
+		return nil, awserrors.Errorf(awserrors.ErrorInvalidIPAddressInUse, "Address %s is in use.", record.PublicIp)
 	}
 
 	// Release IP back to IPAM pool. User-driven release of an already-detached

spinifex/domains/ec2/eip/service_impl_test.go

@@ -169,8 +169,11 @@ func TestEIP_ReleaseWhileAssociated(t *testing.T) {
 	_, err = svc.ReleaseAddress(context.Background(), &ec2.ReleaseAddressInput{
 		AllocationId: aws.String(allocID),
 	}, testAccountID)
-	assert.Error(t, err)
-	assert.Contains(t, err.Error(), "InvalidAddress.Locked")
+	require.Error(t, err)
+	code, message, ok := awserrors.ResolveErrorDetail(err)
+	require.True(t, ok, "error must carry a registered code: %v", err)
+	assert.Equal(t, awserrors.ErrorInvalidIPAddressInUse, code)
+	assert.Equal(t, "Address "+record.PublicIp+" is in use.", message)
 }
 
 func TestEIP_ReleaseByInstanceID_ReclaimsAssociatedEIP(t *testing.T) {

spinifex/domains/ec2/vpc/service_impl.go

@@ -91,6 +91,7 @@ type VPCServiceImpl struct {
 	defaultsKV jetstream.KeyValue // one default-resource claim per account
 	rtbKV      jetstream.KeyValue // route table bucket for auto-creating main route table
 	igwKV      jetstream.KeyValue // internet gateway bucket for the DeleteVpc dependency check
+	eigwKV     jetstream.KeyValue // egress-only internet gateway bucket for the DeleteVpc dependency check
 	ipam       *IPAM
 
 	// Optional: injected after construction for public IP cleanup in DeleteNetworkInterface.
@@ -224,6 +225,10 @@ func NewVPCServiceImplWithNATS(ctx context.Context, cfg *config.Config, natsConn
 	if err != nil {
 		return nil, fmt.Errorf("failed to create KV bucket spinifex-igw: %w", err)
 	}
+	eigwKV, err := kvutil.GetOrCreateBucket(ctx, js, "spinifex-eigw", 10)
+	if err != nil {
+		return nil, fmt.Errorf("failed to create KV bucket spinifex-eigw: %w", err)
+	}
 
 	ipam, err := NewIPAM(ctx, js)
 	if err != nil {
@@ -248,6 +253,7 @@ func NewVPCServiceImplWithNATS(ctx context.Context, cfg *config.Config, natsConn
 		defaultsKV: defaultsKV,
 		rtbKV:      rtbKV,
 		igwKV:      igwKV,
+		eigwKV:     eigwKV,
 		ipam:       ipam,
 	}, nil
 }
@@ -492,10 +498,13 @@ func (s *VPCServiceImpl) DeleteVpc(ctx context.Context, input *ec2.DeleteVpcInpu
 		}
 	}
 
-	// Reject if an internet gateway still names this VPC, as AWS does. Without
-	// this the gateway outlives the only VPC id that can detach it, and it can
-	// then never be detached or deleted.
-	if err := s.rejectAttachedIGW(ctx, accountID, vpcID); err != nil {
+	// Reject if an internet gateway or egress-only internet gateway still names
+	// this VPC, as AWS does. Without this the gateway outlives the only VPC id
+	// that can detach it, and it can then never be detached or deleted.
+	if err := rejectAttachedGateway(ctx, s.igwKV, accountID, vpcID); err != nil {
+		return nil, err
+	}
+	if err := rejectAttachedGateway(ctx, s.eigwKV, accountID, vpcID); err != nil {
 		return nil, err
 	}
 
@@ -538,22 +547,22 @@ func (s *VPCServiceImpl) DeleteVpc(ctx context.Context, input *ec2.DeleteVpcInpu
 	return &ec2.DeleteVpcOutput{}, nil
 }
 
-// rejectAttachedIGW returns DependencyViolation when an internet gateway record
+// rejectAttachedGateway returns DependencyViolation when a gateway record in kv
 // still names vpcID. It reads the raw record rather than a describe projection,
 // because an attachment awaiting confirmation is hidden from the AWS surface but
 // still blocks the delete.
-func (s *VPCServiceImpl) rejectAttachedIGW(ctx context.Context, accountID, vpcID string) error {
-	if s.igwKV == nil {
+func rejectAttachedGateway(ctx context.Context, kv jetstream.KeyValue, accountID, vpcID string) error {
+	if kv == nil {
 		return nil
 	}
 
 	prefix := accountID + "."
-	keys, err := s.igwKV.Keys(ctx)
+	keys, err := kv.Keys(ctx)
 	if err != nil {
 		if errors.Is(err, jetstream.ErrNoKeysFound) {
 			return nil
 		}
-		slog.WarnContext(ctx, "DeleteVpc: IGW key listing failed", "vpcId", vpcID, "err", err)
+		slog.WarnContext(ctx, "DeleteVpc: gateway key listing failed", "bucket", kv.Bucket(), "vpcId", vpcID, "err", err)
 		return errors.New(awserrors.ErrorServerInternal)
 	}
 
@@ -561,7 +570,7 @@ func (s *VPCServiceImpl) rejectAttachedIGW(ctx context.Context, accountID, vpcID
 		if k == kvutil.VersionKey || !strings.HasPrefix(k, prefix) {
 			continue
 		}
-		entry, err := s.igwKV.Get(ctx, k)
+		entry, err := kv.Get(ctx, k)
 		if err != nil {
 			// Deleted between Keys() and Get() is fine to skip. Any other read
 			// error is fail-closed: an attachment this cannot see must not let
@@ -569,19 +578,18 @@ func (s *VPCServiceImpl) rejectAttachedIGW(ctx context.Context, accountID, vpcID
 			if errors.Is(err, jetstream.ErrKeyNotFound) {
 				continue
 			}
-			slog.WarnContext(ctx, "DeleteVpc: IGW read failed", "key", k, "err", err)
+			slog.WarnContext(ctx, "DeleteVpc: gateway read failed", "bucket", kv.Bucket(), "key", k, "err", err)
 			return errors.New(awserrors.ErrorServerInternal)
 		}
-		var igw struct {
-			InternetGatewayId string `json:"internet_gateway_id"`
-			VpcId             string `json:"vpc_id"`
+		var gw struct {
+			VpcId string `json:"vpc_id"`
 		}
-		if err := json.Unmarshal(entry.Value(), &igw); err != nil {
-			slog.WarnContext(ctx, "DeleteVpc: IGW unmarshal failed", "key", k, "err", err)
+		if err := json.Unmarshal(entry.Value(), &gw); err != nil {
+			slog.WarnContext(ctx, "DeleteVpc: gateway unmarshal failed", "bucket", kv.Bucket(), "key", k, "err", err)
 			return errors.New(awserrors.ErrorServerInternal)
 		}
-		if igw.VpcId == vpcID {
-			slog.WarnContext(ctx, "DeleteVpc: VPC has an attached internet gateway", "vpcId", vpcID, "internetGatewayId", igw.InternetGatewayId)
+		if gw.VpcId == vpcID {
+			slog.WarnContext(ctx, "DeleteVpc: VPC has an attached gateway", "bucket", kv.Bucket(), "vpcId", vpcID, "key", k)
 			return awserrors.HasDependencies("vpc", vpcID)
 		}
 	}
@@ -896,7 +904,8 @@ func (s *VPCServiceImpl) checkSubnetResidents(ctx context.Context, accountID, su
 		}
 		var record ENIRecord
 		if err := json.Unmarshal(entry.Value(), &record); err != nil {
-			continue
+			slog.ErrorContext(ctx, "DeleteSubnet: ENI unmarshal failed, blocking delete to avoid orphaning ports", "subnetId", subnetID, "key", key, "err", err)
+			return errors.New(awserrors.ErrorServerInternal)
 		}
 		if record.SubnetId == subnetID && eniIsLiveAttachment(&record) {
 			return awserrors.HasDependencies("subnet", subnetID)

spinifex/domains/ec2/vpc/service_impl_test.go

@@ -864,6 +864,26 @@ func TestDeleteVpc_RejectsAttachedInternetGateway(t *testing.T) {
 	require.Len(t, desc.Vpcs, 1, "VPC must persist so the caller can detach the gateway and retry")
 }
 
+// AWS refuses DeleteVpc while an egress-only internet gateway names the VPC;
+// deleting it anyway leaves the gateway pointing at a VPC that no longer exists.
+func TestDeleteVpc_RejectsEgressOnlyInternetGateway(t *testing.T) {
+	t.Parallel()
+	svc := setupTestVPCService(t)
+	vpcID := createTestVPC(t, svc, "10.66.0.0/16")
+
+	_, err := svc.eigwKV.Put(t.Context(), testAccountID+".eigw-attached",
+		[]byte(`{"egress_only_internet_gateway_id":"eigw-attached","vpc_id":"`+vpcID+`","state":"attached"}`))
+	require.NoError(t, err)
+
+	_, err = svc.DeleteVpc(context.Background(), &ec2.DeleteVpcInput{VpcId: aws.String(vpcID)}, testAccountID)
+	require.Error(t, err)
+	requireAWSError(t, err, awserrors.ErrorDependencyViolation, "The vpc '"+vpcID+"' has dependencies and cannot be deleted.")
+
+	desc, err := svc.DescribeVpcs(context.Background(), &ec2.DescribeVpcsInput{VpcIds: []*string{aws.String(vpcID)}}, testAccountID)
+	require.NoError(t, err)
+	require.Len(t, desc.Vpcs, 1, "VPC must persist so the caller can delete the gateway and retry")
+}
+
 // A gateway attached to some other VPC must not block this one.
 func TestDeleteVpc_IgnoresInternetGatewayOnAnotherVPC(t *testing.T) {
 	t.Parallel()
@@ -994,6 +1014,28 @@ func TestDeleteSubnet_RouteTableCheckFailsClosedOnCorruptRTB(t *testing.T) {
 	require.Len(t, desc.Subnets, 1, "subnet must persist so a KV read failure never strands an association unseen")
 }
 
+// TestDeleteSubnet_ResidentCheckFailsClosedOnCorruptENI asserts an ENI
+// record that cannot be decoded blocks DeleteSubnet, since it may be a live
+// port in this subnet.
+func TestDeleteSubnet_ResidentCheckFailsClosedOnCorruptENI(t *testing.T) {
+	t.Parallel()
+	svc := setupTestVPCService(t)
+	vpcID := createTestVPC(t, svc, "10.66.0.0/16")
+	subnetID := createTestSubnet(t, svc, vpcID, "10.66.1.0/24")
+	eniID := createTestENI(t, svc, subnetID)
+
+	_, err := svc.eniKV.Put(t.Context(), testAccountID+"."+eniID, []byte("{not json"))
+	require.NoError(t, err)
+
+	_, err = svc.DeleteSubnet(context.Background(), &ec2.DeleteSubnetInput{SubnetId: aws.String(subnetID)}, testAccountID)
+	require.Error(t, err)
+	assert.ErrorContains(t, err, "ServerInternal")
+
+	desc, err := svc.DescribeSubnets(context.Background(), &ec2.DescribeSubnetsInput{SubnetIds: []*string{aws.String(subnetID)}}, testAccountID)
+	require.NoError(t, err)
+	require.Len(t, desc.Subnets, 1)
+}
+
 // TestEnsureDefaultVPC_NoVpcdResponder simulates the daemon-startup race where
 // EnsureDefaultVPC runs before vpcd has subscribed. The SG step is best-effort,
 // so subnet and RTB must still land in KV when vpcd is absent.

spinifex/domains/ecr/awsapi/repository.go

@@ -29,10 +29,7 @@ func (e RepositoryEndpoint) RegistryURIHost(accountID string) string {
 	if host == "" {
 		host = accountID + ".dkr.ecr." + e.Region + "." + e.ServicesDomain
 	}
-	if e.RegistryPort != "" && e.RegistryPort != "443" {
-		return host + ":" + e.RegistryPort
-	}
-	return host
+	return ecr.RegistryURIHost(host, e.RegistryPort)
 }
 
 // RepositoryURI returns the pull/push URI for an account-scoped repository.

spinifex/domains/ecr/meta.go

@@ -78,6 +78,31 @@ type ManifestMeta struct {
 	Size         int64     `json:"size"`
 	PushedAt     time.Time `json:"pushedAt"`
 	ChildDigests []string  `json:"childDigests,omitempty"`
+	// SubjectDigest and ArtifactType are set only for manifests that declare a
+	// subject, and feed the OCI referrers listing for that subject.
+	SubjectDigest string            `json:"subjectDigest,omitempty"`
+	ArtifactType  string            `json:"artifactType,omitempty"`
+	Annotations   map[string]string `json:"annotations,omitempty"`
+}
+
+// ReferrerDescriptor is one entry of a subject's referrers listing: the OCI
+// descriptor of a manifest that names the subject.
+type ReferrerDescriptor struct {
+	MediaType    string            `json:"mediaType"`
+	Digest       string            `json:"digest"`
+	Size         int64             `json:"size"`
+	ArtifactType string            `json:"artifactType,omitempty"`
+	Annotations  map[string]string `json:"annotations,omitempty"`
+}
+
+func (m ManifestMeta) referrerDescriptor() ReferrerDescriptor {
+	return ReferrerDescriptor{
+		MediaType:    m.MediaType,
+		Digest:       m.Digest,
+		Size:         m.Size,
+		ArtifactType: m.ArtifactType,
+		Annotations:  m.Annotations,
+	}
 }
 
 // UploadState tracks an in-progress chunked blob upload. The sha256 hash state
@@ -124,6 +149,10 @@ type MetaStore interface {
 	GetManifestMeta(ctx context.Context, accountID, repo, digest string) (ManifestMeta, error)
 	DeleteManifestMeta(ctx context.Context, accountID, repo, digest string) error
 
+	// ListReferrers returns the manifests in repo that declare subjectDigest as
+	// their subject, ordered by digest. A subject with none yields an empty slice.
+	ListReferrers(ctx context.Context, accountID, repo, subjectDigest string) ([]ReferrerDescriptor, error)
+
 	PutUpload(ctx context.Context, accountID, uploadID string, state UploadState) (uint64, error)
 	GetUpload(ctx context.Context, accountID, uploadID string) (UploadState, uint64, error)
 	UpdateUpload(ctx context.Context, accountID, uploadID string, state UploadState, rev uint64) (uint64, error)
@@ -234,11 +263,11 @@ func (s *KVMetaStore) DeleteRepo(ctx context.Context, accountID, repo string) er
 		}
 		return err
 	}
-	tagsPrefix, manifestsPrefix := KVTagsPrefix(repo), KVManifestsPrefix(repo)
+	tagsPrefix, manifestsPrefix, referrersPrefix := KVTagsPrefix(repo), KVManifestsPrefix(repo), KVReferrersPrefix(repo)
 	metaKey, policyKey, lifecycleKey := KVRepoMetaKey(repo), KVRepoPolicyKey(repo), KVLifecyclePolicyKey(repo)
 	for _, k := range keys {
 		if k == metaKey || k == policyKey || k == lifecycleKey ||
-			strings.HasPrefix(k, tagsPrefix) || strings.HasPrefix(k, manifestsPrefix) {
+			strings.HasPrefix(k, tagsPrefix) || strings.HasPrefix(k, manifestsPrefix) || strings.HasPrefix(k, referrersPrefix) {
 			if err := kv.Delete(ctx, k); err != nil {
 				return mapKVErr(err)
 			}
@@ -415,10 +444,55 @@ func (s *KVMetaStore) PutManifestMeta(ctx context.Context, accountID, repo strin
 	if err != nil {
 		return err
 	}
-	_, err = kv.Put(ctx, KVManifestKey(repo, meta.Digest), data)
+	if _, err = kv.Put(ctx, KVManifestKey(repo, meta.Digest), data); err != nil {
+		return err
+	}
+	if meta.SubjectDigest == "" {
+		return nil
+	}
+	entry, err := json.Marshal(meta.referrerDescriptor())
+	if err != nil {
+		return err
+	}
+	_, err = kv.Put(ctx, KVReferrerKey(repo, meta.SubjectDigest, meta.Digest), entry)
 	return err
 }
 
+func (s *KVMetaStore) ListReferrers(ctx context.Context, accountID, repo, subjectDigest string) ([]ReferrerDescriptor, error) {
+	kv, err := s.bucket(ctx, accountID)
+	if err != nil {
+		return nil, err
+	}
+	keys, err := kv.Keys(ctx)
+	if err != nil {
+		if errors.Is(err, jetstream.ErrNoKeysFound) {
+			return []ReferrerDescriptor{}, nil
+		}
+		return nil, err
+	}
+	prefix := KVSubjectReferrersPrefix(repo, subjectDigest)
+	out := []ReferrerDescriptor{}
+	for _, k := range keys {
+		if !strings.HasPrefix(k, prefix) {
+			continue
+		}
+		entry, err := kv.Get(ctx, k)
+		if err != nil {
+			if errors.Is(err, jetstream.ErrKeyNotFound) {
+				continue
+			}
+			return nil, err
+		}
+		var d ReferrerDescriptor
+		if err := json.Unmarshal(entry.Value(), &d); err != nil {
+			return nil, err
+		}
+		out = append(out, d)
+	}
+	sort.Slice(out, func(i, j int) bool { return out[i].Digest < out[j].Digest })
+	return out, nil
+}
+
 func (s *KVMetaStore) GetManifestMeta(ctx context.Context, accountID, repo, digest string) (ManifestMeta, error) {
 	kv, err := s.bucket(ctx, accountID)
 	if err != nil {
@@ -440,9 +514,19 @@ func (s *KVMetaStore) DeleteManifestMeta(ctx context.Context, accountID, repo, d
 	if err != nil {
 		return err
 	}
-	if _, err := kv.Get(ctx, KVManifestKey(repo, digest)); err != nil {
+	entry, err := kv.Get(ctx, KVManifestKey(repo, digest))
+	if err != nil {
 		return mapKVErr(err)
 	}
+	var m ManifestMeta
+	if err := json.Unmarshal(entry.Value(), &m); err != nil {
+		return err
+	}
+	if m.SubjectDigest != "" {
+		if err := kv.Delete(ctx, KVReferrerKey(repo, m.SubjectDigest, digest)); err != nil {
+			return mapKVErr(err)
+		}
+	}
 	return kv.Delete(ctx, KVManifestKey(repo, digest))
 }
 
@@ -738,6 +822,19 @@ func (m *MemoryMetaStore) DeleteManifestMeta(_ context.Context, accountID, repo,
 	return nil
 }
 
+func (m *MemoryMetaStore) ListReferrers(_ context.Context, accountID, repo, subjectDigest string) ([]ReferrerDescriptor, error) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	out := []ReferrerDescriptor{}
+	for k, meta := range m.manifests[accountID] {
+		if r, _, ok := strings.Cut(k, "|"); ok && r == repo && meta.SubjectDigest == subjectDigest {
+			out = append(out, meta.referrerDescriptor())
+		}
+	}
+	sort.Slice(out, func(i, j int) bool { return out[i].Digest < out[j].Digest })
+	return out, nil
+}
+
 func (m *MemoryMetaStore) PutUpload(_ context.Context, accountID, uploadID string, state UploadState) (uint64, error) {
 	m.mu.Lock()
 	defer m.mu.Unlock()

spinifex/domains/ecr/meta_nats.go

@@ -184,6 +184,14 @@ func (s *NATSMetaStore) DeleteManifestMeta(ctx context.Context, accountID, repo,
 	return nil
 }
 
+func (s *NATSMetaStore) ListReferrers(ctx context.Context, accountID, repo, subjectDigest string) ([]ReferrerDescriptor, error) {
+	resp, err := natsmsg.NATSRequest[ReferrerListResponse](ctx, s.conn, SubjectReferrerList, &ReferrerListRequest{Repo: repo, Subject: subjectDigest}, metaRequestTimeout, accountID)
+	if err != nil {
+		return nil, err
+	}
+	return resp.Referrers, nil
+}
+
 func (s *NATSMetaStore) PutUpload(ctx context.Context, accountID, uploadID string, state UploadState) (uint64, error) {
 	resp, err := natsmsg.NATSRequest[UploadCreateResponse](ctx, s.conn, SubjectUploadCreate, &UploadCreateRequest{UploadID: uploadID, State: state}, metaRequestTimeout, accountID)
 	if err != nil {

spinifex/domains/ecr/meta_test.go

@@ -223,3 +223,12 @@ func TestMemoryMetaStore_LifecyclePolicy(t *testing.T) {
 	_, err = m.GetLifecyclePolicy(context.Background(), acct, "team/app")
 	assert.ErrorIs(t, err, ErrNotFound)
 }
+
+func TestManifestMeta_DecodesRecordWithoutReferrerFields(t *testing.T) {
+	var m ManifestMeta
+	require.NoError(t, json.Unmarshal([]byte(`{"digest":"sha256:ccc","mediaType":"application/vnd.oci.image.manifest.v1+json","size":10,"pushedAt":"2026-01-01T00:00:00Z","childDigests":["sha256:ddd"]}`), &m))
+	assert.Equal(t, "sha256:ccc", m.Digest)
+	assert.Equal(t, []string{"sha256:ddd"}, m.ChildDigests)
+	assert.Empty(t, m.SubjectDigest)
+	assert.Empty(t, m.ArtifactType)
+}

spinifex/domains/ecr/referrers_test.go

@@ -0,0 +1,94 @@
+package ecr
+
+import (
+	"context"
+	"testing"
+
+	"github.com/mulgadc/spinifex/internal/testkit"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+const (
+	refSubject = "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+	refSigA    = "sha256:1111111111111111111111111111111111111111111111111111111111111111"
+	refSigB    = "sha256:2222222222222222222222222222222222222222222222222222222222222222"
+)
+
+func referrerMeta(digest, subject string) ManifestMeta {
+	return ManifestMeta{
+		Digest:        digest,
+		MediaType:     "application/vnd.oci.image.manifest.v1+json",
+		Size:          42,
+		SubjectDigest: subject,
+		ArtifactType:  "application/vnd.example.sig.v1",
+		Annotations:   map[string]string{"org.example.key": "v"},
+	}
+}
+
+// referrerStores runs a test against every MetaStore implementation that keeps
+// its own referrers state.
+func referrerStores(t *testing.T) map[string]MetaStore {
+	t.Helper()
+	_, nc, _ := testutil.StartTestJetStream(t)
+	return map[string]MetaStore{
+		"memory": NewMemoryMetaStore(),
+		"kv":     NewKVMetaStore(testutil.NewJetStream(t, nc)),
+	}
+}
+
+func TestListReferrers(t *testing.T) {
+	ctx := context.Background()
+	for name, store := range referrerStores(t) {
+		t.Run(name, func(t *testing.T) {
+			acct, repo := "111111111111", "team/app"
+
+			got, err := store.ListReferrers(ctx, acct, repo, refSubject)
+			require.NoError(t, err)
+			assert.Empty(t, got, "a subject with no referrers lists empty")
+
+			require.NoError(t, store.PutManifestMeta(ctx, acct, repo, referrerMeta(refSigB, refSubject)))
+			require.NoError(t, store.PutManifestMeta(ctx, acct, repo, referrerMeta(refSigA, refSubject)))
+			// Re-putting the same digest must not duplicate the entry.
+			require.NoError(t, store.PutManifestMeta(ctx, acct, repo, referrerMeta(refSigA, refSubject)))
+
+			got, err = store.ListReferrers(ctx, acct, repo, refSubject)
+			require.NoError(t, err)
+			require.Len(t, got, 2)
+			assert.Equal(t, refSigA, got[0].Digest, "entries are ordered by digest")
+			assert.Equal(t, refSigB, got[1].Digest)
+			assert.Equal(t, ReferrerDescriptor{
+				MediaType:    "application/vnd.oci.image.manifest.v1+json",
+				Digest:       refSigA,
+				Size:         42,
+				ArtifactType: "application/vnd.example.sig.v1",
+				Annotations:  map[string]string{"org.example.key": "v"},
+			}, got[0])
+
+			otherRepo, err := store.ListReferrers(ctx, acct, "team/other", refSubject)
+			require.NoError(t, err)
+			assert.Empty(t, otherRepo, "referrers do not cross repositories")
+			otherAcct, err := store.ListReferrers(ctx, "222222222222", repo, refSubject)
+			require.NoError(t, err)
+			assert.Empty(t, otherAcct, "referrers do not cross accounts")
+
+			require.NoError(t, store.DeleteManifestMeta(ctx, acct, repo, refSigA))
+			got, err = store.ListReferrers(ctx, acct, repo, refSubject)
+			require.NoError(t, err)
+			require.Len(t, got, 1)
+			assert.Equal(t, refSigB, got[0].Digest)
+		})
+	}
+}
+
+func TestListReferrers_ManifestWithoutSubjectIsNotListed(t *testing.T) {
+	ctx := context.Background()
+	for name, store := range referrerStores(t) {
+		t.Run(name, func(t *testing.T) {
+			require.NoError(t, store.PutManifestMeta(ctx, "111111111111", "r", referrerMeta(refSigA, "")))
+			got, err := store.ListReferrers(ctx, "111111111111", "r", refSubject)
+			require.NoError(t, err)
+			assert.Empty(t, got)
+		})
+	}
+}

spinifex/domains/ecr/registry/authorization.go

@@ -99,6 +99,8 @@ func ClassifyOperation(method, urlPath string, query url.Values) (ClassifiedOper
 		return classifyBlobOperation(method, name, ref, query)
 	case "manifests":
 		return classifyManifestOperation(method, name, ref)
+	case "referrers":
+		return classifyReferrersOperation(method, name, ref)
 	default:
 		return ClassifiedOperation{}, false
 	}
@@ -165,6 +167,15 @@ func classifyBlobOperation(method, name, ref string, query url.Values) (Classifi
 	}
 }
 
+// classifyReferrersOperation authorizes the referrers listing as a read of the
+// repository's images.
+func classifyReferrersOperation(method, name, subject string) (ClassifiedOperation, bool) {
+	if method != http.MethodGet || subject == "" {
+		return ClassifiedOperation{}, false
+	}
+	return ClassifiedOperation{Repo: name, Requirements: []ActionRequirement{{ActionBatchGetImage, ScopeDestination}}}, true
+}
+
 func classifyManifestOperation(method, name, reference string) (ClassifiedOperation, bool) {
 	if reference == "" {
 		return ClassifiedOperation{}, false

spinifex/domains/ecr/registry/authorization_test.go

@@ -132,6 +132,15 @@ func TestClassifyOperation_RouteMatrix(t *testing.T) {
 				Requirements: []ActionRequirement{{ActionBatchDeleteImage, ScopeDestination}},
 			},
 		},
+		{
+			name:   "GET referrers is a read of the repository images",
+			method: http.MethodGet,
+			path:   "/v2/team/app/referrers/sha256:abc",
+			want: ClassifiedOperation{
+				Repo:         "team/app",
+				Requirements: []ActionRequirement{{ActionBatchGetImage, ScopeDestination}},
+			},
+		},
 		{
 			name:   "cross-repo mount requires destination initiate + source read",
 			method: http.MethodPost,
@@ -205,6 +214,9 @@ func TestClassifyOperation_UnsupportedNeverClassifies(t *testing.T) {
 		{"manifest POST unsupported", http.MethodPost, "/v2/team/app/manifests/latest"},
 		{"manifest empty reference", http.MethodGet, "/v2/team/app/manifests/"},
 		{"catalog wrong method", http.MethodPost, "/v2/_catalog"},
+		{"referrers PUT unsupported", http.MethodPut, "/v2/team/app/referrers/sha256:abc"},
+		{"referrers DELETE unsupported", http.MethodDelete, "/v2/team/app/referrers/sha256:abc"},
+		{"referrers empty subject", http.MethodGet, "/v2/team/app/referrers/"},
 	}
 	for _, c := range cases {
 		t.Run(c.name, func(t *testing.T) {

spinifex/domains/ecr/registry/registry.go

@@ -142,6 +142,8 @@ func (reg *Registry) serve(w http.ResponseWriter, r *http.Request) {
 		reg.routeManifests(w, r, name, ref)
 	case "tags":
 		reg.routeTags(w, r, name, ref)
+	case "referrers":
+		reg.getReferrers(w, r, name, ref)
 	default:
 		WriteError(w, http.StatusNotFound, "NAME_UNKNOWN", "unrecognized registry path")
 	}
@@ -151,7 +153,7 @@ func (reg *Registry) serve(w http.ResponseWriter, r *http.Request) {
 // kind is one of blobs/manifests/tags. ref is the remainder (digest, reference,
 // "uploads", "uploads/{uuid}", or "list"). ok is false if no marker is found.
 func splitV2Path(path string) (name, kind, ref string, ok bool) {
-	for _, marker := range []string{"/blobs/", "/manifests/", "/tags/"} {
+	for _, marker := range []string{"/blobs/", "/manifests/", "/tags/", "/referrers/"} {
 		if before, after, found := strings.Cut(path, marker); found {
 			kind = strings.Trim(marker, "/")
 			return before, kind, after, before != ""
@@ -590,7 +592,7 @@ func (reg *Registry) putManifest(w http.ResponseWriter, r *http.Request, name, r
 		return
 	}
 
-	digest, err := reg.StoreManifest(ctx, reg.AccountID, name, reference, r.Header.Get("Content-Type"), body)
+	digest, subject, err := reg.storeManifest(ctx, reg.AccountID, name, reference, r.Header.Get("Content-Type"), body)
 	if err != nil {
 		if mErr, ok := errors.AsType[*ManifestStoreError](err); ok {
 			WriteError(w, mErr.Status, mErr.Code, mErr.Msg)
@@ -602,6 +604,9 @@ func (reg *Registry) putManifest(w http.ResponseWriter, r *http.Request, name, r
 
 	w.Header().Set("Docker-Content-Digest", digest)
 	w.Header().Set("Location", fmt.Sprintf("/v2/%s/manifests/%s", name, digest))
+	if subject != "" {
+		w.Header().Set("Oci-Subject", subject)
+	}
 	w.WriteHeader(http.StatusCreated)
 }
 
@@ -635,9 +640,15 @@ func (reg *Registry) deleteManifest(ctx context.Context, w http.ResponseWriter,
 
 // manifestDoc is the subset of an OCI image manifest the validator inspects.
 type manifestDoc struct {
-	MediaType string `json:"mediaType"`
-	Config    struct {
+	MediaType    string            `json:"mediaType"`
+	ArtifactType string            `json:"artifactType"`
+	Annotations  map[string]string `json:"annotations"`
+	Subject      *struct {
 		Digest string `json:"digest"`
+	} `json:"subject"`
+	Config struct {
+		MediaType string `json:"mediaType"`
+		Digest    string `json:"digest"`
 	} `json:"config"`
 	Layers []struct {
 		Digest string `json:"digest"`
@@ -648,6 +659,30 @@ type manifestDoc struct {
 	} `json:"manifests"`
 }
 
+// referrerFields returns the subject digest, artifactType and annotations of a
+// manifest that declares a subject. mediaType is the type it is stored under;
+// an image manifest without an artifactType uses its config mediaType.
+func referrerFields(mediaType string, body []byte) (subject, artifactType string, annotations map[string]string, err error) {
+	var doc manifestDoc
+	if err := json.Unmarshal(body, &doc); err != nil {
+		return "", "", nil, fmt.Errorf("manifest is not valid JSON: %w", err)
+	}
+	if doc.Subject == nil {
+		return "", "", nil, nil
+	}
+	if !ecr.ValidateDigest(doc.Subject.Digest) {
+		return "", "", nil, errors.New("subject references malformed digest")
+	}
+	if mediaType == "" {
+		mediaType = doc.MediaType
+	}
+	artifactType = doc.ArtifactType
+	if artifactType == "" && mediaType != mediaTypeOCIIndex && mediaType != mediaTypeDockerManifestList {
+		artifactType = doc.Config.MediaType
+	}
+	return doc.Subject.Digest, artifactType, doc.Annotations, nil
+}
+
 // validateManifest performs type-aware shallow validation, returning the child
 // digests recorded in the manifest metadata. For an image manifest every config
 // and layer blob must exist; for an index every child manifest must exist with a
@@ -697,6 +732,57 @@ func (reg *Registry) validateManifest(ctx context.Context, name, contentType str
 	}
 }
 
+// ---- referrers ----
+
+// referrersIndex is the OCI image index the referrers API answers with.
+type referrersIndex struct {
+	SchemaVersion int                      `json:"schemaVersion"`
+	MediaType     string                   `json:"mediaType"`
+	Manifests     []ecr.ReferrerDescriptor `json:"manifests"`
+}
+
+// getReferrers answers GET /v2/<name>/referrers/<digest> with an index of the
+// manifests that name the digest as their subject. A subject with none, or one
+// that is not stored, yields an empty index.
+func (reg *Registry) getReferrers(w http.ResponseWriter, r *http.Request, name, subject string) {
+	ctx := r.Context()
+	if !ecr.ValidateDigest(subject) {
+		WriteError(w, http.StatusBadRequest, "DIGEST_INVALID", "malformed subject digest")
+		return
+	}
+	if _, err := reg.Meta.GetRepo(ctx, reg.AccountID, name); err != nil {
+		if errors.Is(err, ecr.ErrNotFound) {
+			WriteError(w, http.StatusNotFound, "NAME_UNKNOWN", "repository unknown")
+			return
+		}
+		reg.internal(ctx, w, "get repository", err)
+		return
+	}
+	referrers, err := reg.Meta.ListReferrers(ctx, reg.AccountID, name, subject)
+	if err != nil {
+		reg.internal(ctx, w, "list referrers", err)
+		return
+	}
+	if want := r.URL.Query().Get("artifactType"); want != "" {
+		referrers = slices.DeleteFunc(referrers, func(d ecr.ReferrerDescriptor) bool { return d.ArtifactType != want })
+		w.Header().Set("Oci-Filters-Applied", "artifactType")
+	}
+	if referrers == nil {
+		referrers = []ecr.ReferrerDescriptor{}
+	}
+	out, err := json.Marshal(referrersIndex{SchemaVersion: 2, MediaType: mediaTypeOCIIndex, Manifests: referrers})
+	if err != nil {
+		reg.internal(ctx, w, "encode referrers", err)
+		return
+	}
+	w.Header().Set("Content-Type", mediaTypeOCIIndex)
+	w.Header().Set("Content-Length", strconv.Itoa(len(out)))
+	w.WriteHeader(http.StatusOK)
+	if _, err := w.Write(out); err != nil {
+		slog.ErrorContext(ctx, "ECR/OCI: referrers write failed", "err", err)
+	}
+}
+
 // ---- tags ----
 
 func (reg *Registry) routeTags(w http.ResponseWriter, r *http.Request, name, ref string) {

spinifex/domains/ecr/registry/registry_image.go

@@ -6,6 +6,7 @@ import (
 	"errors"
 	"io"
 	"log/slog"
+	"mime"
 	"net/http"
 	"slices"
 	"strings"
@@ -52,20 +53,29 @@ type ImageRecord struct {
 // case it is detected. On a validation/size failure it returns a
 // *ManifestStoreError carrying the OCI code.
 func (reg *Registry) StoreManifest(ctx context.Context, account, repo, ref, contentType string, body []byte) (string, error) {
+	digest, _, err := reg.storeManifest(ctx, account, repo, ref, contentType, body)
+	return digest, err
+}
+
+// storeManifest is StoreManifest that also returns the subject digest the
+// manifest declares ("" when it has none).
+func (reg *Registry) storeManifest(ctx context.Context, account, repo, ref, contentType string, body []byte) (string, string, error) {
 	scoped := reg.forAccount(account)
 	if err := scoped.requireRepo(ctx, repo); err != nil {
-		return "", err
+		return "", "", err
 	}
 	if len(body) > maxManifestBytes {
-		return "", &ManifestStoreError{http.StatusRequestEntityTooLarge, "MANIFEST_INVALID", "manifest too large"}
+		return "", "", &ManifestStoreError{http.StatusRequestEntityTooLarge, "MANIFEST_INVALID", "manifest too large"}
 	}
 	if contentType == "" {
 		contentType = detectManifestType(body)
+	} else if mt, _, err := mime.ParseMediaType(contentType); err == nil {
+		contentType = mt
 	}
 
 	digest := "sha256:" + hex.EncodeToString(sha256Sum(body))
 	if ecr.ValidateDigest(ref) && ref != digest {
-		return "", &ManifestStoreError{http.StatusBadRequest, "DIGEST_INVALID", "reference digest does not match content"}
+		return "", "", &ManifestStoreError{http.StatusBadRequest, "DIGEST_INVALID", "reference digest does not match content"}
 	}
 
 	// On an IMMUTABLE repo, re-tagging an existing tag onto a different digest is
@@ -73,22 +83,27 @@ func (reg *Registry) StoreManifest(ctx context.Context, account, repo, ref, cont
 	if ref != "" && !ecr.ValidateDigest(ref) {
 		meta, err := scoped.Meta.GetRepo(ctx, account, repo)
 		if err != nil {
-			return "", err
+			return "", "", err
 		}
 		if meta.TagMutability() == ecr.TagMutabilityImmutable {
 			existing, err := scoped.Meta.GetTag(ctx, account, repo, ref)
 			switch {
 			case err == nil && existing != digest:
-				return "", &ManifestStoreError{http.StatusConflict, "TAG_IMMUTABLE", "tag is immutable and already resolves to a different digest"}
+				return "", "", &ManifestStoreError{http.StatusConflict, "TAG_IMMUTABLE", "tag is immutable and already resolves to a different digest"}
 			case err != nil && !errors.Is(err, ecr.ErrNotFound):
-				return "", err
+				return "", "", err
 			}
 		}
 	}
 
 	children, code, vErr := scoped.validateManifest(ctx, repo, contentType, body)
 	if vErr != nil {
-		return "", &ManifestStoreError{http.StatusBadRequest, code, vErr.Error()}
+		return "", "", &ManifestStoreError{http.StatusBadRequest, code, vErr.Error()}
+	}
+
+	subject, artifactType, annotations, rErr := referrerFields(contentType, body)
+	if rErr != nil {
+		return "", "", &ManifestStoreError{http.StatusBadRequest, "MANIFEST_INVALID", rErr.Error()}
 	}
 
 	if _, err := scoped.Store.PutObject(ctx, &s3.PutObjectInput{
@@ -97,7 +112,7 @@ func (reg *Registry) StoreManifest(ctx context.Context, account, repo, ref, cont
 		Body:        aws.ReadSeekCloser(strings.NewReader(string(body))),
 		ContentType: aws.String(contentType),
 	}); err != nil {
-		return "", err
+		return "", "", err
 	}
 
 	if err := scoped.Meta.PutManifestMeta(ctx, account, repo, ecr.ManifestMeta{
@@ -106,16 +121,20 @@ func (reg *Registry) StoreManifest(ctx context.Context, account, repo, ref, cont
 		Size:         int64(len(body)),
 		PushedAt:     time.Now().UTC(),
 		ChildDigests: children,
+
+		SubjectDigest: subject,
+		ArtifactType:  artifactType,
+		Annotations:   annotations,
 	}); err != nil {
-		return "", err
+		return "", "", err
 	}
 
 	if ref != "" && !ecr.ValidateDigest(ref) {
 		if err := scoped.Meta.PutTag(ctx, account, repo, ref, digest); err != nil {
-			return "", err
+			return "", "", err
 		}
 	}
-	return digest, nil
+	return digest, subject, nil
 }
 
 // GetManifest returns the verbatim manifest bytes, media type and digest for a
@@ -230,40 +249,66 @@ func (reg *Registry) DeleteImage(ctx context.Context, account, repo, tag, digest
 		return resolved, nil
 	}
 
-	meta, err := scoped.Meta.GetManifestMeta(ctx, account, repo, digest)
+	if err := scoped.deleteManifestCascade(ctx, account, repo, digest, map[string]bool{}); err != nil {
+		return "", err
+	}
+	return digest, nil
+}
+
+// deleteManifestCascade deletes one manifest by digest with its tags, then the
+// referrers that name it as subject, each through this same path. Referrer
+// failures are logged, not returned: the subject is already deleted and ECR
+// removes referrers asynchronously. visited guards against subject cycles.
+func (reg *Registry) deleteManifestCascade(ctx context.Context, account, repo, digest string, visited map[string]bool) error {
+	visited[digest] = true
+	meta, err := reg.Meta.GetManifestMeta(ctx, account, repo, digest)
 	if err != nil {
 		if errors.Is(err, ecr.ErrNotFound) {
-			return "", ErrImageNotFound
+			return ErrImageNotFound
 		}
-		return "", err
+		return err
 	}
-	tags, err := scoped.Meta.ListTags(ctx, account, repo)
+	tags, err := reg.Meta.ListTags(ctx, account, repo)
 	if err != nil {
-		return "", err
+		return err
 	}
 	for _, t := range tags {
-		d, err := scoped.Meta.GetTag(ctx, account, repo, t)
+		d, err := reg.Meta.GetTag(ctx, account, repo, t)
 		if err != nil {
 			if errors.Is(err, ecr.ErrNotFound) {
 				continue
 			}
-			return "", err
+			return err
 		}
 		if d == digest {
-			if err := scoped.Meta.DeleteTag(ctx, account, repo, t); err != nil {
-				return "", err
+			if err := reg.Meta.DeleteTag(ctx, account, repo, t); err != nil {
+				return err
 			}
 		}
 	}
-	if err := scoped.Meta.DeleteManifestMeta(ctx, account, repo, digest); err != nil {
-		return "", err
+	if err := reg.Meta.DeleteManifestMeta(ctx, account, repo, digest); err != nil {
+		return err
 	}
 
 	// The KV record is gone, so the image is logically deleted. Predastore
 	// reclaim is best-effort: a failure leaks bytes (a capacity nuisance) but
 	// does not undo the delete.
-	scoped.reclaimManifest(ctx, account, repo, digest, meta.ChildDigests)
-	return digest, nil
+	reg.reclaimManifest(ctx, account, repo, digest, meta.ChildDigests)
+
+	referrers, err := reg.Meta.ListReferrers(ctx, account, repo, digest)
+	if err != nil {
+		slog.WarnContext(ctx, "ECR/referrers: listing referrers for cascade failed", "repo", repo, "subject", digest, "err", err)
+		return nil
+	}
+	for _, ref := range referrers {
+		if visited[ref.Digest] {
+			continue
+		}
+		if err := reg.deleteManifestCascade(ctx, account, repo, ref.Digest, visited); err != nil && !errors.Is(err, ErrImageNotFound) {
+			slog.WarnContext(ctx, "ECR/referrers: referrer delete failed", "repo", repo, "subject", digest, "referrer", ref.Digest, "err", err)
+		}
+	}
+	return nil
 }
 
 // reclaimManifest deletes a manifest's predastore object and any child blob no

spinifex/domains/ecr/registry/registry_referrer_test.go

@@ -0,0 +1,409 @@
+package ecrregistry
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/mulgadc/spinifex/spinifex/domains/ecr"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+const (
+	referrerRepo     = "team/app"
+	sigConfigType    = "application/vnd.example.sig.config.v1+json"
+	sigArtifactType  = "application/vnd.example.sig.v1"
+	absentSubjectDgt = "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+)
+
+func imageWithSubject(cfgDigest, subject, artifactType string) string {
+	extra := ""
+	if artifactType != "" {
+		extra = fmt.Sprintf(`"artifactType":%q,`, artifactType)
+	}
+	return fmt.Sprintf(`{"schemaVersion":2,"mediaType":%q,%s"config":{"mediaType":%q,"digest":%q,"size":3},"layers":[],"subject":{"mediaType":%q,"digest":%q,"size":10}}`,
+		mediaTypeOCIManifest, extra, sigConfigType, cfgDigest, mediaTypeOCIManifest, subject)
+}
+
+func storedMeta(t *testing.T, reg *Registry, body string) (subject, artifactType string) {
+	t.Helper()
+	m, err := reg.Meta.GetManifestMeta(context.Background(), testAccount, referrerRepo, digestOf([]byte(body)))
+	require.NoError(t, err)
+	return m.SubjectDigest, m.ArtifactType
+}
+
+func TestPutManifest_SubjectRecordedAndAdvertised(t *testing.T) {
+	reg := newTestRegistry()
+	cfg := pushBlob(t, reg, referrerRepo, []byte("cfg"))
+
+	tests := []struct {
+		name         string
+		artifactType string
+		wantType     string
+	}{
+		{"artifactType falls back to the config mediaType", "", sigConfigType},
+		{"explicit artifactType wins over the config mediaType", sigArtifactType, sigArtifactType},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			body := imageWithSubject(cfg, absentSubjectDgt, tt.artifactType)
+			w := do(reg, http.MethodPut, "/v2/"+referrerRepo+"/manifests/"+digestOf([]byte(body)), []byte(body),
+				map[string]string{"Content-Type": mediaTypeOCIManifest})
+			require.Equal(t, http.StatusCreated, w.Code, "body: %s", w.Body.String())
+			assert.Equal(t, absentSubjectDgt, w.Header().Get("Oci-Subject"))
+
+			subject, artifactType := storedMeta(t, reg, body)
+			assert.Equal(t, absentSubjectDgt, subject)
+			assert.Equal(t, tt.wantType, artifactType)
+		})
+	}
+}
+
+func TestPutManifest_IndexSubject(t *testing.T) {
+	reg := newTestRegistry()
+	seedRepo(t, reg, referrerRepo)
+
+	body := fmt.Sprintf(`{"schemaVersion":2,"mediaType":%q,"manifests":[],"subject":{"mediaType":%q,"digest":%q,"size":10}}`,
+		mediaTypeOCIIndex, mediaTypeOCIManifest, absentSubjectDgt)
+	w := do(reg, http.MethodPut, "/v2/"+referrerRepo+"/manifests/"+digestOf([]byte(body)), []byte(body),
+		map[string]string{"Content-Type": mediaTypeOCIIndex})
+	require.Equal(t, http.StatusCreated, w.Code, "body: %s", w.Body.String())
+	assert.Equal(t, absentSubjectDgt, w.Header().Get("Oci-Subject"))
+
+	subject, artifactType := storedMeta(t, reg, body)
+	assert.Equal(t, absentSubjectDgt, subject)
+	assert.Empty(t, artifactType, "an index has no config mediaType to fall back on")
+}
+
+func TestPutManifest_NoSubjectLeavesFieldsEmpty(t *testing.T) {
+	reg := newTestRegistry()
+	cfg := pushBlob(t, reg, referrerRepo, []byte("cfg"))
+	body := fmt.Sprintf(`{"schemaVersion":2,"mediaType":%q,"artifactType":%q,"config":{"mediaType":%q,"digest":%q,"size":3},"layers":[]}`,
+		mediaTypeOCIManifest, sigArtifactType, sigConfigType, cfg)
+
+	w := do(reg, http.MethodPut, "/v2/"+referrerRepo+"/manifests/v1", []byte(body),
+		map[string]string{"Content-Type": mediaTypeOCIManifest})
+	require.Equal(t, http.StatusCreated, w.Code, "body: %s", w.Body.String())
+	assert.Empty(t, w.Header().Get("Oci-Subject"))
+
+	subject, artifactType := storedMeta(t, reg, body)
+	assert.Empty(t, subject)
+	assert.Empty(t, artifactType, "artifactType is recorded only for manifests that declare a subject")
+}
+
+func TestPutManifest_MalformedSubjectDigestRejected(t *testing.T) {
+	reg := newTestRegistry()
+	cfg := pushBlob(t, reg, referrerRepo, []byte("cfg"))
+	body := imageWithSubject(cfg, "sha256:nothex", "")
+
+	w := do(reg, http.MethodPut, "/v2/"+referrerRepo+"/manifests/v1", []byte(body),
+		map[string]string{"Content-Type": mediaTypeOCIManifest})
+	assert.Equal(t, http.StatusBadRequest, w.Code)
+	assertCode(t, w, "MANIFEST_INVALID")
+
+	_, err := reg.Meta.GetManifestMeta(context.Background(), testAccount, referrerRepo, digestOf([]byte(body)))
+	assert.Error(t, err, "a rejected manifest must not leave a record behind")
+}
+
+func TestPutManifest_IndexWithContentTypeParameterGetsNoConfigArtifactType(t *testing.T) {
+	reg := newTestRegistry()
+	seedRepo(t, reg, referrerRepo)
+
+	body := fmt.Sprintf(`{"schemaVersion":2,"mediaType":%q,"manifests":[],"subject":{"mediaType":%q,"digest":%q,"size":10}}`,
+		mediaTypeOCIIndex, mediaTypeOCIManifest, absentSubjectDgt)
+	w := do(reg, http.MethodPut, "/v2/"+referrerRepo+"/manifests/v1", []byte(body),
+		map[string]string{"Content-Type": mediaTypeOCIIndex + "; charset=utf-8"})
+	require.Equal(t, http.StatusCreated, w.Code, "body: %s", w.Body.String())
+	assert.Equal(t, absentSubjectDgt, w.Header().Get("Oci-Subject"))
+
+	m, err := reg.Meta.GetManifestMeta(context.Background(), testAccount, referrerRepo, digestOf([]byte(body)))
+	require.NoError(t, err)
+	assert.Equal(t, mediaTypeOCIIndex, m.MediaType, "the stored type must not carry the parameter")
+	assert.Empty(t, m.ArtifactType)
+}
+
+func TestDeleteReferrerManifest_RemovesItFromTheSubjectListing(t *testing.T) {
+	reg := newTestRegistry()
+	cfg := pushBlob(t, reg, referrerRepo, []byte("cfg"))
+	body := imageWithSubject(cfg, absentSubjectDgt, sigArtifactType)
+	dg := digestOf([]byte(body))
+	w := do(reg, http.MethodPut, "/v2/"+referrerRepo+"/manifests/"+dg, []byte(body),
+		map[string]string{"Content-Type": mediaTypeOCIManifest})
+	require.Equal(t, http.StatusCreated, w.Code, "body: %s", w.Body.String())
+
+	listed, err := reg.Meta.ListReferrers(context.Background(), testAccount, referrerRepo, absentSubjectDgt)
+	require.NoError(t, err)
+	require.Len(t, listed, 1)
+	assert.Equal(t, dg, listed[0].Digest)
+	assert.Equal(t, sigArtifactType, listed[0].ArtifactType)
+
+	w = do(reg, http.MethodDelete, "/v2/"+referrerRepo+"/manifests/"+dg, nil, nil)
+	require.Equal(t, http.StatusAccepted, w.Code, "body: %s", w.Body.String())
+
+	listed, err = reg.Meta.ListReferrers(context.Background(), testAccount, referrerRepo, absentSubjectDgt)
+	require.NoError(t, err)
+	assert.Empty(t, listed)
+}
+
+// pushReferrer stores a signature-shaped manifest that names subject and returns its digest.
+func pushReferrer(t *testing.T, reg *Registry, repo, subject, artifactType, blobData string) string {
+	t.Helper()
+	cfg := pushBlob(t, reg, repo, []byte(blobData))
+	body := imageWithSubject(cfg, subject, artifactType)
+	dg := digestOf([]byte(body))
+	w := do(reg, http.MethodPut, "/v2/"+repo+"/manifests/"+dg, []byte(body),
+		map[string]string{"Content-Type": mediaTypeOCIManifest})
+	require.Equal(t, http.StatusCreated, w.Code, "body: %s", w.Body.String())
+	return dg
+}
+
+type referrersBody struct {
+	SchemaVersion int    `json:"schemaVersion"`
+	MediaType     string `json:"mediaType"`
+	Manifests     []struct {
+		MediaType    string `json:"mediaType"`
+		Digest       string `json:"digest"`
+		ArtifactType string `json:"artifactType"`
+	} `json:"manifests"`
+}
+
+func getReferrersBody(t *testing.T, reg *Registry, path string) (*httptest.ResponseRecorder, referrersBody) {
+	t.Helper()
+	w := do(reg, http.MethodGet, path, nil, nil)
+	var out referrersBody
+	if w.Code == http.StatusOK {
+		require.NoError(t, json.Unmarshal(w.Body.Bytes(), &out))
+	}
+	return w, out
+}
+
+func TestGetReferrers_ListsReferrersAsAnOCIIndex(t *testing.T) {
+	reg := newTestRegistry()
+	dgA := pushReferrer(t, reg, referrerRepo, absentSubjectDgt, sigArtifactType, "cfgA")
+	dgB := pushReferrer(t, reg, referrerRepo, absentSubjectDgt, "application/vnd.example.sbom.v1", "cfgB")
+
+	w, out := getReferrersBody(t, reg, "/v2/"+referrerRepo+"/referrers/"+absentSubjectDgt)
+	require.Equal(t, http.StatusOK, w.Code, "body: %s", w.Body.String())
+	assert.Equal(t, "application/vnd.oci.image.index.v1+json", w.Header().Get("Content-Type"))
+	assert.Empty(t, w.Header().Get("Oci-Filters-Applied"))
+	assert.Equal(t, 2, out.SchemaVersion)
+	assert.Equal(t, mediaTypeOCIIndex, out.MediaType)
+	got := map[string]string{}
+	for _, m := range out.Manifests {
+		assert.Equal(t, mediaTypeOCIManifest, m.MediaType)
+		got[m.Digest] = m.ArtifactType
+	}
+	assert.Equal(t, map[string]string{dgA: sigArtifactType, dgB: "application/vnd.example.sbom.v1"}, got)
+}
+
+func TestGetReferrers_NoReferrersIsAnEmptyIndexNotAnError(t *testing.T) {
+	reg := newTestRegistry()
+	seedRepo(t, reg, referrerRepo)
+
+	w := do(reg, http.MethodGet, "/v2/"+referrerRepo+"/referrers/"+absentSubjectDgt, nil, nil)
+	require.Equal(t, http.StatusOK, w.Code, "body: %s", w.Body.String())
+	assert.Equal(t, "application/vnd.oci.image.index.v1+json", w.Header().Get("Content-Type"))
+	assert.JSONEq(t, fmt.Sprintf(`{"schemaVersion":2,"mediaType":%q,"manifests":[]}`, mediaTypeOCIIndex), w.Body.String())
+}
+
+func TestGetReferrers_ArtifactTypeFilter(t *testing.T) {
+	reg := newTestRegistry()
+	dgSig := pushReferrer(t, reg, referrerRepo, absentSubjectDgt, sigArtifactType, "cfgA")
+	pushReferrer(t, reg, referrerRepo, absentSubjectDgt, "application/vnd.example.sbom.v1", "cfgB")
+
+	w, out := getReferrersBody(t, reg, "/v2/"+referrerRepo+"/referrers/"+absentSubjectDgt+"?artifactType="+sigArtifactType)
+	require.Equal(t, http.StatusOK, w.Code)
+	assert.Equal(t, "artifactType", w.Header().Get("Oci-Filters-Applied"))
+	require.Len(t, out.Manifests, 1)
+	assert.Equal(t, dgSig, out.Manifests[0].Digest)
+
+	w, out = getReferrersBody(t, reg, "/v2/"+referrerRepo+"/referrers/"+absentSubjectDgt+"?artifactType=application/none")
+	require.Equal(t, http.StatusOK, w.Code)
+	assert.Equal(t, "artifactType", w.Header().Get("Oci-Filters-Applied"), "the header reports the filter even when nothing matches")
+	assert.Empty(t, out.Manifests)
+	assert.Contains(t, w.Body.String(), `"manifests":[]`)
+}
+
+func TestGetReferrers_MalformedDigest(t *testing.T) {
+	reg := newTestRegistry()
+	seedRepo(t, reg, referrerRepo)
+
+	w := do(reg, http.MethodGet, "/v2/"+referrerRepo+"/referrers/not-a-digest", nil, nil)
+	assert.Equal(t, http.StatusBadRequest, w.Code)
+	assertCode(t, w, "DIGEST_INVALID")
+}
+
+func TestGetReferrers_UnknownRepository(t *testing.T) {
+	reg := newTestRegistry()
+	w := do(reg, http.MethodGet, "/v2/team/ghost/referrers/"+absentSubjectDgt, nil, nil)
+	assert.Equal(t, http.StatusNotFound, w.Code)
+	assertCode(t, w, "NAME_UNKNOWN")
+}
+
+func TestGetReferrers_DoesNotCrossRepositories(t *testing.T) {
+	reg := newTestRegistry()
+	pushReferrer(t, reg, referrerRepo, absentSubjectDgt, sigArtifactType, "cfgA")
+	seedRepo(t, reg, "team/other")
+
+	w, out := getReferrersBody(t, reg, "/v2/team/other/referrers/"+absentSubjectDgt)
+	require.Equal(t, http.StatusOK, w.Code)
+	assert.Empty(t, out.Manifests)
+}
+
+func TestReferrersRoute_OnlyGET(t *testing.T) {
+	reg := newTestRegistry()
+	seedRepo(t, reg, referrerRepo)
+	for _, method := range []string{http.MethodPut, http.MethodDelete, http.MethodPost, http.MethodPatch} {
+		w := do(reg, method, "/v2/"+referrerRepo+"/referrers/"+absentSubjectDgt, []byte("{}"), nil)
+		assert.Equal(t, http.StatusNotFound, w.Code, method)
+		assertCode(t, w, "NAME_UNKNOWN")
+	}
+}
+
+type failingReferrerStore struct{ ecr.MetaStore }
+
+func (failingReferrerStore) ListReferrers(context.Context, string, string, string) ([]ecr.ReferrerDescriptor, error) {
+	return nil, errors.New("no responders available for ecr.referrer.list")
+}
+
+func TestGetReferrers_MetadataFailureIsA5xxNotA404(t *testing.T) {
+	reg := newTestRegistry()
+	seedRepo(t, reg, referrerRepo)
+	reg.Meta = failingReferrerStore{reg.Meta}
+
+	w := do(reg, http.MethodGet, "/v2/"+referrerRepo+"/referrers/"+absentSubjectDgt, nil, nil)
+	assert.Equal(t, http.StatusInternalServerError, w.Code)
+}
+
+// pushSubjectImage stores a plain image (no subject) and returns its digest.
+func pushSubjectImage(t *testing.T, reg *Registry, repo, blobData string) string {
+	t.Helper()
+	cfg := pushBlob(t, reg, repo, []byte(blobData))
+	body := fmt.Sprintf(`{"schemaVersion":2,"mediaType":%q,"config":{"mediaType":%q,"digest":%q,"size":3},"layers":[]}`,
+		mediaTypeOCIManifest, sigConfigType, cfg)
+	dg := digestOf([]byte(body))
+	w := do(reg, http.MethodPut, "/v2/"+repo+"/manifests/"+dg, []byte(body),
+		map[string]string{"Content-Type": mediaTypeOCIManifest})
+	require.Equal(t, http.StatusCreated, w.Code, "body: %s", w.Body.String())
+	return dg
+}
+
+func manifestExists(reg *Registry, repo, dg string) bool {
+	_, err := reg.Meta.GetManifestMeta(context.Background(), testAccount, repo, dg)
+	return err == nil
+}
+
+func referrerDigests(t *testing.T, reg *Registry, repo, subject string) []string {
+	t.Helper()
+	refs, err := reg.Meta.ListReferrers(context.Background(), testAccount, repo, subject)
+	require.NoError(t, err)
+	out := make([]string, 0, len(refs))
+	for _, r := range refs {
+		out = append(out, r.Digest)
+	}
+	return out
+}
+
+func TestDeleteManifest_SubjectDeleteRemovesItsReferrers(t *testing.T) {
+	reg := newTestRegistry()
+	subject := pushSubjectImage(t, reg, referrerRepo, "subject")
+	sig := pushReferrer(t, reg, referrerRepo, subject, sigArtifactType, "sig")
+	sbom := pushReferrer(t, reg, referrerRepo, subject, "application/vnd.example.sbom.v1", "sbom")
+	require.ElementsMatch(t, []string{sig, sbom}, referrerDigests(t, reg, referrerRepo, subject))
+
+	w := do(reg, http.MethodDelete, "/v2/"+referrerRepo+"/manifests/"+subject, nil, nil)
+	require.Equal(t, http.StatusAccepted, w.Code, "body: %s", w.Body.String())
+
+	assert.False(t, manifestExists(reg, referrerRepo, subject))
+	assert.False(t, manifestExists(reg, referrerRepo, sig))
+	assert.False(t, manifestExists(reg, referrerRepo, sbom))
+	assert.Empty(t, referrerDigests(t, reg, referrerRepo, subject), "index keys go with the referrers")
+}
+
+func TestDeleteManifest_NestedReferrersAreRemoved(t *testing.T) {
+	reg := newTestRegistry()
+	subject := pushSubjectImage(t, reg, referrerRepo, "subject")
+	sig := pushReferrer(t, reg, referrerRepo, subject, sigArtifactType, "sig")
+	sigOfSig := pushReferrer(t, reg, referrerRepo, sig, sigArtifactType, "sig-of-sig")
+
+	_, err := reg.DeleteImage(context.Background(), testAccount, referrerRepo, "", subject)
+	require.NoError(t, err)
+
+	for _, dg := range []string{subject, sig, sigOfSig} {
+		assert.False(t, manifestExists(reg, referrerRepo, dg), dg)
+	}
+	assert.Empty(t, referrerDigests(t, reg, referrerRepo, sig))
+}
+
+func TestDeleteManifest_ReferrerDeleteLeavesTheSubject(t *testing.T) {
+	reg := newTestRegistry()
+	subject := pushSubjectImage(t, reg, referrerRepo, "subject")
+	sig := pushReferrer(t, reg, referrerRepo, subject, sigArtifactType, "sig")
+	other := pushReferrer(t, reg, referrerRepo, subject, "application/vnd.example.sbom.v1", "sbom")
+
+	_, err := reg.DeleteImage(context.Background(), testAccount, referrerRepo, "", sig)
+	require.NoError(t, err)
+
+	assert.True(t, manifestExists(reg, referrerRepo, subject))
+	assert.True(t, manifestExists(reg, referrerRepo, other))
+	assert.Equal(t, []string{other}, referrerDigests(t, reg, referrerRepo, subject))
+}
+
+func TestDeleteManifest_TagDeleteDoesNotCascade(t *testing.T) {
+	reg := newTestRegistry()
+	subject := pushSubjectImage(t, reg, referrerRepo, "subject")
+	sig := pushReferrer(t, reg, referrerRepo, subject, sigArtifactType, "sig")
+	require.NoError(t, reg.Meta.PutTag(context.Background(), testAccount, referrerRepo, "v1", subject))
+
+	w := do(reg, http.MethodDelete, "/v2/"+referrerRepo+"/manifests/v1", nil, nil)
+	require.Equal(t, http.StatusAccepted, w.Code)
+
+	assert.True(t, manifestExists(reg, referrerRepo, subject), "a tag delete only removes the pointer")
+	assert.True(t, manifestExists(reg, referrerRepo, sig))
+}
+
+func TestDeleteManifest_CascadeStaysInsideTheRepository(t *testing.T) {
+	reg := newTestRegistry()
+	subject := pushSubjectImage(t, reg, referrerRepo, "subject")
+	pushReferrer(t, reg, referrerRepo, subject, sigArtifactType, "sig")
+	elsewhere := pushReferrer(t, reg, "team/other", subject, sigArtifactType, "sig-other")
+
+	_, err := reg.DeleteImage(context.Background(), testAccount, referrerRepo, "", subject)
+	require.NoError(t, err)
+
+	assert.True(t, manifestExists(reg, "team/other", elsewhere))
+	assert.Equal(t, []string{elsewhere}, referrerDigests(t, reg, "team/other", subject))
+}
+
+type failingDeleteStore struct {
+	ecr.MetaStore
+
+	failDigest string
+}
+
+func (s failingDeleteStore) DeleteManifestMeta(ctx context.Context, account, repo, digest string) error {
+	if digest == s.failDigest {
+		return errors.New("kv unavailable")
+	}
+	return s.MetaStore.DeleteManifestMeta(ctx, account, repo, digest)
+}
+
+func TestDeleteManifest_ReferrerFailureDoesNotFailTheSubjectDelete(t *testing.T) {
+	reg := newTestRegistry()
+	subject := pushSubjectImage(t, reg, referrerRepo, "subject")
+	stuck := pushReferrer(t, reg, referrerRepo, subject, sigArtifactType, "stuck")
+	fine := pushReferrer(t, reg, referrerRepo, subject, "application/vnd.example.sbom.v1", "fine")
+	reg.Meta = failingDeleteStore{MetaStore: reg.Meta, failDigest: stuck}
+
+	got, err := reg.DeleteImage(context.Background(), testAccount, referrerRepo, "", subject)
+	require.NoError(t, err)
+	assert.Equal(t, subject, got)
+	assert.False(t, manifestExists(reg, referrerRepo, subject))
+	assert.False(t, manifestExists(reg, referrerRepo, fine), "one stuck referrer does not stop the others")
+	assert.True(t, manifestExists(reg, referrerRepo, stuck))
+}

spinifex/domains/ecr/registry_address.go

@@ -0,0 +1,34 @@
+package ecr
+
+import "net"
+
+// RegistryAddress returns the registry host and port a gateway node advertises.
+// A concrete AWSGW bind host wins, then a concrete AdvertiseIP; an empty host
+// means the per-account parity name applies. The port comes from the bind.
+func RegistryAddress(awsgwHost, advertiseIP string) (host, port string) {
+	if h, p, err := net.SplitHostPort(awsgwHost); err == nil {
+		port = p
+		if isConcreteRegistryHost(h) {
+			host = h
+		}
+	}
+	if host == "" && isConcreteRegistryHost(advertiseIP) {
+		host = advertiseIP
+	}
+	return host, port
+}
+
+// RegistryURIHost joins a registry host and port the way a repositoryUri
+// carries them: the standard HTTPS port 443 is omitted.
+func RegistryURIHost(host, port string) string {
+	if port != "" && port != "443" {
+		return host + ":" + port
+	}
+	return host
+}
+
+// isConcreteRegistryHost reports whether host is a non-empty, non-wildcard
+// literal, so a registry URI never carries 0.0.0.0 or ::.
+func isConcreteRegistryHost(host string) bool {
+	return host != "" && host != "0.0.0.0" && host != "::"
+}

spinifex/domains/ecr/registry_address_test.go

@@ -0,0 +1,48 @@
+package ecr
+
+import "testing"
+
+func TestRegistryAddress(t *testing.T) {
+	tests := []struct {
+		name      string
+		bind      string
+		advertise string
+		wantHost  string
+		wantPort  string
+	}{
+		{"wildcard bind falls back to advertise", "0.0.0.0:9999", "192.168.0.12", "192.168.0.12", "9999"},
+		{"concrete bind equal to advertise", "10.0.0.5:9999", "10.0.0.5", "10.0.0.5", "9999"},
+		{"concrete bind wins over a different advertise", "10.0.0.5:9999", "192.168.0.12", "10.0.0.5", "9999"},
+		{"ipv6 wildcard falls back to advertise", "[::]:9999", "192.168.0.12", "192.168.0.12", "9999"},
+		{"wildcard bind and empty advertise leaves the parity name", "0.0.0.0:9999", "", "", "9999"},
+		{"wildcard bind and wildcard advertise leaves the parity name", "0.0.0.0:9999", "0.0.0.0", "", "9999"},
+		{"unparseable bind falls back to advertise with no port", "not-a-hostport", "192.168.0.12", "192.168.0.12", ""},
+		{"empty bind and empty advertise", "", "", "", ""},
+		{"port is taken from the bind", "10.0.0.5:443", "", "10.0.0.5", "443"},
+		{"loopback bind counts as concrete", "127.0.0.1:9999", "192.168.0.12", "127.0.0.1", "9999"},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			host, port := RegistryAddress(tt.bind, tt.advertise)
+			if host != tt.wantHost || port != tt.wantPort {
+				t.Errorf("RegistryAddress(%q, %q) = (%q, %q), want (%q, %q)",
+					tt.bind, tt.advertise, host, port, tt.wantHost, tt.wantPort)
+			}
+		})
+	}
+}
+
+func TestRegistryURIHost(t *testing.T) {
+	tests := []struct {
+		host, port, want string
+	}{
+		{"10.0.0.1", "9999", "10.0.0.1:9999"},
+		{"10.0.0.1", "443", "10.0.0.1"},
+		{"10.0.0.1", "", "10.0.0.1"},
+	}
+	for _, tt := range tests {
+		if got := RegistryURIHost(tt.host, tt.port); got != tt.want {
+			t.Errorf("RegistryURIHost(%q, %q) = %q, want %q", tt.host, tt.port, got, tt.want)
+		}
+	}
+}

spinifex/domains/ecr/service.go

@@ -28,6 +28,7 @@ const (
 	SubjectManifestDescribe = "ecr.manifest.describe"
 	SubjectManifestList     = "ecr.manifest.list"
 	SubjectManifestDelete   = "ecr.manifest.delete"
+	SubjectReferrerList     = "ecr.referrer.list"
 
 	SubjectUploadCreate = "ecr.upload.create"
 	SubjectUploadGet    = "ecr.upload.get"
@@ -223,6 +224,17 @@ type ManifestListResponse struct {
 	Digests []string `json:"digests"`
 }
 
+// ReferrerListRequest is the payload for SubjectReferrerList, naming the subject whose referrers to list.
+type ReferrerListRequest struct {
+	Repo    string `json:"repo"`
+	Subject string `json:"subject"`
+}
+
+// ReferrerListResponse is the reply to SubjectReferrerList, holding one descriptor per referrer.
+type ReferrerListResponse struct {
+	Referrers []ReferrerDescriptor `json:"referrers"`
+}
+
 // ManifestDeleteRequest is the payload for SubjectManifestDelete, naming the manifest metadata record to remove.
 type ManifestDeleteRequest struct {
 	Repo   string `json:"repo"`
@@ -313,6 +325,7 @@ type MetaService interface {
 	ManifestDescribe(ctx context.Context, req *ManifestDescribeRequest, accountID string) (*ManifestDescribeResponse, error)
 	ManifestList(ctx context.Context, req *ManifestListRequest, accountID string) (*ManifestListResponse, error)
 	ManifestDelete(ctx context.Context, req *ManifestDeleteRequest, accountID string) (*ManifestDeleteResponse, error)
+	ReferrerList(ctx context.Context, req *ReferrerListRequest, accountID string) (*ReferrerListResponse, error)
 
 	UploadCreate(ctx context.Context, req *UploadCreateRequest, accountID string) (*UploadCreateResponse, error)
 	UploadGet(ctx context.Context, req *UploadGetRequest, accountID string) (*UploadGetResponse, error)

spinifex/domains/ecr/service_impl.go

@@ -185,6 +185,14 @@ func (s *MetaServiceImpl) ManifestDelete(ctx context.Context, req *ManifestDelet
 	return &ManifestDeleteResponse{Found: true}, nil
 }
 
+func (s *MetaServiceImpl) ReferrerList(ctx context.Context, req *ReferrerListRequest, accountID string) (*ReferrerListResponse, error) {
+	referrers, err := s.store.ListReferrers(ctx, accountID, req.Repo, req.Subject)
+	if err != nil {
+		return nil, err
+	}
+	return &ReferrerListResponse{Referrers: referrers}, nil
+}
+
 func (s *MetaServiceImpl) ManifestDescribe(ctx context.Context, req *ManifestDescribeRequest, accountID string) (*ManifestDescribeResponse, error) {
 	meta, err := s.store.GetManifestMeta(ctx, accountID, req.Repo, req.Digest)
 	if errors.Is(err, ErrNotFound) {

spinifex/domains/ecr/storage.go

@@ -89,6 +89,7 @@ func KVAccountBucket(accountID string) string {
 //	repos/{name}/meta
 //	repos/{name}/tags/{tag}
 //	repos/{name}/manifests/{digest}
+//	repos/{name}/referrers/{subject}/{referrer}
 //	uploads/{uuid}
 const (
 	kvRepoMetaKeyFmt     = "repos/%s/meta"
@@ -98,6 +99,8 @@ const (
 	kvTagKeyFmt          = "repos/%s/tags/%s"
 	kvManifestKeyFmt     = "repos/%s/manifests/%s"
 	kvManifestsPrefixFmt = "repos/%s/manifests/"
+	kvReferrersPrefixFmt = "repos/%s/referrers/"
+	kvReferrerKeyFmt     = "repos/%s/referrers/%s/%s"
 	kvReposPrefix        = "repos/"
 	kvUploadKeyFmt       = "uploads/%s"
 )
@@ -128,6 +131,21 @@ func KVManifestKey(repo, digest string) string {
 // manifest metadata records.
 func KVManifestsPrefix(repo string) string { return fmt.Sprintf(kvManifestsPrefixFmt, repo) }
 
+// KVReferrersPrefix returns the KV key prefix enumerating every referrer entry
+// of a repository.
+func KVReferrersPrefix(repo string) string { return fmt.Sprintf(kvReferrersPrefixFmt, repo) }
+
+// KVSubjectReferrersPrefix returns the KV key prefix enumerating the referrers
+// of one subject manifest.
+func KVSubjectReferrersPrefix(repo, subjectDigest string) string {
+	return KVReferrersPrefix(repo) + DigestToken(subjectDigest) + "/"
+}
+
+// KVReferrerKey returns the KV key of one referrer entry under its subject.
+func KVReferrerKey(repo, subjectDigest, referrerDigest string) string {
+	return fmt.Sprintf(kvReferrerKeyFmt, repo, DigestToken(subjectDigest), DigestToken(referrerDigest))
+}
+
 // KVReposPrefix is the prefix under which every repo meta key lives. Used by
 // the catalog listing to enumerate account repositories.
 const KVReposPrefix = kvReposPrefix

spinifex/domains/eks/cluster/k3s_server_vm.go

@@ -566,6 +566,7 @@ func buildK3sUserData(in K3sServerInput) string {
 		"EKS_KONNECTIVITY_HOST="+konnHost,
 		"EKS_KONNECTIVITY_SANS="+strings.Join(konnSANs, ","),
 		"EKS_KONNECTIVITY_SERVER_COUNT="+strconv.Itoa(konnCount),
+		"EKS_WEBHOOK_READY_ADDR=0.0.0.0:"+strconv.FormatInt(webhookReadyPort, 10),
 	)
 	// A replacement CP prunes the terminated member it supersedes from etcd once
 	// it has joined (k3s-first-boot deletes the dead Node → embedded-etcd drops

spinifex/domains/eks/cluster/k3s_server_vm_test.go

@@ -4,6 +4,8 @@ import (
 	"context"
 	"errors"
 	"fmt"
+	"os"
+	"slices"
 	"strings"
 	"sync"
 	"testing"
@@ -489,6 +491,29 @@ func TestBuildK3sUserData_KonnectivityEnv(t *testing.T) {
 	assert.Contains(t, ud, "EKS_KONNECTIVITY_SANS=10.32.100.4,203.0.113.9,"+in.NLBDNS)
 }
 
+func TestBuildK3sUserData_WebhookReadinessListener(t *testing.T) {
+	t.Parallel()
+	ud := buildK3sUserData(validK3sInput())
+	assert.Contains(t, ud, "EKS_WEBHOOK_READY_ADDR=0.0.0.0:8444",
+		"the webhook must serve /readyz beyond loopback for the NLB health check")
+}
+
+// The env line alone does nothing: the webhook service only passes through
+// the variables its init script exports.
+func TestWebhookInitExportsReadyAddr(t *testing.T) {
+	t.Parallel()
+	b, err := os.ReadFile("../../../../scripts/images/eks-node/eks-token-webhook.initd")
+	require.NoError(t, err)
+	var exported bool
+	for line := range strings.SplitSeq(string(b), "\n") {
+		f := strings.Fields(line)
+		if len(f) > 1 && f[0] == "export" && slices.Contains(f[1:], "EKS_WEBHOOK_READY_ADDR") {
+			exported = true
+		}
+	}
+	assert.True(t, exported, "eks-token-webhook.initd must export EKS_WEBHOOK_READY_ADDR or the readiness listener never starts")
+}
+
 func TestBuildK3sUserData_KonnServerCountDefaultsToOne(t *testing.T) {
 	t.Parallel()
 	in := validK3sInput()

spinifex/domains/eks/cluster/lifecycle_invariants_test.go

@@ -7,23 +7,21 @@ import (
 
 	"github.com/aws/aws-sdk-go/aws"
 	"github.com/aws/aws-sdk-go/service/elbv2"
+	awserrors "github.com/mulgadc/spinifex/spinifex/foundation/aws/errors"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 )
 
-// TestRLC1_EKSDeleteClusterIdempotentOnAbsent enforces the Common Resource
-// Lifecycle Contract rule #1 (idempotent delete), ADR-0003:S2 (behaviour; mechanism from historic PROP-LIFECYCLE-006 §1): DeleteCluster of
-// a cluster already swept from KV returns success, not ResourceNotFound, so a
-// tofu destroy retry / double-targeted graph node converges instead of failing
-// the run. The live-reference teardown is unaffected — only true absence is
-// idempotent.
-func TestRLC1_EKSDeleteClusterIdempotentOnAbsent(t *testing.T) {
+// TestRLC1_EKSDeleteClusterNotFoundOnAbsent pins AWS DeleteCluster semantics
+// (ADR-0003:S3): a cluster already swept from KV returns ResourceNotFound and
+// triggers no teardown. Internal callers treat that code as already deleted.
+func TestRLC1_EKSDeleteClusterNotFoundOnAbsent(t *testing.T) {
 	t.Parallel()
 	f := newEKSServiceFixture(t)
 
 	out, err := f.svc.DeleteCluster(context.Background(), deleteInput("absent"), testAccountID)
-	require.NoErrorf(t, err, "ADR-0003:S2 behaviour; historic PROP-LIFECYCLE-006 §1 mechanism: DeleteCluster on an absent cluster must return success, not ResourceNotFound (RLC rule #1)")
-	require.NotNil(t, out, "ADR-0003:S2 behaviour; historic PROP-LIFECYCLE-006 §1 mechanism: DeleteCluster must return a non-nil output on absent")
+	require.EqualErrorf(t, err, awserrors.ErrorEKSResourceNotFound, "ADR-0003:S3: DeleteCluster on an absent cluster must return ResourceNotFoundException, as AWS does")
+	assert.Nil(t, out)
 	assert.Empty(t, f.inst.terminateCalls, "an absent cluster must trigger no billable teardown")
 }
 

spinifex/domains/eks/cluster/nlb.go

@@ -7,6 +7,7 @@ import (
 	"errors"
 	"fmt"
 	"log/slog"
+	"strconv"
 	"strings"
 
 	"github.com/aws/aws-sdk-go/aws"
@@ -62,6 +63,31 @@ const lbcClusterOwnershipTagKey = "elbv2.k8s.aws/cluster"
 // agent (HA apiserver->pod/kubelet/webhook egress, EKS-parity).
 const konnectivityAgentPort int64 = 8132
 
+// webhookReadyPort is the plain-HTTP port on which the control-plane VM's token
+// webhook serves only /readyz, so the NLB can tell whether the VM can authenticate.
+const webhookReadyPort int64 = 8444
+
+// Health check of the apiserver target group: the control-plane token webhook's
+// /readyz, which also requires the local apiserver, so a CP that cannot
+// authenticate leaves rotation. Timeout stays below the interval.
+const (
+	cpHealthCheckPath      = "/readyz"
+	cpHealthCheckInterval  = 5
+	cpHealthCheckTimeout   = 3
+	cpHealthCheckThreshold = 2
+)
+
+// cpReadinessHealthCheck applies the readiness health check to a TG create.
+func cpReadinessHealthCheck(in *elbv2.CreateTargetGroupInput) {
+	in.HealthCheckProtocol = aws.String(elbv2.ProtocolEnumHttp)
+	in.HealthCheckPort = aws.String(strconv.FormatInt(webhookReadyPort, 10))
+	in.HealthCheckPath = aws.String(cpHealthCheckPath)
+	in.HealthCheckIntervalSeconds = aws.Int64(cpHealthCheckInterval)
+	in.HealthCheckTimeoutSeconds = aws.Int64(cpHealthCheckTimeout)
+	in.HealthyThresholdCount = aws.Int64(cpHealthCheckThreshold)
+	in.UnhealthyThresholdCount = aws.Int64(cpHealthCheckThreshold)
+}
+
 // ClusterNLB is the NLB resource tuple produced by EnsureClusterNLB; ARNs are persisted to ClusterMeta.
 type ClusterNLB struct {
 	LoadBalancerArn string
@@ -162,7 +188,7 @@ func EnsureClusterNLB(ctx context.Context, nlbp nlbProvisioner, accountID, clust
 		return nil, err
 	}
 	var err error
-	if out.TargetGroupArn, err = ensureClusterTG(ctx, nlbp, accountID, clusterName, tgName, k3sAPIServerPort); err != nil {
+	if out.TargetGroupArn, err = ensureClusterTG(ctx, nlbp, accountID, clusterName, tgName, k3sAPIServerPort, cpReadinessHealthCheck); err != nil {
 		return nil, err
 	}
 	if out.ListenerArn, err = ensureClusterListener(ctx, nlbp, accountID, lbName, out.LoadBalancerArn, out.TargetGroupArn, clusterNLBListenPort); err != nil {
@@ -176,7 +202,7 @@ func EnsureClusterNLB(ctx context.Context, nlbp nlbProvisioner, accountID, clust
 	}
 	// Konnectivity TG + listener :8132 → the 3 CP konnectivity-servers. Worker
 	// konnectivity-agents dial this VIP; server-count fan-out reaches all 3 servers.
-	if out.KonnTargetGroupArn, err = ensureClusterTG(ctx, nlbp, accountID, clusterName, konnTGName, konnectivityAgentPort); err != nil {
+	if out.KonnTargetGroupArn, err = ensureClusterTG(ctx, nlbp, accountID, clusterName, konnTGName, konnectivityAgentPort, nil); err != nil {
 		return nil, err
 	}
 	if _, err = ensureClusterListener(ctx, nlbp, accountID, lbName, out.LoadBalancerArn, out.KonnTargetGroupArn, konnectivityAgentPort); err != nil {
@@ -406,8 +432,9 @@ func ensureClusterLB(ctx context.Context, nlbp nlbProvisioner, accountID, cluste
 }
 
 // ensureClusterTG creates (or reuses) a TCP target group of the given name
-// forwarding to port, returning its ARN. Idempotent on name.
-func ensureClusterTG(ctx context.Context, nlbp nlbProvisioner, accountID, clusterName, tgName string, port int64) (string, error) {
+// forwarding to port, returning its ARN. Idempotent on name. configure, if set,
+// adjusts the create input; an existing group is returned as is, unmodified.
+func ensureClusterTG(ctx context.Context, nlbp nlbProvisioner, accountID, clusterName, tgName string, port int64, configure func(*elbv2.CreateTargetGroupInput)) (string, error) {
 	if tg, err := lookupTGByName(ctx, nlbp, accountID, tgName); err != nil {
 		return "", err
 	} else if tg != nil {
@@ -418,7 +445,7 @@ func ensureClusterTG(ctx context.Context, nlbp nlbProvisioner, accountID, cluste
 		return arn, nil
 	}
 
-	created, err := nlbp.CreateTargetGroup(ctx, &elbv2.CreateTargetGroupInput{
+	in := &elbv2.CreateTargetGroupInput{
 		Name:       aws.String(tgName),
 		Protocol:   aws.String(elbv2.ProtocolEnumTcp),
 		Port:       aws.Int64(port),
@@ -427,7 +454,11 @@ func ensureClusterTG(ctx context.Context, nlbp nlbProvisioner, accountID, cluste
 			{Key: aws.String(tags.ManagedByKey), Value: aws.String(tags.ManagedByEKS)},
 			{Key: aws.String(clusterEKSClusterTagKey), Value: aws.String(clusterName)},
 		},
-	}, accountID)
+	}
+	if configure != nil {
+		configure(in)
+	}
+	created, err := nlbp.CreateTargetGroup(ctx, in, accountID)
 	if err != nil {
 		return "", fmt.Errorf("create TG %s: %w", tgName, err)
 	}

spinifex/domains/eks/cluster/nlb_test.go

@@ -201,6 +201,14 @@ func (f *fakeNLBProvisioner) CreateTargetGroup(_ context.Context, input *elbv2.C
 				Protocol:        input.Protocol,
 				Port:            input.Port,
 				TargetType:      input.TargetType,
+
+				HealthCheckProtocol:        input.HealthCheckProtocol,
+				HealthCheckPort:            input.HealthCheckPort,
+				HealthCheckPath:            input.HealthCheckPath,
+				HealthCheckIntervalSeconds: input.HealthCheckIntervalSeconds,
+				HealthCheckTimeoutSeconds:  input.HealthCheckTimeoutSeconds,
+				HealthyThresholdCount:      input.HealthyThresholdCount,
+				UnhealthyThresholdCount:    input.UnhealthyThresholdCount,
 			}},
 		}
 	}
@@ -434,6 +442,55 @@ func TestEnsureClusterNLB_InternetFacingUsesPublicAddress(t *testing.T) {
 	assert.Equal(t, elbv2.LoadBalancerSchemeEnumInternetFacing, aws.StringValue(nlbp.createLBSyncCalls[0].Scheme))
 }
 
+func TestEnsureClusterNLB_APIServerTGHealthChecksWebhookReadiness(t *testing.T) {
+	t.Parallel()
+	nlbp := newFakeNLBProvisioner()
+	_, err := EnsureClusterNLB(context.Background(), nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, false, nil, nil)
+	require.NoError(t, err)
+
+	api := nlbp.tgByName["eks-alpha-cp"]
+	require.NotNil(t, api)
+	assert.Equal(t, elbv2.ProtocolEnumTcp, aws.StringValue(api.Protocol), "traffic stays TCP")
+	assert.Equal(t, elbv2.ProtocolEnumHttp, aws.StringValue(api.HealthCheckProtocol))
+	assert.Equal(t, "8444", aws.StringValue(api.HealthCheckPort))
+	assert.Equal(t, "/readyz", aws.StringValue(api.HealthCheckPath))
+	assert.Equal(t, int64(5), aws.Int64Value(api.HealthCheckIntervalSeconds))
+	assert.Equal(t, int64(2), aws.Int64Value(api.HealthyThresholdCount))
+	assert.Equal(t, int64(2), aws.Int64Value(api.UnhealthyThresholdCount))
+	assert.Less(t, aws.Int64Value(api.HealthCheckTimeoutSeconds), aws.Int64Value(api.HealthCheckIntervalSeconds),
+		"the probe must time out before the next one starts")
+
+	konn := nlbp.tgByName["eks-alpha-konn"]
+	require.NotNil(t, konn)
+	assert.Nil(t, konn.HealthCheckProtocol, "konnectivity TG keeps the default health check")
+	assert.Nil(t, konn.HealthCheckPort)
+	assert.Nil(t, konn.HealthCheckPath)
+}
+
+func TestEnsureClusterNLB_ExistingAPIServerTGIsLeftAsIs(t *testing.T) {
+	t.Parallel()
+	nlbp := newFakeNLBProvisioner()
+	legacy := &elbv2.TargetGroup{
+		TargetGroupArn:  aws.String("arn:aws:elasticloadbalancing:us-east-1:111122223333:targetgroup/eks-alpha-cp/legacy"),
+		TargetGroupName: aws.String("eks-alpha-cp"),
+		Protocol:        aws.String(elbv2.ProtocolEnumTcp),
+		Port:            aws.Int64(k3sAPIServerPort),
+	}
+	nlbp.tgByName["eks-alpha-cp"] = legacy
+
+	out, err := EnsureClusterNLB(context.Background(), nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, false, nil, nil)
+	require.NoError(t, err)
+
+	assert.Equal(t, aws.StringValue(legacy.TargetGroupArn), out.TargetGroupArn)
+	for _, in := range nlbp.createTGCalls {
+		assert.NotEqual(t, "eks-alpha-cp", aws.StringValue(in.Name), "an existing TG is not re-created")
+	}
+	stored := nlbp.tgByName["eks-alpha-cp"]
+	assert.Nil(t, stored.HealthCheckProtocol, "an existing TG keeps its original health check")
+	assert.Nil(t, stored.HealthCheckPort)
+	assert.Nil(t, stored.HealthCheckPath)
+}
+
 func TestEnsureClusterNLB_IdempotentReusesExisting(t *testing.T) {
 	t.Parallel()
 	nlbp := newFakeNLBProvisioner()

spinifex/domains/eks/cluster/nodegroup.go

@@ -655,6 +655,11 @@ func (s *EKSServiceImpl) launchOneWorker(ctx context.Context, rec *NodegroupReco
 			"gatewayBaseURL", s.deps.GatewayBaseURL, "registryHost", registryHost)
 	}
 
+	var registryHosts []string
+	if s.deps.RegistryHosts != nil {
+		registryHosts = s.deps.RegistryHosts()
+	}
+
 	userData := buildAgentUserData(agentUserDataInput{
 		ClusterName:   rec.ClusterName,
 		NodegroupName: rec.Name,
@@ -667,6 +672,7 @@ func (s *EKSServiceImpl) launchOneWorker(ctx context.Context, rec *NodegroupReco
 		AccountID:     accountID,
 		RegistryHost:  registryHost,
 		GatewayIP:     gatewayIP,
+		RegistryHosts: registryHosts,
 		GPUEnabled:    rec.GPUEnabled,
 		GPUVendor:     rec.GPUVendor,
 	})

spinifex/domains/eks/cluster/nodegroup_userdata.go

@@ -29,24 +29,65 @@ type agentUserDataInput struct {
 	RegistryHost  string
 	GatewayIP     string
 
+	// RegistryHosts lists every gateway's registry host[:port] as it appears in a
+	// repositoryUri, local node first. Each is trusted and pulled from directly, with
+	// the others as fallbacks. Empty keeps the single GatewayIP endpoint.
+	RegistryHosts []string
+
 	// GPUEnabled/GPUVendor gate the GPU node label + default scheduling taint;
 	// set from the owning NodegroupRecord (see gpuFieldsForInstanceTypes).
 	GPUEnabled bool
 	GPUVendor  string
 }
 
-// registryMirrorHosts returns the distinct registry hosts a worker must accept on
-// a pod image ref: the port-less parity name, the port-ful parity name (the value
-// DescribeRepositories returns), and the gateway IP:port when the gateway
-// advertises an IP. containerd matches mirror keys and the credential provider
-// matches matchImages verbatim against the image host, so every form must be
-// listed; all resolve to the single dialed gateway endpoint.
-func registryMirrorHosts(registryHost, endpointHost string) []string {
-	hosts := []string{registryHost, registryHost + ":9999"}
-	if endpointHost != registryHost {
-		hosts = append(hosts, endpointHost+":9999")
+// registryGateways returns the gateway host[:port] endpoints a worker may dial, in
+// preference order: every advertised gateway, then GatewayIP:9999 when it is not
+// already listed. With neither, the parity hostname at :9999 is the only endpoint.
+func registryGateways(in agentUserDataInput) []string {
+	var gws []string
+	seen := map[string]bool{}
+	add := func(h string) {
+		if h != "" && !seen[h] {
+			seen[h] = true
+			gws = append(gws, h)
+		}
 	}
-	return hosts
+	for _, h := range in.RegistryHosts {
+		add(h)
+	}
+	if in.GatewayIP != "" {
+		add(in.GatewayIP + ":9999")
+	}
+	if len(gws) == 0 {
+		add(in.RegistryHost + ":9999")
+	}
+	return gws
+}
+
+// registryMirrors maps every host a pod image ref may carry to the endpoints
+// containerd tries for it, and matches the credential provider's matchImages
+// verbatim against the same keys. The parity names try the gateways in order; a
+// gateway key tries itself first so a down gateway still falls back to the rest.
+func registryMirrors(registryHost string, gateways []string) (keys []string, endpoints map[string][]string) {
+	endpoints = map[string][]string{}
+	for _, k := range []string{registryHost, registryHost + ":9999"} {
+		keys = append(keys, k)
+		endpoints[k] = gateways
+	}
+	for _, g := range gateways {
+		if _, ok := endpoints[g]; ok {
+			continue
+		}
+		keys = append(keys, g)
+		ordered := []string{g}
+		for _, o := range gateways {
+			if o != g {
+				ordered = append(ordered, o)
+			}
+		}
+		endpoints[g] = ordered
+	}
+	return keys, endpoints
 }
 
 // buildAgentUserData renders the cloud-config YAML for a nodegroup worker VM.
@@ -81,32 +122,27 @@ func buildAgentUserData(in agentUserDataInput) string {
 	}
 	agentBody := strings.Join(agentLines, "\n")
 
-	// The mirror endpoint is dialed by IP when the gateway IP is known: the gateway
-	// cert carries an IP SAN, so TLS validates without DNS, and there is no internal
-	// resolver yet (cloud-init manage_etc_hosts rewrites /etc/hosts each boot). A pod
-	// image ref may carry any host the API hands out — the port-less parity name, the
-	// port-ful parity name (the repositoryUri value), or the gateway IP:port — so each
-	// is keyed as a mirror redirecting to the one endpoint host:port actually dialed,
-	// whose CA the configs tls block trusts.
-	endpointHost := in.RegistryHost
-	if in.GatewayIP != "" {
-		endpointHost = in.GatewayIP
-	}
-	endpoint := "https://" + endpointHost + ":9999"
+	// Gateways are dialed by IP: the gateway cert carries IP SANs so TLS validates
+	// without DNS, and there is no internal resolver yet (cloud-init
+	// manage_etc_hosts rewrites /etc/hosts each boot). Every endpoint is trusted via
+	// the configs tls block below.
+	gateways := registryGateways(in)
+	mirrorKeys, mirrorEndpoints := registryMirrors(in.RegistryHost, gateways)
 	registryLines := []string{"mirrors:"}
-	for _, host := range registryMirrorHosts(in.RegistryHost, endpointHost) {
+	for _, key := range mirrorKeys {
+		registryLines = append(registryLines, "  \""+key+"\":", "    endpoint:")
+		for _, ep := range mirrorEndpoints[key] {
+			registryLines = append(registryLines, "      - \"https://"+ep+"\"")
+		}
+	}
+	registryLines = append(registryLines, "configs:")
+	for _, g := range gateways {
 		registryLines = append(registryLines,
-			"  \""+host+"\":",
-			"    endpoint:",
-			"      - \""+endpoint+"\"",
+			"  \""+g+"\":",
+			"    tls:",
+			"      ca_file: "+k3sGatewayCAPath,
 		)
 	}
-	registryLines = append(registryLines,
-		"configs:",
-		"  \""+endpointHost+":9999\":",
-		"    tls:",
-		"      ca_file: "+k3sGatewayCAPath,
-	)
 	registriesYAML := strings.Join(registryLines, "\n")
 
 	credProviderDropin := strings.Join([]string{
@@ -122,7 +158,7 @@ func buildAgentUserData(in agentUserDataInput) string {
 		"  - name: ecr-credential-provider",
 		"    matchImages:",
 	}
-	for _, host := range registryMirrorHosts(in.RegistryHost, endpointHost) {
+	for _, host := range mirrorKeys {
 		credLines = append(credLines, "      - \""+host+"\"")
 	}
 	credLines = append(credLines,

spinifex/domains/eks/cluster/nodegroup_userdata_registry_test.go

@@ -0,0 +1,157 @@
+package cluster
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"go.yaml.in/yaml/v3"
+)
+
+type renderedRegistries struct {
+	Mirrors map[string]struct {
+		Endpoint []string `yaml:"endpoint"`
+	} `yaml:"mirrors"`
+	Configs map[string]struct {
+		TLS struct {
+			CAFile string `yaml:"ca_file"`
+		} `yaml:"tls"`
+	} `yaml:"configs"`
+}
+
+type renderedCredProvider struct {
+	Providers []struct {
+		Name        string   `yaml:"name"`
+		MatchImages []string `yaml:"matchImages"`
+	} `yaml:"providers"`
+}
+
+// renderWorkerFiles builds the userdata and returns the parsed registries.yaml and
+// credential-provider config, so assertions see the structure containerd and the
+// kubelet will, not substrings.
+func renderWorkerFiles(t *testing.T, in agentUserDataInput) (renderedRegistries, renderedCredProvider) {
+	t.Helper()
+	var cloud struct {
+		WriteFiles []struct {
+			Path    string `yaml:"path"`
+			Content string `yaml:"content"`
+		} `yaml:"write_files"`
+	}
+	require.NoError(t, yaml.Unmarshal([]byte(buildAgentUserData(in)), &cloud))
+
+	var reg renderedRegistries
+	var cred renderedCredProvider
+	var sawReg, sawCred bool
+	for _, f := range cloud.WriteFiles {
+		switch f.Path {
+		case "/etc/rancher/k3s/registries.yaml":
+			require.NoError(t, yaml.Unmarshal([]byte(f.Content), &reg))
+			sawReg = true
+		case "/etc/spinifex-eks/credential-provider-config.yaml":
+			require.NoError(t, yaml.Unmarshal([]byte(f.Content), &cred))
+			sawCred = true
+		}
+	}
+	require.True(t, sawReg && sawCred, "registries.yaml and credential-provider config must be rendered")
+	return reg, cred
+}
+
+func registryTestInput() agentUserDataInput {
+	in := baseTestInput()
+	in.GatewayURL = "https://192.168.1.228:9999"
+	in.GatewayCACert = "-----BEGIN CERTIFICATE-----\nMIIB\n-----END CERTIFICATE-----"
+	return in
+}
+
+const parity = "123456789012.dkr.ecr.ap-southeast-2.spinifex.internal"
+
+func TestBuildAgentUserData_RegistryHostsFallbackOrder(t *testing.T) {
+	t.Parallel()
+	in := registryTestInput()
+	in.GatewayIP = "192.168.1.228"
+	in.RegistryHosts = []string{"192.168.1.228:9999", "192.168.1.229:9999", "192.168.1.230:9999", "192.168.1.231:9999"}
+
+	reg, cred := renderWorkerFiles(t, in)
+
+	all := []string{
+		"https://192.168.1.228:9999", "https://192.168.1.229:9999",
+		"https://192.168.1.230:9999", "https://192.168.1.231:9999",
+	}
+	// The issue's case: a repositoryUri on .231 is keyed, dials .231 first, then the rest.
+	assert.Equal(t, []string{
+		"https://192.168.1.231:9999", "https://192.168.1.228:9999",
+		"https://192.168.1.229:9999", "https://192.168.1.230:9999",
+	}, reg.Mirrors["192.168.1.231:9999"].Endpoint)
+	assert.Equal(t, all, reg.Mirrors[parity].Endpoint)
+	assert.Equal(t, all, reg.Mirrors[parity+":9999"].Endpoint)
+
+	for _, gw := range []string{"192.168.1.228:9999", "192.168.1.229:9999", "192.168.1.230:9999", "192.168.1.231:9999"} {
+		assert.Equal(t, "https://"+gw, reg.Mirrors[gw].Endpoint[0], "mirror %s dials itself first", gw)
+		assert.Equal(t, "/etc/spinifex-eks/gateway-ca.pem", reg.Configs[gw].TLS.CAFile, "CA for %s", gw)
+	}
+	assert.Len(t, reg.Mirrors, 6)
+	assert.Len(t, reg.Configs, 4)
+
+	require.Len(t, cred.Providers, 1)
+	assert.ElementsMatch(t, []string{
+		parity, parity + ":9999",
+		"192.168.1.228:9999", "192.168.1.229:9999", "192.168.1.230:9999", "192.168.1.231:9999",
+	}, cred.Providers[0].MatchImages)
+}
+
+func TestBuildAgentUserData_EmptyRegistryHostsKeepsSingleGatewayEndpoint(t *testing.T) {
+	t.Parallel()
+	in := registryTestInput()
+	in.GatewayIP = "10.15.8.1"
+
+	reg, cred := renderWorkerFiles(t, in)
+
+	single := []string{"https://10.15.8.1:9999"}
+	assert.Equal(t, single, reg.Mirrors[parity].Endpoint)
+	assert.Equal(t, single, reg.Mirrors[parity+":9999"].Endpoint)
+	assert.Equal(t, single, reg.Mirrors["10.15.8.1:9999"].Endpoint)
+	assert.Len(t, reg.Mirrors, 3)
+	require.Len(t, reg.Configs, 1)
+	assert.Equal(t, "/etc/spinifex-eks/gateway-ca.pem", reg.Configs["10.15.8.1:9999"].TLS.CAFile)
+	assert.Equal(t, []string{parity, parity + ":9999", "10.15.8.1:9999"}, cred.Providers[0].MatchImages)
+}
+
+func TestBuildAgentUserData_EmptyRegistryHostsAndNoGatewayIPUsesHostname(t *testing.T) {
+	t.Parallel()
+	reg, cred := renderWorkerFiles(t, registryTestInput())
+
+	host := []string{"https://" + parity + ":9999"}
+	assert.Equal(t, host, reg.Mirrors[parity].Endpoint)
+	assert.Equal(t, host, reg.Mirrors[parity+":9999"].Endpoint)
+	assert.Len(t, reg.Mirrors, 2)
+	require.Len(t, reg.Configs, 1)
+	assert.Contains(t, reg.Configs, parity+":9999")
+	assert.Equal(t, []string{parity, parity + ":9999"}, cred.Providers[0].MatchImages)
+}
+
+func TestBuildAgentUserData_GatewayIPNotInRegistryHostsIsKept(t *testing.T) {
+	t.Parallel()
+	in := registryTestInput()
+	in.GatewayIP = "10.15.8.1"
+	in.RegistryHosts = []string{"192.168.1.229:9999", "192.168.1.230:9999"}
+
+	reg, cred := renderWorkerFiles(t, in)
+
+	assert.Equal(t, []string{
+		"https://192.168.1.229:9999", "https://192.168.1.230:9999", "https://10.15.8.1:9999",
+	}, reg.Mirrors[parity].Endpoint)
+	assert.Equal(t, "https://10.15.8.1:9999", reg.Mirrors["10.15.8.1:9999"].Endpoint[0])
+	assert.Contains(t, reg.Configs, "10.15.8.1:9999")
+	assert.Contains(t, cred.Providers[0].MatchImages, "10.15.8.1:9999")
+}
+
+func TestBuildAgentUserData_RegistryHostWithoutPortIsDialedOn443(t *testing.T) {
+	t.Parallel()
+	in := registryTestInput()
+	in.RegistryHosts = []string{"10.0.0.1", "10.0.0.2:9999"}
+
+	reg, _ := renderWorkerFiles(t, in)
+
+	assert.Equal(t, []string{"https://10.0.0.1", "https://10.0.0.2:9999"}, reg.Mirrors["10.0.0.1"].Endpoint)
+	assert.Equal(t, "/etc/spinifex-eks/gateway-ca.pem", reg.Configs["10.0.0.1"].TLS.CAFile)
+}

spinifex/domains/eks/cluster/security_groups.go

@@ -397,8 +397,8 @@ func EnsureNodegroupSGRules(ctx context.Context, sgp sgProvisioner, accountID, c
 }
 
 // EnsureControlPlaneIngress admits the NLB→CP hops from the VPC CIDR: the
-// apiserver (k3sAPIServerPort) and the konnectivity-server agent port
-// (konnectivityAgentPort). Public access is gated separately at the NLB
+// apiserver (k3sAPIServerPort), the konnectivity-server agent port
+// (konnectivityAgentPort) and the webhook readiness port. Public access is gated separately at the NLB
 // front-end. Idempotent.
 func EnsureControlPlaneIngress(ctx context.Context, sgp sgProvisioner, accountID, cpSGID, vpcCIDR string) error {
 	if cpSGID == "" {
@@ -414,6 +414,7 @@ func EnsureControlPlaneIngress(ctx context.Context, sgp sgProvisioner, accountID
 	}{
 		{k3sAPIServerPort, "EKS NLB to apiserver"},
 		{konnectivityAgentPort, "EKS NLB to konnectivity-server"},
+		{webhookReadyPort, "EKS NLB to token webhook readiness"},
 	} {
 		perm := &ec2.IpPermission{
 			IpProtocol: aws.String("tcp"),

spinifex/domains/eks/cluster/security_groups_test.go

@@ -457,8 +457,8 @@ func TestEnsureControlPlaneIngress_AuthorizesAPIServerFromVPCCIDR(t *testing.T)
 	err := EnsureControlPlaneIngress(context.Background(), sgp, "111122223333", "sg-cp-001", "10.0.0.0/16")
 	require.NoError(t, err)
 
-	// NLB → apiserver (:6443) and NLB → konnectivity-server (:8132).
-	require.Len(t, sgp.authorizeCalls, 2)
+	// NLB → apiserver (:6443), konnectivity-server (:8132) and webhook readiness (:8444).
+	require.Len(t, sgp.authorizeCalls, 3)
 	ports := map[int64]bool{}
 	for _, in := range sgp.authorizeCalls {
 		assert.Equal(t, "sg-cp-001", aws.StringValue(in.GroupId))
@@ -472,6 +472,7 @@ func TestEnsureControlPlaneIngress_AuthorizesAPIServerFromVPCCIDR(t *testing.T)
 	}
 	assert.True(t, ports[k3sAPIServerPort], "admits :6443")
 	assert.True(t, ports[konnectivityAgentPort], "admits :8132")
+	assert.True(t, ports[webhookReadyPort], "admits the webhook readiness port from the VPC CIDR")
 }
 
 func TestEnsureControlPlaneIngress_EmptyInputsRejected(t *testing.T) {

spinifex/domains/eks/cluster/service_impl.go

@@ -59,6 +59,10 @@ type EKSServiceDeps struct {
 	// used to compose the worker's ECR registry host.
 	InternalSuffix string
 
+	// RegistryHosts returns every gateway's ECR registry host:port as it appears
+	// in a repositoryUri, local node first. Nil means only the parity name.
+	RegistryHosts func() []string
+
 	// Gateway broker config for the K3s server VM (SigV4 HTTPS to AWSGW).
 	// SystemGatewayURL is the mgmt-reachable endpoint; GatewayCACert signs its TLS cert.
 	SystemGatewayURL string
@@ -1115,10 +1119,7 @@ func (s *EKSServiceImpl) DeleteCluster(ctx context.Context, input *eks.DeleteClu
 	meta, err := GetClusterMeta(ctx, acctKV, name)
 	if err != nil {
 		if errors.Is(err, ErrClusterNotFound) {
-			// Idempotent: a cluster already swept from KV is "deleted". Returning
-			// success lets a tofu destroy retry / double-targeted delete converge
-			// instead of failing the run (Common Resource Lifecycle Contract #1).
-			return &eks.DeleteClusterOutput{}, nil
+			return nil, errors.New(awserrors.ErrorEKSResourceNotFound)
 		}
 		return nil, err
 	}

spinifex/domains/eks/cluster/service_impl_test.go

@@ -2,6 +2,7 @@ package cluster
 
 import (
 	"context"
+	"encoding/json"
 	"errors"
 	"testing"
 
@@ -11,6 +12,7 @@ import (
 	"github.com/mulgadc/spinifex/spinifex/domains/eks/access"
 	iamdomain "github.com/mulgadc/spinifex/spinifex/domains/iam"
 	"github.com/mulgadc/spinifex/spinifex/foundation/aws/errors"
+	"github.com/nats-io/nats.go"
 	"github.com/nats-io/nats.go/jetstream"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
@@ -165,16 +167,29 @@ func TestDescribeCluster_NotFoundWithFullDeps(t *testing.T) {
 }
 
 // DeleteCluster on an absent cluster must reach the KV lookup with full deps
-// wired (the shim path short-circuits to ServiceUnavailable before the meta
-// read) and return idempotent success (Common Resource Lifecycle Contract #1),
-// not a teardown of nothing — so a tofu destroy retry converges.
+// wired and return ResourceNotFoundException through the public NATS path, as
+// AWS does, without tearing anything down.
 func TestDeleteCluster_NotFoundWithFullDeps(t *testing.T) {
 	t.Parallel()
 	f := newEKSServiceFixture(t)
-
-	out, err := f.svc.DeleteCluster(context.Background(), deleteInput("ghost"), testAccountID)
+	nc := f.svc.deps.NATSConn
+	sub, err := nc.Subscribe("eks.DeleteCluster", func(msg *nats.Msg) {
+		input := &eks.DeleteClusterInput{}
+		require.NoError(t, json.Unmarshal(msg.Data, input))
+		out, svcErr := f.svc.DeleteCluster(context.Background(), input, testAccountID)
+		if svcErr != nil {
+			_ = msg.Respond(awserrors.GenerateErrorPayload(awserrors.ValidErrorCodeFromError(svcErr)))
+			return
+		}
+		payload, _ := json.Marshal(out)
+		_ = msg.Respond(payload)
+	})
 	require.NoError(t, err)
-	require.NotNil(t, out)
+	t.Cleanup(func() { _ = sub.Unsubscribe() })
+
+	out, err := NewNATSEKSService(nc).DeleteCluster(context.Background(), deleteInput("ghost"), testAccountID)
+	require.EqualError(t, err, awserrors.ErrorEKSResourceNotFound)
+	assert.Nil(t, out)
 	assert.Empty(t, f.inst.terminateCalls, "absent cluster must trigger no teardown")
 }
 

spinifex/domains/elbv2/haproxy.go

@@ -52,6 +52,12 @@ backend {{.Name}}
 {{- range .Servers}}
     server {{.Name}} {{.Addr}}:{{.Port}} check{{if .Secure}} check-ssl ssl verify none{{end}} inter {{.CheckInterval}}s fall {{.Fall}} rise {{.Rise}}
 {{- end}}
+{{- if .Backups}}
+    option allbackups
+{{- range .Backups}}
+    server {{.Name}} {{.Addr}}:{{.Port}} backup{{if .Secure}} ssl verify none{{end}}
+{{- end}}
+{{- end}}
 {{- end}}
 {{end}}
 `
@@ -107,8 +113,11 @@ type HAProxyBackend struct {
 	// sends "Host: <ip>" on the scheme's default port and "<ip>:<port>" otherwise.
 	HealthCheckDefaultPort int64
 	Servers                []HAProxyServer
-	FixedResponse          *HAProxyFixedResponse
-	Redirect               *HAProxyRedirect
+	// Backups are unchecked copies of Servers that HAProxy routes to only when
+	// every checked server is down, so an all-unhealthy group fails open.
+	Backups       []HAProxyServer
+	FixedResponse *HAProxyFixedResponse
+	Redirect      *HAProxyRedirect
 }
 
 // HAProxyFixedResponse renders an `http-request return` directive. All fields are validated before use.
@@ -204,8 +213,9 @@ func buildHAProxyConfig(lb *LoadBalancerRecord, listeners []*ListenerRecord, tgB
 			if target.PrivateIP == "" || target.HealthState == TargetHealthDraining {
 				continue
 			}
+			name := sanitizeName("srv", target.Id)
 			backend.Servers = append(backend.Servers, HAProxyServer{
-				Name:          sanitizeName("srv", target.Id),
+				Name:          name,
 				Addr:          target.PrivateIP,
 				Port:          target.Port,
 				CheckInterval: tg.HealthCheck.IntervalSeconds,
@@ -213,6 +223,12 @@ func buildHAProxyConfig(lb *LoadBalancerRecord, listeners []*ListenerRecord, tgB
 				Rise:          tg.HealthCheck.HealthyThreshold,
 				Secure:        secure,
 			})
+			backend.Backups = append(backend.Backups, HAProxyServer{
+				Name:   name + lbagent.FailoverServerSuffix,
+				Addr:   target.PrivateIP,
+				Port:   target.Port,
+				Secure: secure,
+			})
 		}
 		cfg.Backends = append(cfg.Backends, backend)
 		backendAdded[backendName] = struct{}{}

spinifex/domains/elbv2/haproxy_test.go

@@ -551,3 +551,74 @@ func TestGenerateALBConfig_StillWorks(t *testing.T) {
 	assert.NotContains(t, config, "mode tcp")
 	assert.NotContains(t, config, "option tcplog")
 }
+
+func albFailoverConfig(t *testing.T, targets ...Target) string {
+	t.Helper()
+	lb := &LoadBalancerRecord{
+		LoadBalancerArn: "arn:aws:elasticloadbalancing:us-east-1:123:loadbalancer/app/my-alb/lb-abc123",
+		LoadBalancerID:  "lb-abc123",
+		Type:            LoadBalancerTypeApplication,
+	}
+	tgArn := "arn:aws:elasticloadbalancing:us-east-1:123:targetgroup/my-tg/tg-222"
+	listeners := []*ListenerRecord{{
+		ListenerArn:     "arn:lst",
+		LoadBalancerArn: lb.LoadBalancerArn,
+		Protocol:        ProtocolHTTP,
+		Port:            80,
+		DefaultActions:  []ListenerAction{{Type: ActionTypeForward, TargetGroupArn: tgArn}},
+	}}
+	tgByArn := map[string]*TargetGroupRecord{tgArn: {
+		TargetGroupArn: tgArn,
+		HealthCheck:    HealthCheckConfig{Path: "/health", Matcher: "200", IntervalSeconds: 30, UnhealthyThreshold: 2, HealthyThreshold: 5},
+		Targets:        targets,
+	}}
+	config, _, err := GenerateHAProxyConfigWithCerts(lb, listeners, tgByArn, nil, "10.0.1.5", nil)
+	require.NoError(t, err)
+	return config
+}
+
+func TestGenerateHAProxy_FailOpenBackupCopies(t *testing.T) {
+	t.Parallel()
+	config := albFailoverConfig(t,
+		Target{Id: "i-aaa111", Port: 8080, PrivateIP: "10.0.1.10", HealthState: TargetHealthHealthy},
+		Target{Id: "i-bbb222", Port: 8080, PrivateIP: "10.0.1.11", HealthState: TargetHealthUnhealthy},
+		Target{Id: "i-drain", Port: 8080, PrivateIP: "10.0.1.12", HealthState: TargetHealthDraining},
+		Target{Id: "i-noip", Port: 8080, PrivateIP: "", HealthState: TargetHealthHealthy},
+	)
+
+	assert.Contains(t, config, "    option allbackups\n")
+	// Checked servers are unchanged; the unhealthy one stays so haproxy's own check decides.
+	assert.Contains(t, config, "server srv_i-aaa111 10.0.1.10:8080 check inter 30s fall 2 rise 5\n")
+	assert.Contains(t, config, "server srv_i-bbb222 10.0.1.11:8080 check inter 30s fall 2 rise 5\n")
+	// Backup copies: same address, distinct name, no check.
+	assert.Contains(t, config, "server srv_i-aaa111.fo 10.0.1.10:8080 backup\n")
+	assert.Contains(t, config, "server srv_i-bbb222.fo 10.0.1.11:8080 backup\n")
+	// Draining and IP-less targets get neither a server nor a copy.
+	assert.NotContains(t, config, "10.0.1.12")
+	assert.NotContains(t, config, "srv_i-noip")
+}
+
+func TestGenerateHAProxy_BackupCopyKeepsTLSToHTTPSTargets(t *testing.T) {
+	t.Parallel()
+	lb := &LoadBalancerRecord{LoadBalancerID: "lb-abc123", Type: LoadBalancerTypeApplication}
+	tgArn := "arn:tg-https"
+	listeners := []*ListenerRecord{{
+		ListenerArn: "arn:lst", Protocol: ProtocolHTTP, Port: 80,
+		DefaultActions: []ListenerAction{{Type: ActionTypeForward, TargetGroupArn: tgArn}},
+	}}
+	tgByArn := map[string]*TargetGroupRecord{tgArn: {
+		TargetGroupArn: tgArn,
+		Protocol:       ProtocolHTTPS,
+		HealthCheck:    HealthCheckConfig{Path: "/", Matcher: "200", IntervalSeconds: 30, UnhealthyThreshold: 2, HealthyThreshold: 2},
+		Targets:        []Target{{Id: "i-1", Port: 8443, PrivateIP: "10.0.1.10", HealthState: TargetHealthHealthy}},
+	}}
+	config, _, err := GenerateHAProxyConfigWithCerts(lb, listeners, tgByArn, nil, "10.0.1.5", nil)
+	require.NoError(t, err)
+	assert.Contains(t, config, "server srv_i-1.fo 10.0.1.10:8443 backup ssl verify none\n")
+}
+
+func TestGenerateHAProxy_NoServersNoAllBackups(t *testing.T) {
+	t.Parallel()
+	config := albFailoverConfig(t, Target{Id: "i-drain", Port: 8080, PrivateIP: "10.0.1.12", HealthState: TargetHealthDraining})
+	assert.NotContains(t, config, "allbackups")
+}

spinifex/domains/elbv2/health_checker.go

@@ -15,6 +15,10 @@ import (
 type healthChecker struct {
 	store *Store
 
+	// onChanged, when set, is called with the IDs of target groups whose
+	// target states were just persisted, so load balancers can re-render.
+	onChanged func(ctx context.Context, tgIDs []string)
+
 	mu       sync.Mutex
 	counters map[string]*targetCounter // key: "tgID:targetId:port"
 }
@@ -118,6 +122,7 @@ func (hc *healthChecker) handleHealthReportDirect(ctx context.Context, report lb
 	// Apply the new states field-by-field under CAS. A whole-record Put here
 	// would discard any target registered since this scan read the group, which
 	// is the one write on this key that runs on its own schedule.
+	var persisted []string
 	for _, tg := range changedTGs {
 		states := make(map[string]Target, len(tg.Targets))
 		for _, t := range tg.Targets {
@@ -139,7 +144,14 @@ func (hc *healthChecker) handleHealthReportDirect(ctx context.Context, report lb
 		})
 		if err != nil && !errors.Is(err, ErrTargetGroupNotFound) {
 			slog.ErrorContext(ctx, "healthChecker: failed to persist target group", "tgId", tg.TargetGroupID, "err", err)
+			continue
 		}
+		if err == nil {
+			persisted = append(persisted, tg.TargetGroupID)
+		}
+	}
+	if len(persisted) > 0 && hc.onChanged != nil {
+		hc.onChanged(ctx, persisted)
 	}
 }
 

spinifex/domains/elbv2/health_rerender_test.go

@@ -0,0 +1,392 @@
+package elbv2
+
+import (
+	"testing"
+	"time"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/service/elbv2"
+	lbagent "github.com/mulgadc/spinifex/spinifex/agents/elbv2"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+const rerenderTGArn = "arn:aws:elasticloadbalancing:us-east-1:1:targetgroup/db/tg-rr"
+
+// putRerenderNLB stores an NLB with a TCP listener forwarding to rerenderTGArn
+// and renders its initial config.
+func putRerenderNLB(t *testing.T, svc *ELBv2ServiceImpl, lbID string) {
+	t.Helper()
+	lbArn := "arn:aws:elasticloadbalancing:us-east-1:1:loadbalancer/net/" + lbID + "/" + lbID
+	lb := &LoadBalancerRecord{
+		LoadBalancerArn: lbArn,
+		LoadBalancerID:  lbID,
+		Name:            lbID,
+		Type:            LoadBalancerTypeNetwork,
+		State:           StateActive,
+		AccountID:       testAccountID,
+		InstanceID:      "i-" + lbID,
+	}
+	require.NoError(t, svc.store.PutLoadBalancer(t.Context(), lb))
+	l := tcpListener(lbArn+"/l1", lbID+"-l1", ProtocolTCP, 5432, rerenderTGArn)
+	l.LoadBalancerArn = lbArn
+	require.NoError(t, svc.store.PutListener(t.Context(), l))
+	require.NoError(t, svc.updateStoredConfig(t.Context(), lb))
+}
+
+func setupRerender(t *testing.T, lbIDs ...string) *ELBv2ServiceImpl {
+	t.Helper()
+	svc := setupTestService(t)
+	hcCfg := nlbHealthCheck()
+	hcCfg.HealthyThreshold = 1
+	hcCfg.UnhealthyThreshold = 1
+	require.NoError(t, svc.store.PutTargetGroup(t.Context(), &TargetGroupRecord{
+		TargetGroupArn: rerenderTGArn,
+		TargetGroupID:  "tg-rr",
+		Protocol:       ProtocolTCP,
+		Port:           5432,
+		HealthCheck:    hcCfg,
+		Targets: []Target{
+			{Id: "i-ok", Port: 5432, PrivateIP: "10.0.1.10", HealthState: TargetHealthHealthy},
+			{Id: "i-bad", Port: 5432, PrivateIP: "10.0.1.11", HealthState: TargetHealthHealthy},
+		},
+	}))
+	for _, id := range lbIDs {
+		putRerenderNLB(t, svc, id)
+	}
+	return svc
+}
+
+func rerenderReport(lbID, badStatus string) lbagent.HealthReport {
+	return lbagent.HealthReport{
+		LBID: lbID,
+		Servers: []lbagent.ServerStatus{
+			{Server: sanitizeName("srv", "i-ok"), Status: "UP"},
+			{Server: sanitizeName("srv", "i-bad"), Status: badStatus},
+		},
+	}
+}
+
+func getLB(t *testing.T, svc *ELBv2ServiceImpl, id string) *LoadBalancerRecord {
+	t.Helper()
+	lb, err := svc.store.GetLoadBalancer(t.Context(), id)
+	require.NoError(t, err)
+	require.NotNil(t, lb)
+	return lb
+}
+
+func TestHealthTransition_ReRendersConfigWithoutUnhealthyTarget(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a")
+	before := getLB(t, svc, "lb-a")
+	require.Contains(t, before.ConfigText, "10.0.1.11")
+
+	svc.hc.handleHealthReportDirect(t.Context(), rerenderReport("lb-a", "DOWN"))
+
+	after := getLB(t, svc, "lb-a")
+	assert.NotEqual(t, before.ConfigHash, after.ConfigHash)
+	assert.NotContains(t, after.ConfigText, "10.0.1.11")
+	assert.Contains(t, after.ConfigText, "10.0.1.10")
+}
+
+func TestHealthReport_SteadyStateLeavesLBUntouched(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a")
+	before := getLB(t, svc, "lb-a")
+
+	svc.hc.handleHealthReportDirect(t.Context(), rerenderReport("lb-a", "UP"))
+
+	assert.Equal(t, before, getLB(t, svc, "lb-a"))
+}
+
+func TestHealthTransition_SharedTargetGroupReRendersEveryLB(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a", "lb-b")
+
+	svc.hc.handleHealthReportDirect(t.Context(), rerenderReport("lb-a", "DOWN"))
+
+	for _, id := range []string{"lb-a", "lb-b"} {
+		assert.NotContains(t, getLB(t, svc, id).ConfigText, "10.0.1.11", id)
+	}
+}
+
+func TestHealthTransition_RecoveryReAddsTarget(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a")
+	svc.hc.handleHealthReportDirect(t.Context(), rerenderReport("lb-a", "DOWN"))
+	require.NotContains(t, getLB(t, svc, "lb-a").ConfigText, "10.0.1.11")
+
+	svc.hc.handleHealthReportDirect(t.Context(), rerenderReport("lb-a", "UP"))
+
+	assert.Contains(t, getLB(t, svc, "lb-a").ConfigText, "10.0.1.11")
+}
+
+func TestUpdateLoadBalancer_RetriesAfterConcurrentWriteAndKeepsBoth(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a")
+	attempts := 0
+	heartbeat := time.Now().UTC().Truncate(time.Second)
+
+	_, err := svc.store.UpdateLoadBalancer(t.Context(), "lb-a", func(rec *LoadBalancerRecord) (bool, error) {
+		attempts++
+		if attempts == 1 {
+			_, werr := svc.store.UpdateLoadBalancer(t.Context(), "lb-a", func(r *LoadBalancerRecord) (bool, error) {
+				r.ConfigHash = "rendered-elsewhere"
+				return true, nil
+			})
+			require.NoError(t, werr)
+		}
+		rec.LastHeartbeat = heartbeat
+		return true, nil
+	})
+	require.NoError(t, err)
+
+	assert.Equal(t, 2, attempts)
+	got := getLB(t, svc, "lb-a")
+	assert.Equal(t, "rendered-elsewhere", got.ConfigHash)
+	assert.True(t, heartbeat.Equal(got.LastHeartbeat))
+}
+
+func TestUpdateLoadBalancer_MissingRecordIsNotCreated(t *testing.T) {
+	t.Parallel()
+	svc := setupTestService(t)
+	_, err := svc.store.UpdateLoadBalancer(t.Context(), "lb-gone", func(rec *LoadBalancerRecord) (bool, error) {
+		return true, nil
+	})
+	require.ErrorIs(t, err, ErrLoadBalancerNotFound)
+	got, gerr := svc.store.GetLoadBalancer(t.Context(), "lb-gone")
+	require.NoError(t, gerr)
+	assert.Nil(t, got)
+}
+
+func TestReRender_KeepsHeartbeatWrittenAfterItReadTheLB(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a")
+	stale := getLB(t, svc, "lb-a")
+
+	heartbeat := time.Now().UTC().Truncate(time.Second)
+	_, err := svc.store.UpdateLoadBalancer(t.Context(), "lb-a", func(rec *LoadBalancerRecord) (bool, error) {
+		rec.LastHeartbeat = heartbeat
+		return true, nil
+	})
+	require.NoError(t, err)
+
+	_, err = svc.store.UpdateTargetGroup(t.Context(), "tg-rr", func(tg *TargetGroupRecord) (bool, error) {
+		tg.Targets[1].HealthState = TargetHealthUnhealthy
+		return true, nil
+	})
+	require.NoError(t, err)
+	require.NoError(t, svc.updateStoredConfig(t.Context(), stale))
+
+	got := getLB(t, svc, "lb-a")
+	assert.True(t, heartbeat.Equal(got.LastHeartbeat), "re-render must not roll back the heartbeat")
+	assert.NotContains(t, got.ConfigText, "10.0.1.11")
+}
+
+func TestHeartbeatRefresh_KeepsReRenderedConfig(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a")
+	svc.hc.handleHealthReportDirect(t.Context(), rerenderReport("lb-a", "DOWN"))
+	rendered := getLB(t, svc, "lb-a")
+	require.NotContains(t, rendered.ConfigText, "10.0.1.11")
+
+	_, err := svc.LBAgentHeartbeat(t.Context(), &LBAgentHeartbeatInput{LBID: aws.String("lb-a")}, testAccountID)
+	require.NoError(t, err)
+
+	got := getLB(t, svc, "lb-a")
+	assert.Equal(t, rendered.ConfigHash, got.ConfigHash)
+	assert.Equal(t, rendered.ConfigText, got.ConfigText)
+	assert.True(t, got.LastHeartbeat.After(rendered.LastHeartbeat), "stale heartbeat must be refreshed")
+}
+
+const rerenderLBArn = "arn:aws:elasticloadbalancing:us-east-1:1:loadbalancer/net/lb-a/lb-a"
+
+// bumpConfigHash simulates a re-render landing after another writer read the LB.
+func bumpConfigHash(t *testing.T, svc *ELBv2ServiceImpl, lbID string) {
+	t.Helper()
+	_, err := svc.store.UpdateLoadBalancer(t.Context(), lbID, func(rec *LoadBalancerRecord) (bool, error) {
+		rec.ConfigHash = "rendered-elsewhere"
+		return true, nil
+	})
+	require.NoError(t, err)
+}
+
+func TestLBAttributeSave_KeepsConfigHashWrittenAfterRead(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a")
+	res := svc.lbAttrResource(t.Context(), rerenderLBArn, testAccountID, "test")
+	stale, err := res.fetch(rerenderLBArn)
+	require.NoError(t, err)
+	require.NotNil(t, stale)
+
+	bumpConfigHash(t, svc, "lb-a")
+	stale.Attributes = map[string]string{"deletion_protection.enabled": "true"}
+	require.NoError(t, res.save(stale))
+
+	got := getLB(t, svc, "lb-a")
+	assert.Equal(t, "rendered-elsewhere", got.ConfigHash)
+	assert.Equal(t, "true", got.Attributes["deletion_protection.enabled"])
+}
+
+func TestLBTagSave_KeepsConfigHashWrittenAfterRead(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a")
+	h, _, found, err := svc.resolveTaggable(t.Context(), rerenderLBArn)
+	require.NoError(t, err)
+	require.True(t, found)
+
+	bumpConfigHash(t, svc, "lb-a")
+	require.NoError(t, h.save(map[string]string{"env": "test"}))
+
+	got := getLB(t, svc, "lb-a")
+	assert.Equal(t, "rendered-elsewhere", got.ConfigHash)
+	assert.Equal(t, "test", got.Tags["env"])
+}
+
+// The next three read and write inside one call, so the hash is bumped first:
+// they prove only the fields they own change, not the interleaving itself.
+func TestSetIpAddressType_LeavesConfigHashAlone(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a")
+	bumpConfigHash(t, svc, "lb-a")
+
+	_, err := svc.SetIpAddressType(t.Context(), &elbv2.SetIpAddressTypeInput{
+		LoadBalancerArn: aws.String(rerenderLBArn),
+		IpAddressType:   aws.String(IPAddressTypeIPv4),
+	}, testAccountID)
+	require.NoError(t, err)
+
+	got := getLB(t, svc, "lb-a")
+	assert.Equal(t, "rendered-elsewhere", got.ConfigHash)
+	assert.Equal(t, IPAddressTypeIPv4, got.IPAddressType)
+}
+
+func TestSetSecurityGroups_LeavesConfigHashAlone(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a")
+	_, err := svc.store.UpdateLoadBalancer(t.Context(), "lb-a", func(rec *LoadBalancerRecord) (bool, error) {
+		rec.SecurityGroups = []string{"sg-old"}
+		rec.ConfigHash = "rendered-elsewhere"
+		return true, nil
+	})
+	require.NoError(t, err)
+
+	_, err = svc.SetSecurityGroups(t.Context(), &elbv2.SetSecurityGroupsInput{
+		LoadBalancerArn: aws.String(rerenderLBArn),
+		SecurityGroups:  []*string{aws.String("sg-new")},
+	}, testAccountID)
+	require.NoError(t, err)
+
+	got := getLB(t, svc, "lb-a")
+	assert.Equal(t, "rendered-elsewhere", got.ConfigHash)
+	assert.Equal(t, []string{"sg-new"}, got.SecurityGroups)
+}
+
+func TestSetLoadBalancerIngressCIDRs_LeavesConfigHashAlone(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a")
+	bumpConfigHash(t, svc, "lb-a")
+
+	require.NoError(t, svc.SetLoadBalancerIngressCIDRs(rerenderLBArn, []string{"10.0.0.0/8"}, testAccountID))
+
+	got := getLB(t, svc, "lb-a")
+	assert.Equal(t, "rendered-elsewhere", got.ConfigHash)
+	assert.Equal(t, []string{"10.0.0.0/8"}, got.NLBIngressCIDRs)
+}
+
+const modifyTGArn = "arn:aws:elasticloadbalancing:us-east-1:1:targetgroup/mod/tg-mod"
+
+// putModifyFixture stores a target group with an HTTP health check, an NLB
+// and an ALB that both forward to it, and renders their initial configs.
+func putModifyFixture(t *testing.T, svc *ELBv2ServiceImpl, protocol string) {
+	t.Helper()
+	hc := nlbHealthCheck()
+	hc.Protocol = ProtocolHTTP
+	hc.Path = "/old"
+	hc.Matcher = "200"
+	require.NoError(t, svc.store.PutTargetGroup(t.Context(), &TargetGroupRecord{
+		TargetGroupArn: modifyTGArn,
+		TargetGroupID:  "tg-mod",
+		AccountID:      testAccountID,
+		Protocol:       protocol,
+		Port:           8080,
+		HealthCheck:    hc,
+		Targets:        []Target{{Id: "i-ok", Port: 8080, PrivateIP: "10.0.1.10", HealthState: TargetHealthHealthy}},
+	}))
+}
+
+func putModifyLB(t *testing.T, svc *ELBv2ServiceImpl, lbID, lbType, protocol string) {
+	t.Helper()
+	lbArn := "arn:aws:elasticloadbalancing:us-east-1:1:loadbalancer/x/" + lbID + "/" + lbID
+	lb := &LoadBalancerRecord{
+		LoadBalancerArn: lbArn,
+		LoadBalancerID:  lbID,
+		Name:            lbID,
+		Type:            lbType,
+		State:           StateActive,
+		AccountID:       testAccountID,
+		InstanceID:      "i-" + lbID,
+	}
+	require.NoError(t, svc.store.PutLoadBalancer(t.Context(), lb))
+	l := tcpListener(lbArn+"/l1", lbID+"-l1", protocol, 8080, modifyTGArn)
+	l.LoadBalancerArn = lbArn
+	require.NoError(t, svc.store.PutListener(t.Context(), l))
+	require.NoError(t, svc.updateStoredConfig(t.Context(), lb))
+}
+
+func modifyPath(t *testing.T, svc *ELBv2ServiceImpl, path string) {
+	t.Helper()
+	_, err := svc.ModifyTargetGroup(t.Context(), &elbv2.ModifyTargetGroupInput{
+		TargetGroupArn:  aws.String(modifyTGArn),
+		HealthCheckPath: aws.String(path),
+	}, testAccountID)
+	require.NoError(t, err)
+}
+
+func TestModifyTargetGroup_ReRendersALBWithTheNewHealthPath(t *testing.T) {
+	t.Parallel()
+	svc := setupTestService(t)
+	putModifyFixture(t, svc, ProtocolHTTP)
+	putModifyLB(t, svc, "alb-m", LoadBalancerTypeApplication, ProtocolHTTP)
+	before := getLB(t, svc, "alb-m")
+	require.Contains(t, before.ConfigText, "uri /old")
+
+	modifyPath(t, svc, "/new")
+
+	after := getLB(t, svc, "alb-m")
+	assert.Contains(t, after.ConfigText, "uri /new")
+	assert.NotContains(t, after.ConfigText, "uri /old")
+	assert.NotEqual(t, before.ConfigHash, after.ConfigHash)
+}
+
+func TestModifyTargetGroup_NLBProbeChangeMovesTheConfigHash(t *testing.T) {
+	t.Parallel()
+	svc := setupTestService(t)
+	putModifyFixture(t, svc, ProtocolTCP)
+	putModifyLB(t, svc, "nlb-m", LoadBalancerTypeNetwork, ProtocolTCP)
+	before := getLB(t, svc, "nlb-m")
+	require.Len(t, before.HealthTargets, 1)
+	require.Equal(t, "/old", before.HealthTargets[0].Path)
+
+	modifyPath(t, svc, "/new")
+
+	// nginx has no probe settings, so only the hash input carries this change
+	// to the agent, which re-fetches health targets when the hash moves.
+	after := getLB(t, svc, "nlb-m")
+	require.Len(t, after.HealthTargets, 1)
+	assert.Equal(t, "/new", after.HealthTargets[0].Path)
+	assert.Equal(t, before.ConfigText, after.ConfigText)
+	assert.NotEqual(t, before.ConfigHash, after.ConfigHash)
+}
+
+func TestModifyTargetGroup_UnchangedSettingsLeaveTheConfigHashAlone(t *testing.T) {
+	t.Parallel()
+	svc := setupTestService(t)
+	putModifyFixture(t, svc, ProtocolHTTP)
+	putModifyLB(t, svc, "alb-same", LoadBalancerTypeApplication, ProtocolHTTP)
+	before := getLB(t, svc, "alb-same")
+
+	modifyPath(t, svc, "/old")
+
+	assert.Equal(t, before.ConfigHash, getLB(t, svc, "alb-same").ConfigHash)
+}

spinifex/domains/elbv2/nginx.go

@@ -148,10 +148,7 @@ func generateNLBStreamConfig(lb *LoadBalancerRecord, listeners []*ListenerRecord
 		}
 		us := nginxUpstream{Name: name}
 		if tg, ok := tgByArn[tgArn]; ok {
-			for _, t := range tg.Targets {
-				if t.PrivateIP == "" || t.HealthState == TargetHealthDraining {
-					continue
-				}
+			for _, t := range routableTargets(tg.Targets) {
 				us.Servers = append(us.Servers, nginxUpstreamServer{
 					Addr:        t.PrivateIP,
 					Port:        t.Port,
@@ -329,6 +326,7 @@ func buildNLBHealthTargets(tgByArn map[string]*TargetGroupRecord) []HealthTarget
 	var out []HealthTargetSpec
 	for _, tg := range tgByArn {
 		for _, t := range tg.Targets {
+			// Unhealthy targets stay probed so they can recover.
 			if t.PrivateIP == "" || t.HealthState == TargetHealthDraining {
 				continue
 			}

spinifex/domains/elbv2/nginx_test.go

@@ -307,3 +307,48 @@ func TestCertDNSNames_DropsNamesThatAreNotHostnames(t *testing.T) {
 	assert.Equal(t, []string{"ok.example.com", "upper.example.com"}, certDNSNames(pemText))
 	assert.Nil(t, certDNSNames("not a pem"), "unparseable material yields no names")
 }
+
+func TestGenerateNLBStream_UnhealthyTargetLeavesUpstreamButStaysProbed(t *testing.T) {
+	t.Parallel()
+	lb := &LoadBalancerRecord{LoadBalancerID: "lb-unh", Type: LoadBalancerTypeNetwork}
+	tgArn := "arn:aws:elasticloadbalancing:us-east-1:1:targetgroup/db/tg-db"
+	listeners := []*ListenerRecord{tcpListener("arn:lst", "lst", ProtocolTCP, 5432, tgArn)}
+	tgByArn := map[string]*TargetGroupRecord{
+		tgArn: nlbTG(tgArn,
+			Target{Id: "i-ok", Port: 5432, PrivateIP: "10.0.1.10", HealthState: TargetHealthHealthy},
+			Target{Id: "i-bad", Port: 5432, PrivateIP: "10.0.1.11", HealthState: TargetHealthUnhealthy},
+		),
+	}
+
+	config, _, err := GenerateHAProxyConfigWithCerts(lb, listeners, tgByArn, nil, "10.0.1.5", nil)
+	require.NoError(t, err)
+	assert.Contains(t, config, "server 10.0.1.10:5432 ")
+	assert.NotContains(t, config, "10.0.1.11")
+
+	probed := buildNLBHealthTargets(tgByArn)
+	var addrs []string
+	for _, p := range probed {
+		addrs = append(addrs, p.Address)
+	}
+	assert.ElementsMatch(t, []string{"10.0.1.10:5432", "10.0.1.11:5432"}, addrs)
+}
+
+func TestGenerateNLBStream_AllUnhealthyFailsOpen(t *testing.T) {
+	t.Parallel()
+	lb := &LoadBalancerRecord{LoadBalancerID: "lb-allunh", Type: LoadBalancerTypeNetwork}
+	tgArn := "arn:aws:elasticloadbalancing:us-east-1:1:targetgroup/db/tg-db"
+	listeners := []*ListenerRecord{tcpListener("arn:lst", "lst", ProtocolTCP, 5432, tgArn)}
+	tgByArn := map[string]*TargetGroupRecord{
+		tgArn: nlbTG(tgArn,
+			Target{Id: "i-1", Port: 5432, PrivateIP: "10.0.1.10", HealthState: TargetHealthUnhealthy},
+			Target{Id: "i-2", Port: 5432, PrivateIP: "10.0.1.11", HealthState: TargetHealthUnhealthy},
+			Target{Id: "i-drain", Port: 5432, PrivateIP: "10.0.1.12", HealthState: TargetHealthDraining},
+		),
+	}
+
+	config, _, err := GenerateHAProxyConfigWithCerts(lb, listeners, tgByArn, nil, "10.0.1.5", nil)
+	require.NoError(t, err)
+	assert.Contains(t, config, "server 10.0.1.10:5432 ")
+	assert.Contains(t, config, "server 10.0.1.11:5432 ")
+	assert.NotContains(t, config, "10.0.1.12")
+}

spinifex/domains/elbv2/routable_targets.go

@@ -0,0 +1,21 @@
+package elbv2
+
+// routableTargets returns the targets a load balancer may send traffic to.
+// Draining, unhealthy and IP-less targets are excluded; when that leaves none,
+// every non-draining target with an IP is returned (fail open, as AWS does).
+func routableTargets(targets []Target) []Target {
+	var healthy, eligible []Target
+	for _, t := range targets {
+		if t.PrivateIP == "" || t.HealthState == TargetHealthDraining {
+			continue
+		}
+		eligible = append(eligible, t)
+		if t.HealthState != TargetHealthUnhealthy {
+			healthy = append(healthy, t)
+		}
+	}
+	if len(healthy) > 0 {
+		return healthy
+	}
+	return eligible
+}

spinifex/domains/elbv2/routable_targets_test.go

@@ -0,0 +1,52 @@
+package elbv2
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestRoutableTargets(t *testing.T) {
+	tgt := func(id, ip, state string) Target {
+		return Target{Id: id, PrivateIP: ip, HealthState: state}
+	}
+	ids := func(ts []Target) []string {
+		var out []string
+		for _, x := range ts {
+			out = append(out, x.Id)
+		}
+		return out
+	}
+
+	tests := []struct {
+		name    string
+		targets []Target
+		want    []string
+	}{
+		{"all healthy", []Target{tgt("a", "10.0.0.1", TargetHealthHealthy), tgt("b", "10.0.0.2", TargetHealthHealthy)}, []string{"a", "b"}},
+		{"one unhealthy is dropped", []Target{tgt("a", "10.0.0.1", TargetHealthHealthy), tgt("b", "10.0.0.2", TargetHealthUnhealthy)}, []string{"a"}},
+		{"initial stays routable", []Target{tgt("a", "10.0.0.1", TargetHealthInitial), tgt("b", "10.0.0.2", TargetHealthUnhealthy)}, []string{"a"}},
+		{"draining is dropped", []Target{tgt("a", "10.0.0.1", TargetHealthHealthy), tgt("b", "10.0.0.2", TargetHealthDraining)}, []string{"a"}},
+		{"no ip is dropped", []Target{tgt("a", "10.0.0.1", TargetHealthHealthy), tgt("b", "", TargetHealthHealthy)}, []string{"a"}},
+		{"mixed", []Target{
+			tgt("a", "10.0.0.1", TargetHealthHealthy),
+			tgt("b", "10.0.0.2", TargetHealthUnhealthy),
+			tgt("c", "10.0.0.3", TargetHealthInitial),
+			tgt("d", "10.0.0.4", TargetHealthDraining),
+			tgt("e", "", TargetHealthHealthy),
+		}, []string{"a", "c"}},
+		{"all unhealthy fails open but keeps draining and no-ip out", []Target{
+			tgt("a", "10.0.0.1", TargetHealthUnhealthy),
+			tgt("b", "10.0.0.2", TargetHealthUnhealthy),
+			tgt("c", "10.0.0.3", TargetHealthDraining),
+			tgt("d", "", TargetHealthUnhealthy),
+		}, []string{"a", "b"}},
+		{"all draining is empty", []Target{tgt("a", "10.0.0.1", TargetHealthDraining)}, nil},
+		{"empty group", nil, nil},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			assert.Equal(t, tc.want, ids(routableTargets(tc.targets)))
+		})
+	}
+}

spinifex/domains/elbv2/service_impl.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"crypto/sha256"
 	"encoding/hex"
+	"encoding/json"
 	"errors"
 	"fmt"
 	"log/slog"
@@ -183,7 +184,7 @@ func NewELBv2ServiceImplWithNATS(cfg *config.Config, nc *nats.Conn, masterKey []
 
 	hc := newHealthChecker(store)
 
-	return &ELBv2ServiceImpl{
+	svc := &ELBv2ServiceImpl{
 		config:         cfg,
 		store:          store,
 		acmStore:       acmStore,
@@ -195,7 +196,9 @@ func NewELBv2ServiceImplWithNATS(cfg *config.Config, nc *nats.Conn, masterKey []
 		cancel:         cancel,
 		hc:             hc,
 		reaperInterval: lbReaperInterval,
-	}, nil
+	}
+	hc.onChanged = svc.reRenderForTargetGroups
+	return svc, nil
 }
 
 // Close cancels background goroutines.
@@ -715,18 +718,35 @@ func (s *ELBv2ServiceImpl) updateStoredConfig(ctx context.Context, lb *LoadBalan
 		return fmt.Errorf("generate config: %w", err)
 	}
 
-	hash := configCertHash(configContent, certFiles)
+	// NLBs: the agent actively probes targets (nginx has no upstream probing).
+	var healthTargets []HealthTargetSpec
+	if lb.Type == LoadBalancerTypeNetwork {
+		healthTargets = buildNLBHealthTargets(tgByArn)
+	}
+	// The agent re-fetches only when the hash moves, so probe settings are part of it.
+	hashed := configContent
+	if len(healthTargets) > 0 {
+		probes, err := json.Marshal(healthTargets)
+		if err != nil {
+			return fmt.Errorf("marshal health targets: %w", err)
+		}
+		hashed += "\n# probes " + string(probes)
+	}
+	hash := configCertHash(hashed, certFiles)
 	lb.ConfigText = configContent
 	lb.ConfigHash = hash
 	lb.CertFiles = certFiles
-	// NLBs: the agent actively probes targets (nginx has no upstream probing).
-	if lb.Type == LoadBalancerTypeNetwork {
-		lb.HealthTargets = buildNLBHealthTargets(tgByArn)
-	} else {
-		lb.HealthTargets = nil
-	}
+	lb.HealthTargets = healthTargets
 
-	if err := s.store.PutLoadBalancer(ctx, lb); err != nil {
+	// Only the rendered fields are written, under CAS, so a concurrent heartbeat
+	// or re-render of the same LB is never overwritten with a stale record.
+	if _, err := s.store.UpdateLoadBalancer(ctx, lb.LoadBalancerID, func(rec *LoadBalancerRecord) (bool, error) {
+		rec.ConfigText = lb.ConfigText
+		rec.ConfigHash = lb.ConfigHash
+		rec.CertFiles = lb.CertFiles
+		rec.HealthTargets = lb.HealthTargets
+		return true, nil
+	}); err != nil {
 		slog.ErrorContext(ctx, "updateStoredConfig: failed to persist LB", "lbId", lb.LoadBalancerID, "err", err)
 		return fmt.Errorf("persist LB: %w", err)
 	}
@@ -736,6 +756,46 @@ func (s *ELBv2ServiceImpl) updateStoredConfig(ctx context.Context, lb *LoadBalan
 	return nil
 }
 
+// reRenderForTargetGroups regenerates the stored config of every load balancer
+// that references one of tgIDs, so a target health change reaches the data plane.
+// The LB is re-read just before rendering and failures are logged, never returned.
+func (s *ELBv2ServiceImpl) reRenderForTargetGroups(ctx context.Context, tgIDs []string) {
+	changed := make(map[string]struct{}, len(tgIDs))
+	for _, id := range tgIDs {
+		changed[id] = struct{}{}
+	}
+	lbs, err := s.store.ListLoadBalancers(ctx)
+	if err != nil {
+		slog.ErrorContext(ctx, "reRenderForTargetGroups: failed to list load balancers", "err", err)
+		return
+	}
+	for _, listed := range lbs {
+		tgs, err := s.store.TargetGroupsForLB(ctx, listed.LoadBalancerID)
+		if err != nil {
+			slog.ErrorContext(ctx, "reRenderForTargetGroups: failed to resolve target groups", "lbId", listed.LoadBalancerID, "err", err)
+			continue
+		}
+		uses := false
+		for _, tg := range tgs {
+			if _, ok := changed[tg.TargetGroupID]; ok {
+				uses = true
+				break
+			}
+		}
+		if !uses {
+			continue
+		}
+		lb, err := s.store.GetLoadBalancer(ctx, listed.LoadBalancerID)
+		if err != nil || lb == nil {
+			slog.ErrorContext(ctx, "reRenderForTargetGroups: failed to re-read load balancer", "lbId", listed.LoadBalancerID, "err", err)
+			continue
+		}
+		if err := s.updateStoredConfig(ctx, lb); err != nil {
+			slog.ErrorContext(ctx, "reRenderForTargetGroups: re-render failed", "lbId", lb.LoadBalancerID, "err", err)
+		}
+	}
+}
+
 // resolveListenerCerts resolves each distinct certificate ARN to its combined PEM.
 // Returns nil for HTTP-only listeners.
 func (s *ELBv2ServiceImpl) resolveListenerCerts(ctx context.Context, listeners []*ListenerRecord, accountID string) (map[string]string, error) {
@@ -1068,8 +1128,21 @@ func (s *ELBv2ServiceImpl) LBAgentHeartbeat(ctx context.Context, input *LBAgentH
 	heartbeatStale := now.Sub(lb.LastHeartbeat) >= heartbeatPersistInterval
 	lb.LastHeartbeat = now
 
+	// Both writes touch only the fields this heartbeat owns, under CAS, so a
+	// concurrent config re-render of the same LB is never rolled back.
 	switch {
 	case stateChanged:
+		if _, err := s.store.UpdateLoadBalancer(ctx, lbID, func(rec *LoadBalancerRecord) (bool, error) {
+			if rec.State == StateProvisioning {
+				rec.State = StateActive
+				rec.StateReason = ""
+			}
+			rec.LastHeartbeat = now
+			return true, nil
+		}); err != nil {
+			slog.ErrorContext(ctx, "LBAgentHeartbeat: failed to persist activation", "lbId", lbID, "err", err)
+			return nil, errors.New(awserrors.ErrorServerInternal)
+		}
 		// Build data-plane config on activation: reactive calls during create
 		// no-op while InstanceID is empty, so we build it now from existing listeners/targets.
 		if err := s.updateStoredConfig(ctx, lb); err != nil {
@@ -1078,7 +1151,10 @@ func (s *ELBv2ServiceImpl) LBAgentHeartbeat(ctx context.Context, input *LBAgentH
 		}
 	case heartbeatStale:
 		// Refresh the heartbeat timestamp only; avoid writing the full record on every tick.
-		if err := s.store.PutLoadBalancer(ctx, lb); err != nil {
+		if _, err := s.store.UpdateLoadBalancer(ctx, lbID, func(rec *LoadBalancerRecord) (bool, error) {
+			rec.LastHeartbeat = now
+			return true, nil
+		}); err != nil {
 			slog.ErrorContext(ctx, "LBAgentHeartbeat: failed to persist LB", "lbId", lbID, "err", err)
 			return nil, errors.New(awserrors.ErrorServerInternal)
 		}
@@ -2095,6 +2171,8 @@ func (s *ELBv2ServiceImpl) ModifyTargetGroup(ctx context.Context, input *elbv2.M
 		return nil, err
 	}
 
+	s.reRenderForTargetGroups(ctx, []string{tg.TargetGroupID})
+
 	slog.InfoContext(ctx, "ModifyTargetGroup completed", "arn", tg.TargetGroupArn, "accountID", accountID)
 
 	return &elbv2.ModifyTargetGroupOutput{

spinifex/domains/elbv2/service_impl_attributes.go

@@ -190,8 +190,15 @@ func (s *ELBv2ServiceImpl) lbAttrResource(ctx context.Context, arn, accountID, o
 		opName:    opName,
 		notFound:  awserrors.ErrorELBv2LoadBalancerNotFound,
 		// See tgAttrResource: the generic helpers carry no context.
-		fetch:       func(arn string) (*LoadBalancerRecord, error) { return s.store.GetLoadBalancerByArn(ctx, arn) },
-		save:        func(lb *LoadBalancerRecord) error { return s.store.PutLoadBalancer(ctx, lb) },
+		fetch: func(arn string) (*LoadBalancerRecord, error) { return s.store.GetLoadBalancerByArn(ctx, arn) },
+		// CAS the attribute field alone so a concurrent config re-render survives.
+		save: func(lb *LoadBalancerRecord) error {
+			_, err := s.store.UpdateLoadBalancer(ctx, lb.LoadBalancerID, func(rec *LoadBalancerRecord) (bool, error) {
+				rec.Attributes = lb.Attributes
+				return true, nil
+			})
+			return err
+		},
 		accountIDOf: func(r *LoadBalancerRecord) string { return r.AccountID },
 		defaults:    func(r *LoadBalancerRecord) map[string]string { return DefaultLoadBalancerAttributes(r.Type) },
 		attrsOf:     func(r *LoadBalancerRecord) map[string]string { return r.Attributes },

spinifex/domains/elbv2/service_impl_lb_network.go

@@ -35,7 +35,10 @@ func (s *ELBv2ServiceImpl) SetIpAddressType(ctx context.Context, input *elbv2.Se
 
 	if lb.IPAddressType != IPAddressTypeIPv4 {
 		lb.IPAddressType = IPAddressTypeIPv4
-		if err := s.store.PutLoadBalancer(ctx, lb); err != nil {
+		if _, err := s.store.UpdateLoadBalancer(ctx, lb.LoadBalancerID, func(rec *LoadBalancerRecord) (bool, error) {
+			rec.IPAddressType = lb.IPAddressType
+			return true, nil
+		}); err != nil {
 			slog.ErrorContext(ctx, "SetIpAddressType: failed to persist LB", "arn", *input.LoadBalancerArn, "err", err)
 			return nil, errors.New(awserrors.ErrorServerInternal)
 		}
@@ -98,7 +101,10 @@ func (s *ELBv2ServiceImpl) SetSecurityGroups(ctx context.Context, input *elbv2.S
 	}
 
 	lb.SecurityGroups = sgs
-	if err := s.store.PutLoadBalancer(ctx, lb); err != nil {
+	if _, err := s.store.UpdateLoadBalancer(ctx, lb.LoadBalancerID, func(rec *LoadBalancerRecord) (bool, error) {
+		rec.SecurityGroups = sgs
+		return true, nil
+	}); err != nil {
 		slog.ErrorContext(ctx, "SetSecurityGroups: failed to persist LB", "arn", *input.LoadBalancerArn, "err", err)
 		return nil, errors.New(awserrors.ErrorServerInternal)
 	}
@@ -155,7 +161,11 @@ func (s *ELBv2ServiceImpl) SetSubnets(ctx context.Context, input *elbv2.SetSubne
 	if s.VPCService == nil {
 		lb.Subnets = desired
 		lb.AvailZones = rebuildAvailZones(desired, lb.AvailZones, nil)
-		if err := s.store.PutLoadBalancer(ctx, lb); err != nil {
+		if _, err := s.store.UpdateLoadBalancer(ctx, lb.LoadBalancerID, func(rec *LoadBalancerRecord) (bool, error) {
+			rec.Subnets = lb.Subnets
+			rec.AvailZones = lb.AvailZones
+			return true, nil
+		}); err != nil {
 			slog.ErrorContext(ctx, "SetSubnets: failed to persist LB", "arn", *input.LoadBalancerArn, "err", err)
 			return nil, errors.New(awserrors.ErrorServerInternal)
 		}
@@ -244,7 +254,17 @@ func (s *ELBv2ServiceImpl) SetSubnets(ctx context.Context, input *elbv2.SetSubne
 	lb.HostPorts = launch.hostPorts
 	lb.State, lb.StateReason = s.lbStateAfterLaunch(launch, lb.Scheme)
 
-	if err := s.store.PutLoadBalancer(ctx, lb); err != nil {
+	if _, err := s.store.UpdateLoadBalancer(ctx, lb.LoadBalancerID, func(rec *LoadBalancerRecord) (bool, error) {
+		rec.Subnets = lb.Subnets
+		rec.ENIs = lb.ENIs
+		rec.AvailZones = lb.AvailZones
+		rec.InstanceID = lb.InstanceID
+		rec.VPCIP = lb.VPCIP
+		rec.HostPorts = lb.HostPorts
+		rec.State = lb.State
+		rec.StateReason = lb.StateReason
+		return true, nil
+	}); err != nil {
 		slog.ErrorContext(ctx, "SetSubnets: failed to persist LB", "arn", *input.LoadBalancerArn, "err", err)
 		return nil, errors.New(awserrors.ErrorServerInternal)
 	}

spinifex/domains/elbv2/service_impl_nlb_sg.go

@@ -245,7 +245,10 @@ func (s *ELBv2ServiceImpl) SetLoadBalancerIngressCIDRs(lbArn string, cidrs []str
 		}
 	}
 
-	if err := s.store.PutLoadBalancer(s.ctx, lb); err != nil {
+	if _, err := s.store.UpdateLoadBalancer(s.ctx, lb.LoadBalancerID, func(rec *LoadBalancerRecord) (bool, error) {
+		rec.NLBIngressCIDRs = lb.NLBIngressCIDRs
+		return true, nil
+	}); err != nil {
 		slog.Error("SetLoadBalancerIngressCIDRs: persist failed", "arn", lbArn, "err", err)
 		return errors.New(awserrors.ErrorServerInternal)
 	}

spinifex/domains/elbv2/service_impl_tags.go

@@ -39,8 +39,12 @@ func (s *ELBv2ServiceImpl) resolveTaggable(ctx context.Context, arn string) (h t
 		if lb != nil {
 			found = true
 			h = taggable{tags: lb.Tags, owner: lb.AccountID, save: func(t map[string]string) error {
-				lb.Tags = t
-				return s.store.PutLoadBalancer(ctx, lb)
+				// CAS the tag field alone so a concurrent config re-render survives.
+				_, uerr := s.store.UpdateLoadBalancer(ctx, lb.LoadBalancerID, func(rec *LoadBalancerRecord) (bool, error) {
+					rec.Tags = t
+					return true, nil
+				})
+				return uerr
 			}}
 		}
 	case resourcearn.ELBv2TargetGroup:

spinifex/domains/elbv2/store.go

@@ -181,6 +181,23 @@ func (s *Store) PutLoadBalancer(ctx context.Context, lb *LoadBalancerRecord) err
 	return err
 }
 
+// ErrLoadBalancerNotFound is what a load-balancer CAS update reports when the
+// record is absent, so a deleted LB is never resurrected by a late writer.
+var ErrLoadBalancerNotFound = errors.New("elbv2: load balancer not found")
+
+// UpdateLoadBalancer applies mutate to one load balancer under optimistic
+// concurrency: a writer that loses the race re-reads and retries instead of
+// overwriting the winner. mutate returns false to commit nothing.
+func (s *Store) UpdateLoadBalancer(ctx context.Context, lbID string, mutate func(*LoadBalancerRecord) (bool, error)) (*LoadBalancerRecord, error) {
+	return kvutil.Update(ctx, s.kv, KeyPrefixLB+lbID, kvutil.CASConfig{
+		Attempts: targetGroupCASAttempts,
+		NotFound: ErrLoadBalancerNotFound,
+		Exhausted: func(_ string, attempts int) error {
+			return fmt.Errorf("elbv2: load balancer %s contended, CAS exhausted after %d attempts", lbID, attempts)
+		},
+	}, mutate)
+}
+
 // GetLoadBalancer retrieves a load balancer by its short ID.
 func (s *Store) GetLoadBalancer(ctx context.Context, lbID string) (*LoadBalancerRecord, error) {
 	entry, err := s.kv.Get(ctx, KeyPrefixLB+lbID)

spinifex/domains/network/external/igw.go

@@ -5,7 +5,9 @@ import (
 	"errors"
 	"fmt"
 	"log/slog"
+	"maps"
 	"net/netip"
+	"strconv"
 
 	networkids "github.com/mulgadc/spinifex/spinifex/domains/network/identifiers"
 	"github.com/mulgadc/spinifex/spinifex/domains/network/ovn"
@@ -79,6 +81,9 @@ type IGWManagerConfig struct {
 	// upstream nexthop, removing the dependency on lazy dynamic ARP. Optional;
 	// nil skips seeding.
 	NexthopSeed func(ctx context.Context, lrpName, nexthopIP string) error
+	// GatewayMTU is written to the gateway LRP's gateway_mtu option so OVN
+	// emits ICMP frag-needed for oversized packets. 0 = do not manage the option.
+	GatewayMTU int
 }
 
 type igwManager struct {
@@ -92,6 +97,7 @@ type igwManager struct {
 	barrier       FlowsBarrier
 	routedIngress *RoutedIngressHooks
 	nexthopSeed   func(ctx context.Context, lrpName, nexthopIP string) error
+	gatewayMTU    int
 }
 
 var _ IGWManager = (*igwManager)(nil)
@@ -125,6 +131,7 @@ func NewIGWManager(cfg IGWManagerConfig) (IGWManager, error) {
 		barrier:       barrier,
 		routedIngress: cfg.RoutedIngress,
 		nexthopSeed:   cfg.NexthopSeed,
+		gatewayMTU:    cfg.GatewayMTU,
 	}, nil
 }
 
@@ -164,6 +171,9 @@ func (m *igwManager) AttachIGW(ctx context.Context, spec IGWSpec) error {
 		if err := m.ensureNATAdvertisement(ctx, existing); err != nil {
 			return err
 		}
+		if err := m.ensureGatewayMTU(ctx, gwPortName); err != nil {
+			return err
+		}
 		// OVN state survives reboots but host routes do not: re-run the routed
 		// leg so reconcile's replayed attach re-ensures ingress + SNAT.
 		if m.natMode == policy.NATModeRouted {
@@ -193,15 +203,22 @@ func (m *igwManager) AttachIGW(ctx context.Context, spec IGWSpec) error {
 		if gwLrpIP != "" {
 			lrpExtIDs[gatewayIPExtIDKey] = gwLrpIP
 		}
+		var lrpOpts map[string]string
+		if m.gatewayMTU > 0 {
+			lrpOpts = map[string]string{gatewayMTUOption: strconv.Itoa(m.gatewayMTU)}
+		}
 		if err := m.ovn.CreateLogicalRouterPort(ctx, routerName, &nbdb.LogicalRouterPort{
 			Name:        gwPortName,
 			MAC:         networkids.HashMAC(gwPortName),
 			Networks:    []string{gwNetwork},
 			ExternalIDs: lrpExtIDs,
+			Options:     lrpOpts,
 		}); err != nil {
 			return fmt.Errorf("create gateway router port %s: %w", gwPortName, err)
 		}
 		createdLRP = true
+	} else if err := m.ensureGatewayMTU(ctx, gwPortName); err != nil {
+		return err
 	}
 
 	// nat-addresses is only honoured on a router-type port; OVN silently ignores
@@ -404,6 +421,35 @@ func (m *igwManager) ensureNATAdvertisement(ctx context.Context, port *nbdb.Logi
 	return nil
 }
 
+// gatewayMTUOption makes OVN answer packets larger than this with ICMP
+// frag-needed instead of silently dropping them at the gateway router.
+const gatewayMTUOption = "gateway_mtu"
+
+// ensureGatewayMTU converges the gateway_mtu option on a gateway LRP, leaving
+// its other options untouched. A no-op when no gateway MTU is configured.
+func (m *igwManager) ensureGatewayMTU(ctx context.Context, lrpName string) error {
+	if m.gatewayMTU == 0 {
+		return nil
+	}
+	lrp, err := m.ovn.GetLogicalRouterPort(ctx, lrpName)
+	if err != nil {
+		return fmt.Errorf("get gateway port %s to set %s: %w", lrpName, gatewayMTUOption, err)
+	}
+	want := strconv.Itoa(m.gatewayMTU)
+	if lrp.Options[gatewayMTUOption] == want {
+		return nil
+	}
+	opts := make(map[string]string, len(lrp.Options)+1)
+	maps.Copy(opts, lrp.Options)
+	opts[gatewayMTUOption] = want
+	lrp.Options = opts
+	if err := m.ovn.UpdateLogicalRouterPort(ctx, lrp); err != nil {
+		return fmt.Errorf("set %s on %s: %w", gatewayMTUOption, lrpName, err)
+	}
+	slog.Info("external: set gateway MTU on gateway port", "port", lrpName, "mtu", m.gatewayMTU)
+	return nil
+}
+
 // ensureSharedExternal idempotently creates the singleton shared external switch
 // and its single localnet port. NAT advertisement lives on the per-VPC gateway
 // port, not here — OVN ignores nat-addresses on a localnet port.

spinifex/domains/network/external/igw_test.go

@@ -9,6 +9,7 @@ import (
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 
+	"github.com/mulgadc/spinifex/spinifex/domains/network/ovn"
 	"github.com/mulgadc/spinifex/spinifex/domains/network/ovn/mock"
 	"github.com/mulgadc/spinifex/spinifex/domains/network/ovn/nbdb"
 	"github.com/mulgadc/spinifex/spinifex/domains/network/policy"
@@ -785,3 +786,121 @@ func TestAttachIGW_NexthopSeed_NilHookSkipped(t *testing.T) {
 
 	require.NoError(t, mgr.AttachIGW(ctx, IGWSpec{VPCID: "vpc-1", InternetGatewayID: "igw-1"}))
 }
+
+// failingLRPUpdateClient fails UpdateLogicalRouterPort; the mock has no hook.
+type failingLRPUpdateClient struct{ *mock.Client }
+
+func (failingLRPUpdateClient) UpdateLogicalRouterPort(context.Context, *nbdb.LogicalRouterPort) error {
+	return errors.New("nb unavailable")
+}
+
+func newGatewayMTUManager(t *testing.T, c ovn.Client, mtu int) IGWManager {
+	t.Helper()
+	nm, err := policy.NewNATManager(c, policy.NATModeDistributed)
+	require.NoError(t, err)
+	mgr, err := NewIGWManager(IGWManagerConfig{
+		OVN:        c,
+		Routes:     policy.NewRouteManager(c),
+		NAT:        nm,
+		Allocator:  LinkLocalAllocator{},
+		Chassis:    []string{"chassis-a"},
+		NATMode:    policy.NATModeDistributed,
+		GatewayMTU: mtu,
+	})
+	require.NoError(t, err)
+	return mgr
+}
+
+func seedGatewayLRP(t *testing.T, m *mock.Client, vpcID string, opts map[string]string) {
+	t.Helper()
+	require.NoError(t, m.CreateLogicalRouterPort(context.Background(), topology.VPCRouter(vpcID), &nbdb.LogicalRouterPort{
+		Name:     topology.GatewayRouterPort(vpcID),
+		MAC:      "02:00:00:00:00:01",
+		Networks: []string{"169.254.0.1/30"},
+		Options:  opts,
+	}))
+}
+
+func seedGatewaySwitchPort(t *testing.T, m *mock.Client, vpcID string) {
+	t.Helper()
+	ctx := context.Background()
+	_, _, err := m.EnsureLogicalSwitch(ctx, &nbdb.LogicalSwitch{Name: topology.ExternalSwitchShared()})
+	require.NoError(t, err)
+	require.NoError(t, m.CreateLogicalSwitchPort(ctx, topology.ExternalSwitchShared(), &nbdb.LogicalSwitchPort{
+		Name:    topology.GatewaySwitchPort(vpcID),
+		Type:    "router",
+		Options: map[string]string{"router-port": topology.GatewayRouterPort(vpcID), natAdvertiseOption: "router"},
+	}))
+}
+
+func gatewayLRPOptions(t *testing.T, m *mock.Client, vpcID string) map[string]string {
+	t.Helper()
+	lrp, err := m.GetLogicalRouterPort(context.Background(), topology.GatewayRouterPort(vpcID))
+	require.NoError(t, err)
+	return lrp.Options
+}
+
+func TestAttachIGW_SetsGatewayMTUOnNewAttach(t *testing.T) {
+	m := mock.New()
+	seedVPCRouter(t, m, "vpc-1", "10.0.0.0/16")
+	mgr := newGatewayMTUManager(t, m, 1442)
+	require.NoError(t, mgr.AttachIGW(context.Background(), IGWSpec{VPCID: "vpc-1", InternetGatewayID: "igw-1"}))
+	assert.Equal(t, "1442", gatewayLRPOptions(t, m, "vpc-1")[gatewayMTUOption])
+}
+
+func TestAttachIGW_ConvergesGatewayMTUOnAnAttachedVPC(t *testing.T) {
+	for name, opts := range map[string]map[string]string{
+		"missing": nil,
+		"stale":   {gatewayMTUOption: "1408"},
+	} {
+		t.Run(name, func(t *testing.T) {
+			m := mock.New()
+			seedVPCRouter(t, m, "vpc-1", "10.0.0.0/16")
+			seedGatewayLRP(t, m, "vpc-1", opts)
+			seedGatewaySwitchPort(t, m, "vpc-1")
+			mgr := newGatewayMTUManager(t, m, 1442)
+			require.NoError(t, mgr.AttachIGW(context.Background(), IGWSpec{VPCID: "vpc-1", InternetGatewayID: "igw-1"}))
+			assert.Equal(t, "1442", gatewayLRPOptions(t, m, "vpc-1")[gatewayMTUOption])
+		})
+	}
+}
+
+func TestAttachIGW_SetsGatewayMTUOnExistingLRPWithoutSwitchPort(t *testing.T) {
+	m := mock.New()
+	seedVPCRouter(t, m, "vpc-1", "10.0.0.0/16")
+	seedGatewayLRP(t, m, "vpc-1", nil)
+	mgr := newGatewayMTUManager(t, m, 1442)
+	require.NoError(t, mgr.AttachIGW(context.Background(), IGWSpec{VPCID: "vpc-1", InternetGatewayID: "igw-1"}))
+	assert.Equal(t, "1442", gatewayLRPOptions(t, m, "vpc-1")[gatewayMTUOption])
+}
+
+func TestAttachIGW_ZeroGatewayMTUManagesNothing(t *testing.T) {
+	m := mock.New()
+	seedVPCRouter(t, m, "vpc-1", "10.0.0.0/16")
+	mgr := newGatewayMTUManager(t, m, 0)
+	require.NoError(t, mgr.AttachIGW(context.Background(), IGWSpec{VPCID: "vpc-1", InternetGatewayID: "igw-1"}))
+	assert.NotContains(t, gatewayLRPOptions(t, m, "vpc-1"), gatewayMTUOption)
+}
+
+func TestAttachIGW_GatewayMTUPreservesOtherLRPOptions(t *testing.T) {
+	m := mock.New()
+	seedVPCRouter(t, m, "vpc-1", "10.0.0.0/16")
+	seedGatewayLRP(t, m, "vpc-1", map[string]string{"reside-on-redirect-chassis": "true"})
+	seedGatewaySwitchPort(t, m, "vpc-1")
+	mgr := newGatewayMTUManager(t, m, 1442)
+	require.NoError(t, mgr.AttachIGW(context.Background(), IGWSpec{VPCID: "vpc-1", InternetGatewayID: "igw-1"}))
+	opts := gatewayLRPOptions(t, m, "vpc-1")
+	assert.Equal(t, "1442", opts[gatewayMTUOption])
+	assert.Equal(t, "true", opts["reside-on-redirect-chassis"])
+}
+
+func TestAttachIGW_GatewayMTUUpdateErrorNamesThePort(t *testing.T) {
+	m := mock.New()
+	seedVPCRouter(t, m, "vpc-1", "10.0.0.0/16")
+	seedGatewayLRP(t, m, "vpc-1", nil)
+	seedGatewaySwitchPort(t, m, "vpc-1")
+	mgr := newGatewayMTUManager(t, failingLRPUpdateClient{m}, 1442)
+	err := mgr.AttachIGW(context.Background(), IGWSpec{VPCID: "vpc-1", InternetGatewayID: "igw-1"})
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), topology.GatewayRouterPort("vpc-1"))
+}

spinifex/domains/network/host/ipsec_cert.go

@@ -48,7 +48,7 @@ func IPSecCertPaths(configDir string) (certPath, keyPath string) {
 // GenerateIPSecPeerCert issues an IKEv2 peer certificate for this node signed
 // by the existing cluster CA. The cert carries the IPSec-IKE EKU so strongSwan
 // accepts it. The dnsName SAN must equal the OVS chassis-id (system-id).
-func GenerateIPSecPeerCert(configDir, caCertPath, caKeyPath, hostname, nodeIP string) error {
+func GenerateIPSecPeerCert(configDir, caCertPath, caKeyPath, hostname, nodeIP string, libreswan bool) error {
 	if hostname == "" {
 		return fmt.Errorf("ipsec peer cert: hostname required")
 	}
@@ -115,8 +115,11 @@ func GenerateIPSecPeerCert(configDir, caCertPath, caKeyPath, hostname, nodeIP st
 		return fmt.Errorf("ipsec peer cert: write key: %w", err)
 	}
 
-	// strongSwan 6.0 only loads CAs placed in /etc/ipsec.d/cacerts at startup;
-	// per-conn cacert= references are insufficient. Install the cluster CA there.
+	// strongSwan 6.0 only loads CAs placed in /etc/ipsec.d/cacerts at startup.
+	// On libreswan ovs-monitor-ipsec imports the CA into NSS from other_config.
+	if libreswan {
+		return nil
+	}
 	if err := installCAIntoCharonTrustStore(caCertPath); err != nil {
 		return fmt.Errorf("ipsec peer cert: install CA into charon trust store: %w", err)
 	}

spinifex/domains/network/host/ipsec_cert_test.go

@@ -68,7 +68,7 @@ func TestGenerateIPSecPeerCert(t *testing.T) {
 	t.Run("happy path", func(t *testing.T) {
 		t.Parallel()
 		dir := t.TempDir()
-		require.NoError(t, GenerateIPSecPeerCert(dir, caCertPath, caKeyPath, "node-1", "10.0.0.5"))
+		require.NoError(t, GenerateIPSecPeerCert(dir, caCertPath, caKeyPath, "node-1", "10.0.0.5", false))
 
 		certPath, keyPath := IPSecCertPaths(dir)
 		certPEM, err := os.ReadFile(certPath)
@@ -109,8 +109,8 @@ func TestGenerateIPSecPeerCert(t *testing.T) {
 	t.Run("idempotent overwrite", func(t *testing.T) {
 		t.Parallel()
 		dir := t.TempDir()
-		require.NoError(t, GenerateIPSecPeerCert(dir, caCertPath, caKeyPath, "node-2", "10.0.0.6"))
-		require.NoError(t, GenerateIPSecPeerCert(dir, caCertPath, caKeyPath, "node-2", "10.0.0.7"))
+		require.NoError(t, GenerateIPSecPeerCert(dir, caCertPath, caKeyPath, "node-2", "10.0.0.6", false))
+		require.NoError(t, GenerateIPSecPeerCert(dir, caCertPath, caKeyPath, "node-2", "10.0.0.7", false))
 
 		certPath, _ := IPSecCertPaths(dir)
 		certPEM, _ := os.ReadFile(certPath)
@@ -122,7 +122,7 @@ func TestGenerateIPSecPeerCert(t *testing.T) {
 	t.Run("rejects missing hostname", func(t *testing.T) {
 		t.Parallel()
 		dir := t.TempDir()
-		err := GenerateIPSecPeerCert(dir, caCertPath, caKeyPath, "", "10.0.0.5")
+		err := GenerateIPSecPeerCert(dir, caCertPath, caKeyPath, "", "10.0.0.5", false)
 		require.Error(t, err)
 		assert.Contains(t, err.Error(), "hostname required")
 	})
@@ -130,7 +130,7 @@ func TestGenerateIPSecPeerCert(t *testing.T) {
 	t.Run("rejects missing nodeIP", func(t *testing.T) {
 		t.Parallel()
 		dir := t.TempDir()
-		err := GenerateIPSecPeerCert(dir, caCertPath, caKeyPath, "node-1", "")
+		err := GenerateIPSecPeerCert(dir, caCertPath, caKeyPath, "node-1", "", false)
 		require.Error(t, err)
 		assert.Contains(t, err.Error(), "nodeIP required")
 	})
@@ -138,7 +138,7 @@ func TestGenerateIPSecPeerCert(t *testing.T) {
 	t.Run("rejects invalid nodeIP", func(t *testing.T) {
 		t.Parallel()
 		dir := t.TempDir()
-		err := GenerateIPSecPeerCert(dir, caCertPath, caKeyPath, "node-1", "not-an-ip")
+		err := GenerateIPSecPeerCert(dir, caCertPath, caKeyPath, "node-1", "not-an-ip", false)
 		require.Error(t, err)
 		assert.Contains(t, err.Error(), "invalid nodeIP")
 	})
@@ -146,7 +146,7 @@ func TestGenerateIPSecPeerCert(t *testing.T) {
 	t.Run("missing CA file", func(t *testing.T) {
 		t.Parallel()
 		dir := t.TempDir()
-		err := GenerateIPSecPeerCert(dir, filepath.Join(dir, "ghost-ca.pem"), caKeyPath, "node-1", "10.0.0.5")
+		err := GenerateIPSecPeerCert(dir, filepath.Join(dir, "ghost-ca.pem"), caKeyPath, "node-1", "10.0.0.5", false)
 		require.Error(t, err)
 		assert.Contains(t, err.Error(), "read CA cert")
 	})
@@ -213,3 +213,31 @@ func TestInstallCAIntoCharonTrustStore_SwallowsRereadError(t *testing.T) {
 type stubError string
 
 func (e stubError) Error() string { return string(e) }
+
+// ovs-monitor-ipsec imports the CA into libreswan's NSS database itself, so
+// strongSwan's cacerts directory and rereadcacerts stay untouched there.
+func TestGenerateIPSecPeerCert_LibreswanLeavesCharonTrustStoreAlone(t *testing.T) {
+	origTrustDir := charonCATrustDir
+	charonCATrustDir = filepath.Join(t.TempDir(), "cacerts")
+	t.Cleanup(func() { charonCATrustDir = origTrustDir })
+
+	origReread := charonRereadCAs
+	charonRereadCAs = func() error {
+		t.Error("ipsec rereadcacerts run on a libreswan host")
+		return nil
+	}
+	t.Cleanup(func() { charonRereadCAs = origReread })
+
+	caDir := t.TempDir()
+	caCertPath := filepath.Join(caDir, "ca.pem")
+	caKeyPath := filepath.Join(caDir, "ca.key")
+	writeTestCA(t, caCertPath, caKeyPath)
+
+	dir := t.TempDir()
+	require.NoError(t, GenerateIPSecPeerCert(dir, caCertPath, caKeyPath, "node-1", "10.0.0.5", true))
+
+	certPath, keyPath := IPSecCertPaths(dir)
+	assert.FileExists(t, certPath)
+	assert.FileExists(t, keyPath)
+	assert.NoDirExists(t, charonCATrustDir, "strongSwan trust store created on a libreswan host")
+}

spinifex/domains/network/host/ipsec_enable.go

@@ -68,16 +68,17 @@ type IPSecBarrier interface {
 }
 
 const (
-	// ovsIPSecUnit owns charon: ovs-monitor-ipsec execs the strongSwan starter
-	// itself rather than going through the starter's own unit.
+	// ovsIPSecUnit owns charon on strongSwan: ovs-monitor-ipsec execs the
+	// starter itself rather than going through the starter's own unit.
 	ovsIPSecUnit = "openvswitch-ipsec.service"
 
 	// strongswanStarterUnit would start a second charon competing for UDP
 	// 500/4500. Nothing here needs it, so it stays off in both states.
 	strongswanStarterUnit = "strongswan-starter.service"
 
-	// libreswanUnit runs pluto, which listens on UDP 500/4500 once started. On
-	// Debian the same name is an alias of strongswanStarterUnit.
+	// libreswanUnit runs pluto, which ovs-monitor-ipsec drives but does not
+	// start, so it is on exactly while IPsec is. On Debian the same name is an
+	// alias of strongswanStarterUnit.
 	libreswanUnit = "ipsec.service"
 )
 
@@ -94,17 +95,15 @@ var ipsecStateHelper = "/usr/local/lib/spinifex/spinifex-set-ipsec-state"
 // unconfigured until something restarts the daemon — while another node may
 // already have made encryption mandatory cluster-wide.
 //
-// libreswan reports that the host's IPsec stack is libreswan, which OVN IPsec
-// does not run on yet, so such a host keeps IPsec off whatever the cluster asks.
+// libreswan reports that the host's IPsec stack is libreswan, whose pluto runs
+// under its own unit rather than being execed by ovs-monitor-ipsec.
 func MaintainIPSec(ctx context.Context, configPath string, clusterConfig *config.ClusterConfig, barrier IPSecBarrier, libreswan bool) {
 	// Said once, because a pass that reconciles to nothing logs nothing, which in
 	// a journal is indistinguishable from a node that armed.
 	switch {
 	case clusterConfig == nil:
 		slog.Info("ipsec: cluster membership unknown, leaving the host's IPsec services alone")
-	case ipsecRequested(clusterConfig) && libreswan:
-		slog.Error("ipsec: the cluster requires IPsec, but this host's IPsec stack is libreswan, which Spinifex cannot run OVN IPsec on yet; IPsec stays off here")
-	case !ipsecWanted(clusterConfig, libreswan):
+	case !ipsecRequested(clusterConfig):
 		slog.Info("ipsec: not in use on this cluster, IKE and NAT-T listeners stay stopped")
 	}
 
@@ -119,7 +118,7 @@ func MaintainIPSec(ctx context.Context, configPath string, clusterConfig *config
 			// Only the first success is news. The steady state re-runs every
 			// interval for the lifetime of the daemon, and saying so each time
 			// buries the passes that changed something.
-			if !announced && ipsecWanted(clusterConfig, libreswan) {
+			if !announced && ipsecRequested(clusterConfig) {
 				slog.Info("ipsec: local IPsec configuration complete on this node")
 				announced = true
 			}
@@ -134,12 +133,6 @@ func MaintainIPSec(ctx context.Context, configPath string, clusterConfig *config
 	}
 }
 
-// ipsecWanted reports whether this host turns IPsec on: the cluster asks for
-// it and the host's IPsec stack can run it.
-func ipsecWanted(clusterConfig *config.ClusterConfig, libreswan bool) bool {
-	return ipsecRequested(clusterConfig) && !libreswan
-}
-
 // ipsecRequested reports the cluster's intent. Single-node clusters have no Geneve
 // tunnels to protect, so charon must not be left listening on 500/4500 there
 // even though ipsec_enabled defaults to true.
@@ -157,7 +150,7 @@ func ReconcileOVNIPSec(ctx context.Context, configPath string, clusterConfig *co
 		return nil
 	}
 
-	want := ipsecWanted(clusterConfig, libreswan)
+	want := ipsecRequested(clusterConfig)
 
 	if err := ensureIPSecServices(want, libreswan); err != nil {
 		return err
@@ -165,7 +158,22 @@ func ReconcileOVNIPSec(ctx context.Context, configPath string, clusterConfig *co
 	if !want {
 		return DisableOVNIPSec(ctx, clusterConfig, barrier)
 	}
-	return EnableOVNIPSec(ctx, configPath, clusterConfig, barrier)
+	err := EnableOVNIPSec(ctx, configPath, clusterConfig, barrier)
+	if libreswan {
+		err = errors.Join(err, guardPlaintextGeneve())
+	}
+	return err
+}
+
+// guardPlaintextGeneve re-syncs the helper's nft drop on Geneve, which keeps a
+// tunnel whose SA is down from falling back to plaintext. Every pass, because
+// the tunnel set follows cluster membership and NB_Global.ipsec.
+func guardPlaintextGeneve() error {
+	out, err := hostcommand.SudoCommand(ipsecStateHelper, "guard").CombinedOutput()
+	if err != nil {
+		return fmt.Errorf("guard plaintext geneve: %w: %s", err, strings.TrimSpace(string(out)))
+	}
+	return nil
 }
 
 // DisableOVNIPSec is the off direction, and it has to reach NB_Global. Charon is
@@ -221,14 +229,21 @@ func ensureIPSecServices(want, libreswan bool) error {
 	return nil
 }
 
+// ipsecServicesMatch reads only the units the host's stack has: on libreswan
+// pluto runs under its own unit and follows the wanted state with
+// openvswitch-ipsec, while on Debian ipsec.service is strongSwan's alias.
 func ipsecServicesMatch(want, libreswan bool) bool {
 	if unitIsEnabled(strongswanStarterUnit) || unitIsActive(strongswanStarterUnit) {
 		return false
 	}
-	if !want && libreswan && (unitIsEnabled(libreswanUnit) || unitIsActive(libreswanUnit)) {
+	if libreswan && !unitInState(libreswanUnit, want) {
 		return false
 	}
-	return unitIsEnabled(ovsIPSecUnit) == want && unitIsActive(ovsIPSecUnit) == want
+	return unitInState(ovsIPSecUnit, want)
+}
+
+func unitInState(unit string, want bool) bool {
+	return unitIsEnabled(unit) == want && unitIsActive(unit) == want
 }
 
 // unitIsEnabled treats every state other than enabled as not enabled, which

spinifex/domains/network/host/ipsec_enable_test.go

@@ -43,6 +43,8 @@ type recordingSudo struct {
 	// change on the socket, a missing binary, a timed-out transaction. Each of
 	// those is a real fault.
 	nbError string
+	// guardError makes the helper's guard verb fail with this message.
+	guardError string
 }
 
 func (r *recordingSudo) stub(name string, args ...string) *exec.Cmd {
@@ -60,6 +62,9 @@ func (r *recordingSudo) stub(name string, args ...string) *exec.Cmd {
 		}
 		return exec.Command("true")
 	}
+	if name == ipsecStateHelper && slices.Equal(args, []string{"guard"}) && r.guardError != "" {
+		return exec.Command("sh", "-c", `echo "`+r.guardError+`" >&2; exit 1`)
+	}
 	if name != "systemctl" || len(args) < 2 {
 		return exec.Command("true")
 	}
@@ -497,22 +502,92 @@ func TestReconcileOVNIPSec_LibreswanAlreadyOffIsNoOp(t *testing.T) {
 	assert.Empty(t, recorder.helperRuns())
 }
 
-// OVN IPsec does not run on libreswan yet, so a cluster that asks for it gets
-// the off state on such a host rather than a half-armed one.
-func TestReconcileOVNIPSec_LibreswanStaysOffWhenTheClusterAsksForIPsec(t *testing.T) {
+// ipsecOnLibreswanHost is a libreswan host already in the on state, with
+// strongSwan absent.
+func ipsecOnLibreswanHost() *recordingSudo {
+	return &recordingSudo{
+		enabledOutput: map[string]string{
+			strongswanStarterUnit: "Failed to get unit file state for strongswan-starter.service: No such file or directory\n",
+			ovsIPSecUnit:          "enabled\n",
+			libreswanUnit:         "enabled\n",
+		},
+		activePerUnit: map[string]string{
+			strongswanStarterUnit: "inactive\n",
+			ovsIPSecUnit:          "active\n",
+			libreswanUnit:         "active\n",
+		},
+		nbUnreachable: true,
+	}
+}
+
+func TestReconcileOVNIPSec_LibreswanTurnsIPsecOnWhenTheClusterAsks(t *testing.T) {
 	recorder := ipsecOffHost()
-	recorder.activePerUnit[libreswanUnit] = "active\n"
+	recorder.nbUnreachable = true
+	// The probe after the helper ran; the static stub cannot change state mid-pass.
+	recorder.activePerUnit[ovsIPSecUnit] = "active\n"
 	t.Cleanup(hostcommand.SetSudoCommandForTest(recorder.stub))
 
 	configPath := filepath.Join(ipsecTestConfigDir(t), "spinifex.toml")
 	require.NoError(t, ReconcileOVNIPSec(t.Context(), configPath, multiNodeIPSecConfig(true), allReady(), true))
 
-	assert.Equal(t, [][]string{{ipsecStateHelper, "off"}}, recorder.helperRuns())
-	for _, run := range recorder.runs {
-		assert.NotEqual(t, "ovs-vsctl", run[0], "must not configure OVS IPsec on libreswan: %v", run)
+	assert.Equal(t, [][]string{{ipsecStateHelper, "on"}, {ipsecStateHelper, "guard"}}, recorder.helperRuns())
+	assert.Contains(t, strings.Join(recorder.ovsSets(), "\n"), "ipsec_encapsulation=true")
+}
+
+// ovs-monitor-ipsec does not start pluto on EL, so openvswitch-ipsec running
+// alone is not the on state.
+func TestReconcileOVNIPSec_LibreswanPlutoStoppedIsTurnedOn(t *testing.T) {
+	for _, tc := range []struct{ name, enabled, active string }{
+		{name: "inactive", enabled: "enabled\n", active: "inactive\n"},
+		{name: "masked", enabled: "masked\n", active: "inactive\n"},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			recorder := ipsecOnLibreswanHost()
+			recorder.enabledOutput[libreswanUnit] = tc.enabled
+			recorder.activePerUnit[libreswanUnit] = tc.active
+			t.Cleanup(hostcommand.SetSudoCommandForTest(recorder.stub))
+
+			configPath := filepath.Join(ipsecTestConfigDir(t), "spinifex.toml")
+			require.NoError(t, ReconcileOVNIPSec(t.Context(), configPath, multiNodeIPSecConfig(true), allReady(), true))
+			assert.Equal(t, [][]string{{ipsecStateHelper, "on"}, {ipsecStateHelper, "guard"}}, recorder.helperRuns())
+		})
 	}
 }
 
+// The units are left alone in the steady state. The guard is not: a peer that
+// joined since the last pass has a tunnel the block does not cover yet.
+func TestReconcileOVNIPSec_LibreswanAlreadyOnOnlyResyncsTheGuard(t *testing.T) {
+	recorder := ipsecOnLibreswanHost()
+	t.Cleanup(hostcommand.SetSudoCommandForTest(recorder.stub))
+
+	configPath := filepath.Join(ipsecTestConfigDir(t), "spinifex.toml")
+	require.NoError(t, ReconcileOVNIPSec(t.Context(), configPath, multiNodeIPSecConfig(true), allReady(), true))
+	assert.Equal(t, [][]string{{ipsecStateHelper, "guard"}}, recorder.helperRuns())
+}
+
+// A guard that did not go in leaves plaintext Geneve possible, so the pass must
+// fail and retry rather than report IPsec configured.
+func TestReconcileOVNIPSec_LibreswanGuardFailureFailsThePass(t *testing.T) {
+	recorder := ipsecOnLibreswanHost()
+	recorder.guardError = "ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed"
+	t.Cleanup(hostcommand.SetSudoCommandForTest(recorder.stub))
+
+	configPath := filepath.Join(ipsecTestConfigDir(t), "spinifex.toml")
+	err := ReconcileOVNIPSec(t.Context(), configPath, multiNodeIPSecConfig(true), allReady(), true)
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "database connection failed")
+}
+
+// The block outlives pluto, so a host that turned IPsec off must not keep
+// re-adding it; the helper's off verb is what removes it.
+func TestReconcileOVNIPSec_LibreswanOffNeverGuards(t *testing.T) {
+	recorder := ipsecOnLibreswanHost()
+	t.Cleanup(hostcommand.SetSudoCommandForTest(recorder.stub))
+
+	require.NoError(t, ReconcileOVNIPSec(t.Context(), "/etc/spinifex/spinifex.toml", multiNodeIPSecConfig(false), allReady(), true))
+	assert.Equal(t, [][]string{{ipsecStateHelper, "off"}}, recorder.helperRuns())
+}
+
 // On Debian ipsec.service is an alias of strongswan-starter, which is already
 // checked under its own name; reading the alias again must not change the answer.
 func TestReconcileOVNIPSec_WithoutLibreswanIPsecServiceIsNotRead(t *testing.T) {

spinifex/domains/network/host/mssclamp.go

@@ -0,0 +1,84 @@
+package host
+
+import (
+	"context"
+	"fmt"
+	"log/slog"
+	"strings"
+)
+
+const (
+	mssClampTable      = "spinifex_mss"
+	mssClampV4Overhead = 40
+	mssClampV6Overhead = 60
+)
+
+// mssClampRuleset returns the nft batch replacing our MSS clamp table for
+// mode ("" when the mode has no host hop). It also drops the other family's
+// table, so a mode change cannot leave a stale clamp.
+func mssClampRuleset(mode UplinkMode, gatewayMTU int) string {
+	v4 := gatewayMTU - mssClampV4Overhead
+	v6 := gatewayMTU - mssClampV6Overhead
+	var family, stale, port, v4match, v6match string
+	switch mode {
+	case UplinkModeVeth:
+		// The bridge family drops `ip protocol tcp` from a rule on list, so the v4 and
+		// v6 rules would both match every packet; `ether type` keeps them apart.
+		family, stale, port = "bridge", "inet", VethLinuxEnd
+		v4match, v6match = "ether type ip", "ether type ip6"
+	case UplinkModeRouted:
+		family, stale, port = "inet", "bridge", NATTransitHostEnd
+		v4match, v6match = "meta nfproto ipv4", "meta nfproto ipv6"
+	default:
+		return ""
+	}
+	var b strings.Builder
+	fmt.Fprintf(&b, "add table %s %s\ndelete table %s %s\n", stale, mssClampTable, stale, mssClampTable)
+	fmt.Fprintf(&b, "add table %s %s\ndelete table %s %s\n", family, mssClampTable, family, mssClampTable)
+	fmt.Fprintf(&b, "table %s %s {\n\tchain clamp {\n\t\ttype filter hook forward priority -150; policy accept;\n", family, mssClampTable)
+	for _, rule := range []struct {
+		match string
+		mss   int
+	}{{v4match, v4}, {v6match, v6}} {
+		for _, dir := range []string{"iifname", "oifname"} {
+			fmt.Fprintf(&b, "\t\t%s %q %s tcp flags & (syn | rst) == syn tcp option maxseg size > %d tcp option maxseg size set %d\n",
+				dir, port, rule.match, rule.mss, rule.mss)
+		}
+	}
+	b.WriteString("\t}\n}\n")
+	return b.String()
+}
+
+// mssClampTeardown drops both families' tables; add-then-delete makes an
+// absent table a no-op instead of an error.
+const mssClampTeardown = "add table bridge " + mssClampTable + "\ndelete table bridge " + mssClampTable + "\n" +
+	"add table inet " + mssClampTable + "\ndelete table inet " + mssClampTable + "\n"
+
+// EnsureMSSClamp clamps SYN and SYN-ACK MSS across the host hop of the WAN
+// uplink to gatewayMTU minus the headers, for peers that never see our ICMP.
+// The batch is one nft argument, which nft applies as a single transaction.
+func EnsureMSSClamp(ctx context.Context, r Runner, mode UplinkMode, gatewayMTU int) error {
+	if mode == UplinkModePhysical {
+		if err := RemoveMSSClamp(ctx, r); err != nil {
+			slog.Debug("host: clear stale MSS clamp failed", "mode", mode, "err", err)
+		}
+		return nil
+	}
+	rules := mssClampRuleset(mode, gatewayMTU)
+	if rules == "" || gatewayMTU <= 0 {
+		return nil
+	}
+	if out, err := r.Run(ctx, "nft", rules); err != nil {
+		return fmt.Errorf("install MSS clamp (%s mode): %s: %w", mode, strings.TrimSpace(string(out)), err)
+	}
+	slog.Debug("host: ensured MSS clamp", "mode", mode, "mss_v4", gatewayMTU-mssClampV4Overhead, "mss_v6", gatewayMTU-mssClampV6Overhead)
+	return nil
+}
+
+// RemoveMSSClamp deletes our MSS clamp tables; absent tables are not an error.
+func RemoveMSSClamp(ctx context.Context, r Runner) error {
+	if out, err := r.Run(ctx, "nft", mssClampTeardown); err != nil {
+		return fmt.Errorf("remove MSS clamp: %s: %w", strings.TrimSpace(string(out)), err)
+	}
+	return nil
+}

spinifex/domains/network/host/mssclamp_test.go

@@ -0,0 +1,157 @@
+package host
+
+import (
+	"context"
+	"errors"
+	"strings"
+	"testing"
+)
+
+const vethClamp1408 = `add table inet spinifex_mss
+delete table inet spinifex_mss
+add table bridge spinifex_mss
+delete table bridge spinifex_mss
+table bridge spinifex_mss {
+	chain clamp {
+		type filter hook forward priority -150; policy accept;
+		iifname "veth-wan-br" ether type ip tcp flags & (syn | rst) == syn tcp option maxseg size > 1368 tcp option maxseg size set 1368
+		oifname "veth-wan-br" ether type ip tcp flags & (syn | rst) == syn tcp option maxseg size > 1368 tcp option maxseg size set 1368
+		iifname "veth-wan-br" ether type ip6 tcp flags & (syn | rst) == syn tcp option maxseg size > 1348 tcp option maxseg size set 1348
+		oifname "veth-wan-br" ether type ip6 tcp flags & (syn | rst) == syn tcp option maxseg size > 1348 tcp option maxseg size set 1348
+	}
+}
+`
+
+const routedClamp1442 = `add table bridge spinifex_mss
+delete table bridge spinifex_mss
+add table inet spinifex_mss
+delete table inet spinifex_mss
+table inet spinifex_mss {
+	chain clamp {
+		type filter hook forward priority -150; policy accept;
+		iifname "spx-nat-host" meta nfproto ipv4 tcp flags & (syn | rst) == syn tcp option maxseg size > 1402 tcp option maxseg size set 1402
+		oifname "spx-nat-host" meta nfproto ipv4 tcp flags & (syn | rst) == syn tcp option maxseg size > 1402 tcp option maxseg size set 1402
+		iifname "spx-nat-host" meta nfproto ipv6 tcp flags & (syn | rst) == syn tcp option maxseg size > 1382 tcp option maxseg size set 1382
+		oifname "spx-nat-host" meta nfproto ipv6 tcp flags & (syn | rst) == syn tcp option maxseg size > 1382 tcp option maxseg size set 1382
+	}
+}
+`
+
+func TestMSSClampRuleset(t *testing.T) {
+	tests := []struct {
+		name string
+		mode UplinkMode
+		mtu  int
+		want string
+	}{
+		{"veth ipsec on", UplinkModeVeth, 1408, vethClamp1408},
+		{"routed ipsec off", UplinkModeRouted, 1442, routedClamp1442},
+		{"physical has no host hop", UplinkModePhysical, 1408, ""},
+		{"unknown mode", UplinkModeUnknown, 1408, ""},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if got := mssClampRuleset(tt.mode, tt.mtu); got != tt.want {
+				t.Errorf("ruleset mismatch\ngot:\n%s\nwant:\n%s", got, tt.want)
+			}
+		})
+	}
+}
+
+func TestMSSClampRuleset_FollowsGatewayMTU(t *testing.T) {
+	got := mssClampRuleset(UplinkModeVeth, 1442)
+	for _, want := range []string{"size set 1402", "size set 1382"} {
+		if !strings.Contains(got, want) {
+			t.Errorf("1442 ruleset missing %q:\n%s", want, got)
+		}
+	}
+	if strings.Contains(got, "1368") || strings.Contains(got, "1348") {
+		t.Errorf("1442 ruleset carries the 1408 values:\n%s", got)
+	}
+}
+
+func TestEnsureMSSClamp_AppliesOneTransaction(t *testing.T) {
+	r := newStubRunner()
+	r.expect("nft ", nil, nil)
+	if err := EnsureMSSClamp(context.Background(), r, UplinkModeVeth, 1408); err != nil {
+		t.Fatalf("EnsureMSSClamp: %v", err)
+	}
+	if len(r.calls) != 1 || r.calls[0] != "nft "+vethClamp1408 {
+		t.Errorf("want exactly one nft call carrying the full batch, got %q", r.calls)
+	}
+}
+
+func TestEnsureMSSClamp_NoOpCases(t *testing.T) {
+	tests := map[string]struct {
+		mode UplinkMode
+		mtu  int
+	}{
+		"unknown mode": {UplinkModeUnknown, 1408},
+		"zero gateway": {UplinkModeVeth, 0},
+		"negative":     {UplinkModeRouted, -1},
+	}
+	for name, tc := range tests {
+		t.Run(name, func(t *testing.T) {
+			r := newStubRunner()
+			if err := EnsureMSSClamp(context.Background(), r, tc.mode, tc.mtu); err != nil {
+				t.Fatalf("EnsureMSSClamp: %v", err)
+			}
+			if len(r.calls) != 0 {
+				t.Errorf("expected no nft call, got %q", r.calls)
+			}
+		})
+	}
+}
+
+func TestEnsureMSSClamp_PhysicalClearsStaleTables(t *testing.T) {
+	r := newStubRunner()
+	r.expect("nft ", nil, nil)
+	if err := EnsureMSSClamp(context.Background(), r, UplinkModePhysical, 1408); err != nil {
+		t.Fatalf("EnsureMSSClamp: %v", err)
+	}
+	if len(r.calls) != 1 || r.calls[0] != "nft "+mssClampTeardown {
+		t.Errorf("want one teardown call, got %q", r.calls)
+	}
+	if strings.Contains(r.calls[0], "chain clamp") {
+		t.Errorf("physical mode must not define a table: %s", r.calls[0])
+	}
+}
+
+func TestEnsureMSSClamp_PhysicalToleratesTeardownFailure(t *testing.T) {
+	r := newStubRunner()
+	r.expect("nft ", nil, errors.New("exec: nft not found"))
+	if err := EnsureMSSClamp(context.Background(), r, UplinkModePhysical, 1408); err != nil {
+		t.Errorf("a failed best-effort cleanup must not fail physical mode: %v", err)
+	}
+}
+
+func TestEnsureMSSClamp_WrapsRunnerErrorWithMode(t *testing.T) {
+	r := newStubRunner()
+	r.expect("nft ", []byte("Error: Operation not permitted"), errors.New("exit status 1"))
+	err := EnsureMSSClamp(context.Background(), r, UplinkModeRouted, 1408)
+	if err == nil {
+		t.Fatal("expected an error")
+	}
+	for _, want := range []string{"routed mode", "Operation not permitted", "exit status 1"} {
+		if !strings.Contains(err.Error(), want) {
+			t.Errorf("error %q missing %q", err, want)
+		}
+	}
+}
+
+func TestRemoveMSSClamp(t *testing.T) {
+	r := newStubRunner()
+	r.expect("nft ", nil, nil)
+	if err := RemoveMSSClamp(context.Background(), r); err != nil {
+		t.Fatalf("RemoveMSSClamp: %v", err)
+	}
+	if len(r.calls) != 1 || r.calls[0] != "nft "+mssClampTeardown {
+		t.Errorf("want one teardown call, got %q", r.calls)
+	}
+
+	r = newStubRunner()
+	r.expect("nft ", []byte("denied"), errors.New("exit status 1"))
+	if err := RemoveMSSClamp(context.Background(), r); err == nil || !strings.Contains(err.Error(), "denied") {
+		t.Errorf("runner failure must be returned with output, got %v", err)
+	}
+}

spinifex/domains/network/reconcile/apply_test.go

@@ -7,6 +7,7 @@ import (
 	"net"
 	"net/netip"
 	"slices"
+	"strconv"
 	"strings"
 	"testing"
 
@@ -1628,3 +1629,77 @@ func TestPassResult_SummaryCountsPerClass(t *testing.T) {
 		t.Errorf("summaryKV() = %v, want %v", got, want)
 	}
 }
+
+// A gateway attached before gateway_mtu existed, or under a different IPsec
+// setting, must converge on the next reconcile pass.
+func TestReconcile_ConvergesGatewayMTUOnAnAttachedVPC(t *testing.T) {
+	ctx := context.Background()
+	m := mock.New()
+	sg := policy.NewSecurityGroupManager(m, policy.EgressPolicy{})
+	nat, err := policy.NewNATManager(m, policy.NATModeDistributed)
+	if err != nil {
+		t.Fatalf("NewNATManager: %v", err)
+	}
+	routes := policy.NewRouteManager(m)
+	newRec := func(gatewayMTU int) (Reconciler, external.IGWManager) {
+		igw, err := external.NewIGWManager(external.IGWManagerConfig{
+			OVN: m, Routes: routes, NAT: nat,
+			Allocator:  external.LinkLocalAllocator{},
+			NATMode:    policy.NATModeDistributed,
+			GatewayMTU: gatewayMTU,
+		})
+		if err != nil {
+			t.Fatalf("NewIGWManager: %v", err)
+		}
+		rec, err := New(Config{
+			OVN: m, SG: sg, NAT: nat, Routes: routes, IGW: igw, Topology: topology.NewLiveManager(m),
+			LocalAZ: "us-east-1a", NodeHostname: "test-host",
+		})
+		if err != nil {
+			t.Fatalf("New: %v", err)
+		}
+		return rec, igw
+	}
+	intent := IntentState{
+		VPCs: map[string]topology.VPCSpec{
+			"vpc-a": {VPCID: "vpc-a", CIDR: netip.MustParsePrefix("10.0.0.0/16"), VNI: 100},
+		},
+		Subnets: map[string]topology.SubnetSpec{},
+		Ports:   map[string]topology.PortSpec{},
+		SGs:     map[string]policy.SGSpec{},
+		IGWs:    map[string]external.IGWSpec{"vpc-a": {VPCID: "vpc-a", InternetGatewayID: "igw-a"}},
+		EIPs:    map[string]policy.EIPSpec{},
+		NATGWs:  map[string]policy.NATGWSpec{},
+	}
+	gwMTU := func() (string, bool) {
+		lrp, err := m.GetLogicalRouterPort(ctx, topology.GatewayRouterPort("vpc-a"))
+		if err != nil {
+			t.Fatalf("GetLogicalRouterPort: %v", err)
+		}
+		v, ok := lrp.Options["gateway_mtu"]
+		return v, ok
+	}
+
+	// Attach with the option unmanaged, as a pre-upgrade gateway would be.
+	rec, _ := newRec(0)
+	if err := rec.Reconcile(ctx, intent); err != nil {
+		t.Fatalf("seed reconcile: %v", err)
+	}
+	if v, ok := gwMTU(); ok {
+		t.Fatalf("seeded gateway already has gateway_mtu=%q", v)
+	}
+
+	for _, want := range []string{
+		strconv.Itoa(topology.GatewayMTU(topology.DefaultUnderlayMTU, false)),
+		strconv.Itoa(topology.GatewayMTU(topology.DefaultUnderlayMTU, true)),
+	} {
+		mtu, _ := strconv.Atoi(want)
+		rec, _ := newRec(mtu)
+		if err := rec.Reconcile(ctx, intent); err != nil {
+			t.Fatalf("Reconcile with GatewayMTU %s: %v", want, err)
+		}
+		if got, _ := gwMTU(); got != want {
+			t.Errorf("gateway_mtu = %q after reconcile, want %q", got, want)
+		}
+	}
+}

spinifex/domains/network/topology/dhcp_options.go

@@ -48,6 +48,15 @@ func SubnetMTU(underlayMTU int, ipsecEnabled bool) int {
 	return mtu
 }
 
+// internetPathMTU is the path MTU of the public internet and WAN links.
+const internetPathMTU = 1500
+
+// GatewayMTU returns the MTU for the VPC gateway's external-facing side:
+// the guest MTU, capped at the internet path MTU since no WAN path carries more.
+func GatewayMTU(underlayMTU int, ipsecEnabled bool) int {
+	return min(SubnetMTU(underlayMTU, ipsecEnabled), internetPathMTU)
+}
+
 // BuildSubnetDHCPOptions returns the OVN DHCPOptions map for a subnet. Shared
 // by the live manager and reconciler to prevent dns_server drift. IMDS is not
 // steered via option 121: a guest either routes to it via the gateway, where the

spinifex/domains/network/topology/dhcp_options_test.go

@@ -54,6 +54,24 @@ func TestSubnetMTU_ClampsImplausibleUnderlay(t *testing.T) {
 	}
 }
 
+func TestGatewayMTU(t *testing.T) {
+	tests := []struct {
+		underlay int
+		ipsec    bool
+		want     int
+	}{
+		{1500, false, 1442},
+		{1500, true, 1408},
+		{9000, false, 1500},
+		{9000, true, 1500},
+	}
+	for _, tt := range tests {
+		if got := GatewayMTU(tt.underlay, tt.ipsec); got != tt.want {
+			t.Errorf("GatewayMTU(%d, ipsec=%v) = %d, want %d", tt.underlay, tt.ipsec, got, tt.want)
+		}
+	}
+}
+
 func TestBuildSubnetDHCPOptions_MTUFollowsIPSec(t *testing.T) {
 	on := BuildSubnetDHCPOptions("10.0.1.1", "02:00:00:00:00:01", "{8.8.8.8}", DefaultUnderlayMTU, true)
 	if got := on["mtu"]; got != "1408" {

spinifex/domains/sqs/message/conformance/conformance.go

@@ -0,0 +1,781 @@
+// Package conformance is the definition of a correct SQS message store
+// (ADR-0011:S6). It drives message.Store only through the interface, so the
+// same cases run unmodified against every implementation.
+package conformance
+
+import (
+	"context"
+	"crypto/rand"
+	"encoding/hex"
+	"errors"
+	"fmt"
+	"math"
+	"strings"
+	"sync"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/mulgadc/bluebottle/pkg/safecast"
+	"github.com/mulgadc/spinifex/spinifex/domains/sqs/message"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// Harness is one store under test and the hooks its implementation can offer.
+type Harness struct {
+	Store message.Store
+	// MessageShards and StateShards are how many of each allocation the
+	// store serves, numbered from zero; the cross-shard case needs two
+	// message shards.
+	MessageShards int
+	StateShards   int
+	// Reopen discards Store without closing it, as a crash would, and
+	// returns a store over the same durable data. Nil skips the crash cases.
+	Reopen func(t *testing.T) message.Store
+	// Advance moves the store's clock forward. Nil skips the backstop case.
+	Advance func(d time.Duration)
+}
+
+// Factory builds a fresh harness for one case.
+type Factory func(t *testing.T) Harness
+
+// feedWait bounds how long a case waits for a change it expects.
+const feedWait = 5 * time.Second
+
+// Run executes every contract case against stores built by newHarness. Cases
+// use fresh queue IDs, so one long-lived store may back every case.
+func Run(t *testing.T, newHarness Factory) {
+	cases := []struct {
+		name string
+		run  func(t *testing.T, h Harness)
+	}{
+		{"Limits", testLimits},
+		{"InvalidNamespaceRejected", testInvalidNamespace},
+		{"QueueCreationNeedsNoStoreCall", testNoCreateCall},
+		{"AppendAndGet", testAppendAndGet},
+		{"GetMissing", testGetMissing},
+		{"DeleteIsIdempotentAndPositionsAreNotReused", testDelete},
+		{"DeleteThrough", testDeleteThrough},
+		{"IdempotentAppend", testIdempotentAppend},
+		{"ConditionalState", testConditionalState},
+		{"StateOutlivesBody", testStateOutlivesBody},
+		{"ConcurrentConditionalWrites", testConcurrentConditionalWrites},
+		{"FeedOrderAndResume", testFeedOrderAndResume},
+		{"FeedWakesOnLiveAppend", testFeedLive},
+		{"FeedOmitsDeletedAppends", testFeedOmitsDeleted},
+		{"FeedHonoursContext", testFeedContext},
+		{"FeedOpenHonoursContext", testFeedOpenContext},
+		{"FeedNextAfterCloseIsClosed", testFeedNextAfterClose},
+		{"FeedCloseEndsBlockedNext", testFeedCloseEndsNext},
+		{"FeedResumesAfterClose", testFeedResumesAfterClose},
+		{"ScanPagesInOrder", testScanPagesInOrder},
+		{"ScanAdvancesAcrossDeletedPositions", testScanAcrossDeleted},
+		{"ScanExcludesAppendsAfterBound", testScanBound},
+		{"ScanRejectsBadInput", testScanBadInput},
+		{"NamespaceIsolation", testNamespaceIsolation},
+		{"ShardIsolation", testShardIsolation},
+		{"CountAndEnumerate", testCountAndEnumerate},
+		{"EnumerateStateOnlyNamespace", testEnumerateStateOnly},
+		{"DeleteNamespace", testDeleteNamespace},
+		{"CrashBetweenAppendAndStateWrite", testCrashAfterAppend},
+		{"CrashBetweenTombstoneAndBodyDelete", testCrashAfterTombstone},
+		{"IdempotentAppendReplayAfterCrash", testIdempotentReplayAfterCrash},
+		{"PositionsSurviveCrash", testPositionsSurviveCrash},
+		{"RetentionBackstop", testRetentionBackstop},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			h := newHarness(t)
+			require.NotNil(t, h.Store, "harness returned no store")
+			c.run(t, h)
+		})
+	}
+}
+
+var nsCounter atomic.Uint64
+
+// newNamespace returns a queue ID no other case or run has used, so a shared
+// store needs no cleanup between cases.
+func newNamespace(t *testing.T, messageShard message.Shard) message.Namespace {
+	t.Helper()
+	var b [6]byte
+	_, err := rand.Read(b[:])
+	require.NoError(t, err)
+	return message.Namespace{
+		MessageShard: messageShard,
+		Queue:        message.QueueID(fmt.Sprintf("cf-%s-%d", hex.EncodeToString(b[:]), nsCounter.Add(1))),
+	}
+}
+
+func appendBody(t *testing.T, s message.Store, ns message.Namespace, body string) message.Position {
+	t.Helper()
+	res, err := s.Append(context.Background(), ns, message.AppendInput{Body: []byte(body), Meta: map[string]string{"body": body}})
+	require.NoError(t, err)
+	require.False(t, res.Duplicate, "an append without an idempotency key is never a duplicate")
+	require.NotZero(t, res.Position, "positions are non-zero")
+	return res.Position
+}
+
+func requireCount(t *testing.T, s message.Store, ns message.Namespace, want message.Usage) {
+	t.Helper()
+	got, err := s.Count(context.Background(), ns)
+	require.NoError(t, err)
+	require.Equal(t, want, got)
+}
+
+func testLimits(t *testing.T, h Harness) {
+	l := h.Store.Limits()
+	assert.GreaterOrEqual(t, l.RetentionBackstop, message.MinRetentionBackstop, "ADR-0011:S12: the backstop is beyond the longest queue retention")
+	assert.Positive(t, l.IdempotencyWindow, "idempotent append needs a window")
+}
+
+func testInvalidNamespace(t *testing.T, h Harness) {
+	ctx := context.Background()
+	for _, id := range []string{"", "a.b", "a*", "a>", "a b", "a/b", strings.Repeat("q", message.MaxQueueIDLen+1)} {
+		ns := message.Namespace{Queue: message.QueueID(id)}
+		_, err := h.Store.Append(ctx, ns, message.AppendInput{Body: []byte("x")})
+		assert.ErrorIsf(t, err, message.ErrInvalidArgument, "Append queue ID %q", id)
+		_, err = h.Store.Get(ctx, ns, 1)
+		assert.ErrorIsf(t, err, message.ErrInvalidArgument, "Get queue ID %q", id)
+		_, err = h.Store.PutState(ctx, ns, 1, []byte("x"), 0)
+		assert.ErrorIsf(t, err, message.ErrInvalidArgument, "PutState queue ID %q", id)
+		_, err = h.Store.Count(ctx, ns)
+		assert.ErrorIsf(t, err, message.ErrInvalidArgument, "Count queue ID %q", id)
+		assert.ErrorIsf(t, h.Store.DeleteNamespace(ctx, ns), message.ErrInvalidArgument, "DeleteNamespace queue ID %q", id)
+	}
+}
+
+// testNoCreateCall holds ADR-0011:S9: creating a queue is the domain recording
+// its shard and queue ID, so a namespace's first append needs no setup.
+func testNoCreateCall(t *testing.T, h Harness) {
+	ns := newNamespace(t, 0)
+	requireCount(t, h.Store, ns, message.Usage{})
+	appendBody(t, h.Store, ns, "first")
+	requireCount(t, h.Store, ns, message.Usage{Messages: 1})
+}
+
+func testAppendAndGet(t *testing.T, h Harness) {
+	ctx := context.Background()
+	ns := newNamespace(t, 0)
+	var last message.Position
+	for i := range 5 {
+		body := fmt.Sprintf("body-%d", i)
+		pos := appendBody(t, h.Store, ns, body)
+		require.Greater(t, pos, last, "positions strictly increase in append order")
+		last = pos
+
+		got, err := h.Store.Get(ctx, ns, pos)
+		require.NoError(t, err)
+		assert.Equal(t, pos, got.Position)
+		assert.Equal(t, []byte(body), got.Body)
+		assert.Equal(t, map[string]string{"body": body}, got.Meta)
+	}
+	requireCount(t, h.Store, ns, message.Usage{Messages: 5})
+}
+
+func testGetMissing(t *testing.T, h Harness) {
+	ctx := context.Background()
+	ns := newNamespace(t, 0)
+	_, err := h.Store.Get(ctx, ns, 1)
+	require.ErrorIs(t, err, message.ErrNotFound)
+	_, err = h.Store.GetState(ctx, ns, 1)
+	require.ErrorIs(t, err, message.ErrNotFound)
+}
+
+func testDelete(t *testing.T, h Harness) {
+	ctx := context.Background()
+	ns := newNamespace(t, 0)
+	first := appendBody(t, h.Store, ns, "a")
+	second := appendBody(t, h.Store, ns, "b")
+
+	require.NoError(t, h.Store.Delete(ctx, ns, first))
+	_, err := h.Store.Get(ctx, ns, first)
+	require.ErrorIs(t, err, message.ErrNotFound)
+	require.NoError(t, h.Store.Delete(ctx, ns, first), "deleting an absent message succeeds")
+	requireCount(t, h.Store, ns, message.Usage{Messages: 1})
+
+	_, err = h.Store.Get(ctx, ns, second)
+	require.NoError(t, err, "deleting one message leaves the others")
+
+	third := appendBody(t, h.Store, ns, "c")
+	require.Greater(t, third, second, "a position is never reused after deletion")
+}
+
+func testDeleteThrough(t *testing.T, h Harness) {
+	ctx := context.Background()
+	ns := newNamespace(t, 0)
+	var pos []message.Position
+	for i := range 5 {
+		pos = append(pos, appendBody(t, h.Store, ns, fmt.Sprintf("m%d", i)))
+	}
+	_, err := h.Store.PutState(ctx, ns, pos[0], []byte("claimed"), 0)
+	require.NoError(t, err)
+
+	require.NoError(t, h.Store.DeleteThrough(ctx, ns, pos[2]))
+	for i, p := range pos {
+		_, err := h.Store.Get(ctx, ns, p)
+		if i <= 2 {
+			assert.ErrorIsf(t, err, message.ErrNotFound, "position %d is at or before the range end", p)
+		} else {
+			assert.NoErrorf(t, err, "position %d is after the range end", p)
+		}
+	}
+	_, err = h.Store.GetState(ctx, ns, pos[0])
+	require.NoError(t, err, "a range delete removes messages, not state records")
+	requireCount(t, h.Store, ns, message.Usage{Messages: 2, States: 1})
+}
+
+func testIdempotentAppend(t *testing.T, h Harness) {
+	ctx := context.Background()
+	ns := newNamespace(t, 0)
+	other := newNamespace(t, 0)
+
+	first, err := h.Store.Append(ctx, ns, message.AppendInput{Body: []byte("original"), IdempotencyKey: "move-1"})
+	require.NoError(t, err)
+	require.False(t, first.Duplicate)
+
+	again, err := h.Store.Append(ctx, ns, message.AppendInput{Body: []byte("retried"), IdempotencyKey: "move-1"})
+	require.NoError(t, err)
+	assert.True(t, again.Duplicate, "a repeated key is reported as a duplicate")
+	assert.Equal(t, first.Position, again.Position, "a repeated key returns the first position")
+	got, err := h.Store.Get(ctx, ns, first.Position)
+	require.NoError(t, err)
+	assert.Equal(t, []byte("original"), got.Body, "a repeated key stores nothing new")
+	requireCount(t, h.Store, ns, message.Usage{Messages: 1})
+
+	elsewhere, err := h.Store.Append(ctx, other, message.AppendInput{Body: []byte("x"), IdempotencyKey: "move-1"})
+	require.NoError(t, err)
+	assert.False(t, elsewhere.Duplicate, "idempotency keys are scoped to a namespace")
+	requireCount(t, h.Store, other, message.Usage{Messages: 1})
+}
+
+func testConditionalState(t *testing.T, h Harness) {
+	ctx := context.Background()
+	ns := newNamespace(t, 0)
+	pos := appendBody(t, h.Store, ns, "m")
+
+	r1, err := h.Store.PutState(ctx, ns, pos, []byte("v1"), 0)
+	require.NoError(t, err)
+	require.NotZero(t, r1)
+	_, err = h.Store.PutState(ctx, ns, pos, []byte("again"), 0)
+	require.ErrorIs(t, err, message.ErrRevisionMismatch, "creating an existing record fails")
+
+	r2, err := h.Store.PutState(ctx, ns, pos, []byte("v2"), r1)
+	require.NoError(t, err)
+	require.NotEqual(t, r1, r2)
+	_, err = h.Store.PutState(ctx, ns, pos, []byte("stale"), r1)
+	require.ErrorIs(t, err, message.ErrRevisionMismatch, "a stale revision is rejected")
+
+	got, err := h.Store.GetState(ctx, ns, pos)
+	require.NoError(t, err)
+	assert.Equal(t, message.State{Value: []byte("v2"), Revision: r2}, got)
+
+	require.ErrorIs(t, h.Store.DeleteState(ctx, ns, pos, r1), message.ErrRevisionMismatch)
+	require.NoError(t, h.Store.DeleteState(ctx, ns, pos, r2))
+	_, err = h.Store.GetState(ctx, ns, pos)
+	require.ErrorIs(t, err, message.ErrNotFound)
+	require.ErrorIs(t, h.Store.DeleteState(ctx, ns, pos, r2), message.ErrNotFound)
+	requireCount(t, h.Store, ns, message.Usage{Messages: 1})
+}
+
+// testStateOutlivesBody: a tombstone is written before the body is deleted,
+// so a state record must stand on its own.
+func testStateOutlivesBody(t *testing.T, h Harness) {
+	ctx := context.Background()
+	ns := newNamespace(t, 0)
+	pos := appendBody(t, h.Store, ns, "m")
+	rev, err := h.Store.PutState(ctx, ns, pos, []byte("tombstone"), 0)
+	require.NoError(t, err)
+	require.NoError(t, h.Store.Delete(ctx, ns, pos))
+
+	got, err := h.Store.GetState(ctx, ns, pos)
+	require.NoError(t, err)
+	assert.Equal(t, rev, got.Revision)
+	requireCount(t, h.Store, ns, message.Usage{States: 1})
+}
+
+func testConcurrentConditionalWrites(t *testing.T, h Harness) {
+	ctx := context.Background()
+	ns := newNamespace(t, 0)
+	pos := appendBody(t, h.Store, ns, "m")
+	const writers = 16
+
+	race := func(expected message.Revision) message.Revision {
+		var wg sync.WaitGroup
+		var wins atomic.Int32
+		var won atomic.Uint64
+		errs := make(chan error, writers)
+		for i := range writers {
+			wg.Go(func() {
+				rev, err := h.Store.PutState(ctx, ns, pos, fmt.Appendf(nil, "writer-%d", i), expected)
+				switch {
+				case err == nil:
+					wins.Add(1)
+					won.Store(uint64(rev))
+				case !errors.Is(err, message.ErrRevisionMismatch):
+					errs <- err
+				}
+			})
+		}
+		wg.Wait()
+		close(errs)
+		for err := range errs {
+			require.NoError(t, err, "a losing conditional write fails only with ErrRevisionMismatch")
+		}
+		require.EqualValues(t, 1, wins.Load(), "exactly one of %d writers expecting revision %d wins", writers, expected)
+		return message.Revision(won.Load())
+	}
+
+	created := race(0)
+	updated := race(created)
+	got, err := h.Store.GetState(ctx, ns, pos)
+	require.NoError(t, err)
+	assert.Equal(t, updated, got.Revision)
+}
+
+// nextIn reads the feed until a change in ns arrives, skipping other
+// namespaces sharing the shard.
+func nextIn(t *testing.T, f message.Feed, ns message.Namespace) message.Change {
+	t.Helper()
+	ctx, cancel := context.WithTimeout(context.Background(), feedWait)
+	defer cancel()
+	for {
+		c, err := f.Next(ctx)
+		require.NoError(t, err, "waiting for a change in %s", ns.Queue)
+		if c.Namespace == ns {
+			return c
+		}
+	}
+}
+
+func openFeed(t *testing.T, s message.Store, messageShard message.Shard, after message.Cursor) message.Feed {
+	t.Helper()
+	f, err := s.Feed(context.Background(), messageShard, after)
+	require.NoError(t, err)
+	t.Cleanup(func() { _ = f.Close() })
+	return f
+}
+
+func testFeedOrderAndResume(t *testing.T, h Harness) {
+	ns := newNamespace(t, 0)
+	var pos []message.Position
+	for i := range 3 {
+		pos = append(pos, appendBody(t, h.Store, ns, fmt.Sprintf("m%d", i)))
+	}
+
+	f := openFeed(t, h.Store, 0, 0)
+	var changes []message.Change
+	for range pos {
+		changes = append(changes, nextIn(t, f, ns))
+	}
+	for i, c := range changes {
+		assert.Equal(t, pos[i], c.Position, "changes arrive in position order")
+		assert.Equal(t, map[string]string{"body": fmt.Sprintf("m%d", i)}, c.Meta, "the feed carries metadata")
+	}
+
+	resumed := openFeed(t, h.Store, 0, changes[1].Cursor)
+	assert.Equal(t, pos[2], nextIn(t, resumed, ns).Position, "a cursor resumes after its change")
+}
+
+func testFeedLive(t *testing.T, h Harness) {
+	ns := newNamespace(t, 0)
+	f := openFeed(t, h.Store, 0, 0)
+	pos := appendBody(t, h.Store, ns, "late")
+	assert.Equal(t, pos, nextIn(t, f, ns).Position, "an open feed sees an append made after it opened")
+}
+
+func testFeedOmitsDeleted(t *testing.T, h Harness) {
+	ctx := context.Background()
+	ns := newNamespace(t, 0)
+	gone := appendBody(t, h.Store, ns, "gone")
+	kept := appendBody(t, h.Store, ns, "kept")
+	require.NoError(t, h.Store.Delete(ctx, ns, gone))
+
+	f := openFeed(t, h.Store, 0, 0)
+	c := nextIn(t, f, ns)
+	if c.Position == gone {
+		c = nextIn(t, f, ns)
+	}
+	assert.Equal(t, kept, c.Position, "a deleted append never hides a later one")
+}
+
+func testFeedNextAfterClose(t *testing.T, h Harness) {
+	ns := newNamespace(t, 0)
+	appendBody(t, h.Store, ns, "pending")
+	f := openFeed(t, h.Store, 0, 0)
+	require.NoError(t, f.Close())
+	_, err := f.Next(context.Background())
+	require.ErrorIs(t, err, message.ErrFeedClosed, "a closed feed delivers nothing, even with changes pending")
+	assert.NoError(t, f.Close(), "a repeated Close returns nil")
+}
+
+func testFeedCloseEndsNext(t *testing.T, h Harness) {
+	f := openFeed(t, h.Store, 0, message.Cursor(math.MaxUint64))
+	done := make(chan error, 1)
+	go func() {
+		_, err := f.Next(context.Background())
+		done <- err
+	}()
+	time.Sleep(100 * time.Millisecond)
+	require.NoError(t, f.Close())
+	select {
+	case err := <-done:
+		require.ErrorIs(t, err, message.ErrFeedClosed, "Close ends a Next already waiting")
+	case <-time.After(feedWait):
+		t.Fatal("Next still blocked after Close")
+	}
+}
+
+// testFeedResumesAfterClose: a change withheld by a racing Close is not lost;
+// a feed from the last delivered cursor returns it.
+func testFeedResumesAfterClose(t *testing.T, h Harness) {
+	ns := newNamespace(t, 0)
+	first := appendBody(t, h.Store, ns, "first")
+	second := appendBody(t, h.Store, ns, "second")
+	f := openFeed(t, h.Store, 0, 0)
+	got := nextIn(t, f, ns)
+	require.Equal(t, first, got.Position)
+	require.NoError(t, f.Close())
+	resumed := openFeed(t, h.Store, 0, got.Cursor)
+	assert.Equal(t, second, nextIn(t, resumed, ns).Position)
+}
+
+func testFeedContext(t *testing.T, h Harness) {
+	f := openFeed(t, h.Store, 0, 0)
+	ctx, cancel := context.WithCancel(context.Background())
+	cancel()
+	for {
+		_, err := f.Next(ctx)
+		if err != nil {
+			require.ErrorIs(t, err, context.Canceled)
+			return
+		}
+	}
+}
+
+func testFeedOpenContext(t *testing.T, h Harness) {
+	ctx, cancel := context.WithCancel(context.Background())
+	cancel()
+	f, err := h.Store.Feed(ctx, 0, 0)
+	require.ErrorIs(t, err, context.Canceled, "a feed is not opened after ctx ended")
+	assert.Nil(t, f)
+}
+
+func testNamespaceIsolation(t *testing.T, h Harness) {
+	ctx := context.Background()
+	a, b := newNamespace(t, 0), newNamespace(t, 0)
+	pa := appendBody(t, h.Store, a, "a")
+	_, err := h.Store.PutState(ctx, a, pa, []byte("a-state"), 0)
+	require.NoError(t, err)
+
+	_, err = h.Store.Get(ctx, b, pa)
+	require.ErrorIs(t, err, message.ErrNotFound, "a namespace never reads another's message")
+	_, err = h.Store.GetState(ctx, b, pa)
+	require.ErrorIs(t, err, message.ErrNotFound, "a namespace never reads another's state")
+	requireCount(t, h.Store, b, message.Usage{})
+
+	pb := appendBody(t, h.Store, b, "b")
+	require.NoError(t, h.Store.DeleteNamespace(ctx, a))
+	_, err = h.Store.Get(ctx, b, pb)
+	require.NoError(t, err, "deleting one namespace leaves its shard neighbours")
+}
+
+func testShardIsolation(t *testing.T, h Harness) {
+	if h.MessageShards < 2 {
+		t.Skip("store serves one message shard")
+	}
+	ctx := context.Background()
+	a := newNamespace(t, 0)
+	b := message.Namespace{MessageShard: 1, Queue: a.Queue}
+	if h.StateShards >= 2 {
+		b.StateShard = 1
+	}
+	pa := appendBody(t, h.Store, a, "a")
+	_, err := h.Store.PutState(ctx, a, pa, []byte("a"), 0)
+	require.NoError(t, err)
+	_, err = h.Store.Get(ctx, b, pa)
+	require.ErrorIs(t, err, message.ErrNotFound, "the same queue ID on another shard is another namespace")
+	_, err = h.Store.GetState(ctx, b, pa)
+	require.ErrorIs(t, err, message.ErrNotFound, "state is scoped to the whole namespace")
+	requireCount(t, h.Store, b, message.Usage{})
+
+	f := openFeed(t, h.Store, 1, 0)
+	pb := appendBody(t, h.Store, b, "b")
+	assert.Equal(t, pb, nextIn(t, f, b).Position, "a shard's feed carries only that shard's appends")
+}
+
+// otherStateShard is a state shard other than zero when the store serves
+// one, so a store that drops or zeroes StateShard is caught.
+func otherStateShard(h Harness) message.Shard {
+	if h.StateShards >= 2 {
+		return message.Shard(safecast.IntToUint32(h.StateShards - 1))
+	}
+	return 0
+}
+
+func enumerate(t *testing.T, s message.Store, messageShard message.Shard) map[message.Namespace]message.Usage {
+	t.Helper()
+	listed, err := s.Namespaces(context.Background(), messageShard)
+	require.NoError(t, err)
+	found := map[message.Namespace]message.Usage{}
+	for _, n := range listed {
+		found[n.Namespace] = n.Usage
+	}
+	return found
+}
+
+func testCountAndEnumerate(t *testing.T, h Harness) {
+	ctx := context.Background()
+	ns, unused := newNamespace(t, 0), newNamespace(t, 0)
+	ns.StateShard = otherStateShard(h)
+	p := appendBody(t, h.Store, ns, "a")
+	appendBody(t, h.Store, ns, "b")
+	_, err := h.Store.PutState(ctx, ns, p, []byte("s"), 0)
+	require.NoError(t, err)
+	requireCount(t, h.Store, ns, message.Usage{Messages: 2, States: 1})
+
+	found := enumerate(t, h.Store, 0)
+	assert.Equal(t, message.Usage{Messages: 2, States: 1}, found[ns], "enumeration reports the complete namespace and its usage")
+	for n := range found {
+		assert.NotEqual(t, unused.Queue, n.Queue, "a namespace never written is not enumerated")
+	}
+}
+
+// testEnumerateStateOnly: after a tombstone and the body delete, only state
+// remains, and enumeration must still name its exact allocations so a
+// caller without the queue record can count and clean it.
+func testEnumerateStateOnly(t *testing.T, h Harness) {
+	ctx := context.Background()
+	ns := newNamespace(t, 0)
+	ns.StateShard = otherStateShard(h)
+	p := appendBody(t, h.Store, ns, "m")
+	_, err := h.Store.PutState(ctx, ns, p, []byte("tombstone"), 0)
+	require.NoError(t, err)
+	require.NoError(t, h.Store.Delete(ctx, ns, p))
+
+	found := enumerate(t, h.Store, 0)
+	require.Contains(t, found, ns, "a state-only namespace is enumerated with its exact StateShard")
+	assert.Equal(t, message.Usage{States: 1}, found[ns])
+}
+
+func testDeleteNamespace(t *testing.T, h Harness) {
+	ctx := context.Background()
+	ns := newNamespace(t, 0)
+	p := appendBody(t, h.Store, ns, "a")
+	appendBody(t, h.Store, ns, "b")
+	_, err := h.Store.PutState(ctx, ns, p, []byte("s"), 0)
+	require.NoError(t, err)
+
+	require.NoError(t, h.Store.DeleteNamespace(ctx, ns))
+	requireCount(t, h.Store, ns, message.Usage{})
+	_, err = h.Store.GetState(ctx, ns, p)
+	require.ErrorIs(t, err, message.ErrNotFound, "namespace deletion removes state records")
+	listed, err := h.Store.Namespaces(ctx, 0)
+	require.NoError(t, err)
+	for _, n := range listed {
+		assert.NotEqual(t, ns.Queue, n.Namespace.Queue, "a deleted namespace is not enumerated")
+	}
+	require.NoError(t, h.Store.DeleteNamespace(ctx, ns), "namespace deletion is idempotent")
+}
+
+func requireReopen(t *testing.T, h Harness) message.Store {
+	t.Helper()
+	if h.Reopen == nil {
+		t.Skip("harness cannot simulate a crash")
+	}
+	s := h.Reopen(t)
+	require.NotNil(t, s)
+	return s
+}
+
+// testCrashAfterAppend: the domain appends, then writes the message's state;
+// a crash between them leaves a durable message with no state, which the
+// feed still offers so the domain can recover it.
+func testCrashAfterAppend(t *testing.T, h Harness) {
+	ctx := context.Background()
+	ns := newNamespace(t, 0)
+	pos := appendBody(t, h.Store, ns, "orphan")
+	s := requireReopen(t, h)
+
+	got, err := s.Get(ctx, ns, pos)
+	require.NoError(t, err, "an acknowledged append survives a crash")
+	assert.Equal(t, []byte("orphan"), got.Body)
+	_, err = s.GetState(ctx, ns, pos)
+	require.ErrorIs(t, err, message.ErrNotFound)
+	assert.Equal(t, pos, nextIn(t, openFeed(t, s, 0, 0), ns).Position, "the feed replays the stateless message")
+	_, err = s.PutState(ctx, ns, pos, []byte("recovered"), 0)
+	require.NoError(t, err, "recovery creates the missing state")
+}
+
+// testCrashAfterTombstone: the domain tombstones, then deletes the body; a
+// crash between them must leave the tombstone and allow the delete to replay.
+func testCrashAfterTombstone(t *testing.T, h Harness) {
+	ctx := context.Background()
+	ns := newNamespace(t, 0)
+	pos := appendBody(t, h.Store, ns, "m")
+	rev, err := h.Store.PutState(ctx, ns, pos, []byte("tombstone"), 0)
+	require.NoError(t, err)
+	s := requireReopen(t, h)
+
+	st, err := s.GetState(ctx, ns, pos)
+	require.NoError(t, err, "an acknowledged state write survives a crash")
+	assert.Equal(t, message.State{Value: []byte("tombstone"), Revision: rev}, st)
+	_, err = s.Get(ctx, ns, pos)
+	require.NoError(t, err, "the body is still there to delete")
+
+	require.NoError(t, s.Delete(ctx, ns, pos))
+	require.NoError(t, s.Delete(ctx, ns, pos), "a replayed body delete succeeds")
+	require.NoError(t, s.DeleteState(ctx, ns, pos, rev))
+	requireCount(t, s, ns, message.Usage{})
+}
+
+func testIdempotentReplayAfterCrash(t *testing.T, h Harness) {
+	ctx := context.Background()
+	ns := newNamespace(t, 0)
+	first, err := h.Store.Append(ctx, ns, message.AppendInput{Body: []byte("copy"), IdempotencyKey: "redrive-7"})
+	require.NoError(t, err)
+	s := requireReopen(t, h)
+
+	again, err := s.Append(ctx, ns, message.AppendInput{Body: []byte("copy"), IdempotencyKey: "redrive-7"})
+	require.NoError(t, err)
+	assert.True(t, again.Duplicate, "an idempotency key survives a crash")
+	assert.Equal(t, first.Position, again.Position)
+	requireCount(t, s, ns, message.Usage{Messages: 1})
+}
+
+func testPositionsSurviveCrash(t *testing.T, h Harness) {
+	ns := newNamespace(t, 0)
+	before := appendBody(t, h.Store, ns, "before")
+	s := requireReopen(t, h)
+	after := appendBody(t, s, ns, "after")
+	require.Greater(t, after, before, "positions keep increasing across a crash")
+}
+
+func testRetentionBackstop(t *testing.T, h Harness) {
+	if h.Advance == nil {
+		t.Skip("harness cannot advance the store's clock")
+	}
+	ctx := context.Background()
+	backstop := h.Store.Limits().RetentionBackstop
+	ns := newNamespace(t, 0)
+	old := appendBody(t, h.Store, ns, "old")
+	h.Advance(backstop - time.Hour)
+	young := appendBody(t, h.Store, ns, "young")
+	h.Advance(2 * time.Hour)
+
+	_, err := h.Store.Get(ctx, ns, old)
+	require.ErrorIs(t, err, message.ErrNotFound, "a message past the backstop is dropped")
+	_, err = h.Store.Get(ctx, ns, young)
+	require.NoError(t, err, "a message inside the backstop is kept")
+}
+
+// scanPass runs one complete scan pass of shard at limit per page, failing
+// on any duplicate or out-of-order position, and returns every change.
+func scanPass(t *testing.T, s message.Store, shard message.Shard, limit int, afterFirst func()) []message.Change {
+	t.Helper()
+	var (
+		all  []message.Change
+		from message.ScanCursor
+		last message.Cursor
+	)
+	for page := 0; ; page++ {
+		require.Less(t, page, 10_000, "scan pass never finished")
+		got, err := s.Scan(context.Background(), shard, from, limit)
+		require.NoError(t, err)
+		require.LessOrEqual(t, len(got.Changes), limit)
+		if page > 0 {
+			require.Equal(t, from.Bound, got.Next.Bound, "a pass keeps its bound")
+		}
+		for _, c := range got.Changes {
+			require.Greater(t, c.Cursor, last, "positions come once each, in order")
+			require.LessOrEqual(t, c.Cursor, got.Next.Bound, "nothing after the bound")
+			last = c.Cursor
+		}
+		all = append(all, got.Changes...)
+		if got.Done {
+			require.Equal(t, got.Next.Bound, got.Next.After)
+			return all
+		}
+		require.Greater(t, got.Next.After, from.After, "an unfinished page advances")
+		from = got.Next
+		if page == 0 && afterFirst != nil {
+			afterFirst()
+		}
+	}
+}
+
+func positionsIn(changes []message.Change, nss ...message.Namespace) []message.Position {
+	want := map[message.Namespace]bool{}
+	for _, ns := range nss {
+		want[ns] = true
+	}
+	var out []message.Position
+	for _, c := range changes {
+		if want[c.Namespace] {
+			out = append(out, c.Position)
+		}
+	}
+	return out
+}
+
+func testScanPagesInOrder(t *testing.T, h Harness) {
+	a, b := newNamespace(t, 0), newNamespace(t, 0)
+	b.StateShard = otherStateShard(h)
+	var want []message.Position
+	for i := range 7 {
+		ns := a
+		if i%2 == 1 {
+			ns = b
+		}
+		want = append(want, appendBody(t, h.Store, ns, fmt.Sprintf("m%d", i)))
+	}
+	all := scanPass(t, h.Store, 0, 2, nil)
+	assert.Equal(t, want, positionsIn(all, a, b), "every surviving append, once, in order")
+	for _, c := range all {
+		if c.Namespace == b {
+			assert.Equal(t, b.StateShard, c.Namespace.StateShard, "complete namespace")
+			assert.Equal(t, map[string]string{"body": c.Meta["body"]}, c.Meta, "domain metadata only")
+		}
+	}
+}
+
+func testScanAcrossDeleted(t *testing.T, h Harness) {
+	ctx := context.Background()
+	ns := newNamespace(t, 0)
+	var ps []message.Position
+	for i := range 6 {
+		ps = append(ps, appendBody(t, h.Store, ns, fmt.Sprintf("m%d", i)))
+	}
+	for _, p := range ps[1:5] {
+		require.NoError(t, h.Store.Delete(ctx, ns, p))
+	}
+	assert.Equal(t, []message.Position{ps[0], ps[5]}, positionsIn(scanPass(t, h.Store, 0, 1, nil), ns), "a deleted run is skipped, not a stop")
+
+	require.NoError(t, h.Store.Delete(ctx, ns, ps[5]))
+	assert.Equal(t, []message.Position{ps[0]}, positionsIn(scanPass(t, h.Store, 0, 1, nil), ns), "a deleted tail ends the pass")
+}
+
+func testScanBound(t *testing.T, h Harness) {
+	ns := newNamespace(t, 0)
+	before := []message.Position{appendBody(t, h.Store, ns, "a"), appendBody(t, h.Store, ns, "b"), appendBody(t, h.Store, ns, "c")}
+	var after []message.Position
+	// The second page holds the bound and has room for one more, so a store
+	// that reads past its bound is caught.
+	all := scanPass(t, h.Store, 0, 2, func() {
+		after = append(after, appendBody(t, h.Store, ns, "d"), appendBody(t, h.Store, ns, "e"))
+	})
+	got := positionsIn(all, ns)
+	assert.Equal(t, before, got, "appends after the bound are not in the pass")
+	assert.Equal(t, append(before, after...), positionsIn(scanPass(t, h.Store, 0, 3, nil), ns), "a new pass has a new bound")
+}
+
+func testScanBadInput(t *testing.T, h Harness) {
+	ctx := context.Background()
+	for _, limit := range []int{0, -1} {
+		_, err := h.Store.Scan(ctx, 0, message.ScanCursor{}, limit)
+		assert.ErrorIsf(t, err, message.ErrInvalidArgument, "limit %d", limit)
+	}
+	_, err := h.Store.Scan(ctx, 0, message.ScanCursor{After: 5, Bound: 4}, 1)
+	assert.ErrorIs(t, err, message.ErrInvalidArgument, "a cursor past its bound")
+}

spinifex/domains/sqs/message/conformance/conformance_test.go

@@ -0,0 +1,41 @@
+package conformance_test
+
+import (
+	"testing"
+
+	"github.com/mulgadc/spinifex/spinifex/domains/sqs/message"
+	"github.com/mulgadc/spinifex/spinifex/domains/sqs/message/conformance"
+)
+
+const fakeShards = 2
+
+func freshHarness(t *testing.T) conformance.Harness {
+	t.Helper()
+	d := newMemData()
+	return conformance.Harness{
+		Store:         &memStore{d: d},
+		MessageShards: fakeShards,
+		StateShards:   fakeShards,
+		Reopen:        func(*testing.T) message.Store { return &memStore{d: d} },
+		Advance:       d.advance,
+	}
+}
+
+func TestReferenceFakeConformance(t *testing.T) {
+	conformance.Run(t, freshHarness)
+}
+
+// TestReferenceFakeSharedInstanceConformance stands in for a long-lived store
+// holding other queues' data: every case must still pass when it is not alone.
+func TestReferenceFakeSharedInstanceConformance(t *testing.T) {
+	shared := freshHarness(t)
+	conformance.Run(t, func(*testing.T) conformance.Harness { return shared })
+}
+
+// TestReferenceFakeMinimalHarnessConformance runs with no crash, clock or
+// second-shard hooks, as a store that offers none of them would.
+func TestReferenceFakeMinimalHarnessConformance(t *testing.T) {
+	conformance.Run(t, func(*testing.T) conformance.Harness {
+		return conformance.Harness{Store: &memStore{d: newMemData()}, MessageShards: 1, StateShards: 1}
+	})
+}

spinifex/domains/sqs/message/conformance/memstore_test.go

@@ -0,0 +1,353 @@
+package conformance_test
+
+import (
+	"context"
+	"maps"
+	"slices"
+	"sync"
+	"time"
+
+	"github.com/mulgadc/spinifex/spinifex/domains/sqs/message"
+)
+
+// memData is the durable state a reopened memStore sees again, standing in
+// for what a real store would hold on disk across a crash.
+type memData struct {
+	mu       sync.Mutex
+	limits   message.Limits
+	clock    time.Duration
+	seq      map[message.Shard]uint64
+	messages map[message.Namespace]map[message.Position]memMessage
+	states   map[message.Namespace]map[message.Position]message.State
+	revision uint64
+	keys     map[message.Namespace]map[string]memKey
+	log      map[message.Shard][]message.Change
+	wake     chan struct{}
+}
+
+type memMessage struct {
+	body []byte
+	meta map[string]string
+	at   time.Duration
+}
+
+type memKey struct {
+	pos message.Position
+	at  time.Duration
+}
+
+// memStore is the reference fake: the simplest store that meets the
+// contract, so the suite is proven runnable before any real store exists.
+type memStore struct{ d *memData }
+
+var _ message.Store = (*memStore)(nil)
+
+func newMemData() *memData {
+	return &memData{
+		limits:   message.Limits{RetentionBackstop: message.MinRetentionBackstop, IdempotencyWindow: 2 * time.Minute},
+		seq:      map[message.Shard]uint64{},
+		messages: map[message.Namespace]map[message.Position]memMessage{},
+		states:   map[message.Namespace]map[message.Position]message.State{},
+		keys:     map[message.Namespace]map[string]memKey{},
+		log:      map[message.Shard][]message.Change{},
+		wake:     make(chan struct{}),
+	}
+}
+
+func (d *memData) advance(by time.Duration) {
+	d.mu.Lock()
+	defer d.mu.Unlock()
+	d.clock += by
+}
+
+// check validates only the queue ID: memory has a slot for every shard, so
+// no allocation is ever out of range.
+func (s *memStore) check(ns message.Namespace) error {
+	return ns.Validate()
+}
+
+// expire drops messages past the backstop; callers hold the lock.
+func (s *memStore) expire() {
+	for _, msgs := range s.d.messages {
+		maps.DeleteFunc(msgs, func(_ message.Position, m memMessage) bool {
+			return s.d.clock-m.at >= s.d.limits.RetentionBackstop
+		})
+	}
+}
+
+func (s *memStore) Append(_ context.Context, ns message.Namespace, in message.AppendInput) (message.AppendResult, error) {
+	if err := s.check(ns); err != nil {
+		return message.AppendResult{}, err
+	}
+	d := s.d
+	d.mu.Lock()
+	defer d.mu.Unlock()
+	if in.IdempotencyKey != "" {
+		if k, ok := d.keys[ns][in.IdempotencyKey]; ok && d.clock-k.at < d.limits.IdempotencyWindow {
+			return message.AppendResult{Position: k.pos, Duplicate: true}, nil
+		}
+	}
+	d.seq[ns.MessageShard]++
+	pos := message.Position(d.seq[ns.MessageShard])
+	if d.messages[ns] == nil {
+		d.messages[ns] = map[message.Position]memMessage{}
+	}
+	d.messages[ns][pos] = memMessage{body: slices.Clone(in.Body), meta: maps.Clone(in.Meta), at: d.clock}
+	if in.IdempotencyKey != "" {
+		if d.keys[ns] == nil {
+			d.keys[ns] = map[string]memKey{}
+		}
+		d.keys[ns][in.IdempotencyKey] = memKey{pos: pos, at: d.clock}
+	}
+	d.log[ns.MessageShard] = append(d.log[ns.MessageShard], message.Change{
+		Namespace: ns, Position: pos, Meta: maps.Clone(in.Meta), Cursor: message.Cursor(pos),
+	})
+	close(d.wake)
+	d.wake = make(chan struct{})
+	return message.AppendResult{Position: pos}, nil
+}
+
+func (s *memStore) Get(_ context.Context, ns message.Namespace, pos message.Position) (message.Message, error) {
+	if err := s.check(ns); err != nil {
+		return message.Message{}, err
+	}
+	s.d.mu.Lock()
+	defer s.d.mu.Unlock()
+	s.expire()
+	m, ok := s.d.messages[ns][pos]
+	if !ok {
+		return message.Message{}, message.ErrNotFound
+	}
+	return message.Message{Position: pos, Body: slices.Clone(m.body), Meta: maps.Clone(m.meta)}, nil
+}
+
+func (s *memStore) Delete(_ context.Context, ns message.Namespace, pos message.Position) error {
+	if err := s.check(ns); err != nil {
+		return err
+	}
+	s.d.mu.Lock()
+	defer s.d.mu.Unlock()
+	delete(s.d.messages[ns], pos)
+	return nil
+}
+
+func (s *memStore) DeleteThrough(_ context.Context, ns message.Namespace, through message.Position) error {
+	if err := s.check(ns); err != nil {
+		return err
+	}
+	s.d.mu.Lock()
+	defer s.d.mu.Unlock()
+	maps.DeleteFunc(s.d.messages[ns], func(p message.Position, _ memMessage) bool { return p <= through })
+	return nil
+}
+
+func (s *memStore) GetState(_ context.Context, ns message.Namespace, pos message.Position) (message.State, error) {
+	if err := s.check(ns); err != nil {
+		return message.State{}, err
+	}
+	s.d.mu.Lock()
+	defer s.d.mu.Unlock()
+	st, ok := s.d.states[ns][pos]
+	if !ok {
+		return message.State{}, message.ErrNotFound
+	}
+	return message.State{Value: slices.Clone(st.Value), Revision: st.Revision}, nil
+}
+
+func (s *memStore) PutState(_ context.Context, ns message.Namespace, pos message.Position, value []byte, expected message.Revision) (message.Revision, error) {
+	if err := s.check(ns); err != nil {
+		return 0, err
+	}
+	d := s.d
+	d.mu.Lock()
+	defer d.mu.Unlock()
+	if d.states[ns][pos].Revision != expected {
+		return 0, message.ErrRevisionMismatch
+	}
+	if d.states[ns] == nil {
+		d.states[ns] = map[message.Position]message.State{}
+	}
+	d.revision++
+	rev := message.Revision(d.revision)
+	d.states[ns][pos] = message.State{Value: slices.Clone(value), Revision: rev}
+	return rev, nil
+}
+
+func (s *memStore) DeleteState(_ context.Context, ns message.Namespace, pos message.Position, expected message.Revision) error {
+	if err := s.check(ns); err != nil {
+		return err
+	}
+	s.d.mu.Lock()
+	defer s.d.mu.Unlock()
+	st, ok := s.d.states[ns][pos]
+	switch {
+	case !ok:
+		return message.ErrNotFound
+	case st.Revision != expected:
+		return message.ErrRevisionMismatch
+	}
+	delete(s.d.states[ns], pos)
+	return nil
+}
+
+func (s *memStore) Feed(ctx context.Context, shard message.Shard, after message.Cursor) (message.Feed, error) {
+	if err := ctx.Err(); err != nil {
+		return nil, err
+	}
+	return &memFeed{d: s.d, shard: shard, after: after, done: make(chan struct{})}, nil
+}
+
+func (s *memStore) Scan(_ context.Context, shard message.Shard, from message.ScanCursor, limit int) (message.ScanPage, error) {
+	if err := message.ValidateScan(from, limit); err != nil {
+		return message.ScanPage{}, err
+	}
+	s.d.mu.Lock()
+	defer s.d.mu.Unlock()
+	s.expire()
+	if from.Bound == 0 {
+		from.Bound = message.Cursor(s.d.seq[shard])
+	}
+	page := message.ScanPage{Next: from}
+	for _, c := range s.d.log[shard] {
+		if c.Cursor <= from.After || c.Cursor > from.Bound {
+			continue
+		}
+		if _, ok := s.d.messages[c.Namespace][c.Position]; !ok {
+			continue
+		}
+		if len(page.Changes) == limit {
+			page.Done = page.Next.After >= page.Next.Bound
+			return page, nil
+		}
+		c.Meta = maps.Clone(c.Meta)
+		page.Changes = append(page.Changes, c)
+		page.Next.After = c.Cursor
+	}
+	page.Next.After = from.Bound
+	page.Done = true
+	return page, nil
+}
+
+func (s *memStore) DeleteNamespace(_ context.Context, ns message.Namespace) error {
+	if err := s.check(ns); err != nil {
+		return err
+	}
+	s.d.mu.Lock()
+	defer s.d.mu.Unlock()
+	delete(s.d.messages, ns)
+	delete(s.d.states, ns)
+	delete(s.d.keys, ns)
+	return nil
+}
+
+func (s *memStore) Count(_ context.Context, ns message.Namespace) (message.Usage, error) {
+	if err := s.check(ns); err != nil {
+		return message.Usage{}, err
+	}
+	s.d.mu.Lock()
+	defer s.d.mu.Unlock()
+	s.expire()
+	return s.usage(ns), nil
+}
+
+func (s *memStore) usage(ns message.Namespace) message.Usage {
+	return message.Usage{Messages: uint64(len(s.d.messages[ns])), States: uint64(len(s.d.states[ns]))}
+}
+
+func (s *memStore) Namespaces(_ context.Context, shard message.Shard) ([]message.NamespaceUsage, error) {
+	s.d.mu.Lock()
+	defer s.d.mu.Unlock()
+	s.expire()
+	seen := map[message.Namespace]bool{}
+	var out []message.NamespaceUsage
+	for _, set := range []map[message.Namespace]bool{nsKeys(s.d.messages), nsKeys(s.d.states)} {
+		for ns := range set {
+			if ns.MessageShard != shard || seen[ns] {
+				continue
+			}
+			seen[ns] = true
+			if u := s.usage(ns); u != (message.Usage{}) {
+				out = append(out, message.NamespaceUsage{Namespace: ns, Usage: u})
+			}
+		}
+	}
+	return out, nil
+}
+
+func nsKeys[V any](m map[message.Namespace]V) map[message.Namespace]bool {
+	out := make(map[message.Namespace]bool, len(m))
+	for k := range m {
+		out[k] = true
+	}
+	return out
+}
+
+func (s *memStore) Limits() message.Limits { return s.d.limits }
+
+type memFeed struct {
+	d     *memData
+	shard message.Shard
+	after message.Cursor
+
+	mu     sync.Mutex
+	closed bool
+	done   chan struct{}
+}
+
+func (f *memFeed) Next(ctx context.Context) (message.Change, error) {
+	for {
+		if f.isClosed() {
+			return message.Change{}, message.ErrFeedClosed
+		}
+		if err := ctx.Err(); err != nil {
+			return message.Change{}, err
+		}
+		f.d.mu.Lock()
+		for _, c := range f.d.log[f.shard] {
+			if c.Cursor <= f.after {
+				continue
+			}
+			if _, live := f.d.messages[c.Namespace][c.Position]; live {
+				f.d.mu.Unlock()
+				return f.deliver(c)
+			}
+			f.after = c.Cursor
+		}
+		wake := f.d.wake
+		f.d.mu.Unlock()
+		select {
+		case <-ctx.Done():
+			return message.Change{}, ctx.Err()
+		case <-f.done:
+		case <-wake:
+		}
+	}
+}
+
+// deliver hands c back unless Close has begun, advancing only on delivery.
+func (f *memFeed) deliver(c message.Change) (message.Change, error) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	if f.closed {
+		return message.Change{}, message.ErrFeedClosed
+	}
+	f.after = c.Cursor
+	c.Meta = maps.Clone(c.Meta)
+	return c, nil
+}
+
+func (f *memFeed) isClosed() bool {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	return f.closed
+}
+
+func (f *memFeed) Close() error {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	if !f.closed {
+		f.closed = true
+		close(f.done)
+	}
+	return nil
+}

spinifex/domains/sqs/message/conformance/sweep_test.go

@@ -0,0 +1,99 @@
+package conformance_test
+
+import (
+	"context"
+	"errors"
+	"testing"
+
+	"github.com/mulgadc/spinifex/spinifex/domains/sqs/message"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// terminal stands in for the domain's decoding of its state records.
+func terminal(st message.State) bool {
+	switch string(st.Value) {
+	case "deleted", "expired", "moved":
+		return true
+	}
+	return false
+}
+
+func put(t *testing.T, s message.Store, ns message.Namespace, value string) message.Position {
+	t.Helper()
+	ctx := context.Background()
+	res, err := s.Append(ctx, ns, message.AppendInput{Body: []byte("m")})
+	require.NoError(t, err)
+	if value != "" {
+		_, err := s.PutState(ctx, ns, res.Position, []byte(value), 0)
+		require.NoError(t, err)
+	}
+	return res.Position
+}
+
+func exists(s message.Store, ns message.Namespace, pos message.Position) (body, state bool) {
+	ctx := context.Background()
+	_, err := s.Get(ctx, ns, pos)
+	body = err == nil
+	_, err = s.GetState(ctx, ns, pos)
+	return body, err == nil
+}
+
+func TestSweepOnReferenceFake(t *testing.T) {
+	s := &memStore{d: newMemData()}
+	ctx := context.Background()
+	q, other := message.Namespace{Queue: "q", StateShard: 1}, message.Namespace{MessageShard: 1, Queue: "q"}
+	noState, live := put(t, s, q, ""), put(t, s, q, "inflight")
+	dead := []message.Position{put(t, s, q, "deleted"), put(t, s, q, "expired"), put(t, s, q, "moved")}
+	otherShard := put(t, s, other, "deleted")
+
+	res, err := message.Sweep(ctx, s, 0, terminal, 2)
+	require.NoError(t, err)
+	assert.Equal(t, message.SweepResult{Scanned: 5, Swept: 3}, res)
+	for _, p := range []message.Position{noState, live} {
+		body, _ := exists(s, q, p)
+		assert.True(t, body, "position %d kept", p)
+	}
+	_, liveState := exists(s, q, live)
+	assert.True(t, liveState, "a live record is kept")
+	for _, p := range dead {
+		body, state := exists(s, q, p)
+		assert.False(t, body || state, "tombstoned position %d swept", p)
+	}
+	body, state := exists(s, other, otherShard)
+	assert.True(t, body && state, "another shard is untouched")
+
+	res, err = message.Sweep(ctx, s, 0, terminal, 2)
+	require.NoError(t, err)
+	assert.Equal(t, message.SweepResult{Scanned: 2}, res, "a rerun finds nothing more")
+}
+
+// failStateDelete crashes the sweeper after its body delete succeeds and
+// before its state delete.
+type failStateDelete struct{ message.Store }
+
+var errCrash = errors.New("crash")
+
+func (failStateDelete) DeleteState(context.Context, message.Namespace, message.Position, message.Revision) error {
+	return errCrash
+}
+
+func TestSweepRetryAfterStateDeleteFailsOnReferenceFake(t *testing.T) {
+	s := &memStore{d: newMemData()}
+	ctx := context.Background()
+	q := message.Namespace{Queue: "q"}
+	p := put(t, s, q, "deleted")
+
+	_, err := message.Sweep(ctx, failStateDelete{s}, 0, terminal, 10)
+	require.ErrorIs(t, err, errCrash)
+	body, state := exists(s, q, p)
+	require.False(t, body, "the body delete succeeded")
+	require.True(t, state, "the state delete did not")
+
+	res, err := message.Sweep(ctx, s, 0, terminal, 10)
+	require.NoError(t, err)
+	assert.Equal(t, message.SweepResult{}, res, "the body never resurfaces to be swept again")
+	body, state = exists(s, q, p)
+	assert.False(t, body)
+	assert.True(t, state, "the bodiless record reads as gone and is left to the backstop")
+}

spinifex/domains/sqs/message/invariants_test.go

@@ -0,0 +1,83 @@
+package message_test
+
+import (
+	"go/ast"
+	"go/parser"
+	"go/token"
+	"io/fs"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// forbiddenConsumerState names every JetStream mechanism ADR-0011:S5 bars
+// from SQS state: acknowledgement and redelivery state, durable or
+// configured consumers, and retention that deletes on acknowledgement.
+var forbiddenConsumerState = map[string]string{
+	"AckWait":                "per-consumer redelivery timer",
+	"MaxDeliver":             "consumer delivery limit",
+	"MaxAckPending":          "consumer in-flight limit",
+	"NumDelivered":           "consumer delivery count",
+	"NakWithDelay":           "consumer redelivery",
+	"Nak":                    "consumer redelivery",
+	"InProgress":             "consumer AckWait reset",
+	"Ack":                    "consumer acknowledgement",
+	"AckSync":                "consumer acknowledgement",
+	"DoubleAck":              "consumer acknowledgement",
+	"ManualAck":              "acknowledged subscription",
+	"AckAll":                 "acknowledged subscription",
+	"AckExplicit":            "acknowledged subscription",
+	"Term":                   "consumer acknowledgement",
+	"TermWithReason":         "consumer acknowledgement",
+	"AckExplicitPolicy":      "acknowledged consumer",
+	"AckAllPolicy":           "acknowledged consumer",
+	"Durable":                "durable consumer",
+	"CreateConsumer":         "configured consumer",
+	"CreateOrUpdateConsumer": "configured consumer",
+	"UpdateConsumer":         "configured consumer",
+	"PushConsumer":           "configured consumer",
+	"WorkQueuePolicy":        "retention that deletes on acknowledgement",
+	"InterestPolicy":         "retention that deletes on acknowledgement",
+}
+
+// TestS5_NoConsumerDeliveryStateInSQS holds ADR-0011:S5 across every
+// non-test Go file of the SQS domain: the only consumer allowed is the
+// ephemeral, unacknowledged ordered consumer.
+func TestS5_NoConsumerDeliveryStateInSQS(t *testing.T) {
+	root := ".."
+	var scanned int
+	err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
+		if err != nil || d.IsDir() || !strings.HasSuffix(path, ".go") || strings.HasSuffix(path, "_test.go") {
+			return err
+		}
+		scanned++
+		fset := token.NewFileSet()
+		f, err := parser.ParseFile(fset, path, nil, 0)
+		require.NoError(t, err)
+		ast.Inspect(f, func(n ast.Node) bool {
+			var name string
+			switch n := n.(type) {
+			case *ast.SelectorExpr:
+				name = n.Sel.Name
+			case *ast.KeyValueExpr:
+				if id, ok := n.Key.(*ast.Ident); ok {
+					name = id.Name
+				}
+			default:
+				return true
+			}
+			if why, bad := forbiddenConsumerState[name]; bad {
+				assert.Failf(t, "ADR-0011:S5 breach",
+					"%s uses %s (%s); \"JetStream consumer delivery state never defines an SQS semantic\". Keep the state in the domain's conditional writes and use only the ordered consumer.",
+					fset.Position(n.Pos()), name, why)
+			}
+			return true
+		})
+		return nil
+	})
+	require.NoError(t, err)
+	require.NotZero(t, scanned, "no SQS source found under %s", root)
+}

spinifex/domains/sqs/message/jetstream/body_enum_test.go

@@ -0,0 +1,130 @@
+package jetstream_test
+
+import (
+	"context"
+	"strings"
+	"testing"
+
+	"github.com/mulgadc/spinifex/spinifex/domains/sqs/message"
+	sqsjs "github.com/mulgadc/spinifex/spinifex/domains/sqs/message/jetstream"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// messageStreamRequests splits recorded API subjects for one message stream
+// into leader message gets, stream infos and anything else.
+func messageStreamRequests(subjects []string, stream string) (leaderGets, infos int, other []string) {
+	for _, subj := range subjects {
+		switch {
+		case !strings.HasSuffix(subj, "."+stream) && !strings.Contains(subj, "."+stream+"."):
+		case subj == "$JS.API.STREAM.MSG.GET."+stream:
+			leaderGets++
+		case subj == "$JS.API.STREAM.INFO."+stream:
+			infos++
+		default:
+			other = append(other, subj)
+		}
+	}
+	return leaderGets, infos, other
+}
+
+// TestBodyEnumerationIsLeaderServed pins the route Namespaces and Scan use
+// for bodies: one leader message get per stored body through the bound, with
+// no direct get and no consumer on the message stream.
+func TestBodyEnumerationIsLeaderServed(t *testing.T) {
+	srv, js := startServer(t, t.TempDir())
+	s, err := sqsjs.New(context.Background(), js, testConfig)
+	require.NoError(t, err)
+	ctx := context.Background()
+	a, b := ns(0, 0, "body-a"), ns(0, 1, "body-b")
+	var aPos []message.Position
+	for range 3 {
+		aPos = append(aPos, appendTo(t, s, a, "a"))
+		appendTo(t, s, b, "b")
+	}
+	// Two removed positions leave gaps the enumeration must step over.
+	require.NoError(t, s.Delete(ctx, a, aPos[0]))
+	require.NoError(t, s.Delete(ctx, a, aPos[2]))
+	stream := sqsjs.StreamName(0)
+
+	rec := recordAPI(t, srv.ClientURL())
+	rec.drain(t, js.Conn())
+	got, err := s.Namespaces(ctx, 0)
+	require.NoError(t, err)
+	usage := map[message.QueueID]uint64{}
+	for _, n := range got {
+		usage[n.Namespace.Queue] = n.Usage.Messages
+	}
+	assert.Equal(t, map[message.QueueID]uint64{a.Queue: 1, b.Queue: 3}, usage)
+	gets, infos, rest := messageStreamRequests(rec.drain(t, js.Conn()), stream)
+	// Four stored bodies; the last is the bound, so no further get follows.
+	assert.Equal(t, 4, gets)
+	assert.Equal(t, 1, infos)
+	assert.Empty(t, rest, "Namespaces sent other requests for the message stream")
+
+	page, err := s.Scan(ctx, 0, message.ScanCursor{}, 3)
+	require.NoError(t, err)
+	require.Len(t, page.Changes, 3)
+	assert.False(t, page.Done)
+	gets, infos, rest = messageStreamRequests(rec.drain(t, js.Conn()), stream)
+	assert.Equal(t, 3, gets, "a full page reads one body per change")
+	assert.Equal(t, 1, infos, "the first page captures the bound")
+	assert.Empty(t, rest, "Scan sent other requests for the message stream")
+
+	page, err = s.Scan(ctx, 0, page.Next, 3)
+	require.NoError(t, err)
+	require.Len(t, page.Changes, 1)
+	assert.True(t, page.Done)
+	gets, infos, rest = messageStreamRequests(rec.drain(t, js.Conn()), stream)
+	assert.Equal(t, 1, gets)
+	assert.Zero(t, infos, "later pages keep the first page's bound")
+	assert.Empty(t, rest)
+}
+
+// TestScanStepsOverGapsInOrderWithinTheBound pages through removed positions
+// and neighbouring namespaces in stream order, and stops at the first page's
+// bound even when later bodies exist.
+func TestScanStepsOverGapsInOrderWithinTheBound(t *testing.T) {
+	s, _ := newStore(t)
+	ctx := context.Background()
+	a, b := ns(0, 0, "gap-a"), ns(0, 0, "gap-b")
+	var want []message.Position
+	for i := range 6 {
+		pa := appendTo(t, s, a, "a")
+		pb := appendTo(t, s, b, "b")
+		if i%2 == 0 {
+			require.NoError(t, s.Delete(ctx, a, pa))
+		} else {
+			want = append(want, pa)
+		}
+		want = append(want, pb)
+	}
+	page, err := s.Scan(ctx, 0, message.ScanCursor{}, 4)
+	require.NoError(t, err)
+	// Appended after the bound is fixed, so never part of this pass.
+	appendTo(t, s, a, "late")
+	var got []message.Position
+	for {
+		for _, c := range page.Changes {
+			got = append(got, c.Position)
+			require.Equal(t, message.Cursor(c.Position), c.Cursor)
+		}
+		if page.Done {
+			break
+		}
+		page, err = s.Scan(ctx, 0, page.Next, 4)
+		require.NoError(t, err)
+	}
+	assert.Equal(t, want, got)
+}
+
+func TestBodyEnumerationCancelled(t *testing.T) {
+	s, _ := newStore(t)
+	appendTo(t, s, ns(0, 0, "cancel"), "b")
+	ctx, cancel := context.WithCancel(context.Background())
+	cancel()
+	_, err := s.Namespaces(ctx, 0)
+	require.ErrorIs(t, err, context.Canceled)
+	_, err = s.Scan(ctx, 0, message.ScanCursor{}, 10)
+	require.ErrorIs(t, err, context.Canceled)
+}

spinifex/domains/sqs/message/jetstream/config.go

@@ -0,0 +1,94 @@
+// Package jetstream implements the SQS message store on the dedicated NATS
+// JetStream server: a fixed set of shared message streams with a subject per
+// queue, and a fixed set of shared state buckets (ADR-0011:S9 to S12).
+package jetstream
+
+import (
+	"fmt"
+	"strconv"
+	"time"
+
+	"github.com/mulgadc/spinifex/spinifex/domains/sqs/message"
+	jsapi "github.com/nats-io/nats.go/jetstream"
+)
+
+// DefaultDuplicateWindow is how long a stream remembers a Nats-Msg-Id, which
+// is the window an idempotent append holds for.
+const DefaultDuplicateWindow = 2 * time.Minute
+
+// Config sizes the shared layout new queues are allocated from. Counts are
+// per topology (ADR-0011:S10) and may change; a queue's recorded allocation
+// does not. Replicas follow the cluster size: R1 on one node, R3 on three.
+type Config struct {
+	MessageStreams  int
+	StateBuckets    int
+	DuplicateWindow time.Duration
+}
+
+// DefaultConfig is ADR-0011:S10's default for a topology of nodes: 16
+// message streams and 8 state buckets on one node, 8 and 8 from three.
+func DefaultConfig(nodes int) Config {
+	cfg := Config{MessageStreams: 16, StateBuckets: 8, DuplicateWindow: DefaultDuplicateWindow}
+	if nodes >= 3 {
+		cfg.MessageStreams = 8
+	}
+	return cfg
+}
+
+// Validate rejects a layout the adapter could not serve.
+func (c Config) Validate() error {
+	switch {
+	case c.MessageStreams < 1:
+		return fmt.Errorf("%w: %d message streams", message.ErrInvalidArgument, c.MessageStreams)
+	case c.StateBuckets < 1:
+		return fmt.Errorf("%w: %d state buckets", message.ErrInvalidArgument, c.StateBuckets)
+	case c.DuplicateWindow <= 0 || c.DuplicateWindow > message.MinRetentionBackstop:
+		return fmt.Errorf("%w: duplicate window %s outside (0, %s]", message.ErrInvalidArgument, c.DuplicateWindow, message.MinRetentionBackstop)
+	}
+	return nil
+}
+
+// StreamName is the shared message stream serving a shard.
+func StreamName(shard message.Shard) string {
+	return "SQS_M_" + strconv.FormatUint(uint64(shard), 10)
+}
+
+// BucketName is the shared state bucket serving a state shard.
+func BucketName(shard message.Shard) string {
+	return "sqs_state_" + strconv.FormatUint(uint64(shard), 10)
+}
+
+// Subject is where a namespace's messages are published:
+// sqs.m.<message shard>.<queue>.
+func Subject(ns message.Namespace) string {
+	return "sqs.m." + strconv.FormatUint(uint64(ns.MessageShard), 10) + "." + string(ns.Queue)
+}
+
+// StateKey is a message's state key in the bucket named by its StateShard.
+// A position is unique only within a message stream, so the key carries the
+// message shard too.
+func StateKey(ns message.Namespace, pos message.Position) string {
+	return statePrefix(ns) + "." + strconv.FormatUint(uint64(pos), 10)
+}
+
+// statePrefix is the key prefix every state record of a namespace shares.
+func statePrefix(ns message.Namespace) string {
+	return "q." + strconv.FormatUint(uint64(ns.MessageShard), 10) + "." + string(ns.Queue)
+}
+
+// MessageStreamConfig is one shared message stream: limits retention, the
+// ADR-0011:S12 backstop as MaxAge, and the default persist mode so an
+// acknowledgement follows a sync (ADR-0011:S3).
+func (c Config) MessageStreamConfig(shard message.Shard, replicas int) jsapi.StreamConfig {
+	return jsapi.StreamConfig{
+		Name:        StreamName(shard),
+		Description: "SQS messages, shared by every queue assigned this shard",
+		Subjects:    []string{"sqs.m." + strconv.FormatUint(uint64(shard), 10) + ".>"},
+		Retention:   jsapi.LimitsPolicy,
+		Storage:     jsapi.FileStorage,
+		Replicas:    replicas,
+		MaxAge:      message.MinRetentionBackstop,
+		Duplicates:  c.DuplicateWindow,
+		PersistMode: jsapi.DefaultPersistMode,
+	}
+}

spinifex/domains/sqs/message/jetstream/conformance_test.go

@@ -0,0 +1,49 @@
+package jetstream_test
+
+import (
+	"context"
+	"testing"
+
+	"github.com/mulgadc/spinifex/spinifex/domains/sqs/message"
+	"github.com/mulgadc/spinifex/spinifex/domains/sqs/message/conformance"
+	sqsjs "github.com/mulgadc/spinifex/spinifex/domains/sqs/message/jetstream"
+	"github.com/stretchr/testify/require"
+)
+
+// r1Harness runs one case on its own single-node server. Reopen restarts
+// the server over the same data, as a crash would. Advance is nil because
+// the server's clock cannot be moved, so the backstop case skips; MaxAge
+// is asserted on the stream config directly instead.
+func r1Harness(t *testing.T) conformance.Harness {
+	t.Helper()
+	dir := t.TempDir()
+	srv, js := startServer(t, dir)
+	s, err := sqsjs.New(context.Background(), js, testConfig)
+	require.NoError(t, err)
+	return conformance.Harness{
+		Store:         s,
+		MessageShards: testConfig.MessageStreams,
+		StateShards:   testConfig.StateBuckets,
+		Reopen: func(t *testing.T) message.Store {
+			srv.Shutdown()
+			srv.WaitForShutdown()
+			_, js := startServer(t, dir)
+			s, err := sqsjs.New(context.Background(), js, testConfig)
+			require.NoError(t, err)
+			return s
+		},
+	}
+}
+
+func TestR1Conformance(t *testing.T) {
+	conformance.Run(t, r1Harness)
+}
+
+// TestR1SharedInstanceConformance runs every case on one long-lived store,
+// so each must pass beside other queues' data on the shared streams.
+func TestR1SharedInstanceConformance(t *testing.T) {
+	s, _ := newStore(t)
+	conformance.Run(t, func(*testing.T) conformance.Harness {
+		return conformance.Harness{Store: s, MessageShards: testConfig.MessageStreams, StateShards: testConfig.StateBuckets}
+	})
+}

spinifex/domains/sqs/message/jetstream/feed_cancel_internal_test.go

@@ -0,0 +1,156 @@
+package jetstream
+
+import (
+	"context"
+	"errors"
+	"testing"
+	"time"
+
+	"github.com/mulgadc/spinifex/spinifex/domains/sqs/message"
+	jsapi "github.com/nats-io/nats.go/jetstream"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// noCreate fails the test if a consumer create is attempted.
+type noCreate struct {
+	jsapi.Stream
+
+	t *testing.T
+}
+
+func (n noCreate) OrderedConsumer(context.Context, jsapi.OrderedConsumerConfig) (jsapi.Consumer, error) {
+	n.t.Error("a consumer create was started after ctx ended")
+	return nil, context.Canceled
+}
+
+// emptied leaves f with no consumer, as recovery's detach does.
+func emptied(t *testing.T, f *feed) {
+	t.Helper()
+	require.NoError(t, f.discard(f.detachForTest()))
+}
+
+// TestFeedCancelledBeforeOpenCreatesNothing: Feed with an ended ctx returns
+// its error and creates no consumer.
+func TestFeedCancelledBeforeOpenCreatesNothing(t *testing.T) {
+	s, _ := feedStore(t)
+	ctx, cancel := context.WithCancel(context.Background())
+	cancel()
+	_, err := s.Feed(ctx, 0, 0)
+	require.ErrorIs(t, err, context.Canceled)
+	assert.Empty(t, consumerNames(t, s))
+
+	f := openWedgeable(t, s)
+	emptied(t, f)
+	f.stream = noCreate{Stream: f.stream, t: t}
+	require.ErrorIs(t, f.start(ctx), context.Canceled)
+	_, err = f.Next(ctx)
+	require.ErrorIs(t, err, context.Canceled, "recovery does not start an open after ctx ended")
+}
+
+// TestFeedStartCancelledDuringCreate: a first open whose create finishes after
+// ctx ended releases that consumer and returns ctx's error.
+func TestFeedStartCancelledDuringCreate(t *testing.T) {
+	s, _ := feedStore(t)
+	f := openWedgeable(t, s)
+	emptied(t, f)
+	b := &blockingStream{
+		Stream:  f.stream,
+		entered: make(chan struct{}), release: make(chan struct{}),
+		created: make(chan struct{}), proceed: make(chan struct{}),
+	}
+	f.stream = b
+	ctx, cancel := context.WithCancel(context.Background())
+	done := make(chan error, 1)
+	go func() { done <- f.start(ctx) }()
+	<-b.entered
+	close(b.release)
+	<-b.created
+	require.Len(t, consumerNames(t, s), 1, "the create landed")
+	cancel()
+	close(b.proceed)
+	require.ErrorIs(t, <-done, context.Canceled)
+	assert.Nil(t, f.it, "the late consumer is not kept")
+	assert.Empty(t, consumerNames(t, s), "the late consumer is deleted")
+	assert.Zero(t, f.busy)
+}
+
+// TestFeedRecoveryCancelledDuringCreate: a recovery whose create finishes
+// after ctx ended releases that consumer, returns ctx's error and no change;
+// the stored change is delivered by the next Next.
+func TestFeedRecoveryCancelledDuringCreate(t *testing.T) {
+	s, _ := feedStore(t)
+	f := openWedgeable(t, s)
+	ctx, cancel := context.WithCancel(context.Background())
+	b, next := recoverInto(ctx, t, s, f)
+	close(b.release)
+	<-b.created
+	cancel()
+	close(b.proceed)
+	require.ErrorIs(t, <-next, context.Canceled)
+	assert.Empty(t, consumerNames(t, s), "the late consumer is deleted")
+
+	f.stream = b.Stream
+	c, err := f.Next(context.Background())
+	require.NoError(t, err)
+	assert.Equal(t, message.Position(2), c.Position, "the withheld change is not lost")
+	require.NoError(t, f.Close())
+	assert.Empty(t, consumerNames(t, s))
+}
+
+// TestFeedRecoveryCloseWinsOverCancel: when Close has begun and ctx has ended
+// during a recovery's create, Next returns ErrFeedClosed and Close succeeds
+// only once the late consumer is deleted.
+func TestFeedRecoveryCloseWinsOverCancel(t *testing.T) {
+	s, _ := feedStore(t)
+	f := openWedgeable(t, s)
+	ctx, cancel := context.WithCancel(context.Background())
+	b, next := recoverInto(ctx, t, s, f)
+	closed := make(chan error, 1)
+	go func() { closed <- f.Close() }()
+	require.Eventually(t, f.isClosed, 5*time.Second, 5*time.Millisecond)
+	cancel()
+	close(b.release)
+	close(b.proceed)
+	require.ErrorIs(t, <-next, message.ErrFeedClosed)
+	require.NoError(t, <-closed)
+	assert.Empty(t, consumerNames(t, s))
+}
+
+// TestFeedConsumerHasInactiveBackstop: every feed consumer carries the
+// server-side inactive threshold that removes one an ambiguous create left.
+func TestFeedConsumerHasInactiveBackstop(t *testing.T) {
+	s, _ := feedStore(t)
+	f := openWedgeable(t, s)
+	info, err := f.consumer.Info(context.Background())
+	require.NoError(t, err)
+	assert.Equal(t, time.Minute, info.Config.InactiveThreshold)
+}
+
+var errInjectedCreate = errors.New("injected create failure")
+
+// cancelThenFail ends the caller's ctx during a create, then fails it.
+type cancelThenFail struct {
+	jsapi.Stream
+
+	cancel context.CancelFunc
+}
+
+func (c cancelThenFail) OrderedConsumer(context.Context, jsapi.OrderedConsumerConfig) (jsapi.Consumer, error) {
+	c.cancel()
+	return nil, errInjectedCreate
+}
+
+// TestFeedFailedOpenAfterCancelReportsCancellation: an open that fails after
+// ctx ended still reports ctx's error, alongside the create failure.
+func TestFeedFailedOpenAfterCancelReportsCancellation(t *testing.T) {
+	s, _ := feedStore(t)
+	f := openWedgeable(t, s)
+	emptied(t, f)
+	ctx, cancel := context.WithCancel(context.Background())
+	f.stream = cancelThenFail{Stream: f.stream, cancel: cancel}
+	err := f.start(ctx)
+	require.ErrorIs(t, err, context.Canceled)
+	require.ErrorIs(t, err, errInjectedCreate)
+	assert.Nil(t, f.it)
+}

spinifex/domains/sqs/message/jetstream/feed_cleanup_internal_test.go

@@ -0,0 +1,237 @@
+package jetstream
+
+import (
+	"context"
+	"errors"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/mulgadc/spinifex/spinifex/domains/sqs/message"
+	jsapi "github.com/nats-io/nats.go/jetstream"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+var errInjectedDelete = errors.New("injected delete failure")
+
+// deleteControl can hold every consumer delete until release is closed and
+// fail the next failures of them.
+type deleteControl struct {
+	jsapi.Stream
+
+	entered  chan struct{}
+	release  chan struct{}
+	failures atomic.Int32
+}
+
+func controlDeletes(f *feed, hold bool) *deleteControl {
+	d := &deleteControl{Stream: f.stream, entered: make(chan struct{}, 16)}
+	if hold {
+		d.release = make(chan struct{})
+	}
+	f.stream = d
+	return d
+}
+
+func (d *deleteControl) DeleteConsumer(ctx context.Context, name string) error {
+	d.entered <- struct{}{}
+	if d.release != nil {
+		<-d.release
+	}
+	if d.failures.Add(-1) >= 0 {
+		return errInjectedDelete
+	}
+	return d.Stream.DeleteConsumer(ctx, name)
+}
+
+// returnsWithin reports whether ch delivers within d, and its value.
+func returnsWithin(ch <-chan error, d time.Duration) (error, bool) {
+	select {
+	case err := <-ch:
+		return err, true
+	case <-time.After(d):
+		return nil, false
+	}
+}
+
+// TestFeedConcurrentCloseWaitsForDelete: a second Close does not report
+// success while the first Close's delete is still in progress.
+func TestFeedConcurrentCloseWaitsForDelete(t *testing.T) {
+	s, _ := feedStore(t)
+	f := openWedgeable(t, s)
+	d := controlDeletes(f, true)
+	first, second := make(chan error, 1), make(chan error, 1)
+	go func() { first <- f.Close() }()
+	<-d.entered
+	go func() { second <- f.Close() }()
+	_, done := returnsWithin(second, 200*time.Millisecond)
+	require.False(t, done, "the second Close returned while the delete was held")
+	close(d.release)
+	require.NoError(t, <-first)
+	require.NoError(t, <-second)
+	assert.Empty(t, consumerNames(t, s))
+}
+
+// TestFeedCloseWaitsForRecoveryRelease: Close waits for the release of a
+// consumer a recovery has detached.
+func TestFeedCloseWaitsForRecoveryRelease(t *testing.T) {
+	s, _ := feedStore(t)
+	ns := message.Namespace{Queue: "q"}
+	appendAt(t, s, ns)
+	f := openWedgeable(t, s)
+	_, err := f.Next(context.Background())
+	require.NoError(t, err)
+	appendAt(t, s, ns)
+	d := controlDeletes(f, true)
+	wedge(f)
+	next := make(chan error, 1)
+	go func() {
+		_, err := f.Next(context.Background())
+		next <- err
+	}()
+	<-d.entered
+	closed := make(chan error, 1)
+	go func() { closed <- f.Close() }()
+	_, done := returnsWithin(closed, 200*time.Millisecond)
+	require.False(t, done, "Close returned while the recovery's delete was held")
+	close(d.release)
+	require.NoError(t, <-closed)
+	assert.Empty(t, consumerNames(t, s))
+	require.ErrorIs(t, <-next, message.ErrFeedClosed)
+}
+
+// TestFeedCloseRetriesFailedDelete: a failed delete is an error, not success,
+// and a later Close retries it.
+func TestFeedCloseRetriesFailedDelete(t *testing.T) {
+	s, _ := feedStore(t)
+	f := openWedgeable(t, s)
+	d := controlDeletes(f, false)
+	d.failures.Store(2)
+	require.ErrorIs(t, f.Close(), errInjectedDelete, "the release and Close's own retry both fail")
+	require.Len(t, consumerNames(t, s), 1, "the consumer is still on the server")
+	require.NoError(t, f.Close(), "a later Close retries the delete")
+	assert.Empty(t, consumerNames(t, s))
+	assert.NoError(t, f.Close())
+}
+
+// TestFeedCloseRetriesFailedRecoveryRelease: a recovery's failed delete is
+// retried by Close rather than forgotten.
+func TestFeedCloseRetriesFailedRecoveryRelease(t *testing.T) {
+	s, _ := feedStore(t)
+	ns := message.Namespace{Queue: "q"}
+	appendAt(t, s, ns)
+	f := openWedgeable(t, s)
+	_, err := f.Next(context.Background())
+	require.NoError(t, err)
+	appendAt(t, s, ns)
+	d := controlDeletes(f, false)
+	d.failures.Store(1)
+	wedge(f)
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+	_, err = f.Next(ctx)
+	require.ErrorIs(t, err, errInjectedDelete)
+	require.NoError(t, f.Close())
+	assert.Empty(t, consumerNames(t, s), "the abandoned consumer was deleted by Close")
+}
+
+// TestFeedDeliveryClaimedBeforeCloseIsReturned: Close beginning after Next has
+// claimed a change does not withdraw it, and the feed then reports closed.
+func TestFeedDeliveryClaimedBeforeCloseIsReturned(t *testing.T) {
+	s, _ := feedStore(t)
+	ns := message.Namespace{Queue: "q"}
+	first := appendAt(t, s, ns)
+	second := appendAt(t, s, ns)
+	f := openWedgeable(t, s)
+	f.claimed = func() {
+		f.claimed = nil
+		require.NoError(t, f.Close())
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+	c, err := f.Next(ctx)
+	require.NoError(t, err)
+	assert.Equal(t, first, c.Position, "the change claimed before Close is delivered")
+	_, err = f.Next(ctx)
+	require.ErrorIs(t, err, message.ErrFeedClosed)
+
+	resumed, err := s.Feed(ctx, 0, c.Cursor)
+	require.NoError(t, err)
+	defer func() { _ = resumed.Close() }()
+	got, err := resumed.Next(ctx)
+	require.NoError(t, err)
+	assert.Equal(t, second, got.Position, "resuming after it neither repeats nor skips")
+}
+
+// lostReply creates the consumer but reports the create as failed, as a
+// cancelled or timed-out request does after the server has acted on it.
+type lostReply struct{ jsapi.Stream }
+
+func (l lostReply) OrderedConsumer(_ context.Context, cfg jsapi.OrderedConsumerConfig) (jsapi.Consumer, error) {
+	if _, err := l.Stream.OrderedConsumer(context.Background(), cfg); err != nil {
+		return nil, err
+	}
+	return nil, context.DeadlineExceeded
+}
+
+func TestFeedFailedCreateLeavesNoConsumer(t *testing.T) {
+	s, _ := feedStore(t)
+	f := openWedgeable(t, s)
+	require.NoError(t, f.discard(f.detachForTest()))
+	f.stream = lostReply{f.stream}
+	_, _, err := f.open(context.Background())
+	require.ErrorIs(t, err, context.DeadlineExceeded)
+	assert.Empty(t, consumerNames(t, s), "the create that landed is deleted by name")
+}
+
+// TestFeedCallerDeadlineDoesNotCutCreateShort: real creates under caller
+// deadlines that expire mid-request complete, so none lands unowned.
+func TestFeedCallerDeadlineDoesNotCutCreateShort(t *testing.T) {
+	s, _ := feedStore(t)
+	f := openWedgeable(t, s)
+	require.NoError(t, f.discard(f.detachForTest()))
+	for i := range 100 {
+		ctx, cancel := context.WithTimeout(context.Background(), time.Duration(i%40)*25*time.Microsecond)
+		consumer, it, err := f.open(ctx)
+		cancel()
+		require.NoError(t, err, "create %d", i)
+		require.NoError(t, release(f.stream, consumer, it))
+	}
+	assert.Empty(t, consumerNames(t, s))
+}
+
+// TestFeedCloseRetriesAbandonedCreate: when the delete after a failed create
+// finds nothing, Close deletes the name again, catching a create that landed
+// after that first delete.
+func TestFeedCloseRetriesAbandonedCreate(t *testing.T) {
+	s, _ := feedStore(t)
+	f := openWedgeable(t, s)
+	require.NoError(t, f.discard(f.detachForTest()))
+	late := &lateCreate{Stream: f.stream}
+	f.stream = late
+	_, _, err := f.open(context.Background())
+	require.ErrorIs(t, err, context.DeadlineExceeded)
+	require.NoError(t, late.land())
+	require.Len(t, consumerNames(t, s), 1, "the late create is on the server")
+	require.NoError(t, f.Close())
+	assert.Empty(t, consumerNames(t, s))
+}
+
+// lateCreate reports a create as timed out before making it; land then makes
+// it, as a request the server acts on after the client gave up would.
+type lateCreate struct {
+	jsapi.Stream
+
+	cfg jsapi.OrderedConsumerConfig
+}
+
+func (l *lateCreate) OrderedConsumer(_ context.Context, cfg jsapi.OrderedConsumerConfig) (jsapi.Consumer, error) {
+	l.cfg = cfg
+	return nil, context.DeadlineExceeded
+}
+
+func (l *lateCreate) land() error {
+	_, err := l.Stream.OrderedConsumer(context.Background(), l.cfg)
+	return err
+}

spinifex/domains/sqs/message/jetstream/feed_close_internal_test.go

@@ -0,0 +1,289 @@
+package jetstream
+
+import (
+	"context"
+	"errors"
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/mulgadc/spinifex/spinifex/domains/sqs/message"
+	jsapi "github.com/nats-io/nats.go/jetstream"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// blockingStream holds a consumer create until release, then holds its
+// result, with the consumer already on the server, until proceed.
+type blockingStream struct {
+	jsapi.Stream
+
+	entered, release, created, proceed chan struct{}
+	deleteErr                          error
+}
+
+func (b *blockingStream) OrderedConsumer(ctx context.Context, cfg jsapi.OrderedConsumerConfig) (jsapi.Consumer, error) {
+	close(b.entered)
+	<-b.release
+	c, err := b.Stream.OrderedConsumer(ctx, cfg)
+	close(b.created)
+	<-b.proceed
+	return c, err
+}
+
+func (b *blockingStream) DeleteConsumer(ctx context.Context, name string) error {
+	if b.deleteErr != nil {
+		return b.deleteErr
+	}
+	return b.Stream.DeleteConsumer(ctx, name)
+}
+
+// recoverInto wedges f with a change stored after its cursor and starts a
+// Next under ctx that reopens through a blockingStream; it returns once the
+// open blocks.
+func recoverInto(ctx context.Context, t *testing.T, s *Store, f *feed) (*blockingStream, chan error) {
+	t.Helper()
+	ns := message.Namespace{Queue: "q"}
+	appendAt(t, s, ns)
+	_, err := f.Next(context.Background())
+	require.NoError(t, err)
+	appendAt(t, s, ns)
+	b := &blockingStream{
+		Stream:  f.stream,
+		entered: make(chan struct{}), release: make(chan struct{}),
+		created: make(chan struct{}), proceed: make(chan struct{}),
+	}
+	f.stream = b
+	wedge(f)
+	done := make(chan error, 1)
+	go func() {
+		_, err := f.Next(ctx)
+		done <- err
+	}()
+	<-b.entered
+	return b, done
+}
+
+func TestFeedNextAfterCloseCreatesNoConsumer(t *testing.T) {
+	s, _ := feedStore(t)
+	appendAt(t, s, message.Namespace{Queue: "q"})
+	f := openWedgeable(t, s)
+	require.NoError(t, f.Close())
+	require.Empty(t, consumerNames(t, s))
+
+	_, err := f.Next(context.Background())
+	require.ErrorIs(t, err, message.ErrFeedClosed)
+	assert.Empty(t, consumerNames(t, s), "Next after Close opens nothing")
+	assert.NoError(t, f.Close(), "a repeated Close is a no-op")
+	assert.NoError(t, f.Close())
+}
+
+// TestFeedCloseInterruptsBlockedNext: a Next waiting on a consumer with
+// nothing to deliver returns message.ErrFeedClosed promptly once Close runs.
+func TestFeedCloseInterruptsBlockedNext(t *testing.T) {
+	s, _ := feedStore(t)
+	f := openWedgeable(t, s)
+	f.idleCheck = time.Minute
+	done := make(chan error, 1)
+	go func() {
+		_, err := f.Next(context.Background())
+		done <- err
+	}()
+	time.Sleep(200 * time.Millisecond)
+	require.NoError(t, f.Close())
+	select {
+	case err := <-done:
+		require.ErrorIs(t, err, message.ErrFeedClosed)
+	case <-time.After(5 * time.Second):
+		t.Fatal("Next still blocked after Close")
+	}
+	assert.Empty(t, consumerNames(t, s))
+}
+
+// TestFeedCloseConcurrentWithNext races Close against active Next calls and
+// recoveries; run under -race. Every round ends closed with no consumer.
+func TestFeedCloseConcurrentWithNext(t *testing.T) {
+	s, _ := feedStore(t)
+	ns := message.Namespace{Queue: "q"}
+	for range 3 {
+		appendAt(t, s, ns)
+	}
+	for round := range 20 {
+		f := openWedgeable(t, s)
+		f.idleCheck = 20 * time.Millisecond
+		var wg sync.WaitGroup
+		var last error
+		wg.Go(func() {
+			for {
+				_, err := f.Next(context.Background())
+				if err != nil {
+					last = err
+					return
+				}
+			}
+		})
+		time.Sleep(time.Duration(round%5) * 15 * time.Millisecond)
+		var closers sync.WaitGroup
+		for range 3 {
+			closers.Go(func() { assert.NoError(t, f.Close()) })
+		}
+		closers.Wait()
+		wg.Wait()
+		require.ErrorIs(t, last, message.ErrFeedClosed, "round %d", round)
+		require.Eventually(t, func() bool { return len(consumerNames(t, s)) == 0 }, 5*time.Second, 20*time.Millisecond, "round %d", round)
+	}
+}
+
+// stallingStream holds every message get until its context ends, as a slow
+// or unreachable leader would.
+type stallingStream struct {
+	jsapi.Stream
+
+	entered chan struct{}
+}
+
+func (s *stallingStream) GetMsg(ctx context.Context, _ uint64, _ ...jsapi.GetMsgOpt) (*jsapi.RawStreamMsg, error) {
+	close(s.entered)
+	<-ctx.Done()
+	return nil, ctx.Err()
+}
+
+// TestFeedCloseInterruptsRecoveryProbe: Close ends a Next blocked in the
+// idle check's stream read, not only one waiting on the consumer.
+func TestFeedCloseInterruptsRecoveryProbe(t *testing.T) {
+	s, _ := feedStore(t)
+	f := openWedgeable(t, s)
+	stall := &stallingStream{Stream: f.stream, entered: make(chan struct{})}
+	f.stream = stall
+	done := make(chan error, 1)
+	go func() {
+		_, err := f.Next(context.Background())
+		done <- err
+	}()
+	<-stall.entered
+	// Remove the iterator first, so only cancellation can end the read.
+	consumer, it := f.detach()
+	require.NoError(t, f.discard(consumer, it))
+	require.NoError(t, f.Close())
+	select {
+	case err := <-done:
+		require.ErrorIs(t, err, message.ErrFeedClosed)
+	case <-time.After(5 * time.Second):
+		t.Fatal("Next still blocked in the idle check after Close")
+	}
+}
+
+// TestFeedCloseWaitsForInFlightOpen: Close waits for a reopen already
+// creating a consumer, and that consumer is gone when Close returns nil.
+func TestFeedCloseWaitsForInFlightOpen(t *testing.T) {
+	s, _ := feedStore(t)
+	f := openWedgeable(t, s)
+	b, next := recoverInto(context.Background(), t, s, f)
+	closed := make(chan error, 1)
+	go func() { closed <- f.Close() }()
+	select {
+	case <-closed:
+		t.Fatal("Close returned while a consumer create was still in progress")
+	case <-time.After(200 * time.Millisecond):
+	}
+	close(b.release)
+	close(b.proceed)
+	select {
+	case err := <-closed:
+		require.NoError(t, err)
+	case <-time.After(5 * time.Second):
+		t.Fatal("Close did not return after the open finished")
+	}
+	assert.Empty(t, consumerNames(t, s), "no consumer survives a successful Close")
+	require.ErrorIs(t, <-next, message.ErrFeedClosed)
+}
+
+// TestFeedCloseReportsStuckOpen: past closeWait, Close returns an error
+// rather than success; a repeated Close waits again and succeeds once the
+// late consumer has been deleted.
+func TestFeedCloseReportsStuckOpen(t *testing.T) {
+	s, _ := feedStore(t)
+	f := openWedgeable(t, s)
+	f.closeWait = 200 * time.Millisecond
+	b, next := recoverInto(context.Background(), t, s, f)
+	err := f.Close()
+	require.ErrorIs(t, err, errCleanupPending)
+	close(b.release)
+	<-b.created
+	f.closeWait = 5 * time.Second
+	again := make(chan error, 1)
+	go func() { again <- f.Close() }()
+	select {
+	case <-again:
+		t.Fatal("a repeated Close returned while the late consumer still existed")
+	case <-time.After(200 * time.Millisecond):
+	}
+	close(b.proceed)
+	require.NoError(t, <-again)
+	assert.Empty(t, consumerNames(t, s))
+	require.ErrorIs(t, <-next, message.ErrFeedClosed)
+}
+
+// TestFeedCloseReportsLateDeleteFailure: if the consumer a racing reopen
+// created cannot be deleted, Close returns that error, not success.
+func TestFeedCloseReportsLateDeleteFailure(t *testing.T) {
+	s, _ := feedStore(t)
+	f := openWedgeable(t, s)
+	b, next := recoverInto(context.Background(), t, s, f)
+	injected := errors.New("injected delete failure")
+	b.deleteErr = injected
+	closed := make(chan error, 1)
+	go func() { closed <- f.Close() }()
+	for !f.isClosed() {
+		time.Sleep(time.Millisecond)
+	}
+	close(b.release)
+	close(b.proceed)
+	require.ErrorIs(t, <-closed, injected)
+	require.ErrorIs(t, <-next, message.ErrFeedClosed)
+}
+
+// closeOnDelivery takes a real delivery, then lets Close begin before Next
+// hands it back.
+type closeOnDelivery struct {
+	jsapi.MessagesContext
+
+	f *feed
+}
+
+func (c *closeOnDelivery) Next(opts ...jsapi.NextOpt) (jsapi.Msg, error) {
+	msg, err := c.MessagesContext.Next(opts...)
+	if err == nil {
+		go func() { _ = c.f.Close() }()
+		for !c.f.isClosed() {
+			time.Sleep(time.Millisecond)
+		}
+	}
+	return msg, err
+}
+
+// TestFeedWithholdsDeliveryRacingClose: a change received after Close began
+// is not returned, and a feed from the last delivered cursor still gets it.
+func TestFeedWithholdsDeliveryRacingClose(t *testing.T) {
+	s, _ := feedStore(t)
+	ns := message.Namespace{Queue: "q"}
+	first := appendAt(t, s, ns)
+	second := appendAt(t, s, ns)
+	f := openWedgeable(t, s)
+	c, err := f.Next(context.Background())
+	require.NoError(t, err)
+	require.Equal(t, first, c.Position)
+	f.it = &closeOnDelivery{MessagesContext: f.it, f: f}
+
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+	_, err = f.Next(ctx)
+	require.ErrorIs(t, err, message.ErrFeedClosed)
+
+	resumed, err := s.Feed(ctx, 0, c.Cursor)
+	require.NoError(t, err)
+	defer func() { _ = resumed.Close() }()
+	got, err := resumed.Next(ctx)
+	require.NoError(t, err)
+	assert.Equal(t, second, got.Position, "the withheld change is not lost")
+}

spinifex/domains/sqs/message/jetstream/feed_internal_test.go

@@ -0,0 +1,153 @@
+package jetstream
+
+import (
+	"context"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/mulgadc/spinifex/spinifex/domains/sqs/message"
+	"github.com/mulgadc/spinifex/spinifex/foundation/state/clustersize"
+	"github.com/nats-io/nats-server/v2/server"
+	"github.com/nats-io/nats.go"
+	jsapi "github.com/nats-io/nats.go/jetstream"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// wedgedIterator models a consumer whose pull has stopped delivering: every
+// Next waits out its window and returns nothing, as the live one does.
+type wedgedIterator struct {
+	wait    time.Duration
+	nexts   atomic.Int32
+	stopped atomic.Bool
+}
+
+var _ jsapi.MessagesContext = (*wedgedIterator)(nil)
+
+func (w *wedgedIterator) Next(...jsapi.NextOpt) (jsapi.Msg, error) {
+	w.nexts.Add(1)
+	time.Sleep(w.wait)
+	return nil, context.DeadlineExceeded
+}
+
+func (w *wedgedIterator) Stop()  { w.stopped.Store(true) }
+func (w *wedgedIterator) Drain() { w.stopped.Store(true) }
+
+func feedStore(t *testing.T) (*Store, jsapi.JetStream) {
+	t.Helper()
+	srv, err := server.NewServer(&server.Options{
+		Host: "127.0.0.1", Port: -1, JetStream: true, StoreDir: t.TempDir(), NoLog: true, NoSigs: true,
+	})
+	require.NoError(t, err)
+	go srv.Start()
+	require.True(t, srv.ReadyForConnections(10*time.Second))
+	t.Cleanup(srv.Shutdown)
+	nc, err := nats.Connect(srv.ClientURL())
+	require.NoError(t, err)
+	t.Cleanup(nc.Close)
+	js, err := jsapi.New(nc)
+	require.NoError(t, err)
+	clustersize.DeclareForTest(t, 1)
+	s, err := New(context.Background(), js, Config{MessageStreams: 1, StateBuckets: 1, DuplicateWindow: time.Minute})
+	require.NoError(t, err)
+	return s, js
+}
+
+func appendAt(t *testing.T, s *Store, ns message.Namespace) message.Position {
+	t.Helper()
+	res, err := s.Append(context.Background(), ns, message.AppendInput{Body: []byte("m")})
+	require.NoError(t, err)
+	return res.Position
+}
+
+// openWedgeable opens a feed with a short idle check, so recovery is quick.
+func openWedgeable(t *testing.T, s *Store) *feed {
+	t.Helper()
+	fd, err := s.Feed(context.Background(), 0, 0)
+	require.NoError(t, err)
+	t.Cleanup(func() { _ = fd.Close() })
+	f := fd.(*feed)
+	f.idleCheck = 100 * time.Millisecond
+	return f
+}
+
+// wedge swaps the feed's live iterator for one that never delivers.
+func wedge(f *feed) *wedgedIterator {
+	f.it.Stop()
+	w := &wedgedIterator{wait: f.idleCheck}
+	f.it = w
+	return w
+}
+
+func consumerNames(t *testing.T, s *Store) []string {
+	t.Helper()
+	stream, err := s.js.Stream(context.Background(), StreamName(0))
+	require.NoError(t, err)
+	var names []string
+	lister := stream.ConsumerNames(context.Background())
+	for name := range lister.Name() {
+		names = append(names, name)
+	}
+	require.NoError(t, lister.Err())
+	return names
+}
+
+// TestFeedRecoversFromWedgedPull: with a change stored after the cursor and a
+// consumer that delivers nothing, Next replaces the consumer from the cursor
+// and returns that change within one idle check plus the reopen.
+func TestFeedRecoversFromWedgedPull(t *testing.T) {
+	s, _ := feedStore(t)
+	ns := message.Namespace{Queue: "q"}
+	first, second := appendAt(t, s, ns), appendAt(t, s, ns)
+	f := openWedgeable(t, s)
+
+	c, err := f.Next(context.Background())
+	require.NoError(t, err)
+	require.Equal(t, first, c.Position)
+	wedged := f.consumer.CachedInfo().Name
+	w := wedge(f)
+
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+	start := time.Now()
+	c, err = f.Next(ctx)
+	require.NoError(t, err)
+	assert.Equal(t, second, c.Position, "the next change after the cursor, none skipped or repeated")
+	assert.Less(t, time.Since(start), 2*f.idleCheck+time.Second, "recovery is bounded by the idle check and one reopen")
+	assert.True(t, w.stopped.Load(), "the wedged iterator is stopped")
+	assert.EqualValues(t, 1, w.nexts.Load(), "one idle check finds the stored change")
+	assert.NotContains(t, consumerNames(t, s), wedged, "the wedged consumer is deleted, not left to linger")
+	assert.Len(t, consumerNames(t, s), 1)
+
+	live := appendAt(t, s, ns)
+	c, err = f.Next(ctx)
+	require.NoError(t, err)
+	assert.Equal(t, live, c.Position, "the replacement consumer delivers live appends")
+}
+
+// TestFeedIdleWithNothingStoredKeepsConsumer: an idle feed caught up with the
+// stream waits on its consumer and never rebuilds it.
+func TestFeedIdleWithNothingStoredKeepsConsumer(t *testing.T) {
+	s, _ := feedStore(t)
+	ns := message.Namespace{Queue: "q"}
+	appendAt(t, s, ns)
+	f := openWedgeable(t, s)
+	_, err := f.Next(context.Background())
+	require.NoError(t, err)
+	name := f.consumer.CachedInfo().Name
+	w := wedge(f)
+
+	ctx, cancel := context.WithTimeout(context.Background(), 350*time.Millisecond)
+	defer cancel()
+	_, err = f.Next(ctx)
+	require.ErrorIs(t, err, context.DeadlineExceeded)
+	assert.GreaterOrEqual(t, w.nexts.Load(), int32(3), "the feed keeps waiting, one idle check after another")
+	assert.False(t, w.stopped.Load(), "nothing stored after the cursor, so no rebuild")
+	assert.Equal(t, []string{name}, consumerNames(t, s))
+}
+
+// detachForTest takes the live consumer as tracked work, for discard.
+func (f *feed) detachForTest() (jsapi.Consumer, jsapi.MessagesContext) {
+	return f.detach()
+}