feat: serve the OCI referrers API for ECR repositories (#1319)

* feat: record manifest subject and artifactType and return OCI-Subject on push

* fix: resolve the manifest media type once so a parameterised Content-Type cannot misclassify an index

* feat: index referrers per subject and serve them through the metadata service

* feat: serve the OCI referrers API for ECR repositories

* test: cover the ECR referrers route in the gateway authorization matrices

* feat: delete a subject image's referrers with it

* fix: satisfy nilerr in the referrers manifest parser

* fix: return the manifest parse error and format the referrers code

* docs: describe the ECR OCI referrers API and cosign support

* docs: name oras cp and crane copy as the way to carry v3 signatures to an ECR mirror

* docs: present oras cp and crane copy as one remedy for copied v3 signatures

cmd/eks-token-webhook/main.go

@@ -29,6 +29,7 @@ func loadConfig() (tokenwebhook.Config, error) {
 		CertPath:    envOr("EKS_WEBHOOK_CERT", "/etc/spinifex-eks/token-webhook.crt"),
 		KeyPath:     envOr("EKS_WEBHOOK_KEY", "/etc/spinifex-eks/token-webhook.key"),
 		Kubeconfig:  envOr("EKS_WEBHOOK_KUBECONFIG", "/etc/spinifex-eks/token-webhook.kubeconfig"),
+		ReadyAddr:   os.Getenv("EKS_WEBHOOK_READY_ADDR"),
 	}
 	// Static SigV4 creds are optional: when absent, eksgw.New signs with the
 	// AWS SDK chain (IMDS instance-role creds), the same path the CP VM's

docs/DESIGN.md

@@ -229,6 +229,20 @@ type EBSRequest struct {
 
 Viperblock responds with an NBD URI that QEMU uses to access the block device.
 
+#### Direct-boot microVM data volumes
+
+A direct-boot microVM keeps the launcher's saved `vm.Config` as a clean base, and `startQEMU` adds data volumes only to a launch-only copy of it (`runtime/compute/vm/lifecycle.go`).
+At each start the daemon takes one snapshot of the instance's already-mounted data requests, probes each NBD or host block export from that snapshot, and builds the command from the same snapshot, so the launched endpoint is the one that was checked and is never written back into the saved base.
+Each data volume becomes a `virtio-blk-device` on the MMIO bus with the same id, node name, I/O thread, serial and `werror=stop,rerror=stop` policy as the PCI device, and takes no PCIe hot-plug port.
+
+Boot and EFI requests are rejected on this path, and a direct-boot instance with no data requests launches its base unchanged.
+Disk-boot (q35) instances keep the existing PCI hot-plug path.
+
+Hot-attaching a volume to a running direct-boot microVM is not supported, because the MMIO bus has no PCIe root port to plug into.
+`AttachVolume` refuses it before mounting the volume or changing any instance or volume state, and the EC2 API returns `UnsupportedOperation`.
+
+This covers only requests that are already mounted when the instance starts; nothing yet supplies data volumes to a system instance, and the path is verified with fake-QEMU unit tests, not with a real EBS volume in a guest.
+
 ### Predastore (S3)
 
 Object storage used for:

docs/containers/ecr/README.md

@@ -45,6 +45,7 @@ Each repository holds image manifests and their layers. Images are stored in obj
 
 - Repository lifecycle — `CreateRepository`, `DeleteRepository`, `DescribeRepositories`.
 - OCI push and pull — blobs, manifests, and tags, including cross-repository blob mounts.
+- OCI 1.1 referrers — the referrers API, so cosign v3 signatures and other artifacts attached to an image can be listed (see [OCI referrers and image signing](#oci-referrers-and-image-signing)).
 - Token auth — `GetAuthorizationToken` plus bearer/basic auth on `/v2/`.
 - Repository policies — set, get, and delete a repository policy document.
 
@@ -195,6 +196,52 @@ Limits:
 Measured on a single-node environment: pulls by `repositoryUri` and by the account's ECR DNS name, a pull rejected with `x509: certificate signed by unknown authority` when the gateway host was not in the worker config, and a 31 second pull through an unreachable first endpoint.
 Measured on a 3-node environment: a pod pulling each of the three gateways' `repositoryUri` on a worker launched with this change (two of the three failed with `x509` on a worker launched without it), and a pull of the stopped gateway's own `repositoryUri` that succeeded through another gateway within one retry (the pull itself took 147 ms after the first endpoint refused).
 
+## OCI referrers and image signing
+
+A manifest can name another manifest as its `subject`, which is how tools such as cosign v3 attach signatures and SBOMs to an image.
+The registry records the subject when the manifest is pushed and answers with an `OCI-Subject` header on the `201`.
+The `subject` does not have to exist in the repository when the referrer is pushed.
+
+`GET /v2/<name>/referrers/<digest>` lists the manifests that name `<digest>` as their subject.
+The request is authorised as `ecr:BatchGetImage` on the repository.
+The response is `200` with `Content-Type: application/vnd.oci.image.index.v1+json` and an index whose `manifests` hold each referrer's `mediaType`, `digest`, `size`, `artifactType` and annotations.
+A digest with no referrers, or one that is not stored, returns an empty index.
+A malformed digest returns `400` with `DIGEST_INVALID`.
+
+Adding `?artifactType=<type>` returns only referrers of that type and sets `OCI-Filters-Applied: artifactType`.
+The header is also set when nothing matches.
+An image manifest without an `artifactType` is listed under its config `mediaType`; an index without one has none.
+
+Deleting an image by digest (`DELETE /v2/<name>/manifests/<digest>`, `aws ecr batch-delete-image` with `imageDigest`) also deletes its referrers, and their referrers in turn.
+Deleting by tag removes only the tag and leaves the image and its referrers.
+
+cosign v3 sign and verify work against the registry, including on a child manifest of an image index copied with `cosign copy`.
+cosign v2 keeps writing its `sha256-<digest>.sig` tags and is unaffected.
+`GET` on a manifest still returns `406` when the request's `Accept` header does not include the stored `mediaType`.
+
+Limits:
+
+- `cosign copy` does not copy v3 signatures to the destination repository.
+  This is client behaviour.
+  To keep the original signature, copy with `oras cp -r <source> <mirror>`, which copies the image and its referrers in one command, or `crane copy` the signature manifest by digest after the image.
+  Otherwise, sign the copy again.
+- Referrers pushed before this version are not listed.
+  Push them again to add them to the index.
+- A crash between deleting an image and deleting its referrers can leave referrers behind that nothing lists.
+- The referrers listing is not paginated.
+- `HEAD` on the referrers route is not served.
+
+Measured on a 3-node environment with cosign v3.1.4 and v2.6.5:
+
+- cosign v3.1.4 signed a fresh image index, with one `GET /referrers` answered `200`, no `sha256-<digest>` tag written and no `406`.
+- Verify passed.
+- A copied index's child manifest signed and verified, which returned `406` before this change.
+- The listing returned the signature with its `artifactType`, the filter returned one entry for the signature's type and none for another type.
+- `aws ecr batch-delete-image` of the signed image left an empty listing and `404` for both the signature and the image.
+- cosign v2.6.5 signed and verified a fresh image.
+
+Not measured: `oras`, `notation`, a listing with many referrers, and a crash in the middle of a delete.
+
 ## Troubleshooting
 
 **`docker login` fails with an authorization error.** The token is account-scoped and short-lived. Re-run `aws ecr get-login-password` to mint a fresh token, and confirm the registry host in `docker login` matches your account's endpoint (`aws ecr describe-repositories` returns it).

docs/containers/eks/README.md

@@ -60,6 +60,36 @@ Authentication uses **EKS access entries** (the API authentication mode) — IAM
 - **The worker image is fixed.** Node groups always boot Spinifex's `eks-node` image. `ami_type` is recorded but does not select the image, and the image must be registered on the cluster before you create a node group.
 - **Default Kubernetes version is `1.32`.**
 
+### Control plane readiness and kubectl authentication
+
+Each control-plane VM authenticates `kubectl` tokens through a webhook that calls the AWS gateway on the host the VM runs on.
+When that gateway is stopped, the webhook cannot validate tokens and `kubectl` reports `Unauthorized` for the share of requests that the load balancer sends to that control plane.
+
+The webhook serves `/readyz` on port 8444 of each control-plane VM.
+It returns 200 only when the host gateway answers a TLS handshake and the local API server accepts a TCP connection, and 503 otherwise.
+The API server target group health-checks `HTTP :8444/readyz` every 5 seconds with a healthy and unhealthy threshold of 2.
+The target group still forwards TCP on 6443.
+A control plane that cannot authenticate leaves rotation, and returns when its gateway is back.
+
+Measured on a three-node cluster, one control plane per host, with `kubectl get ns` run continuously after stopping one gateway:
+
+| Gateway stopped | Result |
+| --- | --- |
+| Host without the cluster load balancer VM, first run | 4 of 20 calls failed in the first round, 0 of 20 in the next round 54 s after the stop |
+| Host without the cluster load balancer VM, second run | 2 failures in 100 s, at 3 s and 8 s after the stop |
+| Host running the cluster load balancer VM | Not fixed: 20 of 60 calls failed over 100 s |
+
+Before this change the same test failed 3, 4 and 3 calls of 10 per stopped gateway.
+
+Limits:
+
+- Only clusters created with the current `eks-node` image get the readiness listener, and only clusters created from now on get the health check on the target group.
+  Older clusters keep the TCP check on 6443 and are not changed.
+- A cluster created with an older image gets the health check on its target group but no listener on 8444, so every control plane fails the check and the load balancer fails open to all of them.
+- A stopped gateway on the host that runs the cluster load balancer VM is not covered.
+  The load balancer VM reports target health through its own host's gateway, so while that gateway is down the unhealthy control plane stays in rotation.
+- Not measured: more than one control plane per host, a physical-node failure instead of a stopped gateway, and the target health state as `DescribeTargetHealth` reports it, because the load balancer and control-plane VMs belong to the system account.
+
 ## Prerequisites
 
 > [!IMPORTANT]
@@ -256,6 +286,10 @@ The workbooks form a ladder. **eks-quickstart** is a NodePort demo; **eks-https-
 
 **`kubectl` cannot connect.** Re-run `aws eks update-kubeconfig --name <name>` to refresh the endpoint and CA, and confirm the cluster is `ACTIVE`. `aws sts get-caller-identity` verifies your credentials reach the Spinifex endpoint.
 
+**`kubectl` returns `Unauthorized` for some requests and works for others.**
+One control plane cannot reach its host's AWS gateway, so check `systemctl status spinifex-awsgw` on each node.
+Control plane readiness and kubectl authentication above lists which clusters recover on their own.
+
 **Workers stay `NotReady`.** Almost always an egress or IAM problem:
 - Confirm the workers can pull images — a default route to an Internet Gateway (public) or NAT Gateway (private).
 - Confirm the node role has `AmazonEKSWorkerNodePolicy`, `AmazonEKS_CNI_Policy`, and `AmazonEC2ContainerRegistryReadOnly`.

docs/coverage/elbv2/README.md

@@ -24,6 +24,34 @@ Spinifex implements **36 operations** in the ELBv2 `2015-12-01` API model.
 
 The data plane is a system-managed load balancer VM, launched automatically during `CreateLoadBalancer`. Application Load Balancers run HAProxy for the L7 surface — rules, fixed responses and redirects over HTTP and HTTPS. Network Load Balancers run nginx `stream` for the L4 surface — TCP, UDP, TLS and TCP_UDP — because HAProxy cannot load-balance UDP. The agent selects the engine from the configuration the control plane delivers, so the choice follows the load balancer type and is not separately configurable.
 
+### Target health and routing
+
+Traffic goes only to targets that are healthy or still `initial`.
+A target that has failed its health check is removed from the data plane on the next render and put back when it passes again.
+When every target in a group is unhealthy, the load balancer fails open and sends traffic to all of them rather than refusing it.
+A Network Load Balancer renders every registered server in that case, and an Application Load Balancer falls back to unchecked backup servers that HAProxy uses only when every checked server is down.
+`DescribeTargetHealth` keeps reporting the targets as unhealthy while this happens.
+Draining targets and targets without an address never receive traffic.
+
+An `initial` target receives traffic before its first successful check.
+This differs from AWS, where a target takes no traffic until it passes.
+Health state is reset to `initial` for every target when the control plane restarts, so excluding `initial` targets would drop all traffic after a restart.
+
+Changing the health check of a target group with `ModifyTargetGroup` takes effect on the next render of every load balancer that uses it.
+The time from a backend failing to traffic stopping is the health check interval times the unhealthy threshold, plus the agent poll of about 5 seconds and the heartbeat that reports the result.
+Recovery takes the interval times the healthy threshold, plus the same delays.
+
+Measured on a three-node cluster with a 5 second interval and thresholds of 2:
+
+| Event | Time |
+| --- | --- |
+| Backend starts failing to reported `unhealthy` | 21 s |
+| Backend recovers to reported `healthy` | 15 s, serving again 6 s later |
+| Every backend failing to reported `unhealthy` | 12 s, fail open immediate |
+
+A TCP connection held through a Network Load Balancer survived four configuration reloads while the other backend changed state, because the agent applies nginx with a graceful reload and HAProxy with a graceful restart.
+The HAProxy configuration with the backup servers was accepted by a live Application Load Balancer, but its rendered text was not dumped and checked by hand.
+
 ### Operations
 
 | Operation |

docs/coverage/elbv2/intro.md

@@ -1,3 +1,31 @@
 ### Two data planes
 
 The data plane is a system-managed load balancer VM, launched automatically during `CreateLoadBalancer`. Application Load Balancers run HAProxy for the L7 surface — rules, fixed responses and redirects over HTTP and HTTPS. Network Load Balancers run nginx `stream` for the L4 surface — TCP, UDP, TLS and TCP_UDP — because HAProxy cannot load-balance UDP. The agent selects the engine from the configuration the control plane delivers, so the choice follows the load balancer type and is not separately configurable.
+
+### Target health and routing
+
+Traffic goes only to targets that are healthy or still `initial`.
+A target that has failed its health check is removed from the data plane on the next render and put back when it passes again.
+When every target in a group is unhealthy, the load balancer fails open and sends traffic to all of them rather than refusing it.
+A Network Load Balancer renders every registered server in that case, and an Application Load Balancer falls back to unchecked backup servers that HAProxy uses only when every checked server is down.
+`DescribeTargetHealth` keeps reporting the targets as unhealthy while this happens.
+Draining targets and targets without an address never receive traffic.
+
+An `initial` target receives traffic before its first successful check.
+This differs from AWS, where a target takes no traffic until it passes.
+Health state is reset to `initial` for every target when the control plane restarts, so excluding `initial` targets would drop all traffic after a restart.
+
+Changing the health check of a target group with `ModifyTargetGroup` takes effect on the next render of every load balancer that uses it.
+The time from a backend failing to traffic stopping is the health check interval times the unhealthy threshold, plus the agent poll of about 5 seconds and the heartbeat that reports the result.
+Recovery takes the interval times the healthy threshold, plus the same delays.
+
+Measured on a three-node cluster with a 5 second interval and thresholds of 2:
+
+| Event | Time |
+| --- | --- |
+| Backend starts failing to reported `unhealthy` | 21 s |
+| Backend recovers to reported `healthy` | 15 s, serving again 6 s later |
+| Every backend failing to reported `unhealthy` | 12 s, fail open immediate |
+
+A TCP connection held through a Network Load Balancer survived four configuration reloads while the other backend changed state, because the agent applies nginx with a graceful reload and HAProxy with a graceful restart.
+The HAProxy configuration with the backup servers was accepted by a live Application Load Balancer, but its rendered text was not dumped and checked by hand.

docs/testing/e2e-coverage.md

@@ -112,6 +112,9 @@ CA installed and downloadable, fingerprint matches config, hostnames present in
 
 Internal and internet-facing ALB and NLB, HTTPS listeners, listener rules, `ModifyListener`, and NLB UDP.
 
+Unhealthy-target routing: an NLB with an HTTP health check on a TCP target group stops sending traffic to a target that fails it while still accepting TCP, sends traffic again once it recovers, and fails open when every target is unhealthy.
+An ALB whose targets all fail their health check keeps serving from them instead of answering 503.
+
 ### `eks`, `ecs`, `ecr` — container services
 
 `eks`: cluster create/delete, access entries, addon delivery, `GetToken`, kubeconfig artifacts, the EBS CSI volume path, and an IRSA pod wired with a projected SA token. `ecs`: clusters, container instances, the three network modes (`awsvpc`, `bridge`, `host`), task-role credentials fetched from the container credentials endpoint, a service behind an ELB, and ENI release on terminate. `ecr`: control-plane CRUD plus a real OCI data plane — docker login/build/push/pull round trips, multi-layer images, and raw blob HTTP.

scripts/images/eks-node/eks-token-webhook.initd

@@ -25,7 +25,7 @@ error_log="/dev/console"
 if [ -f /etc/spinifex-eks/first-boot.env ]; then
     # shellcheck disable=SC1091
     . /etc/spinifex-eks/first-boot.env
-    export EKS_GATEWAY_URL EKS_GATEWAY_CA EKS_ACCESS_KEY EKS_SECRET_KEY EKS_REGION EKS_ACCOUNT_ID EKS_CLUSTER_NAME
+    export EKS_GATEWAY_URL EKS_GATEWAY_CA EKS_ACCESS_KEY EKS_SECRET_KEY EKS_REGION EKS_ACCOUNT_ID EKS_CLUSTER_NAME EKS_WEBHOOK_READY_ADDR
 fi
 
 depend() {

spinifex/agents/eks/tokenwebhook/readiness.go

@@ -0,0 +1,128 @@
+package tokenwebhook
+
+import (
+	"context"
+	"crypto/tls"
+	"crypto/x509"
+	"fmt"
+	"net"
+	"net/http"
+	"net/url"
+	"os"
+	"sync"
+	"time"
+
+	"github.com/mulgadc/bluebottle/pkg/tlsconfig"
+)
+
+const (
+	readinessTimeout = 2 * time.Second
+	readinessTTL     = 2 * time.Second
+)
+
+// defaultAPIServerAddr is the local k3s apiserver the readiness check dials.
+const defaultAPIServerAddr = "127.0.0.1:6443"
+
+// gatewayProbe reports whether this control plane can serve: the gateway it
+// relays to accepts a TLS handshake and the local apiserver accepts a TCP
+// connection. It sends no credentials and caches the result so frequent load
+// balancer probes never reach either more than once per ttl.
+type gatewayProbe struct {
+	addr    string
+	apiAddr string
+	tlsCfg  *tls.Config
+	timeout time.Duration
+	ttl     time.Duration
+	now     func() time.Time
+
+	mu      sync.Mutex
+	checked time.Time
+	ok      bool
+}
+
+func newGatewayProbe(gatewayURL, caPath, apiAddr string) (*gatewayProbe, error) {
+	u, err := url.Parse(gatewayURL)
+	if err != nil {
+		return nil, fmt.Errorf("parse gateway URL %q: %w", gatewayURL, err)
+	}
+	host, port := u.Hostname(), u.Port()
+	if host == "" {
+		return nil, fmt.Errorf("gateway URL %q has no host", gatewayURL)
+	}
+	if port == "" {
+		port = "443"
+	}
+	cfg := &tls.Config{MinVersion: tls.VersionTLS13, CurvePreferences: tlsconfig.Curves}
+	if caPath != "" {
+		pem, err := os.ReadFile(caPath)
+		if err != nil {
+			return nil, fmt.Errorf("read gateway CA %q: %w", caPath, err)
+		}
+		pool := x509.NewCertPool()
+		if !pool.AppendCertsFromPEM(pem) {
+			return nil, fmt.Errorf("gateway CA %q has no usable certificates", caPath)
+		}
+		cfg.RootCAs = pool
+	}
+	if apiAddr == "" {
+		apiAddr = defaultAPIServerAddr
+	}
+	return &gatewayProbe{
+		addr:    net.JoinHostPort(host, port),
+		apiAddr: apiAddr,
+		tlsCfg:  cfg,
+		timeout: readinessTimeout,
+		ttl:     readinessTTL,
+		now:     time.Now,
+	}, nil
+}
+
+// ready returns the cached result while it is younger than ttl, else probes.
+// Concurrent callers wait for one probe and share its result.
+func (p *gatewayProbe) ready() bool {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	if !p.checked.IsZero() && p.now().Sub(p.checked) < p.ttl {
+		return p.ok
+	}
+	p.ok = p.probe()
+	p.checked = p.now()
+	return p.ok
+}
+
+func (p *gatewayProbe) probe() bool {
+	ctx, cancel := context.WithTimeout(context.Background(), p.timeout)
+	defer cancel()
+	d := tls.Dialer{NetDialer: &net.Dialer{}, Config: p.tlsCfg}
+	conn, err := d.DialContext(ctx, "tcp", p.addr)
+	if err != nil {
+		return false
+	}
+	_ = conn.Close()
+
+	// A control plane whose apiserver is down must leave rotation too.
+	api, err := (&net.Dialer{}).DialContext(ctx, "tcp", p.apiAddr)
+	if err != nil {
+		return false
+	}
+	_ = api.Close()
+	return true
+}
+
+// handle serves 200 when the gateway and apiserver are reachable, else 503.
+func (p *gatewayProbe) handle(w http.ResponseWriter, _ *http.Request) {
+	if p.ready() {
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte("ok"))
+		return
+	}
+	http.Error(w, "not ready", http.StatusServiceUnavailable)
+}
+
+// readinessHandler serves only /readyz; the listener it backs is plain HTTP and
+// reachable beyond loopback, so it must never expose /authenticate.
+func readinessHandler(p *gatewayProbe) http.Handler {
+	mux := http.NewServeMux()
+	mux.HandleFunc("/readyz", p.handle)
+	return mux
+}

spinifex/agents/eks/tokenwebhook/readiness_test.go

@@ -0,0 +1,251 @@
+package tokenwebhook
+
+import (
+	"crypto/tls"
+	"net"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// tlsGateway is a loopback TLS listener standing in for the gateway. It counts
+// accepted connections and can be stopped and restarted on the same address.
+type tlsGateway struct {
+	t       *testing.T
+	api     *fakeAPIServer
+	addr    string
+	cert    tls.Certificate
+	caPath  string
+	ln      net.Listener
+	accepts atomic.Int32
+}
+
+func newTLSGateway(t *testing.T) *tlsGateway {
+	t.Helper()
+	dir := t.TempDir()
+	cert, certPEM, err := ensureServingCert(filepath.Join(dir, "c.pem"), filepath.Join(dir, "k.pem"))
+	require.NoError(t, err)
+	caPath := filepath.Join(dir, "ca.pem")
+	require.NoError(t, os.WriteFile(caPath, certPEM, 0o600))
+
+	g := &tlsGateway{t: t, cert: cert, caPath: caPath, api: newFakeAPIServer(t)}
+	g.start("127.0.0.1:0")
+	t.Cleanup(g.stop)
+	return g
+}
+
+func (g *tlsGateway) start(addr string) {
+	g.t.Helper()
+	ln, err := tls.Listen("tcp", addr, serverTLSConfig(g.cert))
+	require.NoError(g.t, err)
+	g.ln = ln
+	g.addr = ln.Addr().String()
+	go func() {
+		for {
+			c, err := ln.Accept()
+			if err != nil {
+				return
+			}
+			g.accepts.Add(1)
+			go func() {
+				if tc, ok := c.(*tls.Conn); ok {
+					_ = tc.Handshake()
+				}
+				_ = c.Close()
+			}()
+		}
+	}()
+}
+
+func (g *tlsGateway) stop() {
+	if g.ln != nil {
+		_ = g.ln.Close()
+		g.ln = nil
+	}
+}
+
+// fakeAPIServer is a plain TCP listener standing in for the local apiserver.
+type fakeAPIServer struct {
+	addr string
+	ln   net.Listener
+}
+
+func newFakeAPIServer(t *testing.T) *fakeAPIServer {
+	t.Helper()
+	a := &fakeAPIServer{}
+	a.start("127.0.0.1:0", t)
+	t.Cleanup(a.stop)
+	return a
+}
+
+func (a *fakeAPIServer) start(addr string, t *testing.T) {
+	t.Helper()
+	ln, err := net.Listen("tcp", addr)
+	require.NoError(t, err)
+	a.ln = ln
+	a.addr = ln.Addr().String()
+	go func() {
+		for {
+			c, err := ln.Accept()
+			if err != nil {
+				return
+			}
+			_ = c.Close()
+		}
+	}()
+}
+
+func (a *fakeAPIServer) stop() {
+	if a.ln != nil {
+		_ = a.ln.Close()
+		a.ln = nil
+	}
+}
+
+func (g *tlsGateway) probe(t *testing.T) *gatewayProbe {
+	t.Helper()
+	p, err := newGatewayProbe("https://"+g.addr, g.caPath, g.api.addr)
+	require.NoError(t, err)
+	return p
+}
+
+func TestGatewayProbe_UpAndRefused(t *testing.T) {
+	g := newTLSGateway(t)
+	p := g.probe(t)
+	p.ttl = 0
+
+	assert.True(t, p.ready())
+
+	g.stop()
+	assert.False(t, p.ready(), "a closed gateway port is not ready")
+}
+
+func TestGatewayProbe_RecoversAfterDown(t *testing.T) {
+	g := newTLSGateway(t)
+	p := g.probe(t)
+	p.ttl = 0
+	addr := g.addr
+
+	g.stop()
+	require.False(t, p.ready())
+
+	g.start(addr)
+	assert.True(t, p.ready(), "ready again once the gateway is back")
+}
+
+func TestGatewayProbe_TimesOutOnStalledHandshake(t *testing.T) {
+	g := newTLSGateway(t)
+	// A plain listener that accepts and never answers the TLS handshake.
+	ln, err := net.Listen("tcp", "127.0.0.1:0")
+	require.NoError(t, err)
+	t.Cleanup(func() { _ = ln.Close() })
+	go func() {
+		for {
+			c, err := ln.Accept()
+			if err != nil {
+				return
+			}
+			t.Cleanup(func() { _ = c.Close() })
+		}
+	}()
+
+	p, err := newGatewayProbe("https://"+ln.Addr().String(), g.caPath, g.api.addr)
+	require.NoError(t, err)
+	p.timeout = 200 * time.Millisecond
+	p.ttl = 0
+
+	start := time.Now()
+	assert.False(t, p.ready())
+	assert.Less(t, time.Since(start), 2*time.Second, "the probe gives up at its timeout")
+}
+
+func TestGatewayProbe_RejectsUntrustedCertificate(t *testing.T) {
+	g := newTLSGateway(t)
+	other := newTLSGateway(t)
+	p, err := newGatewayProbe("https://"+g.addr, other.caPath, g.api.addr)
+	require.NoError(t, err)
+	p.ttl = 0
+
+	assert.False(t, p.ready(), "a certificate not signed by the configured CA is not ready")
+}
+
+func TestGatewayProbe_CachesWithinTTL(t *testing.T) {
+	g := newTLSGateway(t)
+	p := g.probe(t)
+	clock := time.Unix(1_000_000, 0)
+	p.now = func() time.Time { return clock }
+	p.ttl = 2 * time.Second
+
+	require.True(t, p.ready())
+	require.True(t, p.ready())
+	require.True(t, p.ready())
+	assert.Equal(t, int32(1), g.accepts.Load(), "calls inside the ttl share one probe")
+
+	g.stop()
+	assert.True(t, p.ready(), "a cached answer is served while the ttl holds")
+
+	clock = clock.Add(3 * time.Second)
+	assert.False(t, p.ready(), "after the ttl the gateway is probed again")
+}
+
+func TestGatewayProbe_HandlerStatus(t *testing.T) {
+	g := newTLSGateway(t)
+	p := g.probe(t)
+	p.ttl = 0
+
+	w := httptest.NewRecorder()
+	p.handle(w, httptest.NewRequest(http.MethodGet, "/readyz", nil))
+	assert.Equal(t, http.StatusOK, w.Code)
+
+	g.stop()
+	w = httptest.NewRecorder()
+	p.handle(w, httptest.NewRequest(http.MethodGet, "/readyz", nil))
+	assert.Equal(t, http.StatusServiceUnavailable, w.Code)
+}
+
+func TestNewGatewayProbe_DefaultsPortAndRejectsBadInput(t *testing.T) {
+	p, err := newGatewayProbe("https://gw.example", "", "")
+	require.NoError(t, err)
+	assert.Equal(t, "gw.example:443", p.addr)
+	assert.Equal(t, defaultAPIServerAddr, p.apiAddr)
+
+	_, err = newGatewayProbe("https://", "", "")
+	require.Error(t, err)
+	_, err = newGatewayProbe("https://gw.example:9999", "/no/such/ca.pem", "")
+	require.Error(t, err)
+}
+
+func TestGatewayProbe_NotReadyWhenAPIServerDown(t *testing.T) {
+	g := newTLSGateway(t)
+	p := g.probe(t)
+	p.ttl = 0
+	require.True(t, p.ready())
+
+	g.api.stop()
+	assert.False(t, p.ready(), "a dead apiserver must not look ready while the gateway is up")
+
+	w := httptest.NewRecorder()
+	p.handle(w, httptest.NewRequest(http.MethodGet, "/readyz", nil))
+	assert.Equal(t, http.StatusServiceUnavailable, w.Code)
+}
+
+func TestReadinessHandler_ServesOnlyReadyz(t *testing.T) {
+	g := newTLSGateway(t)
+	h := readinessHandler(g.probe(t))
+
+	for _, path := range []string{"/authenticate", "/healthz", "/"} {
+		w := httptest.NewRecorder()
+		h.ServeHTTP(w, httptest.NewRequest(http.MethodPost, path, nil))
+		assert.Equalf(t, http.StatusNotFound, w.Code, "%s must not be served on the readiness listener", path)
+	}
+	w := httptest.NewRecorder()
+	h.ServeHTTP(w, httptest.NewRequest(http.MethodGet, "/readyz", nil))
+	assert.Equal(t, http.StatusOK, w.Code)
+}

spinifex/agents/eks/tokenwebhook/run_test.go

@@ -128,3 +128,58 @@ func TestRunServesUntilCancelled(t *testing.T) {
 		t.Fatal("Run did not return after cancel")
 	}
 }
+
+func TestRunServesReadinessOnSeparateListener(t *testing.T) {
+	g := newTLSGateway(t)
+	cfg := runConfig(t, freeAddr(t))
+	cfg.GatewayURL = "https://" + g.addr
+	cfg.GatewayCA = g.caPath
+	cfg.APIServerAddr = g.api.addr
+	cfg.ReadyAddr = freeAddr(t)
+
+	ctx, cancel := context.WithCancel(context.Background())
+	done := make(chan error, 1)
+	go func() { done <- Run(ctx, cfg) }()
+
+	client := &http.Client{Timeout: 3 * time.Second}
+	require.Eventually(t, func() bool {
+		resp, err := client.Get("http://" + cfg.ReadyAddr + "/readyz")
+		if err != nil {
+			return false
+		}
+		_ = resp.Body.Close()
+		return resp.StatusCode == http.StatusOK
+	}, 10*time.Second, 50*time.Millisecond, "readiness listener never reported ready")
+
+	resp, err := client.Post("http://"+cfg.ReadyAddr+"/authenticate", "application/json", strings.NewReader("{}"))
+	require.NoError(t, err)
+	_ = resp.Body.Close()
+	assert.Equal(t, http.StatusNotFound, resp.StatusCode, "the readiness listener must not serve /authenticate")
+
+	cancel()
+	select {
+	case err := <-done:
+		require.NoError(t, err)
+	case <-time.After(5 * time.Second):
+		t.Fatal("Run did not return after cancel")
+	}
+}
+
+func TestRunReportsReadinessListenFailure(t *testing.T) {
+	l, err := net.Listen("tcp", "127.0.0.1:0")
+	require.NoError(t, err)
+	defer l.Close()
+
+	cfg := runConfig(t, freeAddr(t))
+	cfg.ReadyAddr = l.Addr().String()
+
+	done := make(chan error, 1)
+	go func() { done <- Run(context.Background(), cfg) }()
+	select {
+	case err := <-done:
+		require.Error(t, err)
+		assert.True(t, strings.HasPrefix(err.Error(), "serve readiness: "), err.Error())
+	case <-time.After(10 * time.Second):
+		t.Fatal("Run kept serving after the readiness listener failed")
+	}
+}

spinifex/agents/eks/tokenwebhook/webhook.go

@@ -43,6 +43,10 @@ type Config struct {
 	CertPath    string
 	KeyPath     string
 	Kubeconfig  string
+	// ReadyAddr, when set, is a plain-HTTP listener serving only /readyz for
+	// load balancer health checks. APIServerAddr defaults to the local k3s.
+	ReadyAddr     string
+	APIServerAddr string
 }
 
 // Run writes the serving certificate and apiserver kubeconfig, then serves
@@ -63,6 +67,11 @@ func Run(ctx context.Context, cfg Config) error {
 		return fmt.Errorf("build gateway client: %w", err)
 	}
 
+	probe, err := newGatewayProbe(cfg.GatewayURL, cfg.GatewayCA, cfg.APIServerAddr)
+	if err != nil {
+		return fmt.Errorf("build readiness probe: %w", err)
+	}
+
 	authr := &authenticator{
 		accountID:   cfg.AccountID,
 		clusterName: cfg.ClusterName,
@@ -74,6 +83,7 @@ func Run(ctx context.Context, cfg Config) error {
 		w.WriteHeader(http.StatusOK)
 		_, _ = w.Write([]byte("ok"))
 	})
+	mux.HandleFunc("/readyz", probe.handle)
 	mux.HandleFunc("/authenticate", authr.handle)
 
 	server := &http.Server{
@@ -82,12 +92,38 @@ func Run(ctx context.Context, cfg Config) error {
 		ReadHeaderTimeout: 5 * time.Second,
 		TLSConfig:         serverTLSConfig(tlsCert),
 	}
-	stop := context.AfterFunc(ctx, func() { _ = server.Close() })
+	readyErr := make(chan error, 1)
+	var readySrv *http.Server
+	if cfg.ReadyAddr != "" {
+		readySrv = &http.Server{
+			Addr:              cfg.ReadyAddr,
+			Handler:           readinessHandler(probe),
+			ReadHeaderTimeout: 5 * time.Second,
+		}
+		go func() {
+			if err := readySrv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
+				readyErr <- fmt.Errorf("serve readiness: %w", err)
+				_ = server.Close()
+			}
+		}()
+		defer func() { _ = readySrv.Close() }()
+	}
+	stop := context.AfterFunc(ctx, func() {
+		_ = server.Close()
+		if readySrv != nil {
+			_ = readySrv.Close()
+		}
+	})
 	defer stop()
-	slog.Info("eks-token-webhook listening", "addr", cfg.Addr, "cluster", cfg.ClusterName)
+	slog.Info("eks-token-webhook listening", "addr", cfg.Addr, "ready_addr", cfg.ReadyAddr, "cluster", cfg.ClusterName)
 	if err := server.ListenAndServeTLS("", ""); err != nil && !errors.Is(err, http.ErrServerClosed) {
 		return fmt.Errorf("serve: %w", err)
 	}
+	select {
+	case err := <-readyErr:
+		return err
+	default:
+	}
 	return nil
 }
 
@@ -101,8 +137,8 @@ type tokenReviewRequest struct {
 
 // gatewayReviewer returns the per-call relay: SigV4-POST {accountId,token} to
 // the gateway and decode the resolved identity. A non-2xx / transport error is
-// surfaced (handle() turns it into a 5xx so the apiserver retries) rather than a
-// silent deny, distinguishing "webhook broker down" from "token rejected".
+// surfaced (handle() turns it into a 5xx) rather than a silent deny, which
+// distinguishes "webhook broker down" from "token rejected" in the logs.
 func gatewayReviewer(client *eksgw.Client, clusterName, accountID string) func(string) (ekscluster.WebhookTokenReviewResult, error) {
 	path := "/clusters/" + clusterName + "/token-review"
 	return func(token string) (ekscluster.WebhookTokenReviewResult, error) {
@@ -142,8 +178,8 @@ func (a *authenticator) handle(w http.ResponseWriter, r *http.Request) {
 
 	res, err := a.review(review.Spec.Token)
 	if err != nil {
-		// Broker fault (gateway down, transport error): a 5xx tells the apiserver
-		// to retry rather than treating a transient outage as a hard deny.
+		// Broker fault (gateway down, transport error). The apiserver treats any
+		// webhook error as an authentication failure and does not retry it.
 		slog.Error("TokenReview relay failed", "cluster", a.clusterName, "err", err)
 		http.Error(w, "token review unavailable", http.StatusServiceUnavailable)
 		return

spinifex/agents/elbv2/health.go

@@ -22,6 +22,11 @@ type HealthReport struct {
 	Servers []ServerStatus `json:"servers"`
 }
 
+// FailoverServerSuffix marks the unchecked backup copy of a server that HAProxy
+// uses only when every checked server is down. '.' cannot appear in a sanitized
+// target name, so a copy never collides with a real server.
+const FailoverServerSuffix = ".fo"
+
 // queryHAProxyStats reads backend server health from the HAProxy stats socket.
 // Parses CSV "show stat" output; cols 0/1/17 = pxname/svname/status.
 func queryHAProxyStats(socketPath string) ([]ServerStatus, error) {
@@ -53,7 +58,7 @@ func queryHAProxyStats(socketPath string) ([]ServerStatus, error) {
 		}
 
 		svname := fields[1]
-		if svname == "FRONTEND" || svname == "BACKEND" {
+		if svname == "FRONTEND" || svname == "BACKEND" || strings.HasSuffix(svname, FailoverServerSuffix) {
 			continue
 		}
 

spinifex/agents/elbv2/health_test.go

@@ -58,3 +58,38 @@ func TestQueryHAProxyStats_SocketNotFound(t *testing.T) {
 		t.Error("expected error for non-existent socket")
 	}
 }
+
+func TestQueryHAProxyStats_IgnoresFailoverCopies(t *testing.T) {
+	dir := t.TempDir()
+	sock := filepath.Join(dir, "haproxy.sock")
+
+	ln, err := net.Listen("unix", sock)
+	if err != nil {
+		t.Fatalf("listen: %v", err)
+	}
+	defer ln.Close()
+
+	go func() {
+		conn, err := ln.Accept()
+		if err != nil {
+			return
+		}
+		defer conn.Close()
+
+		buf := make([]byte, 256)
+		conn.Read(buf)
+
+		csv := "# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight\n"
+		csv += "bk_tg1,srv_i-aaa111,0,0,0,0,0,5,0,0,0,0,0,0,0,0,0,DOWN,1\n"
+		csv += "bk_tg1,srv_i-aaa111" + FailoverServerSuffix + ",0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,UP,1\n"
+		fmt.Fprint(conn, csv)
+	}()
+
+	servers, err := queryHAProxyStats(sock)
+	if err != nil {
+		t.Fatalf("queryHAProxyStats: %v", err)
+	}
+	if len(servers) != 1 || servers[0].Server != "srv_i-aaa111" || servers[0].Status != "DOWN" {
+		t.Fatalf("got %+v, want only srv_i-aaa111/DOWN", servers)
+	}
+}

spinifex/domains/ecr/meta.go

@@ -78,6 +78,31 @@ type ManifestMeta struct {
 	Size         int64     `json:"size"`
 	PushedAt     time.Time `json:"pushedAt"`
 	ChildDigests []string  `json:"childDigests,omitempty"`
+	// SubjectDigest and ArtifactType are set only for manifests that declare a
+	// subject, and feed the OCI referrers listing for that subject.
+	SubjectDigest string            `json:"subjectDigest,omitempty"`
+	ArtifactType  string            `json:"artifactType,omitempty"`
+	Annotations   map[string]string `json:"annotations,omitempty"`
+}
+
+// ReferrerDescriptor is one entry of a subject's referrers listing: the OCI
+// descriptor of a manifest that names the subject.
+type ReferrerDescriptor struct {
+	MediaType    string            `json:"mediaType"`
+	Digest       string            `json:"digest"`
+	Size         int64             `json:"size"`
+	ArtifactType string            `json:"artifactType,omitempty"`
+	Annotations  map[string]string `json:"annotations,omitempty"`
+}
+
+func (m ManifestMeta) referrerDescriptor() ReferrerDescriptor {
+	return ReferrerDescriptor{
+		MediaType:    m.MediaType,
+		Digest:       m.Digest,
+		Size:         m.Size,
+		ArtifactType: m.ArtifactType,
+		Annotations:  m.Annotations,
+	}
 }
 
 // UploadState tracks an in-progress chunked blob upload. The sha256 hash state
@@ -124,6 +149,10 @@ type MetaStore interface {
 	GetManifestMeta(ctx context.Context, accountID, repo, digest string) (ManifestMeta, error)
 	DeleteManifestMeta(ctx context.Context, accountID, repo, digest string) error
 
+	// ListReferrers returns the manifests in repo that declare subjectDigest as
+	// their subject, ordered by digest. A subject with none yields an empty slice.
+	ListReferrers(ctx context.Context, accountID, repo, subjectDigest string) ([]ReferrerDescriptor, error)
+
 	PutUpload(ctx context.Context, accountID, uploadID string, state UploadState) (uint64, error)
 	GetUpload(ctx context.Context, accountID, uploadID string) (UploadState, uint64, error)
 	UpdateUpload(ctx context.Context, accountID, uploadID string, state UploadState, rev uint64) (uint64, error)
@@ -234,11 +263,11 @@ func (s *KVMetaStore) DeleteRepo(ctx context.Context, accountID, repo string) er
 		}
 		return err
 	}
-	tagsPrefix, manifestsPrefix := KVTagsPrefix(repo), KVManifestsPrefix(repo)
+	tagsPrefix, manifestsPrefix, referrersPrefix := KVTagsPrefix(repo), KVManifestsPrefix(repo), KVReferrersPrefix(repo)
 	metaKey, policyKey, lifecycleKey := KVRepoMetaKey(repo), KVRepoPolicyKey(repo), KVLifecyclePolicyKey(repo)
 	for _, k := range keys {
 		if k == metaKey || k == policyKey || k == lifecycleKey ||
-			strings.HasPrefix(k, tagsPrefix) || strings.HasPrefix(k, manifestsPrefix) {
+			strings.HasPrefix(k, tagsPrefix) || strings.HasPrefix(k, manifestsPrefix) || strings.HasPrefix(k, referrersPrefix) {
 			if err := kv.Delete(ctx, k); err != nil {
 				return mapKVErr(err)
 			}
@@ -415,10 +444,55 @@ func (s *KVMetaStore) PutManifestMeta(ctx context.Context, accountID, repo strin
 	if err != nil {
 		return err
 	}
-	_, err = kv.Put(ctx, KVManifestKey(repo, meta.Digest), data)
+	if _, err = kv.Put(ctx, KVManifestKey(repo, meta.Digest), data); err != nil {
+		return err
+	}
+	if meta.SubjectDigest == "" {
+		return nil
+	}
+	entry, err := json.Marshal(meta.referrerDescriptor())
+	if err != nil {
+		return err
+	}
+	_, err = kv.Put(ctx, KVReferrerKey(repo, meta.SubjectDigest, meta.Digest), entry)
 	return err
 }
 
+func (s *KVMetaStore) ListReferrers(ctx context.Context, accountID, repo, subjectDigest string) ([]ReferrerDescriptor, error) {
+	kv, err := s.bucket(ctx, accountID)
+	if err != nil {
+		return nil, err
+	}
+	keys, err := kv.Keys(ctx)
+	if err != nil {
+		if errors.Is(err, jetstream.ErrNoKeysFound) {
+			return []ReferrerDescriptor{}, nil
+		}
+		return nil, err
+	}
+	prefix := KVSubjectReferrersPrefix(repo, subjectDigest)
+	out := []ReferrerDescriptor{}
+	for _, k := range keys {
+		if !strings.HasPrefix(k, prefix) {
+			continue
+		}
+		entry, err := kv.Get(ctx, k)
+		if err != nil {
+			if errors.Is(err, jetstream.ErrKeyNotFound) {
+				continue
+			}
+			return nil, err
+		}
+		var d ReferrerDescriptor
+		if err := json.Unmarshal(entry.Value(), &d); err != nil {
+			return nil, err
+		}
+		out = append(out, d)
+	}
+	sort.Slice(out, func(i, j int) bool { return out[i].Digest < out[j].Digest })
+	return out, nil
+}
+
 func (s *KVMetaStore) GetManifestMeta(ctx context.Context, accountID, repo, digest string) (ManifestMeta, error) {
 	kv, err := s.bucket(ctx, accountID)
 	if err != nil {
@@ -440,9 +514,19 @@ func (s *KVMetaStore) DeleteManifestMeta(ctx context.Context, accountID, repo, d
 	if err != nil {
 		return err
 	}
-	if _, err := kv.Get(ctx, KVManifestKey(repo, digest)); err != nil {
+	entry, err := kv.Get(ctx, KVManifestKey(repo, digest))
+	if err != nil {
 		return mapKVErr(err)
 	}
+	var m ManifestMeta
+	if err := json.Unmarshal(entry.Value(), &m); err != nil {
+		return err
+	}
+	if m.SubjectDigest != "" {
+		if err := kv.Delete(ctx, KVReferrerKey(repo, m.SubjectDigest, digest)); err != nil {
+			return mapKVErr(err)
+		}
+	}
 	return kv.Delete(ctx, KVManifestKey(repo, digest))
 }
 
@@ -738,6 +822,19 @@ func (m *MemoryMetaStore) DeleteManifestMeta(_ context.Context, accountID, repo,
 	return nil
 }
 
+func (m *MemoryMetaStore) ListReferrers(_ context.Context, accountID, repo, subjectDigest string) ([]ReferrerDescriptor, error) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	out := []ReferrerDescriptor{}
+	for k, meta := range m.manifests[accountID] {
+		if r, _, ok := strings.Cut(k, "|"); ok && r == repo && meta.SubjectDigest == subjectDigest {
+			out = append(out, meta.referrerDescriptor())
+		}
+	}
+	sort.Slice(out, func(i, j int) bool { return out[i].Digest < out[j].Digest })
+	return out, nil
+}
+
 func (m *MemoryMetaStore) PutUpload(_ context.Context, accountID, uploadID string, state UploadState) (uint64, error) {
 	m.mu.Lock()
 	defer m.mu.Unlock()

spinifex/domains/ecr/meta_nats.go

@@ -184,6 +184,14 @@ func (s *NATSMetaStore) DeleteManifestMeta(ctx context.Context, accountID, repo,
 	return nil
 }
 
+func (s *NATSMetaStore) ListReferrers(ctx context.Context, accountID, repo, subjectDigest string) ([]ReferrerDescriptor, error) {
+	resp, err := natsmsg.NATSRequest[ReferrerListResponse](ctx, s.conn, SubjectReferrerList, &ReferrerListRequest{Repo: repo, Subject: subjectDigest}, metaRequestTimeout, accountID)
+	if err != nil {
+		return nil, err
+	}
+	return resp.Referrers, nil
+}
+
 func (s *NATSMetaStore) PutUpload(ctx context.Context, accountID, uploadID string, state UploadState) (uint64, error) {
 	resp, err := natsmsg.NATSRequest[UploadCreateResponse](ctx, s.conn, SubjectUploadCreate, &UploadCreateRequest{UploadID: uploadID, State: state}, metaRequestTimeout, accountID)
 	if err != nil {

spinifex/domains/ecr/meta_test.go

@@ -223,3 +223,12 @@ func TestMemoryMetaStore_LifecyclePolicy(t *testing.T) {
 	_, err = m.GetLifecyclePolicy(context.Background(), acct, "team/app")
 	assert.ErrorIs(t, err, ErrNotFound)
 }
+
+func TestManifestMeta_DecodesRecordWithoutReferrerFields(t *testing.T) {
+	var m ManifestMeta
+	require.NoError(t, json.Unmarshal([]byte(`{"digest":"sha256:ccc","mediaType":"application/vnd.oci.image.manifest.v1+json","size":10,"pushedAt":"2026-01-01T00:00:00Z","childDigests":["sha256:ddd"]}`), &m))
+	assert.Equal(t, "sha256:ccc", m.Digest)
+	assert.Equal(t, []string{"sha256:ddd"}, m.ChildDigests)
+	assert.Empty(t, m.SubjectDigest)
+	assert.Empty(t, m.ArtifactType)
+}

spinifex/domains/ecr/referrers_test.go

@@ -0,0 +1,94 @@
+package ecr
+
+import (
+	"context"
+	"testing"
+
+	"github.com/mulgadc/spinifex/internal/testkit"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+const (
+	refSubject = "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+	refSigA    = "sha256:1111111111111111111111111111111111111111111111111111111111111111"
+	refSigB    = "sha256:2222222222222222222222222222222222222222222222222222222222222222"
+)
+
+func referrerMeta(digest, subject string) ManifestMeta {
+	return ManifestMeta{
+		Digest:        digest,
+		MediaType:     "application/vnd.oci.image.manifest.v1+json",
+		Size:          42,
+		SubjectDigest: subject,
+		ArtifactType:  "application/vnd.example.sig.v1",
+		Annotations:   map[string]string{"org.example.key": "v"},
+	}
+}
+
+// referrerStores runs a test against every MetaStore implementation that keeps
+// its own referrers state.
+func referrerStores(t *testing.T) map[string]MetaStore {
+	t.Helper()
+	_, nc, _ := testutil.StartTestJetStream(t)
+	return map[string]MetaStore{
+		"memory": NewMemoryMetaStore(),
+		"kv":     NewKVMetaStore(testutil.NewJetStream(t, nc)),
+	}
+}
+
+func TestListReferrers(t *testing.T) {
+	ctx := context.Background()
+	for name, store := range referrerStores(t) {
+		t.Run(name, func(t *testing.T) {
+			acct, repo := "111111111111", "team/app"
+
+			got, err := store.ListReferrers(ctx, acct, repo, refSubject)
+			require.NoError(t, err)
+			assert.Empty(t, got, "a subject with no referrers lists empty")
+
+			require.NoError(t, store.PutManifestMeta(ctx, acct, repo, referrerMeta(refSigB, refSubject)))
+			require.NoError(t, store.PutManifestMeta(ctx, acct, repo, referrerMeta(refSigA, refSubject)))
+			// Re-putting the same digest must not duplicate the entry.
+			require.NoError(t, store.PutManifestMeta(ctx, acct, repo, referrerMeta(refSigA, refSubject)))
+
+			got, err = store.ListReferrers(ctx, acct, repo, refSubject)
+			require.NoError(t, err)
+			require.Len(t, got, 2)
+			assert.Equal(t, refSigA, got[0].Digest, "entries are ordered by digest")
+			assert.Equal(t, refSigB, got[1].Digest)
+			assert.Equal(t, ReferrerDescriptor{
+				MediaType:    "application/vnd.oci.image.manifest.v1+json",
+				Digest:       refSigA,
+				Size:         42,
+				ArtifactType: "application/vnd.example.sig.v1",
+				Annotations:  map[string]string{"org.example.key": "v"},
+			}, got[0])
+
+			otherRepo, err := store.ListReferrers(ctx, acct, "team/other", refSubject)
+			require.NoError(t, err)
+			assert.Empty(t, otherRepo, "referrers do not cross repositories")
+			otherAcct, err := store.ListReferrers(ctx, "222222222222", repo, refSubject)
+			require.NoError(t, err)
+			assert.Empty(t, otherAcct, "referrers do not cross accounts")
+
+			require.NoError(t, store.DeleteManifestMeta(ctx, acct, repo, refSigA))
+			got, err = store.ListReferrers(ctx, acct, repo, refSubject)
+			require.NoError(t, err)
+			require.Len(t, got, 1)
+			assert.Equal(t, refSigB, got[0].Digest)
+		})
+	}
+}
+
+func TestListReferrers_ManifestWithoutSubjectIsNotListed(t *testing.T) {
+	ctx := context.Background()
+	for name, store := range referrerStores(t) {
+		t.Run(name, func(t *testing.T) {
+			require.NoError(t, store.PutManifestMeta(ctx, "111111111111", "r", referrerMeta(refSigA, "")))
+			got, err := store.ListReferrers(ctx, "111111111111", "r", refSubject)
+			require.NoError(t, err)
+			assert.Empty(t, got)
+		})
+	}
+}

spinifex/domains/ecr/registry/authorization.go

@@ -99,6 +99,8 @@ func ClassifyOperation(method, urlPath string, query url.Values) (ClassifiedOper
 		return classifyBlobOperation(method, name, ref, query)
 	case "manifests":
 		return classifyManifestOperation(method, name, ref)
+	case "referrers":
+		return classifyReferrersOperation(method, name, ref)
 	default:
 		return ClassifiedOperation{}, false
 	}
@@ -165,6 +167,15 @@ func classifyBlobOperation(method, name, ref string, query url.Values) (Classifi
 	}
 }
 
+// classifyReferrersOperation authorizes the referrers listing as a read of the
+// repository's images.
+func classifyReferrersOperation(method, name, subject string) (ClassifiedOperation, bool) {
+	if method != http.MethodGet || subject == "" {
+		return ClassifiedOperation{}, false
+	}
+	return ClassifiedOperation{Repo: name, Requirements: []ActionRequirement{{ActionBatchGetImage, ScopeDestination}}}, true
+}
+
 func classifyManifestOperation(method, name, reference string) (ClassifiedOperation, bool) {
 	if reference == "" {
 		return ClassifiedOperation{}, false

spinifex/domains/ecr/registry/authorization_test.go

@@ -132,6 +132,15 @@ func TestClassifyOperation_RouteMatrix(t *testing.T) {
 				Requirements: []ActionRequirement{{ActionBatchDeleteImage, ScopeDestination}},
 			},
 		},
+		{
+			name:   "GET referrers is a read of the repository images",
+			method: http.MethodGet,
+			path:   "/v2/team/app/referrers/sha256:abc",
+			want: ClassifiedOperation{
+				Repo:         "team/app",
+				Requirements: []ActionRequirement{{ActionBatchGetImage, ScopeDestination}},
+			},
+		},
 		{
 			name:   "cross-repo mount requires destination initiate + source read",
 			method: http.MethodPost,
@@ -205,6 +214,9 @@ func TestClassifyOperation_UnsupportedNeverClassifies(t *testing.T) {
 		{"manifest POST unsupported", http.MethodPost, "/v2/team/app/manifests/latest"},
 		{"manifest empty reference", http.MethodGet, "/v2/team/app/manifests/"},
 		{"catalog wrong method", http.MethodPost, "/v2/_catalog"},
+		{"referrers PUT unsupported", http.MethodPut, "/v2/team/app/referrers/sha256:abc"},
+		{"referrers DELETE unsupported", http.MethodDelete, "/v2/team/app/referrers/sha256:abc"},
+		{"referrers empty subject", http.MethodGet, "/v2/team/app/referrers/"},
 	}
 	for _, c := range cases {
 		t.Run(c.name, func(t *testing.T) {

spinifex/domains/ecr/registry/registry.go

@@ -142,6 +142,8 @@ func (reg *Registry) serve(w http.ResponseWriter, r *http.Request) {
 		reg.routeManifests(w, r, name, ref)
 	case "tags":
 		reg.routeTags(w, r, name, ref)
+	case "referrers":
+		reg.getReferrers(w, r, name, ref)
 	default:
 		WriteError(w, http.StatusNotFound, "NAME_UNKNOWN", "unrecognized registry path")
 	}
@@ -151,7 +153,7 @@ func (reg *Registry) serve(w http.ResponseWriter, r *http.Request) {
 // kind is one of blobs/manifests/tags. ref is the remainder (digest, reference,
 // "uploads", "uploads/{uuid}", or "list"). ok is false if no marker is found.
 func splitV2Path(path string) (name, kind, ref string, ok bool) {
-	for _, marker := range []string{"/blobs/", "/manifests/", "/tags/"} {
+	for _, marker := range []string{"/blobs/", "/manifests/", "/tags/", "/referrers/"} {
 		if before, after, found := strings.Cut(path, marker); found {
 			kind = strings.Trim(marker, "/")
 			return before, kind, after, before != ""
@@ -590,7 +592,7 @@ func (reg *Registry) putManifest(w http.ResponseWriter, r *http.Request, name, r
 		return
 	}
 
-	digest, err := reg.StoreManifest(ctx, reg.AccountID, name, reference, r.Header.Get("Content-Type"), body)
+	digest, subject, err := reg.storeManifest(ctx, reg.AccountID, name, reference, r.Header.Get("Content-Type"), body)
 	if err != nil {
 		if mErr, ok := errors.AsType[*ManifestStoreError](err); ok {
 			WriteError(w, mErr.Status, mErr.Code, mErr.Msg)
@@ -602,6 +604,9 @@ func (reg *Registry) putManifest(w http.ResponseWriter, r *http.Request, name, r
 
 	w.Header().Set("Docker-Content-Digest", digest)
 	w.Header().Set("Location", fmt.Sprintf("/v2/%s/manifests/%s", name, digest))
+	if subject != "" {
+		w.Header().Set("OCI-Subject", subject)
+	}
 	w.WriteHeader(http.StatusCreated)
 }
 
@@ -635,9 +640,15 @@ func (reg *Registry) deleteManifest(ctx context.Context, w http.ResponseWriter,
 
 // manifestDoc is the subset of an OCI image manifest the validator inspects.
 type manifestDoc struct {
-	MediaType string `json:"mediaType"`
-	Config    struct {
+	MediaType    string            `json:"mediaType"`
+	ArtifactType string            `json:"artifactType"`
+	Annotations  map[string]string `json:"annotations"`
+	Subject      *struct {
 		Digest string `json:"digest"`
+	} `json:"subject"`
+	Config struct {
+		MediaType string `json:"mediaType"`
+		Digest    string `json:"digest"`
 	} `json:"config"`
 	Layers []struct {
 		Digest string `json:"digest"`
@@ -648,6 +659,30 @@ type manifestDoc struct {
 	} `json:"manifests"`
 }
 
+// referrerFields returns the subject digest, artifactType and annotations of a
+// manifest that declares a subject. mediaType is the type it is stored under;
+// an image manifest without an artifactType uses its config mediaType.
+func referrerFields(mediaType string, body []byte) (subject, artifactType string, annotations map[string]string, err error) {
+	var doc manifestDoc
+	if err := json.Unmarshal(body, &doc); err != nil {
+		return "", "", nil, fmt.Errorf("manifest is not valid JSON: %w", err)
+	}
+	if doc.Subject == nil {
+		return "", "", nil, nil
+	}
+	if !ecr.ValidateDigest(doc.Subject.Digest) {
+		return "", "", nil, errors.New("subject references malformed digest")
+	}
+	if mediaType == "" {
+		mediaType = doc.MediaType
+	}
+	artifactType = doc.ArtifactType
+	if artifactType == "" && mediaType != mediaTypeOCIIndex && mediaType != mediaTypeDockerManifestList {
+		artifactType = doc.Config.MediaType
+	}
+	return doc.Subject.Digest, artifactType, doc.Annotations, nil
+}
+
 // validateManifest performs type-aware shallow validation, returning the child
 // digests recorded in the manifest metadata. For an image manifest every config
 // and layer blob must exist; for an index every child manifest must exist with a
@@ -697,6 +732,57 @@ func (reg *Registry) validateManifest(ctx context.Context, name, contentType str
 	}
 }
 
+// ---- referrers ----
+
+// referrersIndex is the OCI image index the referrers API answers with.
+type referrersIndex struct {
+	SchemaVersion int                      `json:"schemaVersion"`
+	MediaType     string                   `json:"mediaType"`
+	Manifests     []ecr.ReferrerDescriptor `json:"manifests"`
+}
+
+// getReferrers answers GET /v2/<name>/referrers/<digest> with an index of the
+// manifests that name the digest as their subject. A subject with none, or one
+// that is not stored, yields an empty index.
+func (reg *Registry) getReferrers(w http.ResponseWriter, r *http.Request, name, subject string) {
+	ctx := r.Context()
+	if !ecr.ValidateDigest(subject) {
+		WriteError(w, http.StatusBadRequest, "DIGEST_INVALID", "malformed subject digest")
+		return
+	}
+	if _, err := reg.Meta.GetRepo(ctx, reg.AccountID, name); err != nil {
+		if errors.Is(err, ecr.ErrNotFound) {
+			WriteError(w, http.StatusNotFound, "NAME_UNKNOWN", "repository unknown")
+			return
+		}
+		reg.internal(ctx, w, "get repository", err)
+		return
+	}
+	referrers, err := reg.Meta.ListReferrers(ctx, reg.AccountID, name, subject)
+	if err != nil {
+		reg.internal(ctx, w, "list referrers", err)
+		return
+	}
+	if want := r.URL.Query().Get("artifactType"); want != "" {
+		referrers = slices.DeleteFunc(referrers, func(d ecr.ReferrerDescriptor) bool { return d.ArtifactType != want })
+		w.Header().Set("OCI-Filters-Applied", "artifactType")
+	}
+	if referrers == nil {
+		referrers = []ecr.ReferrerDescriptor{}
+	}
+	out, err := json.Marshal(referrersIndex{SchemaVersion: 2, MediaType: mediaTypeOCIIndex, Manifests: referrers})
+	if err != nil {
+		reg.internal(ctx, w, "encode referrers", err)
+		return
+	}
+	w.Header().Set("Content-Type", mediaTypeOCIIndex)
+	w.Header().Set("Content-Length", strconv.Itoa(len(out)))
+	w.WriteHeader(http.StatusOK)
+	if _, err := w.Write(out); err != nil {
+		slog.ErrorContext(ctx, "ECR/OCI: referrers write failed", "err", err)
+	}
+}
+
 // ---- tags ----
 
 func (reg *Registry) routeTags(w http.ResponseWriter, r *http.Request, name, ref string) {

spinifex/domains/ecr/registry/registry_image.go

@@ -6,6 +6,7 @@ import (
 	"errors"
 	"io"
 	"log/slog"
+	"mime"
 	"net/http"
 	"slices"
 	"strings"
@@ -52,20 +53,29 @@ type ImageRecord struct {
 // case it is detected. On a validation/size failure it returns a
 // *ManifestStoreError carrying the OCI code.
 func (reg *Registry) StoreManifest(ctx context.Context, account, repo, ref, contentType string, body []byte) (string, error) {
+	digest, _, err := reg.storeManifest(ctx, account, repo, ref, contentType, body)
+	return digest, err
+}
+
+// storeManifest is StoreManifest that also returns the subject digest the
+// manifest declares ("" when it has none).
+func (reg *Registry) storeManifest(ctx context.Context, account, repo, ref, contentType string, body []byte) (string, string, error) {
 	scoped := reg.forAccount(account)
 	if err := scoped.requireRepo(ctx, repo); err != nil {
-		return "", err
+		return "", "", err
 	}
 	if len(body) > maxManifestBytes {
-		return "", &ManifestStoreError{http.StatusRequestEntityTooLarge, "MANIFEST_INVALID", "manifest too large"}
+		return "", "", &ManifestStoreError{http.StatusRequestEntityTooLarge, "MANIFEST_INVALID", "manifest too large"}
 	}
 	if contentType == "" {
 		contentType = detectManifestType(body)
+	} else if mt, _, err := mime.ParseMediaType(contentType); err == nil {
+		contentType = mt
 	}
 
 	digest := "sha256:" + hex.EncodeToString(sha256Sum(body))
 	if ecr.ValidateDigest(ref) && ref != digest {
-		return "", &ManifestStoreError{http.StatusBadRequest, "DIGEST_INVALID", "reference digest does not match content"}
+		return "", "", &ManifestStoreError{http.StatusBadRequest, "DIGEST_INVALID", "reference digest does not match content"}
 	}
 
 	// On an IMMUTABLE repo, re-tagging an existing tag onto a different digest is
@@ -73,22 +83,27 @@ func (reg *Registry) StoreManifest(ctx context.Context, account, repo, ref, cont
 	if ref != "" && !ecr.ValidateDigest(ref) {
 		meta, err := scoped.Meta.GetRepo(ctx, account, repo)
 		if err != nil {
-			return "", err
+			return "", "", err
 		}
 		if meta.TagMutability() == ecr.TagMutabilityImmutable {
 			existing, err := scoped.Meta.GetTag(ctx, account, repo, ref)
 			switch {
 			case err == nil && existing != digest:
-				return "", &ManifestStoreError{http.StatusConflict, "TAG_IMMUTABLE", "tag is immutable and already resolves to a different digest"}
+				return "", "", &ManifestStoreError{http.StatusConflict, "TAG_IMMUTABLE", "tag is immutable and already resolves to a different digest"}
 			case err != nil && !errors.Is(err, ecr.ErrNotFound):
-				return "", err
+				return "", "", err
 			}
 		}
 	}
 
 	children, code, vErr := scoped.validateManifest(ctx, repo, contentType, body)
 	if vErr != nil {
-		return "", &ManifestStoreError{http.StatusBadRequest, code, vErr.Error()}
+		return "", "", &ManifestStoreError{http.StatusBadRequest, code, vErr.Error()}
+	}
+
+	subject, artifactType, annotations, rErr := referrerFields(contentType, body)
+	if rErr != nil {
+		return "", "", &ManifestStoreError{http.StatusBadRequest, "MANIFEST_INVALID", rErr.Error()}
 	}
 
 	if _, err := scoped.Store.PutObject(ctx, &s3.PutObjectInput{
@@ -97,7 +112,7 @@ func (reg *Registry) StoreManifest(ctx context.Context, account, repo, ref, cont
 		Body:        aws.ReadSeekCloser(strings.NewReader(string(body))),
 		ContentType: aws.String(contentType),
 	}); err != nil {
-		return "", err
+		return "", "", err
 	}
 
 	if err := scoped.Meta.PutManifestMeta(ctx, account, repo, ecr.ManifestMeta{
@@ -106,16 +121,20 @@ func (reg *Registry) StoreManifest(ctx context.Context, account, repo, ref, cont
 		Size:         int64(len(body)),
 		PushedAt:     time.Now().UTC(),
 		ChildDigests: children,
+
+		SubjectDigest: subject,
+		ArtifactType:  artifactType,
+		Annotations:   annotations,
 	}); err != nil {
-		return "", err
+		return "", "", err
 	}
 
 	if ref != "" && !ecr.ValidateDigest(ref) {
 		if err := scoped.Meta.PutTag(ctx, account, repo, ref, digest); err != nil {
-			return "", err
+			return "", "", err
 		}
 	}
-	return digest, nil
+	return digest, subject, nil
 }
 
 // GetManifest returns the verbatim manifest bytes, media type and digest for a
@@ -230,40 +249,66 @@ func (reg *Registry) DeleteImage(ctx context.Context, account, repo, tag, digest
 		return resolved, nil
 	}
 
-	meta, err := scoped.Meta.GetManifestMeta(ctx, account, repo, digest)
+	if err := scoped.deleteManifestCascade(ctx, account, repo, digest, map[string]bool{}); err != nil {
+		return "", err
+	}
+	return digest, nil
+}
+
+// deleteManifestCascade deletes one manifest by digest with its tags, then the
+// referrers that name it as subject, each through this same path. Referrer
+// failures are logged, not returned: the subject is already deleted and ECR
+// removes referrers asynchronously. visited guards against subject cycles.
+func (reg *Registry) deleteManifestCascade(ctx context.Context, account, repo, digest string, visited map[string]bool) error {
+	visited[digest] = true
+	meta, err := reg.Meta.GetManifestMeta(ctx, account, repo, digest)
 	if err != nil {
 		if errors.Is(err, ecr.ErrNotFound) {
-			return "", ErrImageNotFound
+			return ErrImageNotFound
 		}
-		return "", err
+		return err
 	}
-	tags, err := scoped.Meta.ListTags(ctx, account, repo)
+	tags, err := reg.Meta.ListTags(ctx, account, repo)
 	if err != nil {
-		return "", err
+		return err
 	}
 	for _, t := range tags {
-		d, err := scoped.Meta.GetTag(ctx, account, repo, t)
+		d, err := reg.Meta.GetTag(ctx, account, repo, t)
 		if err != nil {
 			if errors.Is(err, ecr.ErrNotFound) {
 				continue
 			}
-			return "", err
+			return err
 		}
 		if d == digest {
-			if err := scoped.Meta.DeleteTag(ctx, account, repo, t); err != nil {
-				return "", err
+			if err := reg.Meta.DeleteTag(ctx, account, repo, t); err != nil {
+				return err
 			}
 		}
 	}
-	if err := scoped.Meta.DeleteManifestMeta(ctx, account, repo, digest); err != nil {
-		return "", err
+	if err := reg.Meta.DeleteManifestMeta(ctx, account, repo, digest); err != nil {
+		return err
 	}
 
 	// The KV record is gone, so the image is logically deleted. Predastore
 	// reclaim is best-effort: a failure leaks bytes (a capacity nuisance) but
 	// does not undo the delete.
-	scoped.reclaimManifest(ctx, account, repo, digest, meta.ChildDigests)
-	return digest, nil
+	reg.reclaimManifest(ctx, account, repo, digest, meta.ChildDigests)
+
+	referrers, err := reg.Meta.ListReferrers(ctx, account, repo, digest)
+	if err != nil {
+		slog.WarnContext(ctx, "ECR/referrers: listing referrers for cascade failed", "repo", repo, "subject", digest, "err", err)
+		return nil
+	}
+	for _, ref := range referrers {
+		if visited[ref.Digest] {
+			continue
+		}
+		if err := reg.deleteManifestCascade(ctx, account, repo, ref.Digest, visited); err != nil && !errors.Is(err, ErrImageNotFound) {
+			slog.WarnContext(ctx, "ECR/referrers: referrer delete failed", "repo", repo, "subject", digest, "referrer", ref.Digest, "err", err)
+		}
+	}
+	return nil
 }
 
 // reclaimManifest deletes a manifest's predastore object and any child blob no

spinifex/domains/ecr/registry/registry_referrer_test.go

@@ -0,0 +1,409 @@
+package ecrregistry
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/mulgadc/spinifex/spinifex/domains/ecr"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+const (
+	referrerRepo     = "team/app"
+	sigConfigType    = "application/vnd.example.sig.config.v1+json"
+	sigArtifactType  = "application/vnd.example.sig.v1"
+	absentSubjectDgt = "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+)
+
+func imageWithSubject(cfgDigest, subject, artifactType string) string {
+	extra := ""
+	if artifactType != "" {
+		extra = fmt.Sprintf(`"artifactType":%q,`, artifactType)
+	}
+	return fmt.Sprintf(`{"schemaVersion":2,"mediaType":%q,%s"config":{"mediaType":%q,"digest":%q,"size":3},"layers":[],"subject":{"mediaType":%q,"digest":%q,"size":10}}`,
+		mediaTypeOCIManifest, extra, sigConfigType, cfgDigest, mediaTypeOCIManifest, subject)
+}
+
+func storedMeta(t *testing.T, reg *Registry, body string) (subject, artifactType string) {
+	t.Helper()
+	m, err := reg.Meta.GetManifestMeta(context.Background(), testAccount, referrerRepo, digestOf([]byte(body)))
+	require.NoError(t, err)
+	return m.SubjectDigest, m.ArtifactType
+}
+
+func TestPutManifest_SubjectRecordedAndAdvertised(t *testing.T) {
+	reg := newTestRegistry()
+	cfg := pushBlob(t, reg, referrerRepo, []byte("cfg"))
+
+	tests := []struct {
+		name         string
+		artifactType string
+		wantType     string
+	}{
+		{"artifactType falls back to the config mediaType", "", sigConfigType},
+		{"explicit artifactType wins over the config mediaType", sigArtifactType, sigArtifactType},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			body := imageWithSubject(cfg, absentSubjectDgt, tt.artifactType)
+			w := do(reg, http.MethodPut, "/v2/"+referrerRepo+"/manifests/"+digestOf([]byte(body)), []byte(body),
+				map[string]string{"Content-Type": mediaTypeOCIManifest})
+			require.Equal(t, http.StatusCreated, w.Code, "body: %s", w.Body.String())
+			assert.Equal(t, absentSubjectDgt, w.Header().Get("OCI-Subject"))
+
+			subject, artifactType := storedMeta(t, reg, body)
+			assert.Equal(t, absentSubjectDgt, subject)
+			assert.Equal(t, tt.wantType, artifactType)
+		})
+	}
+}
+
+func TestPutManifest_IndexSubject(t *testing.T) {
+	reg := newTestRegistry()
+	seedRepo(t, reg, referrerRepo)
+
+	body := fmt.Sprintf(`{"schemaVersion":2,"mediaType":%q,"manifests":[],"subject":{"mediaType":%q,"digest":%q,"size":10}}`,
+		mediaTypeOCIIndex, mediaTypeOCIManifest, absentSubjectDgt)
+	w := do(reg, http.MethodPut, "/v2/"+referrerRepo+"/manifests/"+digestOf([]byte(body)), []byte(body),
+		map[string]string{"Content-Type": mediaTypeOCIIndex})
+	require.Equal(t, http.StatusCreated, w.Code, "body: %s", w.Body.String())
+	assert.Equal(t, absentSubjectDgt, w.Header().Get("OCI-Subject"))
+
+	subject, artifactType := storedMeta(t, reg, body)
+	assert.Equal(t, absentSubjectDgt, subject)
+	assert.Empty(t, artifactType, "an index has no config mediaType to fall back on")
+}
+
+func TestPutManifest_NoSubjectLeavesFieldsEmpty(t *testing.T) {
+	reg := newTestRegistry()
+	cfg := pushBlob(t, reg, referrerRepo, []byte("cfg"))
+	body := fmt.Sprintf(`{"schemaVersion":2,"mediaType":%q,"artifactType":%q,"config":{"mediaType":%q,"digest":%q,"size":3},"layers":[]}`,
+		mediaTypeOCIManifest, sigArtifactType, sigConfigType, cfg)
+
+	w := do(reg, http.MethodPut, "/v2/"+referrerRepo+"/manifests/v1", []byte(body),
+		map[string]string{"Content-Type": mediaTypeOCIManifest})
+	require.Equal(t, http.StatusCreated, w.Code, "body: %s", w.Body.String())
+	assert.Empty(t, w.Header().Get("OCI-Subject"))
+
+	subject, artifactType := storedMeta(t, reg, body)
+	assert.Empty(t, subject)
+	assert.Empty(t, artifactType, "artifactType is recorded only for manifests that declare a subject")
+}
+
+func TestPutManifest_MalformedSubjectDigestRejected(t *testing.T) {
+	reg := newTestRegistry()
+	cfg := pushBlob(t, reg, referrerRepo, []byte("cfg"))
+	body := imageWithSubject(cfg, "sha256:nothex", "")
+
+	w := do(reg, http.MethodPut, "/v2/"+referrerRepo+"/manifests/v1", []byte(body),
+		map[string]string{"Content-Type": mediaTypeOCIManifest})
+	assert.Equal(t, http.StatusBadRequest, w.Code)
+	assertCode(t, w, "MANIFEST_INVALID")
+
+	_, err := reg.Meta.GetManifestMeta(context.Background(), testAccount, referrerRepo, digestOf([]byte(body)))
+	assert.Error(t, err, "a rejected manifest must not leave a record behind")
+}
+
+func TestPutManifest_IndexWithContentTypeParameterGetsNoConfigArtifactType(t *testing.T) {
+	reg := newTestRegistry()
+	seedRepo(t, reg, referrerRepo)
+
+	body := fmt.Sprintf(`{"schemaVersion":2,"mediaType":%q,"manifests":[],"subject":{"mediaType":%q,"digest":%q,"size":10}}`,
+		mediaTypeOCIIndex, mediaTypeOCIManifest, absentSubjectDgt)
+	w := do(reg, http.MethodPut, "/v2/"+referrerRepo+"/manifests/v1", []byte(body),
+		map[string]string{"Content-Type": mediaTypeOCIIndex + "; charset=utf-8"})
+	require.Equal(t, http.StatusCreated, w.Code, "body: %s", w.Body.String())
+	assert.Equal(t, absentSubjectDgt, w.Header().Get("OCI-Subject"))
+
+	m, err := reg.Meta.GetManifestMeta(context.Background(), testAccount, referrerRepo, digestOf([]byte(body)))
+	require.NoError(t, err)
+	assert.Equal(t, mediaTypeOCIIndex, m.MediaType, "the stored type must not carry the parameter")
+	assert.Empty(t, m.ArtifactType)
+}
+
+func TestDeleteReferrerManifest_RemovesItFromTheSubjectListing(t *testing.T) {
+	reg := newTestRegistry()
+	cfg := pushBlob(t, reg, referrerRepo, []byte("cfg"))
+	body := imageWithSubject(cfg, absentSubjectDgt, sigArtifactType)
+	dg := digestOf([]byte(body))
+	w := do(reg, http.MethodPut, "/v2/"+referrerRepo+"/manifests/"+dg, []byte(body),
+		map[string]string{"Content-Type": mediaTypeOCIManifest})
+	require.Equal(t, http.StatusCreated, w.Code, "body: %s", w.Body.String())
+
+	listed, err := reg.Meta.ListReferrers(context.Background(), testAccount, referrerRepo, absentSubjectDgt)
+	require.NoError(t, err)
+	require.Len(t, listed, 1)
+	assert.Equal(t, dg, listed[0].Digest)
+	assert.Equal(t, sigArtifactType, listed[0].ArtifactType)
+
+	w = do(reg, http.MethodDelete, "/v2/"+referrerRepo+"/manifests/"+dg, nil, nil)
+	require.Equal(t, http.StatusAccepted, w.Code, "body: %s", w.Body.String())
+
+	listed, err = reg.Meta.ListReferrers(context.Background(), testAccount, referrerRepo, absentSubjectDgt)
+	require.NoError(t, err)
+	assert.Empty(t, listed)
+}
+
+// pushReferrer stores a signature-shaped manifest that names subject and returns its digest.
+func pushReferrer(t *testing.T, reg *Registry, repo, subject, artifactType, blobData string) string {
+	t.Helper()
+	cfg := pushBlob(t, reg, repo, []byte(blobData))
+	body := imageWithSubject(cfg, subject, artifactType)
+	dg := digestOf([]byte(body))
+	w := do(reg, http.MethodPut, "/v2/"+repo+"/manifests/"+dg, []byte(body),
+		map[string]string{"Content-Type": mediaTypeOCIManifest})
+	require.Equal(t, http.StatusCreated, w.Code, "body: %s", w.Body.String())
+	return dg
+}
+
+type referrersBody struct {
+	SchemaVersion int    `json:"schemaVersion"`
+	MediaType     string `json:"mediaType"`
+	Manifests     []struct {
+		MediaType    string `json:"mediaType"`
+		Digest       string `json:"digest"`
+		ArtifactType string `json:"artifactType"`
+	} `json:"manifests"`
+}
+
+func getReferrersBody(t *testing.T, reg *Registry, path string) (*httptest.ResponseRecorder, referrersBody) {
+	t.Helper()
+	w := do(reg, http.MethodGet, path, nil, nil)
+	var out referrersBody
+	if w.Code == http.StatusOK {
+		require.NoError(t, json.Unmarshal(w.Body.Bytes(), &out))
+	}
+	return w, out
+}
+
+func TestGetReferrers_ListsReferrersAsAnOCIIndex(t *testing.T) {
+	reg := newTestRegistry()
+	dgA := pushReferrer(t, reg, referrerRepo, absentSubjectDgt, sigArtifactType, "cfgA")
+	dgB := pushReferrer(t, reg, referrerRepo, absentSubjectDgt, "application/vnd.example.sbom.v1", "cfgB")
+
+	w, out := getReferrersBody(t, reg, "/v2/"+referrerRepo+"/referrers/"+absentSubjectDgt)
+	require.Equal(t, http.StatusOK, w.Code, "body: %s", w.Body.String())
+	assert.Equal(t, "application/vnd.oci.image.index.v1+json", w.Header().Get("Content-Type"))
+	assert.Empty(t, w.Header().Get("OCI-Filters-Applied"))
+	assert.Equal(t, 2, out.SchemaVersion)
+	assert.Equal(t, mediaTypeOCIIndex, out.MediaType)
+	got := map[string]string{}
+	for _, m := range out.Manifests {
+		assert.Equal(t, mediaTypeOCIManifest, m.MediaType)
+		got[m.Digest] = m.ArtifactType
+	}
+	assert.Equal(t, map[string]string{dgA: sigArtifactType, dgB: "application/vnd.example.sbom.v1"}, got)
+}
+
+func TestGetReferrers_NoReferrersIsAnEmptyIndexNotAnError(t *testing.T) {
+	reg := newTestRegistry()
+	seedRepo(t, reg, referrerRepo)
+
+	w := do(reg, http.MethodGet, "/v2/"+referrerRepo+"/referrers/"+absentSubjectDgt, nil, nil)
+	require.Equal(t, http.StatusOK, w.Code, "body: %s", w.Body.String())
+	assert.Equal(t, "application/vnd.oci.image.index.v1+json", w.Header().Get("Content-Type"))
+	assert.JSONEq(t, fmt.Sprintf(`{"schemaVersion":2,"mediaType":%q,"manifests":[]}`, mediaTypeOCIIndex), w.Body.String())
+}
+
+func TestGetReferrers_ArtifactTypeFilter(t *testing.T) {
+	reg := newTestRegistry()
+	dgSig := pushReferrer(t, reg, referrerRepo, absentSubjectDgt, sigArtifactType, "cfgA")
+	pushReferrer(t, reg, referrerRepo, absentSubjectDgt, "application/vnd.example.sbom.v1", "cfgB")
+
+	w, out := getReferrersBody(t, reg, "/v2/"+referrerRepo+"/referrers/"+absentSubjectDgt+"?artifactType="+sigArtifactType)
+	require.Equal(t, http.StatusOK, w.Code)
+	assert.Equal(t, "artifactType", w.Header().Get("OCI-Filters-Applied"))
+	require.Len(t, out.Manifests, 1)
+	assert.Equal(t, dgSig, out.Manifests[0].Digest)
+
+	w, out = getReferrersBody(t, reg, "/v2/"+referrerRepo+"/referrers/"+absentSubjectDgt+"?artifactType=application/none")
+	require.Equal(t, http.StatusOK, w.Code)
+	assert.Equal(t, "artifactType", w.Header().Get("OCI-Filters-Applied"), "the header reports the filter even when nothing matches")
+	assert.Empty(t, out.Manifests)
+	assert.Contains(t, w.Body.String(), `"manifests":[]`)
+}
+
+func TestGetReferrers_MalformedDigest(t *testing.T) {
+	reg := newTestRegistry()
+	seedRepo(t, reg, referrerRepo)
+
+	w := do(reg, http.MethodGet, "/v2/"+referrerRepo+"/referrers/not-a-digest", nil, nil)
+	assert.Equal(t, http.StatusBadRequest, w.Code)
+	assertCode(t, w, "DIGEST_INVALID")
+}
+
+func TestGetReferrers_UnknownRepository(t *testing.T) {
+	reg := newTestRegistry()
+	w := do(reg, http.MethodGet, "/v2/team/ghost/referrers/"+absentSubjectDgt, nil, nil)
+	assert.Equal(t, http.StatusNotFound, w.Code)
+	assertCode(t, w, "NAME_UNKNOWN")
+}
+
+func TestGetReferrers_DoesNotCrossRepositories(t *testing.T) {
+	reg := newTestRegistry()
+	pushReferrer(t, reg, referrerRepo, absentSubjectDgt, sigArtifactType, "cfgA")
+	seedRepo(t, reg, "team/other")
+
+	w, out := getReferrersBody(t, reg, "/v2/team/other/referrers/"+absentSubjectDgt)
+	require.Equal(t, http.StatusOK, w.Code)
+	assert.Empty(t, out.Manifests)
+}
+
+func TestReferrersRoute_OnlyGET(t *testing.T) {
+	reg := newTestRegistry()
+	seedRepo(t, reg, referrerRepo)
+	for _, method := range []string{http.MethodPut, http.MethodDelete, http.MethodPost, http.MethodPatch} {
+		w := do(reg, method, "/v2/"+referrerRepo+"/referrers/"+absentSubjectDgt, []byte("{}"), nil)
+		assert.Equal(t, http.StatusNotFound, w.Code, method)
+		assertCode(t, w, "NAME_UNKNOWN")
+	}
+}
+
+type failingReferrerStore struct{ ecr.MetaStore }
+
+func (failingReferrerStore) ListReferrers(context.Context, string, string, string) ([]ecr.ReferrerDescriptor, error) {
+	return nil, errors.New("no responders available for ecr.referrer.list")
+}
+
+func TestGetReferrers_MetadataFailureIsA5xxNotA404(t *testing.T) {
+	reg := newTestRegistry()
+	seedRepo(t, reg, referrerRepo)
+	reg.Meta = failingReferrerStore{reg.Meta}
+
+	w := do(reg, http.MethodGet, "/v2/"+referrerRepo+"/referrers/"+absentSubjectDgt, nil, nil)
+	assert.Equal(t, http.StatusInternalServerError, w.Code)
+}
+
+// pushSubjectImage stores a plain image (no subject) and returns its digest.
+func pushSubjectImage(t *testing.T, reg *Registry, repo, blobData string) string {
+	t.Helper()
+	cfg := pushBlob(t, reg, repo, []byte(blobData))
+	body := fmt.Sprintf(`{"schemaVersion":2,"mediaType":%q,"config":{"mediaType":%q,"digest":%q,"size":3},"layers":[]}`,
+		mediaTypeOCIManifest, sigConfigType, cfg)
+	dg := digestOf([]byte(body))
+	w := do(reg, http.MethodPut, "/v2/"+repo+"/manifests/"+dg, []byte(body),
+		map[string]string{"Content-Type": mediaTypeOCIManifest})
+	require.Equal(t, http.StatusCreated, w.Code, "body: %s", w.Body.String())
+	return dg
+}
+
+func manifestExists(reg *Registry, repo, dg string) bool {
+	_, err := reg.Meta.GetManifestMeta(context.Background(), testAccount, repo, dg)
+	return err == nil
+}
+
+func referrerDigests(t *testing.T, reg *Registry, repo, subject string) []string {
+	t.Helper()
+	refs, err := reg.Meta.ListReferrers(context.Background(), testAccount, repo, subject)
+	require.NoError(t, err)
+	out := make([]string, 0, len(refs))
+	for _, r := range refs {
+		out = append(out, r.Digest)
+	}
+	return out
+}
+
+func TestDeleteManifest_SubjectDeleteRemovesItsReferrers(t *testing.T) {
+	reg := newTestRegistry()
+	subject := pushSubjectImage(t, reg, referrerRepo, "subject")
+	sig := pushReferrer(t, reg, referrerRepo, subject, sigArtifactType, "sig")
+	sbom := pushReferrer(t, reg, referrerRepo, subject, "application/vnd.example.sbom.v1", "sbom")
+	require.ElementsMatch(t, []string{sig, sbom}, referrerDigests(t, reg, referrerRepo, subject))
+
+	w := do(reg, http.MethodDelete, "/v2/"+referrerRepo+"/manifests/"+subject, nil, nil)
+	require.Equal(t, http.StatusAccepted, w.Code, "body: %s", w.Body.String())
+
+	assert.False(t, manifestExists(reg, referrerRepo, subject))
+	assert.False(t, manifestExists(reg, referrerRepo, sig))
+	assert.False(t, manifestExists(reg, referrerRepo, sbom))
+	assert.Empty(t, referrerDigests(t, reg, referrerRepo, subject), "index keys go with the referrers")
+}
+
+func TestDeleteManifest_NestedReferrersAreRemoved(t *testing.T) {
+	reg := newTestRegistry()
+	subject := pushSubjectImage(t, reg, referrerRepo, "subject")
+	sig := pushReferrer(t, reg, referrerRepo, subject, sigArtifactType, "sig")
+	sigOfSig := pushReferrer(t, reg, referrerRepo, sig, sigArtifactType, "sig-of-sig")
+
+	_, err := reg.DeleteImage(context.Background(), testAccount, referrerRepo, "", subject)
+	require.NoError(t, err)
+
+	for _, dg := range []string{subject, sig, sigOfSig} {
+		assert.False(t, manifestExists(reg, referrerRepo, dg), dg)
+	}
+	assert.Empty(t, referrerDigests(t, reg, referrerRepo, sig))
+}
+
+func TestDeleteManifest_ReferrerDeleteLeavesTheSubject(t *testing.T) {
+	reg := newTestRegistry()
+	subject := pushSubjectImage(t, reg, referrerRepo, "subject")
+	sig := pushReferrer(t, reg, referrerRepo, subject, sigArtifactType, "sig")
+	other := pushReferrer(t, reg, referrerRepo, subject, "application/vnd.example.sbom.v1", "sbom")
+
+	_, err := reg.DeleteImage(context.Background(), testAccount, referrerRepo, "", sig)
+	require.NoError(t, err)
+
+	assert.True(t, manifestExists(reg, referrerRepo, subject))
+	assert.True(t, manifestExists(reg, referrerRepo, other))
+	assert.Equal(t, []string{other}, referrerDigests(t, reg, referrerRepo, subject))
+}
+
+func TestDeleteManifest_TagDeleteDoesNotCascade(t *testing.T) {
+	reg := newTestRegistry()
+	subject := pushSubjectImage(t, reg, referrerRepo, "subject")
+	sig := pushReferrer(t, reg, referrerRepo, subject, sigArtifactType, "sig")
+	require.NoError(t, reg.Meta.PutTag(context.Background(), testAccount, referrerRepo, "v1", subject))
+
+	w := do(reg, http.MethodDelete, "/v2/"+referrerRepo+"/manifests/v1", nil, nil)
+	require.Equal(t, http.StatusAccepted, w.Code)
+
+	assert.True(t, manifestExists(reg, referrerRepo, subject), "a tag delete only removes the pointer")
+	assert.True(t, manifestExists(reg, referrerRepo, sig))
+}
+
+func TestDeleteManifest_CascadeStaysInsideTheRepository(t *testing.T) {
+	reg := newTestRegistry()
+	subject := pushSubjectImage(t, reg, referrerRepo, "subject")
+	pushReferrer(t, reg, referrerRepo, subject, sigArtifactType, "sig")
+	elsewhere := pushReferrer(t, reg, "team/other", subject, sigArtifactType, "sig-other")
+
+	_, err := reg.DeleteImage(context.Background(), testAccount, referrerRepo, "", subject)
+	require.NoError(t, err)
+
+	assert.True(t, manifestExists(reg, "team/other", elsewhere))
+	assert.Equal(t, []string{elsewhere}, referrerDigests(t, reg, "team/other", subject))
+}
+
+type failingDeleteStore struct {
+	ecr.MetaStore
+
+	failDigest string
+}
+
+func (s failingDeleteStore) DeleteManifestMeta(ctx context.Context, account, repo, digest string) error {
+	if digest == s.failDigest {
+		return errors.New("kv unavailable")
+	}
+	return s.MetaStore.DeleteManifestMeta(ctx, account, repo, digest)
+}
+
+func TestDeleteManifest_ReferrerFailureDoesNotFailTheSubjectDelete(t *testing.T) {
+	reg := newTestRegistry()
+	subject := pushSubjectImage(t, reg, referrerRepo, "subject")
+	stuck := pushReferrer(t, reg, referrerRepo, subject, sigArtifactType, "stuck")
+	fine := pushReferrer(t, reg, referrerRepo, subject, "application/vnd.example.sbom.v1", "fine")
+	reg.Meta = failingDeleteStore{MetaStore: reg.Meta, failDigest: stuck}
+
+	got, err := reg.DeleteImage(context.Background(), testAccount, referrerRepo, "", subject)
+	require.NoError(t, err)
+	assert.Equal(t, subject, got)
+	assert.False(t, manifestExists(reg, referrerRepo, subject))
+	assert.False(t, manifestExists(reg, referrerRepo, fine), "one stuck referrer does not stop the others")
+	assert.True(t, manifestExists(reg, referrerRepo, stuck))
+}

spinifex/domains/ecr/service.go

@@ -28,6 +28,7 @@ const (
 	SubjectManifestDescribe = "ecr.manifest.describe"
 	SubjectManifestList     = "ecr.manifest.list"
 	SubjectManifestDelete   = "ecr.manifest.delete"
+	SubjectReferrerList     = "ecr.referrer.list"
 
 	SubjectUploadCreate = "ecr.upload.create"
 	SubjectUploadGet    = "ecr.upload.get"
@@ -223,6 +224,17 @@ type ManifestListResponse struct {
 	Digests []string `json:"digests"`
 }
 
+// ReferrerListRequest is the payload for SubjectReferrerList, naming the subject whose referrers to list.
+type ReferrerListRequest struct {
+	Repo    string `json:"repo"`
+	Subject string `json:"subject"`
+}
+
+// ReferrerListResponse is the reply to SubjectReferrerList, holding one descriptor per referrer.
+type ReferrerListResponse struct {
+	Referrers []ReferrerDescriptor `json:"referrers"`
+}
+
 // ManifestDeleteRequest is the payload for SubjectManifestDelete, naming the manifest metadata record to remove.
 type ManifestDeleteRequest struct {
 	Repo   string `json:"repo"`
@@ -313,6 +325,7 @@ type MetaService interface {
 	ManifestDescribe(ctx context.Context, req *ManifestDescribeRequest, accountID string) (*ManifestDescribeResponse, error)
 	ManifestList(ctx context.Context, req *ManifestListRequest, accountID string) (*ManifestListResponse, error)
 	ManifestDelete(ctx context.Context, req *ManifestDeleteRequest, accountID string) (*ManifestDeleteResponse, error)
+	ReferrerList(ctx context.Context, req *ReferrerListRequest, accountID string) (*ReferrerListResponse, error)
 
 	UploadCreate(ctx context.Context, req *UploadCreateRequest, accountID string) (*UploadCreateResponse, error)
 	UploadGet(ctx context.Context, req *UploadGetRequest, accountID string) (*UploadGetResponse, error)

spinifex/domains/ecr/service_impl.go

@@ -185,6 +185,14 @@ func (s *MetaServiceImpl) ManifestDelete(ctx context.Context, req *ManifestDelet
 	return &ManifestDeleteResponse{Found: true}, nil
 }
 
+func (s *MetaServiceImpl) ReferrerList(ctx context.Context, req *ReferrerListRequest, accountID string) (*ReferrerListResponse, error) {
+	referrers, err := s.store.ListReferrers(ctx, accountID, req.Repo, req.Subject)
+	if err != nil {
+		return nil, err
+	}
+	return &ReferrerListResponse{Referrers: referrers}, nil
+}
+
 func (s *MetaServiceImpl) ManifestDescribe(ctx context.Context, req *ManifestDescribeRequest, accountID string) (*ManifestDescribeResponse, error) {
 	meta, err := s.store.GetManifestMeta(ctx, accountID, req.Repo, req.Digest)
 	if errors.Is(err, ErrNotFound) {

spinifex/domains/ecr/storage.go

@@ -89,6 +89,7 @@ func KVAccountBucket(accountID string) string {
 //	repos/{name}/meta
 //	repos/{name}/tags/{tag}
 //	repos/{name}/manifests/{digest}
+//	repos/{name}/referrers/{subject}/{referrer}
 //	uploads/{uuid}
 const (
 	kvRepoMetaKeyFmt     = "repos/%s/meta"
@@ -98,6 +99,8 @@ const (
 	kvTagKeyFmt          = "repos/%s/tags/%s"
 	kvManifestKeyFmt     = "repos/%s/manifests/%s"
 	kvManifestsPrefixFmt = "repos/%s/manifests/"
+	kvReferrersPrefixFmt = "repos/%s/referrers/"
+	kvReferrerKeyFmt     = "repos/%s/referrers/%s/%s"
 	kvReposPrefix        = "repos/"
 	kvUploadKeyFmt       = "uploads/%s"
 )
@@ -128,6 +131,21 @@ func KVManifestKey(repo, digest string) string {
 // manifest metadata records.
 func KVManifestsPrefix(repo string) string { return fmt.Sprintf(kvManifestsPrefixFmt, repo) }
 
+// KVReferrersPrefix returns the KV key prefix enumerating every referrer entry
+// of a repository.
+func KVReferrersPrefix(repo string) string { return fmt.Sprintf(kvReferrersPrefixFmt, repo) }
+
+// KVSubjectReferrersPrefix returns the KV key prefix enumerating the referrers
+// of one subject manifest.
+func KVSubjectReferrersPrefix(repo, subjectDigest string) string {
+	return KVReferrersPrefix(repo) + DigestToken(subjectDigest) + "/"
+}
+
+// KVReferrerKey returns the KV key of one referrer entry under its subject.
+func KVReferrerKey(repo, subjectDigest, referrerDigest string) string {
+	return fmt.Sprintf(kvReferrerKeyFmt, repo, DigestToken(subjectDigest), DigestToken(referrerDigest))
+}
+
 // KVReposPrefix is the prefix under which every repo meta key lives. Used by
 // the catalog listing to enumerate account repositories.
 const KVReposPrefix = kvReposPrefix

spinifex/domains/eks/cluster/k3s_server_vm.go

@@ -566,6 +566,7 @@ func buildK3sUserData(in K3sServerInput) string {
 		"EKS_KONNECTIVITY_HOST="+konnHost,
 		"EKS_KONNECTIVITY_SANS="+strings.Join(konnSANs, ","),
 		"EKS_KONNECTIVITY_SERVER_COUNT="+strconv.Itoa(konnCount),
+		"EKS_WEBHOOK_READY_ADDR=0.0.0.0:"+strconv.FormatInt(webhookReadyPort, 10),
 	)
 	// A replacement CP prunes the terminated member it supersedes from etcd once
 	// it has joined (k3s-first-boot deletes the dead Node → embedded-etcd drops

spinifex/domains/eks/cluster/k3s_server_vm_test.go

@@ -4,6 +4,8 @@ import (
 	"context"
 	"errors"
 	"fmt"
+	"os"
+	"slices"
 	"strings"
 	"sync"
 	"testing"
@@ -489,6 +491,29 @@ func TestBuildK3sUserData_KonnectivityEnv(t *testing.T) {
 	assert.Contains(t, ud, "EKS_KONNECTIVITY_SANS=10.32.100.4,203.0.113.9,"+in.NLBDNS)
 }
 
+func TestBuildK3sUserData_WebhookReadinessListener(t *testing.T) {
+	t.Parallel()
+	ud := buildK3sUserData(validK3sInput())
+	assert.Contains(t, ud, "EKS_WEBHOOK_READY_ADDR=0.0.0.0:8444",
+		"the webhook must serve /readyz beyond loopback for the NLB health check")
+}
+
+// The env line alone does nothing: the webhook service only passes through
+// the variables its init script exports.
+func TestWebhookInitExportsReadyAddr(t *testing.T) {
+	t.Parallel()
+	b, err := os.ReadFile("../../../../scripts/images/eks-node/eks-token-webhook.initd")
+	require.NoError(t, err)
+	var exported bool
+	for line := range strings.SplitSeq(string(b), "\n") {
+		f := strings.Fields(line)
+		if len(f) > 1 && f[0] == "export" && slices.Contains(f[1:], "EKS_WEBHOOK_READY_ADDR") {
+			exported = true
+		}
+	}
+	assert.True(t, exported, "eks-token-webhook.initd must export EKS_WEBHOOK_READY_ADDR or the readiness listener never starts")
+}
+
 func TestBuildK3sUserData_KonnServerCountDefaultsToOne(t *testing.T) {
 	t.Parallel()
 	in := validK3sInput()

spinifex/domains/eks/cluster/nlb.go

@@ -7,6 +7,7 @@ import (
 	"errors"
 	"fmt"
 	"log/slog"
+	"strconv"
 	"strings"
 
 	"github.com/aws/aws-sdk-go/aws"
@@ -62,6 +63,31 @@ const lbcClusterOwnershipTagKey = "elbv2.k8s.aws/cluster"
 // agent (HA apiserver->pod/kubelet/webhook egress, EKS-parity).
 const konnectivityAgentPort int64 = 8132
 
+// webhookReadyPort is the plain-HTTP port on which the control-plane VM's token
+// webhook serves only /readyz, so the NLB can tell whether the VM can authenticate.
+const webhookReadyPort int64 = 8444
+
+// Health check of the apiserver target group: the control-plane token webhook's
+// /readyz, which also requires the local apiserver, so a CP that cannot
+// authenticate leaves rotation. Timeout stays below the interval.
+const (
+	cpHealthCheckPath      = "/readyz"
+	cpHealthCheckInterval  = 5
+	cpHealthCheckTimeout   = 3
+	cpHealthCheckThreshold = 2
+)
+
+// cpReadinessHealthCheck applies the readiness health check to a TG create.
+func cpReadinessHealthCheck(in *elbv2.CreateTargetGroupInput) {
+	in.HealthCheckProtocol = aws.String(elbv2.ProtocolEnumHttp)
+	in.HealthCheckPort = aws.String(strconv.FormatInt(webhookReadyPort, 10))
+	in.HealthCheckPath = aws.String(cpHealthCheckPath)
+	in.HealthCheckIntervalSeconds = aws.Int64(cpHealthCheckInterval)
+	in.HealthCheckTimeoutSeconds = aws.Int64(cpHealthCheckTimeout)
+	in.HealthyThresholdCount = aws.Int64(cpHealthCheckThreshold)
+	in.UnhealthyThresholdCount = aws.Int64(cpHealthCheckThreshold)
+}
+
 // ClusterNLB is the NLB resource tuple produced by EnsureClusterNLB; ARNs are persisted to ClusterMeta.
 type ClusterNLB struct {
 	LoadBalancerArn string
@@ -162,7 +188,7 @@ func EnsureClusterNLB(ctx context.Context, nlbp nlbProvisioner, accountID, clust
 		return nil, err
 	}
 	var err error
-	if out.TargetGroupArn, err = ensureClusterTG(ctx, nlbp, accountID, clusterName, tgName, k3sAPIServerPort); err != nil {
+	if out.TargetGroupArn, err = ensureClusterTG(ctx, nlbp, accountID, clusterName, tgName, k3sAPIServerPort, cpReadinessHealthCheck); err != nil {
 		return nil, err
 	}
 	if out.ListenerArn, err = ensureClusterListener(ctx, nlbp, accountID, lbName, out.LoadBalancerArn, out.TargetGroupArn, clusterNLBListenPort); err != nil {
@@ -176,7 +202,7 @@ func EnsureClusterNLB(ctx context.Context, nlbp nlbProvisioner, accountID, clust
 	}
 	// Konnectivity TG + listener :8132 → the 3 CP konnectivity-servers. Worker
 	// konnectivity-agents dial this VIP; server-count fan-out reaches all 3 servers.
-	if out.KonnTargetGroupArn, err = ensureClusterTG(ctx, nlbp, accountID, clusterName, konnTGName, konnectivityAgentPort); err != nil {
+	if out.KonnTargetGroupArn, err = ensureClusterTG(ctx, nlbp, accountID, clusterName, konnTGName, konnectivityAgentPort, nil); err != nil {
 		return nil, err
 	}
 	if _, err = ensureClusterListener(ctx, nlbp, accountID, lbName, out.LoadBalancerArn, out.KonnTargetGroupArn, konnectivityAgentPort); err != nil {
@@ -406,8 +432,9 @@ func ensureClusterLB(ctx context.Context, nlbp nlbProvisioner, accountID, cluste
 }
 
 // ensureClusterTG creates (or reuses) a TCP target group of the given name
-// forwarding to port, returning its ARN. Idempotent on name.
-func ensureClusterTG(ctx context.Context, nlbp nlbProvisioner, accountID, clusterName, tgName string, port int64) (string, error) {
+// forwarding to port, returning its ARN. Idempotent on name. configure, if set,
+// adjusts the create input; an existing group is returned as is, unmodified.
+func ensureClusterTG(ctx context.Context, nlbp nlbProvisioner, accountID, clusterName, tgName string, port int64, configure func(*elbv2.CreateTargetGroupInput)) (string, error) {
 	if tg, err := lookupTGByName(ctx, nlbp, accountID, tgName); err != nil {
 		return "", err
 	} else if tg != nil {
@@ -418,7 +445,7 @@ func ensureClusterTG(ctx context.Context, nlbp nlbProvisioner, accountID, cluste
 		return arn, nil
 	}
 
-	created, err := nlbp.CreateTargetGroup(ctx, &elbv2.CreateTargetGroupInput{
+	in := &elbv2.CreateTargetGroupInput{
 		Name:       aws.String(tgName),
 		Protocol:   aws.String(elbv2.ProtocolEnumTcp),
 		Port:       aws.Int64(port),
@@ -427,7 +454,11 @@ func ensureClusterTG(ctx context.Context, nlbp nlbProvisioner, accountID, cluste
 			{Key: aws.String(tags.ManagedByKey), Value: aws.String(tags.ManagedByEKS)},
 			{Key: aws.String(clusterEKSClusterTagKey), Value: aws.String(clusterName)},
 		},
-	}, accountID)
+	}
+	if configure != nil {
+		configure(in)
+	}
+	created, err := nlbp.CreateTargetGroup(ctx, in, accountID)
 	if err != nil {
 		return "", fmt.Errorf("create TG %s: %w", tgName, err)
 	}

spinifex/domains/eks/cluster/nlb_test.go

@@ -201,6 +201,14 @@ func (f *fakeNLBProvisioner) CreateTargetGroup(_ context.Context, input *elbv2.C
 				Protocol:        input.Protocol,
 				Port:            input.Port,
 				TargetType:      input.TargetType,
+
+				HealthCheckProtocol:        input.HealthCheckProtocol,
+				HealthCheckPort:            input.HealthCheckPort,
+				HealthCheckPath:            input.HealthCheckPath,
+				HealthCheckIntervalSeconds: input.HealthCheckIntervalSeconds,
+				HealthCheckTimeoutSeconds:  input.HealthCheckTimeoutSeconds,
+				HealthyThresholdCount:      input.HealthyThresholdCount,
+				UnhealthyThresholdCount:    input.UnhealthyThresholdCount,
 			}},
 		}
 	}
@@ -434,6 +442,55 @@ func TestEnsureClusterNLB_InternetFacingUsesPublicAddress(t *testing.T) {
 	assert.Equal(t, elbv2.LoadBalancerSchemeEnumInternetFacing, aws.StringValue(nlbp.createLBSyncCalls[0].Scheme))
 }
 
+func TestEnsureClusterNLB_APIServerTGHealthChecksWebhookReadiness(t *testing.T) {
+	t.Parallel()
+	nlbp := newFakeNLBProvisioner()
+	_, err := EnsureClusterNLB(context.Background(), nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, false, nil, nil)
+	require.NoError(t, err)
+
+	api := nlbp.tgByName["eks-alpha-cp"]
+	require.NotNil(t, api)
+	assert.Equal(t, elbv2.ProtocolEnumTcp, aws.StringValue(api.Protocol), "traffic stays TCP")
+	assert.Equal(t, elbv2.ProtocolEnumHttp, aws.StringValue(api.HealthCheckProtocol))
+	assert.Equal(t, "8444", aws.StringValue(api.HealthCheckPort))
+	assert.Equal(t, "/readyz", aws.StringValue(api.HealthCheckPath))
+	assert.Equal(t, int64(5), aws.Int64Value(api.HealthCheckIntervalSeconds))
+	assert.Equal(t, int64(2), aws.Int64Value(api.HealthyThresholdCount))
+	assert.Equal(t, int64(2), aws.Int64Value(api.UnhealthyThresholdCount))
+	assert.Less(t, aws.Int64Value(api.HealthCheckTimeoutSeconds), aws.Int64Value(api.HealthCheckIntervalSeconds),
+		"the probe must time out before the next one starts")
+
+	konn := nlbp.tgByName["eks-alpha-konn"]
+	require.NotNil(t, konn)
+	assert.Nil(t, konn.HealthCheckProtocol, "konnectivity TG keeps the default health check")
+	assert.Nil(t, konn.HealthCheckPort)
+	assert.Nil(t, konn.HealthCheckPath)
+}
+
+func TestEnsureClusterNLB_ExistingAPIServerTGIsLeftAsIs(t *testing.T) {
+	t.Parallel()
+	nlbp := newFakeNLBProvisioner()
+	legacy := &elbv2.TargetGroup{
+		TargetGroupArn:  aws.String("arn:aws:elasticloadbalancing:us-east-1:111122223333:targetgroup/eks-alpha-cp/legacy"),
+		TargetGroupName: aws.String("eks-alpha-cp"),
+		Protocol:        aws.String(elbv2.ProtocolEnumTcp),
+		Port:            aws.Int64(k3sAPIServerPort),
+	}
+	nlbp.tgByName["eks-alpha-cp"] = legacy
+
+	out, err := EnsureClusterNLB(context.Background(), nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, false, nil, nil)
+	require.NoError(t, err)
+
+	assert.Equal(t, aws.StringValue(legacy.TargetGroupArn), out.TargetGroupArn)
+	for _, in := range nlbp.createTGCalls {
+		assert.NotEqual(t, "eks-alpha-cp", aws.StringValue(in.Name), "an existing TG is not re-created")
+	}
+	stored := nlbp.tgByName["eks-alpha-cp"]
+	assert.Nil(t, stored.HealthCheckProtocol, "an existing TG keeps its original health check")
+	assert.Nil(t, stored.HealthCheckPort)
+	assert.Nil(t, stored.HealthCheckPath)
+}
+
 func TestEnsureClusterNLB_IdempotentReusesExisting(t *testing.T) {
 	t.Parallel()
 	nlbp := newFakeNLBProvisioner()

spinifex/domains/eks/cluster/security_groups.go

@@ -397,8 +397,8 @@ func EnsureNodegroupSGRules(ctx context.Context, sgp sgProvisioner, accountID, c
 }
 
 // EnsureControlPlaneIngress admits the NLB→CP hops from the VPC CIDR: the
-// apiserver (k3sAPIServerPort) and the konnectivity-server agent port
-// (konnectivityAgentPort). Public access is gated separately at the NLB
+// apiserver (k3sAPIServerPort), the konnectivity-server agent port
+// (konnectivityAgentPort) and the webhook readiness port. Public access is gated separately at the NLB
 // front-end. Idempotent.
 func EnsureControlPlaneIngress(ctx context.Context, sgp sgProvisioner, accountID, cpSGID, vpcCIDR string) error {
 	if cpSGID == "" {
@@ -414,6 +414,7 @@ func EnsureControlPlaneIngress(ctx context.Context, sgp sgProvisioner, accountID
 	}{
 		{k3sAPIServerPort, "EKS NLB to apiserver"},
 		{konnectivityAgentPort, "EKS NLB to konnectivity-server"},
+		{webhookReadyPort, "EKS NLB to token webhook readiness"},
 	} {
 		perm := &ec2.IpPermission{
 			IpProtocol: aws.String("tcp"),

spinifex/domains/eks/cluster/security_groups_test.go

@@ -457,8 +457,8 @@ func TestEnsureControlPlaneIngress_AuthorizesAPIServerFromVPCCIDR(t *testing.T)
 	err := EnsureControlPlaneIngress(context.Background(), sgp, "111122223333", "sg-cp-001", "10.0.0.0/16")
 	require.NoError(t, err)
 
-	// NLB → apiserver (:6443) and NLB → konnectivity-server (:8132).
-	require.Len(t, sgp.authorizeCalls, 2)
+	// NLB → apiserver (:6443), konnectivity-server (:8132) and webhook readiness (:8444).
+	require.Len(t, sgp.authorizeCalls, 3)
 	ports := map[int64]bool{}
 	for _, in := range sgp.authorizeCalls {
 		assert.Equal(t, "sg-cp-001", aws.StringValue(in.GroupId))
@@ -472,6 +472,7 @@ func TestEnsureControlPlaneIngress_AuthorizesAPIServerFromVPCCIDR(t *testing.T)
 	}
 	assert.True(t, ports[k3sAPIServerPort], "admits :6443")
 	assert.True(t, ports[konnectivityAgentPort], "admits :8132")
+	assert.True(t, ports[webhookReadyPort], "admits the webhook readiness port from the VPC CIDR")
 }
 
 func TestEnsureControlPlaneIngress_EmptyInputsRejected(t *testing.T) {

spinifex/domains/elbv2/haproxy.go

@@ -52,6 +52,12 @@ backend {{.Name}}
 {{- range .Servers}}
     server {{.Name}} {{.Addr}}:{{.Port}} check{{if .Secure}} check-ssl ssl verify none{{end}} inter {{.CheckInterval}}s fall {{.Fall}} rise {{.Rise}}
 {{- end}}
+{{- if .Backups}}
+    option allbackups
+{{- range .Backups}}
+    server {{.Name}} {{.Addr}}:{{.Port}} backup{{if .Secure}} ssl verify none{{end}}
+{{- end}}
+{{- end}}
 {{- end}}
 {{end}}
 `
@@ -107,8 +113,11 @@ type HAProxyBackend struct {
 	// sends "Host: <ip>" on the scheme's default port and "<ip>:<port>" otherwise.
 	HealthCheckDefaultPort int64
 	Servers                []HAProxyServer
-	FixedResponse          *HAProxyFixedResponse
-	Redirect               *HAProxyRedirect
+	// Backups are unchecked copies of Servers that HAProxy routes to only when
+	// every checked server is down, so an all-unhealthy group fails open.
+	Backups       []HAProxyServer
+	FixedResponse *HAProxyFixedResponse
+	Redirect      *HAProxyRedirect
 }
 
 // HAProxyFixedResponse renders an `http-request return` directive. All fields are validated before use.
@@ -204,8 +213,9 @@ func buildHAProxyConfig(lb *LoadBalancerRecord, listeners []*ListenerRecord, tgB
 			if target.PrivateIP == "" || target.HealthState == TargetHealthDraining {
 				continue
 			}
+			name := sanitizeName("srv", target.Id)
 			backend.Servers = append(backend.Servers, HAProxyServer{
-				Name:          sanitizeName("srv", target.Id),
+				Name:          name,
 				Addr:          target.PrivateIP,
 				Port:          target.Port,
 				CheckInterval: tg.HealthCheck.IntervalSeconds,
@@ -213,6 +223,12 @@ func buildHAProxyConfig(lb *LoadBalancerRecord, listeners []*ListenerRecord, tgB
 				Rise:          tg.HealthCheck.HealthyThreshold,
 				Secure:        secure,
 			})
+			backend.Backups = append(backend.Backups, HAProxyServer{
+				Name:   name + lbagent.FailoverServerSuffix,
+				Addr:   target.PrivateIP,
+				Port:   target.Port,
+				Secure: secure,
+			})
 		}
 		cfg.Backends = append(cfg.Backends, backend)
 		backendAdded[backendName] = struct{}{}

spinifex/domains/elbv2/haproxy_test.go

@@ -551,3 +551,74 @@ func TestGenerateALBConfig_StillWorks(t *testing.T) {
 	assert.NotContains(t, config, "mode tcp")
 	assert.NotContains(t, config, "option tcplog")
 }
+
+func albFailoverConfig(t *testing.T, targets ...Target) string {
+	t.Helper()
+	lb := &LoadBalancerRecord{
+		LoadBalancerArn: "arn:aws:elasticloadbalancing:us-east-1:123:loadbalancer/app/my-alb/lb-abc123",
+		LoadBalancerID:  "lb-abc123",
+		Type:            LoadBalancerTypeApplication,
+	}
+	tgArn := "arn:aws:elasticloadbalancing:us-east-1:123:targetgroup/my-tg/tg-222"
+	listeners := []*ListenerRecord{{
+		ListenerArn:     "arn:lst",
+		LoadBalancerArn: lb.LoadBalancerArn,
+		Protocol:        ProtocolHTTP,
+		Port:            80,
+		DefaultActions:  []ListenerAction{{Type: ActionTypeForward, TargetGroupArn: tgArn}},
+	}}
+	tgByArn := map[string]*TargetGroupRecord{tgArn: {
+		TargetGroupArn: tgArn,
+		HealthCheck:    HealthCheckConfig{Path: "/health", Matcher: "200", IntervalSeconds: 30, UnhealthyThreshold: 2, HealthyThreshold: 5},
+		Targets:        targets,
+	}}
+	config, _, err := GenerateHAProxyConfigWithCerts(lb, listeners, tgByArn, nil, "10.0.1.5", nil)
+	require.NoError(t, err)
+	return config
+}
+
+func TestGenerateHAProxy_FailOpenBackupCopies(t *testing.T) {
+	t.Parallel()
+	config := albFailoverConfig(t,
+		Target{Id: "i-aaa111", Port: 8080, PrivateIP: "10.0.1.10", HealthState: TargetHealthHealthy},
+		Target{Id: "i-bbb222", Port: 8080, PrivateIP: "10.0.1.11", HealthState: TargetHealthUnhealthy},
+		Target{Id: "i-drain", Port: 8080, PrivateIP: "10.0.1.12", HealthState: TargetHealthDraining},
+		Target{Id: "i-noip", Port: 8080, PrivateIP: "", HealthState: TargetHealthHealthy},
+	)
+
+	assert.Contains(t, config, "    option allbackups\n")
+	// Checked servers are unchanged; the unhealthy one stays so haproxy's own check decides.
+	assert.Contains(t, config, "server srv_i-aaa111 10.0.1.10:8080 check inter 30s fall 2 rise 5\n")
+	assert.Contains(t, config, "server srv_i-bbb222 10.0.1.11:8080 check inter 30s fall 2 rise 5\n")
+	// Backup copies: same address, distinct name, no check.
+	assert.Contains(t, config, "server srv_i-aaa111.fo 10.0.1.10:8080 backup\n")
+	assert.Contains(t, config, "server srv_i-bbb222.fo 10.0.1.11:8080 backup\n")
+	// Draining and IP-less targets get neither a server nor a copy.
+	assert.NotContains(t, config, "10.0.1.12")
+	assert.NotContains(t, config, "srv_i-noip")
+}
+
+func TestGenerateHAProxy_BackupCopyKeepsTLSToHTTPSTargets(t *testing.T) {
+	t.Parallel()
+	lb := &LoadBalancerRecord{LoadBalancerID: "lb-abc123", Type: LoadBalancerTypeApplication}
+	tgArn := "arn:tg-https"
+	listeners := []*ListenerRecord{{
+		ListenerArn: "arn:lst", Protocol: ProtocolHTTP, Port: 80,
+		DefaultActions: []ListenerAction{{Type: ActionTypeForward, TargetGroupArn: tgArn}},
+	}}
+	tgByArn := map[string]*TargetGroupRecord{tgArn: {
+		TargetGroupArn: tgArn,
+		Protocol:       ProtocolHTTPS,
+		HealthCheck:    HealthCheckConfig{Path: "/", Matcher: "200", IntervalSeconds: 30, UnhealthyThreshold: 2, HealthyThreshold: 2},
+		Targets:        []Target{{Id: "i-1", Port: 8443, PrivateIP: "10.0.1.10", HealthState: TargetHealthHealthy}},
+	}}
+	config, _, err := GenerateHAProxyConfigWithCerts(lb, listeners, tgByArn, nil, "10.0.1.5", nil)
+	require.NoError(t, err)
+	assert.Contains(t, config, "server srv_i-1.fo 10.0.1.10:8443 backup ssl verify none\n")
+}
+
+func TestGenerateHAProxy_NoServersNoAllBackups(t *testing.T) {
+	t.Parallel()
+	config := albFailoverConfig(t, Target{Id: "i-drain", Port: 8080, PrivateIP: "10.0.1.12", HealthState: TargetHealthDraining})
+	assert.NotContains(t, config, "allbackups")
+}

spinifex/domains/elbv2/health_checker.go

@@ -15,6 +15,10 @@ import (
 type healthChecker struct {
 	store *Store
 
+	// onChanged, when set, is called with the IDs of target groups whose
+	// target states were just persisted, so load balancers can re-render.
+	onChanged func(ctx context.Context, tgIDs []string)
+
 	mu       sync.Mutex
 	counters map[string]*targetCounter // key: "tgID:targetId:port"
 }
@@ -118,6 +122,7 @@ func (hc *healthChecker) handleHealthReportDirect(ctx context.Context, report lb
 	// Apply the new states field-by-field under CAS. A whole-record Put here
 	// would discard any target registered since this scan read the group, which
 	// is the one write on this key that runs on its own schedule.
+	var persisted []string
 	for _, tg := range changedTGs {
 		states := make(map[string]Target, len(tg.Targets))
 		for _, t := range tg.Targets {
@@ -139,7 +144,14 @@ func (hc *healthChecker) handleHealthReportDirect(ctx context.Context, report lb
 		})
 		if err != nil && !errors.Is(err, ErrTargetGroupNotFound) {
 			slog.ErrorContext(ctx, "healthChecker: failed to persist target group", "tgId", tg.TargetGroupID, "err", err)
+			continue
 		}
+		if err == nil {
+			persisted = append(persisted, tg.TargetGroupID)
+		}
+	}
+	if len(persisted) > 0 && hc.onChanged != nil {
+		hc.onChanged(ctx, persisted)
 	}
 }
 

spinifex/domains/elbv2/health_rerender_test.go

@@ -0,0 +1,392 @@
+package elbv2
+
+import (
+	"testing"
+	"time"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/service/elbv2"
+	lbagent "github.com/mulgadc/spinifex/spinifex/agents/elbv2"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+const rerenderTGArn = "arn:aws:elasticloadbalancing:us-east-1:1:targetgroup/db/tg-rr"
+
+// putRerenderNLB stores an NLB with a TCP listener forwarding to rerenderTGArn
+// and renders its initial config.
+func putRerenderNLB(t *testing.T, svc *ELBv2ServiceImpl, lbID string) {
+	t.Helper()
+	lbArn := "arn:aws:elasticloadbalancing:us-east-1:1:loadbalancer/net/" + lbID + "/" + lbID
+	lb := &LoadBalancerRecord{
+		LoadBalancerArn: lbArn,
+		LoadBalancerID:  lbID,
+		Name:            lbID,
+		Type:            LoadBalancerTypeNetwork,
+		State:           StateActive,
+		AccountID:       testAccountID,
+		InstanceID:      "i-" + lbID,
+	}
+	require.NoError(t, svc.store.PutLoadBalancer(t.Context(), lb))
+	l := tcpListener(lbArn+"/l1", lbID+"-l1", ProtocolTCP, 5432, rerenderTGArn)
+	l.LoadBalancerArn = lbArn
+	require.NoError(t, svc.store.PutListener(t.Context(), l))
+	require.NoError(t, svc.updateStoredConfig(t.Context(), lb))
+}
+
+func setupRerender(t *testing.T, lbIDs ...string) *ELBv2ServiceImpl {
+	t.Helper()
+	svc := setupTestService(t)
+	hcCfg := nlbHealthCheck()
+	hcCfg.HealthyThreshold = 1
+	hcCfg.UnhealthyThreshold = 1
+	require.NoError(t, svc.store.PutTargetGroup(t.Context(), &TargetGroupRecord{
+		TargetGroupArn: rerenderTGArn,
+		TargetGroupID:  "tg-rr",
+		Protocol:       ProtocolTCP,
+		Port:           5432,
+		HealthCheck:    hcCfg,
+		Targets: []Target{
+			{Id: "i-ok", Port: 5432, PrivateIP: "10.0.1.10", HealthState: TargetHealthHealthy},
+			{Id: "i-bad", Port: 5432, PrivateIP: "10.0.1.11", HealthState: TargetHealthHealthy},
+		},
+	}))
+	for _, id := range lbIDs {
+		putRerenderNLB(t, svc, id)
+	}
+	return svc
+}
+
+func rerenderReport(lbID, badStatus string) lbagent.HealthReport {
+	return lbagent.HealthReport{
+		LBID: lbID,
+		Servers: []lbagent.ServerStatus{
+			{Server: sanitizeName("srv", "i-ok"), Status: "UP"},
+			{Server: sanitizeName("srv", "i-bad"), Status: badStatus},
+		},
+	}
+}
+
+func getLB(t *testing.T, svc *ELBv2ServiceImpl, id string) *LoadBalancerRecord {
+	t.Helper()
+	lb, err := svc.store.GetLoadBalancer(t.Context(), id)
+	require.NoError(t, err)
+	require.NotNil(t, lb)
+	return lb
+}
+
+func TestHealthTransition_ReRendersConfigWithoutUnhealthyTarget(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a")
+	before := getLB(t, svc, "lb-a")
+	require.Contains(t, before.ConfigText, "10.0.1.11")
+
+	svc.hc.handleHealthReportDirect(t.Context(), rerenderReport("lb-a", "DOWN"))
+
+	after := getLB(t, svc, "lb-a")
+	assert.NotEqual(t, before.ConfigHash, after.ConfigHash)
+	assert.NotContains(t, after.ConfigText, "10.0.1.11")
+	assert.Contains(t, after.ConfigText, "10.0.1.10")
+}
+
+func TestHealthReport_SteadyStateLeavesLBUntouched(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a")
+	before := getLB(t, svc, "lb-a")
+
+	svc.hc.handleHealthReportDirect(t.Context(), rerenderReport("lb-a", "UP"))
+
+	assert.Equal(t, before, getLB(t, svc, "lb-a"))
+}
+
+func TestHealthTransition_SharedTargetGroupReRendersEveryLB(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a", "lb-b")
+
+	svc.hc.handleHealthReportDirect(t.Context(), rerenderReport("lb-a", "DOWN"))
+
+	for _, id := range []string{"lb-a", "lb-b"} {
+		assert.NotContains(t, getLB(t, svc, id).ConfigText, "10.0.1.11", id)
+	}
+}
+
+func TestHealthTransition_RecoveryReAddsTarget(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a")
+	svc.hc.handleHealthReportDirect(t.Context(), rerenderReport("lb-a", "DOWN"))
+	require.NotContains(t, getLB(t, svc, "lb-a").ConfigText, "10.0.1.11")
+
+	svc.hc.handleHealthReportDirect(t.Context(), rerenderReport("lb-a", "UP"))
+
+	assert.Contains(t, getLB(t, svc, "lb-a").ConfigText, "10.0.1.11")
+}
+
+func TestUpdateLoadBalancer_RetriesAfterConcurrentWriteAndKeepsBoth(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a")
+	attempts := 0
+	heartbeat := time.Now().UTC().Truncate(time.Second)
+
+	_, err := svc.store.UpdateLoadBalancer(t.Context(), "lb-a", func(rec *LoadBalancerRecord) (bool, error) {
+		attempts++
+		if attempts == 1 {
+			_, werr := svc.store.UpdateLoadBalancer(t.Context(), "lb-a", func(r *LoadBalancerRecord) (bool, error) {
+				r.ConfigHash = "rendered-elsewhere"
+				return true, nil
+			})
+			require.NoError(t, werr)
+		}
+		rec.LastHeartbeat = heartbeat
+		return true, nil
+	})
+	require.NoError(t, err)
+
+	assert.Equal(t, 2, attempts)
+	got := getLB(t, svc, "lb-a")
+	assert.Equal(t, "rendered-elsewhere", got.ConfigHash)
+	assert.True(t, heartbeat.Equal(got.LastHeartbeat))
+}
+
+func TestUpdateLoadBalancer_MissingRecordIsNotCreated(t *testing.T) {
+	t.Parallel()
+	svc := setupTestService(t)
+	_, err := svc.store.UpdateLoadBalancer(t.Context(), "lb-gone", func(rec *LoadBalancerRecord) (bool, error) {
+		return true, nil
+	})
+	require.ErrorIs(t, err, ErrLoadBalancerNotFound)
+	got, gerr := svc.store.GetLoadBalancer(t.Context(), "lb-gone")
+	require.NoError(t, gerr)
+	assert.Nil(t, got)
+}
+
+func TestReRender_KeepsHeartbeatWrittenAfterItReadTheLB(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a")
+	stale := getLB(t, svc, "lb-a")
+
+	heartbeat := time.Now().UTC().Truncate(time.Second)
+	_, err := svc.store.UpdateLoadBalancer(t.Context(), "lb-a", func(rec *LoadBalancerRecord) (bool, error) {
+		rec.LastHeartbeat = heartbeat
+		return true, nil
+	})
+	require.NoError(t, err)
+
+	_, err = svc.store.UpdateTargetGroup(t.Context(), "tg-rr", func(tg *TargetGroupRecord) (bool, error) {
+		tg.Targets[1].HealthState = TargetHealthUnhealthy
+		return true, nil
+	})
+	require.NoError(t, err)
+	require.NoError(t, svc.updateStoredConfig(t.Context(), stale))
+
+	got := getLB(t, svc, "lb-a")
+	assert.True(t, heartbeat.Equal(got.LastHeartbeat), "re-render must not roll back the heartbeat")
+	assert.NotContains(t, got.ConfigText, "10.0.1.11")
+}
+
+func TestHeartbeatRefresh_KeepsReRenderedConfig(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a")
+	svc.hc.handleHealthReportDirect(t.Context(), rerenderReport("lb-a", "DOWN"))
+	rendered := getLB(t, svc, "lb-a")
+	require.NotContains(t, rendered.ConfigText, "10.0.1.11")
+
+	_, err := svc.LBAgentHeartbeat(t.Context(), &LBAgentHeartbeatInput{LBID: aws.String("lb-a")}, testAccountID)
+	require.NoError(t, err)
+
+	got := getLB(t, svc, "lb-a")
+	assert.Equal(t, rendered.ConfigHash, got.ConfigHash)
+	assert.Equal(t, rendered.ConfigText, got.ConfigText)
+	assert.True(t, got.LastHeartbeat.After(rendered.LastHeartbeat), "stale heartbeat must be refreshed")
+}
+
+const rerenderLBArn = "arn:aws:elasticloadbalancing:us-east-1:1:loadbalancer/net/lb-a/lb-a"
+
+// bumpConfigHash simulates a re-render landing after another writer read the LB.
+func bumpConfigHash(t *testing.T, svc *ELBv2ServiceImpl, lbID string) {
+	t.Helper()
+	_, err := svc.store.UpdateLoadBalancer(t.Context(), lbID, func(rec *LoadBalancerRecord) (bool, error) {
+		rec.ConfigHash = "rendered-elsewhere"
+		return true, nil
+	})
+	require.NoError(t, err)
+}
+
+func TestLBAttributeSave_KeepsConfigHashWrittenAfterRead(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a")
+	res := svc.lbAttrResource(t.Context(), rerenderLBArn, testAccountID, "test")
+	stale, err := res.fetch(rerenderLBArn)
+	require.NoError(t, err)
+	require.NotNil(t, stale)
+
+	bumpConfigHash(t, svc, "lb-a")
+	stale.Attributes = map[string]string{"deletion_protection.enabled": "true"}
+	require.NoError(t, res.save(stale))
+
+	got := getLB(t, svc, "lb-a")
+	assert.Equal(t, "rendered-elsewhere", got.ConfigHash)
+	assert.Equal(t, "true", got.Attributes["deletion_protection.enabled"])
+}
+
+func TestLBTagSave_KeepsConfigHashWrittenAfterRead(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a")
+	h, _, found, err := svc.resolveTaggable(t.Context(), rerenderLBArn)
+	require.NoError(t, err)
+	require.True(t, found)
+
+	bumpConfigHash(t, svc, "lb-a")
+	require.NoError(t, h.save(map[string]string{"env": "test"}))
+
+	got := getLB(t, svc, "lb-a")
+	assert.Equal(t, "rendered-elsewhere", got.ConfigHash)
+	assert.Equal(t, "test", got.Tags["env"])
+}
+
+// The next three read and write inside one call, so the hash is bumped first:
+// they prove only the fields they own change, not the interleaving itself.
+func TestSetIpAddressType_LeavesConfigHashAlone(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a")
+	bumpConfigHash(t, svc, "lb-a")
+
+	_, err := svc.SetIpAddressType(t.Context(), &elbv2.SetIpAddressTypeInput{
+		LoadBalancerArn: aws.String(rerenderLBArn),
+		IpAddressType:   aws.String(IPAddressTypeIPv4),
+	}, testAccountID)
+	require.NoError(t, err)
+
+	got := getLB(t, svc, "lb-a")
+	assert.Equal(t, "rendered-elsewhere", got.ConfigHash)
+	assert.Equal(t, IPAddressTypeIPv4, got.IPAddressType)
+}
+
+func TestSetSecurityGroups_LeavesConfigHashAlone(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a")
+	_, err := svc.store.UpdateLoadBalancer(t.Context(), "lb-a", func(rec *LoadBalancerRecord) (bool, error) {
+		rec.SecurityGroups = []string{"sg-old"}
+		rec.ConfigHash = "rendered-elsewhere"
+		return true, nil
+	})
+	require.NoError(t, err)
+
+	_, err = svc.SetSecurityGroups(t.Context(), &elbv2.SetSecurityGroupsInput{
+		LoadBalancerArn: aws.String(rerenderLBArn),
+		SecurityGroups:  []*string{aws.String("sg-new")},
+	}, testAccountID)
+	require.NoError(t, err)
+
+	got := getLB(t, svc, "lb-a")
+	assert.Equal(t, "rendered-elsewhere", got.ConfigHash)
+	assert.Equal(t, []string{"sg-new"}, got.SecurityGroups)
+}
+
+func TestSetLoadBalancerIngressCIDRs_LeavesConfigHashAlone(t *testing.T) {
+	t.Parallel()
+	svc := setupRerender(t, "lb-a")
+	bumpConfigHash(t, svc, "lb-a")
+
+	require.NoError(t, svc.SetLoadBalancerIngressCIDRs(rerenderLBArn, []string{"10.0.0.0/8"}, testAccountID))
+
+	got := getLB(t, svc, "lb-a")
+	assert.Equal(t, "rendered-elsewhere", got.ConfigHash)
+	assert.Equal(t, []string{"10.0.0.0/8"}, got.NLBIngressCIDRs)
+}
+
+const modifyTGArn = "arn:aws:elasticloadbalancing:us-east-1:1:targetgroup/mod/tg-mod"
+
+// putModifyFixture stores a target group with an HTTP health check, an NLB
+// and an ALB that both forward to it, and renders their initial configs.
+func putModifyFixture(t *testing.T, svc *ELBv2ServiceImpl, protocol string) {
+	t.Helper()
+	hc := nlbHealthCheck()
+	hc.Protocol = ProtocolHTTP
+	hc.Path = "/old"
+	hc.Matcher = "200"
+	require.NoError(t, svc.store.PutTargetGroup(t.Context(), &TargetGroupRecord{
+		TargetGroupArn: modifyTGArn,
+		TargetGroupID:  "tg-mod",
+		AccountID:      testAccountID,
+		Protocol:       protocol,
+		Port:           8080,
+		HealthCheck:    hc,
+		Targets:        []Target{{Id: "i-ok", Port: 8080, PrivateIP: "10.0.1.10", HealthState: TargetHealthHealthy}},
+	}))
+}
+
+func putModifyLB(t *testing.T, svc *ELBv2ServiceImpl, lbID, lbType, protocol string) {
+	t.Helper()
+	lbArn := "arn:aws:elasticloadbalancing:us-east-1:1:loadbalancer/x/" + lbID + "/" + lbID
+	lb := &LoadBalancerRecord{
+		LoadBalancerArn: lbArn,
+		LoadBalancerID:  lbID,
+		Name:            lbID,
+		Type:            lbType,
+		State:           StateActive,
+		AccountID:       testAccountID,
+		InstanceID:      "i-" + lbID,
+	}
+	require.NoError(t, svc.store.PutLoadBalancer(t.Context(), lb))
+	l := tcpListener(lbArn+"/l1", lbID+"-l1", protocol, 8080, modifyTGArn)
+	l.LoadBalancerArn = lbArn
+	require.NoError(t, svc.store.PutListener(t.Context(), l))
+	require.NoError(t, svc.updateStoredConfig(t.Context(), lb))
+}
+
+func modifyPath(t *testing.T, svc *ELBv2ServiceImpl, path string) {
+	t.Helper()
+	_, err := svc.ModifyTargetGroup(t.Context(), &elbv2.ModifyTargetGroupInput{
+		TargetGroupArn:  aws.String(modifyTGArn),
+		HealthCheckPath: aws.String(path),
+	}, testAccountID)
+	require.NoError(t, err)
+}
+
+func TestModifyTargetGroup_ReRendersALBWithTheNewHealthPath(t *testing.T) {
+	t.Parallel()
+	svc := setupTestService(t)
+	putModifyFixture(t, svc, ProtocolHTTP)
+	putModifyLB(t, svc, "alb-m", LoadBalancerTypeApplication, ProtocolHTTP)
+	before := getLB(t, svc, "alb-m")
+	require.Contains(t, before.ConfigText, "uri /old")
+
+	modifyPath(t, svc, "/new")
+
+	after := getLB(t, svc, "alb-m")
+	assert.Contains(t, after.ConfigText, "uri /new")
+	assert.NotContains(t, after.ConfigText, "uri /old")
+	assert.NotEqual(t, before.ConfigHash, after.ConfigHash)
+}
+
+func TestModifyTargetGroup_NLBProbeChangeMovesTheConfigHash(t *testing.T) {
+	t.Parallel()
+	svc := setupTestService(t)
+	putModifyFixture(t, svc, ProtocolTCP)
+	putModifyLB(t, svc, "nlb-m", LoadBalancerTypeNetwork, ProtocolTCP)
+	before := getLB(t, svc, "nlb-m")
+	require.Len(t, before.HealthTargets, 1)
+	require.Equal(t, "/old", before.HealthTargets[0].Path)
+
+	modifyPath(t, svc, "/new")
+
+	// nginx has no probe settings, so only the hash input carries this change
+	// to the agent, which re-fetches health targets when the hash moves.
+	after := getLB(t, svc, "nlb-m")
+	require.Len(t, after.HealthTargets, 1)
+	assert.Equal(t, "/new", after.HealthTargets[0].Path)
+	assert.Equal(t, before.ConfigText, after.ConfigText)
+	assert.NotEqual(t, before.ConfigHash, after.ConfigHash)
+}
+
+func TestModifyTargetGroup_UnchangedSettingsLeaveTheConfigHashAlone(t *testing.T) {
+	t.Parallel()
+	svc := setupTestService(t)
+	putModifyFixture(t, svc, ProtocolHTTP)
+	putModifyLB(t, svc, "alb-same", LoadBalancerTypeApplication, ProtocolHTTP)
+	before := getLB(t, svc, "alb-same")
+
+	modifyPath(t, svc, "/old")
+
+	assert.Equal(t, before.ConfigHash, getLB(t, svc, "alb-same").ConfigHash)
+}

spinifex/domains/elbv2/nginx.go

@@ -148,10 +148,7 @@ func generateNLBStreamConfig(lb *LoadBalancerRecord, listeners []*ListenerRecord
 		}
 		us := nginxUpstream{Name: name}
 		if tg, ok := tgByArn[tgArn]; ok {
-			for _, t := range tg.Targets {
-				if t.PrivateIP == "" || t.HealthState == TargetHealthDraining {
-					continue
-				}
+			for _, t := range routableTargets(tg.Targets) {
 				us.Servers = append(us.Servers, nginxUpstreamServer{
 					Addr:        t.PrivateIP,
 					Port:        t.Port,
@@ -329,6 +326,7 @@ func buildNLBHealthTargets(tgByArn map[string]*TargetGroupRecord) []HealthTarget
 	var out []HealthTargetSpec
 	for _, tg := range tgByArn {
 		for _, t := range tg.Targets {
+			// Unhealthy targets stay probed so they can recover.
 			if t.PrivateIP == "" || t.HealthState == TargetHealthDraining {
 				continue
 			}

spinifex/domains/elbv2/nginx_test.go

@@ -307,3 +307,48 @@ func TestCertDNSNames_DropsNamesThatAreNotHostnames(t *testing.T) {
 	assert.Equal(t, []string{"ok.example.com", "upper.example.com"}, certDNSNames(pemText))
 	assert.Nil(t, certDNSNames("not a pem"), "unparseable material yields no names")
 }
+
+func TestGenerateNLBStream_UnhealthyTargetLeavesUpstreamButStaysProbed(t *testing.T) {
+	t.Parallel()
+	lb := &LoadBalancerRecord{LoadBalancerID: "lb-unh", Type: LoadBalancerTypeNetwork}
+	tgArn := "arn:aws:elasticloadbalancing:us-east-1:1:targetgroup/db/tg-db"
+	listeners := []*ListenerRecord{tcpListener("arn:lst", "lst", ProtocolTCP, 5432, tgArn)}
+	tgByArn := map[string]*TargetGroupRecord{
+		tgArn: nlbTG(tgArn,
+			Target{Id: "i-ok", Port: 5432, PrivateIP: "10.0.1.10", HealthState: TargetHealthHealthy},
+			Target{Id: "i-bad", Port: 5432, PrivateIP: "10.0.1.11", HealthState: TargetHealthUnhealthy},
+		),
+	}
+
+	config, _, err := GenerateHAProxyConfigWithCerts(lb, listeners, tgByArn, nil, "10.0.1.5", nil)
+	require.NoError(t, err)
+	assert.Contains(t, config, "server 10.0.1.10:5432 ")
+	assert.NotContains(t, config, "10.0.1.11")
+
+	probed := buildNLBHealthTargets(tgByArn)
+	var addrs []string
+	for _, p := range probed {
+		addrs = append(addrs, p.Address)
+	}
+	assert.ElementsMatch(t, []string{"10.0.1.10:5432", "10.0.1.11:5432"}, addrs)
+}
+
+func TestGenerateNLBStream_AllUnhealthyFailsOpen(t *testing.T) {
+	t.Parallel()
+	lb := &LoadBalancerRecord{LoadBalancerID: "lb-allunh", Type: LoadBalancerTypeNetwork}
+	tgArn := "arn:aws:elasticloadbalancing:us-east-1:1:targetgroup/db/tg-db"
+	listeners := []*ListenerRecord{tcpListener("arn:lst", "lst", ProtocolTCP, 5432, tgArn)}
+	tgByArn := map[string]*TargetGroupRecord{
+		tgArn: nlbTG(tgArn,
+			Target{Id: "i-1", Port: 5432, PrivateIP: "10.0.1.10", HealthState: TargetHealthUnhealthy},
+			Target{Id: "i-2", Port: 5432, PrivateIP: "10.0.1.11", HealthState: TargetHealthUnhealthy},
+			Target{Id: "i-drain", Port: 5432, PrivateIP: "10.0.1.12", HealthState: TargetHealthDraining},
+		),
+	}
+
+	config, _, err := GenerateHAProxyConfigWithCerts(lb, listeners, tgByArn, nil, "10.0.1.5", nil)
+	require.NoError(t, err)
+	assert.Contains(t, config, "server 10.0.1.10:5432 ")
+	assert.Contains(t, config, "server 10.0.1.11:5432 ")
+	assert.NotContains(t, config, "10.0.1.12")
+}

spinifex/domains/elbv2/routable_targets.go

@@ -0,0 +1,21 @@
+package elbv2
+
+// routableTargets returns the targets a load balancer may send traffic to.
+// Draining, unhealthy and IP-less targets are excluded; when that leaves none,
+// every non-draining target with an IP is returned (fail open, as AWS does).
+func routableTargets(targets []Target) []Target {
+	var healthy, eligible []Target
+	for _, t := range targets {
+		if t.PrivateIP == "" || t.HealthState == TargetHealthDraining {
+			continue
+		}
+		eligible = append(eligible, t)
+		if t.HealthState != TargetHealthUnhealthy {
+			healthy = append(healthy, t)
+		}
+	}
+	if len(healthy) > 0 {
+		return healthy
+	}
+	return eligible
+}

spinifex/domains/elbv2/routable_targets_test.go

@@ -0,0 +1,52 @@
+package elbv2
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestRoutableTargets(t *testing.T) {
+	tgt := func(id, ip, state string) Target {
+		return Target{Id: id, PrivateIP: ip, HealthState: state}
+	}
+	ids := func(ts []Target) []string {
+		var out []string
+		for _, x := range ts {
+			out = append(out, x.Id)
+		}
+		return out
+	}
+
+	tests := []struct {
+		name    string
+		targets []Target
+		want    []string
+	}{
+		{"all healthy", []Target{tgt("a", "10.0.0.1", TargetHealthHealthy), tgt("b", "10.0.0.2", TargetHealthHealthy)}, []string{"a", "b"}},
+		{"one unhealthy is dropped", []Target{tgt("a", "10.0.0.1", TargetHealthHealthy), tgt("b", "10.0.0.2", TargetHealthUnhealthy)}, []string{"a"}},
+		{"initial stays routable", []Target{tgt("a", "10.0.0.1", TargetHealthInitial), tgt("b", "10.0.0.2", TargetHealthUnhealthy)}, []string{"a"}},
+		{"draining is dropped", []Target{tgt("a", "10.0.0.1", TargetHealthHealthy), tgt("b", "10.0.0.2", TargetHealthDraining)}, []string{"a"}},
+		{"no ip is dropped", []Target{tgt("a", "10.0.0.1", TargetHealthHealthy), tgt("b", "", TargetHealthHealthy)}, []string{"a"}},
+		{"mixed", []Target{
+			tgt("a", "10.0.0.1", TargetHealthHealthy),
+			tgt("b", "10.0.0.2", TargetHealthUnhealthy),
+			tgt("c", "10.0.0.3", TargetHealthInitial),
+			tgt("d", "10.0.0.4", TargetHealthDraining),
+			tgt("e", "", TargetHealthHealthy),
+		}, []string{"a", "c"}},
+		{"all unhealthy fails open but keeps draining and no-ip out", []Target{
+			tgt("a", "10.0.0.1", TargetHealthUnhealthy),
+			tgt("b", "10.0.0.2", TargetHealthUnhealthy),
+			tgt("c", "10.0.0.3", TargetHealthDraining),
+			tgt("d", "", TargetHealthUnhealthy),
+		}, []string{"a", "b"}},
+		{"all draining is empty", []Target{tgt("a", "10.0.0.1", TargetHealthDraining)}, nil},
+		{"empty group", nil, nil},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			assert.Equal(t, tc.want, ids(routableTargets(tc.targets)))
+		})
+	}
+}

spinifex/domains/elbv2/service_impl.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"crypto/sha256"
 	"encoding/hex"
+	"encoding/json"
 	"errors"
 	"fmt"
 	"log/slog"
@@ -183,7 +184,7 @@ func NewELBv2ServiceImplWithNATS(cfg *config.Config, nc *nats.Conn, masterKey []
 
 	hc := newHealthChecker(store)
 
-	return &ELBv2ServiceImpl{
+	svc := &ELBv2ServiceImpl{
 		config:         cfg,
 		store:          store,
 		acmStore:       acmStore,
@@ -195,7 +196,9 @@ func NewELBv2ServiceImplWithNATS(cfg *config.Config, nc *nats.Conn, masterKey []
 		cancel:         cancel,
 		hc:             hc,
 		reaperInterval: lbReaperInterval,
-	}, nil
+	}
+	hc.onChanged = svc.reRenderForTargetGroups
+	return svc, nil
 }
 
 // Close cancels background goroutines.
@@ -715,18 +718,35 @@ func (s *ELBv2ServiceImpl) updateStoredConfig(ctx context.Context, lb *LoadBalan
 		return fmt.Errorf("generate config: %w", err)
 	}
 
-	hash := configCertHash(configContent, certFiles)
+	// NLBs: the agent actively probes targets (nginx has no upstream probing).
+	var healthTargets []HealthTargetSpec
+	if lb.Type == LoadBalancerTypeNetwork {
+		healthTargets = buildNLBHealthTargets(tgByArn)
+	}
+	// The agent re-fetches only when the hash moves, so probe settings are part of it.
+	hashed := configContent
+	if len(healthTargets) > 0 {
+		probes, err := json.Marshal(healthTargets)
+		if err != nil {
+			return fmt.Errorf("marshal health targets: %w", err)
+		}
+		hashed += "\n# probes " + string(probes)
+	}
+	hash := configCertHash(hashed, certFiles)
 	lb.ConfigText = configContent
 	lb.ConfigHash = hash
 	lb.CertFiles = certFiles
-	// NLBs: the agent actively probes targets (nginx has no upstream probing).
-	if lb.Type == LoadBalancerTypeNetwork {
-		lb.HealthTargets = buildNLBHealthTargets(tgByArn)
-	} else {
-		lb.HealthTargets = nil
-	}
+	lb.HealthTargets = healthTargets
 
-	if err := s.store.PutLoadBalancer(ctx, lb); err != nil {
+	// Only the rendered fields are written, under CAS, so a concurrent heartbeat
+	// or re-render of the same LB is never overwritten with a stale record.
+	if _, err := s.store.UpdateLoadBalancer(ctx, lb.LoadBalancerID, func(rec *LoadBalancerRecord) (bool, error) {
+		rec.ConfigText = lb.ConfigText
+		rec.ConfigHash = lb.ConfigHash
+		rec.CertFiles = lb.CertFiles
+		rec.HealthTargets = lb.HealthTargets
+		return true, nil
+	}); err != nil {
 		slog.ErrorContext(ctx, "updateStoredConfig: failed to persist LB", "lbId", lb.LoadBalancerID, "err", err)
 		return fmt.Errorf("persist LB: %w", err)
 	}
@@ -736,6 +756,46 @@ func (s *ELBv2ServiceImpl) updateStoredConfig(ctx context.Context, lb *LoadBalan
 	return nil
 }
 
+// reRenderForTargetGroups regenerates the stored config of every load balancer
+// that references one of tgIDs, so a target health change reaches the data plane.
+// The LB is re-read just before rendering and failures are logged, never returned.
+func (s *ELBv2ServiceImpl) reRenderForTargetGroups(ctx context.Context, tgIDs []string) {
+	changed := make(map[string]struct{}, len(tgIDs))
+	for _, id := range tgIDs {
+		changed[id] = struct{}{}
+	}
+	lbs, err := s.store.ListLoadBalancers(ctx)
+	if err != nil {
+		slog.ErrorContext(ctx, "reRenderForTargetGroups: failed to list load balancers", "err", err)
+		return
+	}
+	for _, listed := range lbs {
+		tgs, err := s.store.TargetGroupsForLB(ctx, listed.LoadBalancerID)
+		if err != nil {
+			slog.ErrorContext(ctx, "reRenderForTargetGroups: failed to resolve target groups", "lbId", listed.LoadBalancerID, "err", err)
+			continue
+		}
+		uses := false
+		for _, tg := range tgs {
+			if _, ok := changed[tg.TargetGroupID]; ok {
+				uses = true
+				break
+			}
+		}
+		if !uses {
+			continue
+		}
+		lb, err := s.store.GetLoadBalancer(ctx, listed.LoadBalancerID)
+		if err != nil || lb == nil {
+			slog.ErrorContext(ctx, "reRenderForTargetGroups: failed to re-read load balancer", "lbId", listed.LoadBalancerID, "err", err)
+			continue
+		}
+		if err := s.updateStoredConfig(ctx, lb); err != nil {
+			slog.ErrorContext(ctx, "reRenderForTargetGroups: re-render failed", "lbId", lb.LoadBalancerID, "err", err)
+		}
+	}
+}
+
 // resolveListenerCerts resolves each distinct certificate ARN to its combined PEM.
 // Returns nil for HTTP-only listeners.
 func (s *ELBv2ServiceImpl) resolveListenerCerts(ctx context.Context, listeners []*ListenerRecord, accountID string) (map[string]string, error) {
@@ -1068,8 +1128,21 @@ func (s *ELBv2ServiceImpl) LBAgentHeartbeat(ctx context.Context, input *LBAgentH
 	heartbeatStale := now.Sub(lb.LastHeartbeat) >= heartbeatPersistInterval
 	lb.LastHeartbeat = now
 
+	// Both writes touch only the fields this heartbeat owns, under CAS, so a
+	// concurrent config re-render of the same LB is never rolled back.
 	switch {
 	case stateChanged:
+		if _, err := s.store.UpdateLoadBalancer(ctx, lbID, func(rec *LoadBalancerRecord) (bool, error) {
+			if rec.State == StateProvisioning {
+				rec.State = StateActive
+				rec.StateReason = ""
+			}
+			rec.LastHeartbeat = now
+			return true, nil
+		}); err != nil {
+			slog.ErrorContext(ctx, "LBAgentHeartbeat: failed to persist activation", "lbId", lbID, "err", err)
+			return nil, errors.New(awserrors.ErrorServerInternal)
+		}
 		// Build data-plane config on activation: reactive calls during create
 		// no-op while InstanceID is empty, so we build it now from existing listeners/targets.
 		if err := s.updateStoredConfig(ctx, lb); err != nil {
@@ -1078,7 +1151,10 @@ func (s *ELBv2ServiceImpl) LBAgentHeartbeat(ctx context.Context, input *LBAgentH
 		}
 	case heartbeatStale:
 		// Refresh the heartbeat timestamp only; avoid writing the full record on every tick.
-		if err := s.store.PutLoadBalancer(ctx, lb); err != nil {
+		if _, err := s.store.UpdateLoadBalancer(ctx, lbID, func(rec *LoadBalancerRecord) (bool, error) {
+			rec.LastHeartbeat = now
+			return true, nil
+		}); err != nil {
 			slog.ErrorContext(ctx, "LBAgentHeartbeat: failed to persist LB", "lbId", lbID, "err", err)
 			return nil, errors.New(awserrors.ErrorServerInternal)
 		}
@@ -2095,6 +2171,8 @@ func (s *ELBv2ServiceImpl) ModifyTargetGroup(ctx context.Context, input *elbv2.M
 		return nil, err
 	}
 
+	s.reRenderForTargetGroups(ctx, []string{tg.TargetGroupID})
+
 	slog.InfoContext(ctx, "ModifyTargetGroup completed", "arn", tg.TargetGroupArn, "accountID", accountID)
 
 	return &elbv2.ModifyTargetGroupOutput{

spinifex/domains/elbv2/service_impl_attributes.go

@@ -190,8 +190,15 @@ func (s *ELBv2ServiceImpl) lbAttrResource(ctx context.Context, arn, accountID, o
 		opName:    opName,
 		notFound:  awserrors.ErrorELBv2LoadBalancerNotFound,
 		// See tgAttrResource: the generic helpers carry no context.
-		fetch:       func(arn string) (*LoadBalancerRecord, error) { return s.store.GetLoadBalancerByArn(ctx, arn) },
-		save:        func(lb *LoadBalancerRecord) error { return s.store.PutLoadBalancer(ctx, lb) },
+		fetch: func(arn string) (*LoadBalancerRecord, error) { return s.store.GetLoadBalancerByArn(ctx, arn) },
+		// CAS the attribute field alone so a concurrent config re-render survives.
+		save: func(lb *LoadBalancerRecord) error {
+			_, err := s.store.UpdateLoadBalancer(ctx, lb.LoadBalancerID, func(rec *LoadBalancerRecord) (bool, error) {
+				rec.Attributes = lb.Attributes
+				return true, nil
+			})
+			return err
+		},
 		accountIDOf: func(r *LoadBalancerRecord) string { return r.AccountID },
 		defaults:    func(r *LoadBalancerRecord) map[string]string { return DefaultLoadBalancerAttributes(r.Type) },
 		attrsOf:     func(r *LoadBalancerRecord) map[string]string { return r.Attributes },

spinifex/domains/elbv2/service_impl_lb_network.go

@@ -35,7 +35,10 @@ func (s *ELBv2ServiceImpl) SetIpAddressType(ctx context.Context, input *elbv2.Se
 
 	if lb.IPAddressType != IPAddressTypeIPv4 {
 		lb.IPAddressType = IPAddressTypeIPv4
-		if err := s.store.PutLoadBalancer(ctx, lb); err != nil {
+		if _, err := s.store.UpdateLoadBalancer(ctx, lb.LoadBalancerID, func(rec *LoadBalancerRecord) (bool, error) {
+			rec.IPAddressType = lb.IPAddressType
+			return true, nil
+		}); err != nil {
 			slog.ErrorContext(ctx, "SetIpAddressType: failed to persist LB", "arn", *input.LoadBalancerArn, "err", err)
 			return nil, errors.New(awserrors.ErrorServerInternal)
 		}
@@ -98,7 +101,10 @@ func (s *ELBv2ServiceImpl) SetSecurityGroups(ctx context.Context, input *elbv2.S
 	}
 
 	lb.SecurityGroups = sgs
-	if err := s.store.PutLoadBalancer(ctx, lb); err != nil {
+	if _, err := s.store.UpdateLoadBalancer(ctx, lb.LoadBalancerID, func(rec *LoadBalancerRecord) (bool, error) {
+		rec.SecurityGroups = sgs
+		return true, nil
+	}); err != nil {
 		slog.ErrorContext(ctx, "SetSecurityGroups: failed to persist LB", "arn", *input.LoadBalancerArn, "err", err)
 		return nil, errors.New(awserrors.ErrorServerInternal)
 	}
@@ -155,7 +161,11 @@ func (s *ELBv2ServiceImpl) SetSubnets(ctx context.Context, input *elbv2.SetSubne
 	if s.VPCService == nil {
 		lb.Subnets = desired
 		lb.AvailZones = rebuildAvailZones(desired, lb.AvailZones, nil)
-		if err := s.store.PutLoadBalancer(ctx, lb); err != nil {
+		if _, err := s.store.UpdateLoadBalancer(ctx, lb.LoadBalancerID, func(rec *LoadBalancerRecord) (bool, error) {
+			rec.Subnets = lb.Subnets
+			rec.AvailZones = lb.AvailZones
+			return true, nil
+		}); err != nil {
 			slog.ErrorContext(ctx, "SetSubnets: failed to persist LB", "arn", *input.LoadBalancerArn, "err", err)
 			return nil, errors.New(awserrors.ErrorServerInternal)
 		}
@@ -244,7 +254,17 @@ func (s *ELBv2ServiceImpl) SetSubnets(ctx context.Context, input *elbv2.SetSubne
 	lb.HostPorts = launch.hostPorts
 	lb.State, lb.StateReason = s.lbStateAfterLaunch(launch, lb.Scheme)
 
-	if err := s.store.PutLoadBalancer(ctx, lb); err != nil {
+	if _, err := s.store.UpdateLoadBalancer(ctx, lb.LoadBalancerID, func(rec *LoadBalancerRecord) (bool, error) {
+		rec.Subnets = lb.Subnets
+		rec.ENIs = lb.ENIs
+		rec.AvailZones = lb.AvailZones
+		rec.InstanceID = lb.InstanceID
+		rec.VPCIP = lb.VPCIP
+		rec.HostPorts = lb.HostPorts
+		rec.State = lb.State
+		rec.StateReason = lb.StateReason
+		return true, nil
+	}); err != nil {
 		slog.ErrorContext(ctx, "SetSubnets: failed to persist LB", "arn", *input.LoadBalancerArn, "err", err)
 		return nil, errors.New(awserrors.ErrorServerInternal)
 	}

spinifex/domains/elbv2/service_impl_nlb_sg.go

@@ -245,7 +245,10 @@ func (s *ELBv2ServiceImpl) SetLoadBalancerIngressCIDRs(lbArn string, cidrs []str
 		}
 	}
 
-	if err := s.store.PutLoadBalancer(s.ctx, lb); err != nil {
+	if _, err := s.store.UpdateLoadBalancer(s.ctx, lb.LoadBalancerID, func(rec *LoadBalancerRecord) (bool, error) {
+		rec.NLBIngressCIDRs = lb.NLBIngressCIDRs
+		return true, nil
+	}); err != nil {
 		slog.Error("SetLoadBalancerIngressCIDRs: persist failed", "arn", lbArn, "err", err)
 		return errors.New(awserrors.ErrorServerInternal)
 	}

spinifex/domains/elbv2/service_impl_tags.go

@@ -39,8 +39,12 @@ func (s *ELBv2ServiceImpl) resolveTaggable(ctx context.Context, arn string) (h t
 		if lb != nil {
 			found = true
 			h = taggable{tags: lb.Tags, owner: lb.AccountID, save: func(t map[string]string) error {
-				lb.Tags = t
-				return s.store.PutLoadBalancer(ctx, lb)
+				// CAS the tag field alone so a concurrent config re-render survives.
+				_, uerr := s.store.UpdateLoadBalancer(ctx, lb.LoadBalancerID, func(rec *LoadBalancerRecord) (bool, error) {
+					rec.Tags = t
+					return true, nil
+				})
+				return uerr
 			}}
 		}
 	case resourcearn.ELBv2TargetGroup:

spinifex/domains/elbv2/store.go

@@ -181,6 +181,23 @@ func (s *Store) PutLoadBalancer(ctx context.Context, lb *LoadBalancerRecord) err
 	return err
 }
 
+// ErrLoadBalancerNotFound is what a load-balancer CAS update reports when the
+// record is absent, so a deleted LB is never resurrected by a late writer.
+var ErrLoadBalancerNotFound = errors.New("elbv2: load balancer not found")
+
+// UpdateLoadBalancer applies mutate to one load balancer under optimistic
+// concurrency: a writer that loses the race re-reads and retries instead of
+// overwriting the winner. mutate returns false to commit nothing.
+func (s *Store) UpdateLoadBalancer(ctx context.Context, lbID string, mutate func(*LoadBalancerRecord) (bool, error)) (*LoadBalancerRecord, error) {
+	return kvutil.Update(ctx, s.kv, KeyPrefixLB+lbID, kvutil.CASConfig{
+		Attempts: targetGroupCASAttempts,
+		NotFound: ErrLoadBalancerNotFound,
+		Exhausted: func(_ string, attempts int) error {
+			return fmt.Errorf("elbv2: load balancer %s contended, CAS exhausted after %d attempts", lbID, attempts)
+		},
+	}, mutate)
+}
+
 // GetLoadBalancer retrieves a load balancer by its short ID.
 func (s *Store) GetLoadBalancer(ctx context.Context, lbID string) (*LoadBalancerRecord, error) {
 	entry, err := s.kv.Get(ctx, KeyPrefixLB+lbID)

spinifex/gateway/ecr_authorization_test.go

@@ -83,6 +83,7 @@ func TestECROperationAuthorization_PullOnlyMatrix(t *testing.T) {
 		{http.MethodGet, "/v2/team/app/blobs/sha256:abc"},
 		{http.MethodHead, "/v2/team/app/manifests/latest"},
 		{http.MethodGet, "/v2/team/app/manifests/latest"},
+		{http.MethodGet, "/v2/team/app/referrers/sha256:abc"},
 	}
 	for _, c := range allowed {
 		t.Run("allow "+c.method+" "+c.path, func(t *testing.T) {
@@ -133,6 +134,7 @@ func TestECROperationAuthorization_ReadOnlyMatrix(t *testing.T) {
 		{http.MethodGet, "/v2/team/app/tags/list"},
 		{http.MethodHead, "/v2/team/app/blobs/sha256:abc"},
 		{http.MethodGet, "/v2/team/app/manifests/latest"},
+		{http.MethodGet, "/v2/team/app/referrers/sha256:abc"},
 	}
 	for _, c := range allowed {
 		t.Run("allow "+c.method+" "+c.path, func(t *testing.T) {
@@ -183,6 +185,39 @@ func TestECROperationAuthorization_RepoScopedAllowDoesNotCrossRepos(t *testing.T
 		assert.False(t, *called)
 		assert.Equal(t, http.StatusForbidden, w.Code)
 	})
+
+	t.Run("repo A referrers allowed", func(t *testing.T) {
+		next, called := okHandler()
+		w := httptest.NewRecorder()
+		gw.ecrOperationAuthorization(next).ServeHTTP(w, ecrAuthzRequest(http.MethodGet, "/v2/team/a/referrers/sha256:abc", ecrTestUserPrincipal()))
+		assert.True(t, *called)
+		assert.Equal(t, http.StatusOK, w.Code)
+	})
+
+	t.Run("repo B referrers denied", func(t *testing.T) {
+		next, called := okHandler()
+		w := httptest.NewRecorder()
+		gw.ecrOperationAuthorization(next).ServeHTTP(w, ecrAuthzRequest(http.MethodGet, "/v2/team/b/referrers/sha256:abc", ecrTestUserPrincipal()))
+		assert.False(t, *called)
+		assert.Equal(t, http.StatusForbidden, w.Code)
+	})
+}
+
+// TestECROperationAuthorization_ReferrersNeedBatchGetImage verifies the
+// referrers listing is gated on BatchGetImage and on nothing a push grants.
+func TestECROperationAuthorization_ReferrersNeedBatchGetImage(t *testing.T) {
+	repoARN := "arn:aws:ecr:" + ecrTestRegion + ":" + ecrAuthzTestAccount + ":repository/team/app"
+	pushOnly := ecrAllowPolicy(
+		[]string{"ecr:PutImage", "ecr:InitiateLayerUpload", "ecr:ListImages", "ecr:BatchCheckLayerAvailability"},
+		[]string{repoARN},
+	)
+	gw, _ := ecrAuthzTestGateway(t, []iamdomain.PolicyDocument{pushOnly})
+
+	next, called := okHandler()
+	w := httptest.NewRecorder()
+	gw.ecrOperationAuthorization(next).ServeHTTP(w, ecrAuthzRequest(http.MethodGet, "/v2/team/app/referrers/sha256:abc", ecrTestUserPrincipal()))
+	assert.False(t, *called, "a caller without BatchGetImage must not reach the handler")
+	assert.Equal(t, http.StatusForbidden, w.Code)
 }
 
 // TestECROperationAuthorization_ExplicitDenyWins verifies an explicit Deny
@@ -198,6 +233,12 @@ func TestECROperationAuthorization_ExplicitDenyWins(t *testing.T) {
 	gw.ecrOperationAuthorization(next).ServeHTTP(w, ecrAuthzRequest(http.MethodGet, "/v2/team/app/manifests/latest", ecrTestUserPrincipal()))
 	assert.False(t, *called)
 	assert.Equal(t, http.StatusForbidden, w.Code)
+
+	next, called = okHandler()
+	w = httptest.NewRecorder()
+	gw.ecrOperationAuthorization(next).ServeHTTP(w, ecrAuthzRequest(http.MethodGet, "/v2/team/app/referrers/sha256:abc", ecrTestUserPrincipal()))
+	assert.False(t, *called)
+	assert.Equal(t, http.StatusForbidden, w.Code)
 }
 
 // TestECROperationAuthorization_CrossRepoMount verifies a mount requires both

spinifex/runtime/compute/vm/devices.go

@@ -297,6 +297,18 @@ func VolumeBlkDevice(volumeID, nodeName, iothreadID, bus string) Device {
 	return Device{Value: strings.Join(args, ",")}
 }
 
+// volumeMMIOBlkDevice renders a microvm data volume as virtio-blk-device with
+// VolumeBlkDeviceArgs' identity and error policy; MMIO has no PCI bus to name.
+func volumeMMIOBlkDevice(volumeID, nodeName, iothreadID string) Device {
+	args := []string{"virtio-blk-device"}
+	for _, arg := range VolumeBlkDeviceArgs(volumeID, nodeName, iothreadID, "") {
+		if !strings.HasPrefix(arg, "bus=") {
+			args = append(args, arg)
+		}
+	}
+	return Device{Value: strings.Join(args, ",")}
+}
+
 // VolumeBlkDeviceQMPArgs renders the same virtio-blk-pci arguments as a
 // device_add argument map for AttachVolume's QMP hot-attach path. See
 // VolumeBlkDeviceArgs for why werror/rerror are set here.

spinifex/runtime/compute/vm/devices_test.go

@@ -177,6 +177,19 @@ func TestVolumeBlkDevice_OnErrorStop(t *testing.T) {
 	assert.Equal(t, "virtio-blk-pci,id=vdisk-vol-data-a,drive=nbd-vol-data-a,iothread=ioth-vol-data-a,serial=voldataa,bus=hotplug-ebs3,werror=stop,rerror=stop", d.Value)
 }
 
+// TestVolumeMMIOBlkDevice pins the microvm data-volume device: the PCI
+// device's identity, block node, I/O thread and stop-on-error policy on
+// virtio-blk-device, with no PCI bus, which MMIO has no root port for.
+func TestVolumeMMIOBlkDevice(t *testing.T) {
+	d := volumeMMIOBlkDevice("vol-data-a", "nbd-vol-data-a", "ioth-vol-data-a")
+	assert.Equal(t, "virtio-blk-device,id=vdisk-vol-data-a,drive=nbd-vol-data-a,iothread=ioth-vol-data-a,serial=voldataa,werror=stop,rerror=stop", d.Value)
+	assert.NotContains(t, d.Value, "bus=")
+
+	pci := VolumeBlkDevice("vol-data-a", "nbd-vol-data-a", "ioth-vol-data-a", "hotplug-ebs3")
+	assert.Equal(t, "virtio-blk-pci,id=vdisk-vol-data-a,drive=nbd-vol-data-a,iothread=ioth-vol-data-a,serial=voldataa,bus=hotplug-ebs3,werror=stop,rerror=stop", pci.Value,
+		"the PCI device keeps its bus and shape")
+}
+
 // TestVolumeBlkDeviceQMPArgs_OnErrorStop is the hotplug counterpart: the QMP
 // device_add argument map AttachVolume sends must carry the same policy.
 func TestVolumeBlkDeviceQMPArgs_OnErrorStop(t *testing.T) {

spinifex/runtime/compute/vm/lifecycle.go

@@ -401,6 +401,9 @@ const (
 	qemuStartupTimeout = 5 * time.Second
 )
 
+// waitForNBDReady is a test seam over nbd.WaitForNBDReady.
+var waitForNBDReady = nbd.WaitForNBDReady
+
 // startQEMU launches the QEMU process for instance and waits for startup to
 // confirm.
 func (m *Manager) startQEMU(instance *VM) error {
@@ -607,13 +610,29 @@ func (m *Manager) startQEMU(instance *VM) error {
 		refreshTelemetryMeta(instance)
 	}
 
+	// One snapshot feeds both the readiness probes and the direct-boot command,
+	// so a concurrent remount cannot launch an endpoint that was never checked.
+	instance.EBSRequests.Mu.Lock()
+	requests := slices.Clone(instance.EBSRequests.Requests)
+	instance.EBSRequests.Mu.Unlock()
+
+	// A direct-boot Config is the launcher's saved base, so mounted data volumes
+	// go on a launch-only copy and their current endpoints are never saved.
+	launchCfg := &instance.Config
+	if instance.DirectBoot && len(requests) > 0 {
+		composed, err := directBootLaunchConfig(instance.Config, requests, instance.Config.CPUCount)
+		if err != nil {
+			return err
+		}
+		launchCfg = &composed
+	}
+
 	// An NBD export is waited on until it accepts connections; a block export
 	// is local to the host and only has to exist. An invalid request has no
 	// endpoint to probe (buildDrives rejects it on the command-line path).
-	instance.EBSRequests.Mu.Lock()
-	nbdEndpoints := make([]struct{ name, uri string }, 0, len(instance.EBSRequests.Requests))
+	nbdEndpoints := make([]struct{ name, uri string }, 0, len(requests))
 	var blockDevices []struct{ name, path string }
-	for _, req := range instance.EBSRequests.Requests {
+	for _, req := range requests {
 		switch kind, _ := classifyVolumeExport(req); kind {
 		case volumeExportNBD:
 			nbdEndpoints = append(nbdEndpoints, struct{ name, uri string }{req.Name, req.NBDURI})
@@ -621,9 +640,8 @@ func (m *Manager) startQEMU(instance *VM) error {
 			blockDevices = append(blockDevices, struct{ name, path string }{req.Name, req.BlockDevice})
 		}
 	}
-	instance.EBSRequests.Mu.Unlock()
 	for _, ep := range nbdEndpoints {
-		if err := nbd.WaitForNBDReady(ep.uri, nbdReadyTimeout); err != nil {
+		if err := waitForNBDReady(ep.uri, nbdReadyTimeout); err != nil {
 			return fmt.Errorf("nbd endpoint not ready for %s: %w", ep.name, err)
 		}
 	}
@@ -638,7 +656,7 @@ func (m *Manager) startQEMU(instance *VM) error {
 	startupConfirmed := make(chan bool, 1)
 
 	go func() {
-		cmd, err := instance.Config.Execute()
+		cmd, err := launchCfg.Execute()
 		if err != nil {
 			slog.Error("Failed to execute VM", "err", err)
 			processChan <- 0
@@ -1456,6 +1474,32 @@ type driveConfig struct {
 	Blockdevs []Blockdev
 }
 
+// directBootLaunchConfig returns a launch-only copy of a clean direct-boot base
+// with the mounted data volumes appended after its devices. Neither input is
+// modified, so composing again from the same base never accumulates entries.
+func directBootLaunchConfig(base Config, requests []EBSRequest, cpuCount int) (Config, error) {
+	if !IsMMIO(base.MachineType) {
+		return Config{}, fmt.Errorf("direct-boot launch config needs a microvm base, got machine type %q", base.MachineType)
+	}
+	for _, r := range requests {
+		if r.Boot || r.EFI {
+			return Config{}, fmt.Errorf("volume %s: a direct-boot microvm takes data volumes only", r.Name)
+		}
+	}
+	drives, err := buildDrives(slices.Clone(requests), cpuCount, base.MachineType)
+	if err != nil {
+		return Config{}, err
+	}
+	cfg := base
+	cfg.Drives = slices.Clone(base.Drives)
+	cfg.NetDevs = slices.Clone(base.NetDevs)
+	cfg.FwCfg = slices.Clone(base.FwCfg)
+	cfg.IOThreads = append(slices.Clone(base.IOThreads), drives.IOThreads...)
+	cfg.Blockdevs = append(slices.Clone(base.Blockdevs), drives.Blockdevs...)
+	cfg.Devices = append(slices.Clone(base.Devices), drives.Devices...)
+	return cfg, nil
+}
+
 // buildDrives converts EBS requests into QEMU drive/iothread/device/blockdev
 // configs. Returns an error if any volume has no valid export (see
 // classifyVolumeExport), or if the EBS hot-plug port pool is exhausted while
@@ -1535,12 +1579,14 @@ func buildDrives(requests []EBSRequest, cpuCount int, machineType string) (drive
 			})
 
 		default:
+			// microvm has no PCIe root ports, so its data volumes take none.
+			mmio := IsMMIO(machineType)
 			// A data (non-boot, non-EFI) volume. Allocate a stable hot-plug
 			// port when the persisted state predates port accounting, or the
 			// volume was attached at RunInstances time without one;
 			// freeHotplugEBSPort re-scans requests as mutated so far in this
 			// loop, so two unset data volumes in one call cannot collide.
-			if v.HotplugPort == 0 {
+			if v.HotplugPort == 0 && !mmio {
 				port := freeHotplugEBSPort(requests)
 				if port == 0 {
 					return driveConfig{}, fmt.Errorf("no free EBS hot-plug port for volume %s", v.Name)
@@ -1550,7 +1596,6 @@ func buildDrives(requests []EBSRequest, cpuCount int, machineType string) (drive
 
 			nodeName := VolumeNodeName(v.Name)
 			iothreadID := VolumeIOThreadID(v.Name)
-			bus := HotplugEBSBus(v.HotplugPort)
 
 			var blockdev Blockdev
 			if kind == volumeExportBlock {
@@ -1570,7 +1615,11 @@ func buildDrives(requests []EBSRequest, cpuCount int, machineType string) (drive
 
 			cfg.IOThreads = append(cfg.IOThreads, IOThread{ID: iothreadID})
 			cfg.Blockdevs = append(cfg.Blockdevs, blockdev)
-			cfg.Devices = append(cfg.Devices, VolumeBlkDevice(v.Name, nodeName, iothreadID, bus))
+			if mmio {
+				cfg.Devices = append(cfg.Devices, volumeMMIOBlkDevice(v.Name, nodeName, iothreadID))
+			} else {
+				cfg.Devices = append(cfg.Devices, VolumeBlkDevice(v.Name, nodeName, iothreadID, HotplugEBSBus(v.HotplugPort)))
+			}
 			// Deliberately no entry in cfg.Drives: the bare anonymous -drive
 			// this case used to emit (no id=, no node-name=) is exactly what
 			// left the volume undetachable.

spinifex/runtime/compute/vm/lifecycle_test.go

@@ -8,6 +8,8 @@ import (
 	"net"
 	"os"
 	"path/filepath"
+	"slices"
+	"strings"
 	"sync"
 	"sync/atomic"
 	"syscall"
@@ -289,6 +291,57 @@ func TestBuildDrives(t *testing.T) {
 			},
 			wantHotplugPorts: []int{2},
 		},
+		{
+			// microvm has no PCIe root ports, so an unset port stays unset
+			// rather than naming a bus the machine does not have.
+			name: "microvm NBD data volume renders an MMIO device and allocates no hot-plug port",
+			requests: []EBSRequest{
+				{Name: "vol-data-a", NBDURI: "nbd:unix:/tmp/data-a.sock"},
+			},
+			cpuCount:      2,
+			machineType:   "microvm,x-option-roms=off,pic=off",
+			wantIOThreads: []IOThread{{ID: "ioth-vol-data-a"}},
+			wantBlockdevs: []Blockdev{
+				{Value: "driver=nbd,node-name=nbd-vol-data-a,server.type=unix,server.path=/tmp/data-a.sock,export=,reconnect-delay=30"},
+			},
+			wantDevices: []Device{
+				{Value: "virtio-blk-device,id=vdisk-vol-data-a,drive=nbd-vol-data-a,iothread=ioth-vol-data-a,serial=voldataa,werror=stop,rerror=stop"},
+			},
+			wantHotplugPorts: []int{0},
+		},
+		{
+			name: "microvm block data volume keeps its recorded port and host_device node on an MMIO device",
+			requests: []EBSRequest{
+				{Name: "vol-blk", BlockDevice: "/dev/sdx", BlockSizeBytes: 10 << 30, HotplugPort: 2},
+			},
+			cpuCount:      2,
+			machineType:   "microvm",
+			wantIOThreads: []IOThread{{ID: "ioth-vol-blk"}},
+			wantBlockdevs: []Blockdev{
+				{Value: "driver=raw,node-name=nbd-vol-blk,size=10737418240,file.driver=host_device,file.filename=/dev/sdx,file.aio=native,file.cache.direct=on"},
+			},
+			wantDevices: []Device{
+				{Value: "virtio-blk-device,id=vdisk-vol-blk,drive=nbd-vol-blk,iothread=ioth-vol-blk,serial=volblk,werror=stop,rerror=stop"},
+			},
+			wantHotplugPorts: []int{2},
+		},
+		{
+			// q35 is not a microvm: the boundary must stay on PCI allocation.
+			name: "q35 data volume with HotplugPort unset still gets a PCI hot-plug port",
+			requests: []EBSRequest{
+				{Name: "vol-data-a", NBDURI: "nbd:unix:/tmp/data-a.sock"},
+			},
+			cpuCount:      2,
+			machineType:   "q35",
+			wantIOThreads: []IOThread{{ID: "ioth-vol-data-a"}},
+			wantBlockdevs: []Blockdev{
+				{Value: "driver=nbd,node-name=nbd-vol-data-a,server.type=unix,server.path=/tmp/data-a.sock,export=,reconnect-delay=30"},
+			},
+			wantDevices: []Device{
+				{Value: "virtio-blk-pci,id=vdisk-vol-data-a,drive=nbd-vol-data-a,iothread=ioth-vol-data-a,serial=voldataa,bus=hotplug-ebs1,werror=stop,rerror=stop"},
+			},
+			wantHotplugPorts: []int{1},
+		},
 		{
 			name: "NBD boot volume alongside a block data volume",
 			requests: []EBSRequest{
@@ -1092,14 +1145,8 @@ func TestStartQEMU_BlockDeviceMissing(t *testing.T) {
 	m := directBootManager(t, &fakeNetworkPlumber{})
 	missing := filepath.Join(t.TempDir(), "missing-dev")
 
-	instance := &VM{
-		ID:           "i-blk-missing",
-		InstanceType: "t3.nano",
-		DirectBoot:   true,
-	}
-	instance.EBSRequests.Requests = []EBSRequest{
-		{Name: "vol-blk", BlockDevice: missing, BlockSizeBytes: 1 << 20},
-	}
+	instance := directBootInstance("i-blk-missing",
+		EBSRequest{Name: "vol-blk", BlockDevice: missing, BlockSizeBytes: 1 << 20})
 
 	err := m.startQEMU(instance)
 
@@ -1693,3 +1740,438 @@ func TestQMPSuccessClearsTheIOErrorBudget(t *testing.T) {
 	assert.Zero(t, instance.Health.IOErrorResumes)
 	assert.True(t, instance.Health.IOErrorSince.IsZero())
 }
+
+// directBootBase is a launcher-built microvm Config with one NIC and spare
+// capacity in every slice, so an append that aliased it would be caught.
+func directBootBase() Config {
+	devices := make([]Device, 1, 8)
+	devices[0] = Device{Value: "virtio-net-device,netdev=net0,mac=02:00:00:aa:bb:01"}
+	netdevs := make([]NetDev, 1, 8)
+	netdevs[0] = NetDev{Value: "tap,id=net0,ifname=tap0,script=no,downscript=no"}
+	fwcfg := make([]FwCfgEntry, 1, 8)
+	fwcfg[0] = FwCfgEntry{Name: "opt/spinifex/netcfg", File: "/run/netcfg"}
+	return Config{
+		Name:          "i-sys",
+		EnableKVM:     true,
+		NoGraphic:     true,
+		MachineType:   "microvm,pic=off,pit=off,rtc=on,acpi=on,isa-serial=on",
+		CPUType:       "host,-pmu",
+		CPUCount:      2,
+		Memory:        512,
+		BIOS:          "/usr/share/spinifex/microvm/qboot.rom",
+		KernelImage:   "/usr/share/spinifex/microvm/vmlinuz",
+		Initrd:        "/usr/share/spinifex/microvm/initramfs.cpio.gz",
+		KernelCmdline: "console=ttyS0 quiet",
+		Drives:        make([]Drive, 0, 8),
+		IOThreads:     make([]IOThread, 0, 8),
+		Blockdevs:     make([]Blockdev, 0, 8),
+		Devices:       devices,
+		NetDevs:       netdevs,
+		FwCfg:         fwcfg,
+	}
+}
+
+func directBootRequests() []EBSRequest {
+	return []EBSRequest{
+		{Name: "vol-data-a", NBDURI: "nbd:unix:/tmp/data-a.sock"},
+		{Name: "vol-blk", BlockDevice: "/dev/sdx", BlockSizeBytes: 10 << 30, HotplugPort: 2},
+	}
+}
+
+// deepCopyConfig snapshots every slice so a later write through an alias
+// shows up as a difference.
+func deepCopyConfig(c Config) Config {
+	c.Drives = slices.Clone(c.Drives)
+	c.IOThreads = slices.Clone(c.IOThreads)
+	c.Blockdevs = slices.Clone(c.Blockdevs)
+	c.Devices = slices.Clone(c.Devices)
+	c.NetDevs = slices.Clone(c.NetDevs)
+	c.FwCfg = slices.Clone(c.FwCfg)
+	return c
+}
+
+func TestDirectBootLaunchConfig_AppendsDataVolumesAfterExistingDevices(t *testing.T) {
+	base := directBootBase()
+	got, err := directBootLaunchConfig(base, directBootRequests(), 2)
+	require.NoError(t, err)
+
+	assert.Equal(t, []IOThread{{ID: "ioth-vol-data-a"}, {ID: "ioth-vol-blk"}}, got.IOThreads)
+	assert.Equal(t, []Blockdev{
+		{Value: "driver=nbd,node-name=nbd-vol-data-a,server.type=unix,server.path=/tmp/data-a.sock,export=,reconnect-delay=30"},
+		{Value: "driver=raw,node-name=nbd-vol-blk,size=10737418240,file.driver=host_device,file.filename=/dev/sdx,file.aio=native,file.cache.direct=on"},
+	}, got.Blockdevs)
+	assert.Equal(t, []Device{
+		{Value: "virtio-net-device,netdev=net0,mac=02:00:00:aa:bb:01"},
+		{Value: "virtio-blk-device,id=vdisk-vol-data-a,drive=nbd-vol-data-a,iothread=ioth-vol-data-a,serial=voldataa,werror=stop,rerror=stop"},
+		{Value: "virtio-blk-device,id=vdisk-vol-blk,drive=nbd-vol-blk,iothread=ioth-vol-blk,serial=volblk,werror=stop,rerror=stop"},
+	}, got.Devices, "the NIC keeps its MMIO slot ahead of the volumes")
+	assert.Empty(t, got.Drives, "data volumes are blockdevs, never -drive")
+
+	// Only the three volume fields differ; the kernel args and every
+	// unrelated field are the base's.
+	want := deepCopyConfig(base)
+	want.IOThreads, want.Blockdevs, want.Devices = got.IOThreads, got.Blockdevs, got.Devices
+	assert.Equal(t, want, got)
+}
+
+func TestDirectBootLaunchConfig_DoesNotMutateInputs(t *testing.T) {
+	base := directBootBase()
+	baseBefore := deepCopyConfig(base)
+	requests := directBootRequests()
+	requestsBefore := append([]EBSRequest(nil), requests...)
+
+	got, err := directBootLaunchConfig(base, requests, 2)
+	require.NoError(t, err)
+	assert.Equal(t, baseBefore, deepCopyConfig(base), "base fields are unchanged")
+	assert.Equal(t, requestsBefore, requests, "requests are unchanged")
+
+	// Spare capacity: the composition must not have written into the base's
+	// backing arrays, and neither result may see the other's later appends.
+	assert.Equal(t, Device{}, base.Devices[:2][1], "nothing was written past the base's length")
+	assert.Equal(t, IOThread{}, base.IOThreads[:1][0])
+	assert.Equal(t, Blockdev{}, base.Blockdevs[:1][0])
+
+	base.Devices = append(base.Devices, Device{Value: "virtio-rng-device"})
+	base.NetDevs = append(base.NetDevs, NetDev{Value: "tap,id=net9"})
+	base.FwCfg = append(base.FwCfg, FwCfgEntry{Name: "x"})
+	assert.Equal(t, "virtio-blk-device,id=vdisk-vol-data-a,drive=nbd-vol-data-a,iothread=ioth-vol-data-a,serial=voldataa,werror=stop,rerror=stop", got.Devices[1].Value)
+	assert.Len(t, got.NetDevs, 1)
+	assert.Len(t, got.FwCfg, 1)
+
+	got.Devices[0].Value = "changed"
+	got.NetDevs = append(got.NetDevs, NetDev{Value: "tap,id=net8"})
+	got.FwCfg[0].File = "changed"
+	assert.Equal(t, baseBefore.Devices[0], base.Devices[0])
+	assert.Equal(t, NetDev{Value: "tap,id=net9"}, base.NetDevs[1])
+	assert.Equal(t, baseBefore.FwCfg[0], base.FwCfg[0])
+}
+
+func TestDirectBootLaunchConfig_RepeatedCompositionFromTheSameBase(t *testing.T) {
+	base := directBootBase()
+	first, err := directBootLaunchConfig(base, directBootRequests(), 2)
+	require.NoError(t, err)
+	second, err := directBootLaunchConfig(base, directBootRequests(), 2)
+	require.NoError(t, err)
+
+	assert.Equal(t, first, second)
+	assert.Len(t, second.Devices, 3, "one NIC and one device per volume, never accumulated")
+	assert.Len(t, second.Blockdevs, 2)
+	assert.Len(t, second.IOThreads, 2)
+}
+
+func TestDirectBootLaunchConfig_ChangedNBDURIOnALaterLaunch(t *testing.T) {
+	base := directBootBase()
+	requests := []EBSRequest{{Name: "vol-data-a", NBDURI: "nbd:unix:/tmp/data-a.sock"}}
+	_, err := directBootLaunchConfig(base, requests, 2)
+	require.NoError(t, err)
+
+	// A remount publishes a new export; the next launch must use only it.
+	requests[0].NBDURI = "nbd://10.0.0.7:10809"
+	got, err := directBootLaunchConfig(base, requests, 2)
+	require.NoError(t, err)
+	require.Len(t, got.Blockdevs, 1)
+	assert.Contains(t, got.Blockdevs[0].Value, "server.type=inet,server.host=10.0.0.7,server.port=10809")
+	assert.NotContains(t, got.Blockdevs[0].Value, "/tmp/data-a.sock")
+}
+
+func TestDirectBootLaunchConfig_Rejects(t *testing.T) {
+	tests := []struct {
+		name     string
+		base     func() Config
+		requests []EBSRequest
+		wantErr  string
+	}{
+		{
+			name:     "boot volume",
+			base:     directBootBase,
+			requests: []EBSRequest{{Name: "vol-boot", NBDURI: "nbd:unix:/tmp/boot.sock", Boot: true}},
+			wantErr:  "vol-boot",
+		},
+		{
+			name:     "EFI volume",
+			base:     directBootBase,
+			requests: []EBSRequest{{Name: "vol-efi", NBDURI: "nbd:unix:/tmp/efi.sock", EFI: true}},
+			wantErr:  "vol-efi",
+		},
+		{
+			name:     "unmounted data volume",
+			base:     directBootBase,
+			requests: []EBSRequest{{Name: "vol-bad"}},
+			wantErr:  "NBDURI not set for volume vol-bad",
+		},
+		{
+			name:     "both NBD and block exports",
+			base:     directBootBase,
+			requests: []EBSRequest{{Name: "vol-both", NBDURI: "nbd:unix:/tmp/x.sock", BlockDevice: "/dev/sdx", BlockSizeBytes: 1 << 20}},
+			wantErr:  "vol-both",
+		},
+		{
+			name: "non-microvm base",
+			base: func() Config {
+				c := directBootBase()
+				c.MachineType = "q35"
+				return c
+			},
+			requests: []EBSRequest{{Name: "vol-data-a", NBDURI: "nbd:unix:/tmp/data-a.sock"}},
+			wantErr:  "microvm",
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			base := tt.base()
+			before := deepCopyConfig(base)
+			_, err := directBootLaunchConfig(base, tt.requests, 2)
+			require.Error(t, err)
+			assert.Contains(t, err.Error(), tt.wantErr)
+			assert.Equal(t, before, deepCopyConfig(base))
+		})
+	}
+}
+
+// fakeQEMUScript is a stand-in QEMU that records each launch's argv, writes the
+// pidfile and creates the QMP socket path startQEMU waits on.
+const fakeQEMUScript = `#!/bin/sh
+pidfile=""
+qmp=""
+prev=""
+for a in "$@"; do
+	case "$prev" in
+	-pidfile) pidfile="$a" ;;
+	-qmp)
+		if [ -z "$qmp" ]; then
+			q="${a#unix:}"
+			qmp="${q%%,*}"
+		fi
+		;;
+	esac
+	prev="$a"
+done
+{
+	echo "--- launch"
+	for a in "$@"; do printf '%s\n' "$a"; done
+} >>"$0.log"
+echo $$ >"$pidfile"
+: >"$qmp"
+# Stay up like a running guest until the socket goes: a relaunch unlinks it,
+# and test cleanup removes its directory.
+while [ -e "$qmp" ]; do sleep 0.05; done
+`
+
+// withFakeQEMU points QEMU resolution at fakeQEMUScript and returns a reader
+// for the argv of every launch so far.
+func withFakeQEMU(t *testing.T) func() [][]string {
+	t.Helper()
+	path := filepath.Join(t.TempDir(), "qemu-system-x86_64")
+	require.NoError(t, os.WriteFile(path, []byte(fakeQEMUScript), 0o755))
+	withQEMUResolver(t, func(string) (string, error) { return path, nil })
+	return func() [][]string {
+		data, err := os.ReadFile(path + ".log")
+		if errors.Is(err, os.ErrNotExist) {
+			return nil
+		}
+		require.NoError(t, err)
+		var launches [][]string
+		for line := range strings.SplitSeq(strings.TrimRight(string(data), "\n"), "\n") {
+			if line == "--- launch" {
+				launches = append(launches, nil)
+				continue
+			}
+			launches[len(launches)-1] = append(launches[len(launches)-1], line)
+		}
+		return launches
+	}
+}
+
+// nbdListener stands in for a mounted NBD export: startQEMU only needs it to
+// accept a connection.
+func nbdListener(t *testing.T) string {
+	t.Helper()
+	l, err := net.Listen("tcp", "127.0.0.1:0")
+	require.NoError(t, err)
+	t.Cleanup(func() { _ = l.Close() })
+	go func() {
+		for {
+			c, err := l.Accept()
+			if err != nil {
+				return
+			}
+			_ = c.Close()
+		}
+	}()
+	return "nbd://" + l.Addr().String()
+}
+
+// argsContaining returns the argv entries that contain sub.
+func argsContaining(args []string, sub string) []string {
+	var out []string
+	for _, a := range args {
+		if strings.Contains(a, sub) {
+			out = append(out, a)
+		}
+	}
+	return out
+}
+
+func directBootInstance(id string, requests ...EBSRequest) *VM {
+	instance := &VM{
+		ID:           id,
+		InstanceType: "t3.nano",
+		DirectBoot:   true,
+		Config:       directBootBase(),
+	}
+	instance.Config.Architecture = "x86_64"
+	instance.EBSRequests.Requests = requests
+	return instance
+}
+
+// assertCleanBase checks the instance's saved Config carries none of the
+// launch-only volume entries and no NBD endpoint.
+func assertCleanBase(t *testing.T, cfg Config, endpoints ...string) {
+	t.Helper()
+	base := directBootBase()
+	assert.Equal(t, base.Devices, cfg.Devices)
+	assert.Equal(t, base.Blockdevs, cfg.Blockdevs)
+	assert.Equal(t, base.IOThreads, cfg.IOThreads)
+	assert.Equal(t, base.Drives, cfg.Drives)
+	saved, err := json.Marshal(cfg)
+	require.NoError(t, err)
+	for _, ep := range endpoints {
+		assert.NotContains(t, string(saved), strings.TrimPrefix(ep, "nbd://"))
+	}
+}
+
+func TestStartQEMU_DirectBoot_LaunchesMountedDataVolumeWithoutSavingIt(t *testing.T) {
+	m := directBootManager(t, &fakeNetworkPlumber{})
+	launches := withFakeQEMU(t)
+	first := nbdListener(t)
+	second := nbdListener(t)
+
+	instance := directBootInstance("i-db-vol", EBSRequest{Name: "vol-data-a", NBDURI: first})
+	require.NoError(t, m.startQEMU(instance))
+
+	// A later launch from the same saved base, after the volume was remounted
+	// on a different endpoint.
+	instance.EBSRequests.Requests[0].NBDURI = second
+	require.NoError(t, m.startQEMU(instance))
+
+	got := launches()
+	require.Len(t, got, 2)
+	for i, ep := range []string{first, second} {
+		args := got[i]
+		require.Len(t, argsContaining(args, "virtio-blk-device"), 1, "launch %d", i+1)
+		assert.Empty(t, argsContaining(args, "virtio-blk-pci"), "launch %d", i+1)
+		assert.Len(t, argsContaining(args, "ioth-vol-data-a"), 2, "launch %d: one iothread object and its device", i+1)
+		port := ep[strings.LastIndex(ep, ":")+1:]
+		assert.NotEmpty(t, argsContaining(args, port), "launch %d uses its current endpoint", i+1)
+		// The NIC stays ahead of the data disk.
+		nic := slices.IndexFunc(args, func(a string) bool { return strings.HasPrefix(a, "virtio-net-device") })
+		disk := slices.IndexFunc(args, func(a string) bool { return strings.HasPrefix(a, "virtio-blk-device") })
+		assert.Less(t, nic, disk, "launch %d", i+1)
+	}
+	firstPort := first[strings.LastIndex(first, ":")+1:]
+	assert.Empty(t, argsContaining(got[1], firstPort), "second launch must not carry the stale endpoint")
+
+	assertCleanBase(t, instance.Config, first, second)
+}
+
+// TestStartQEMU_DirectBoot_LaunchesTheEndpointItProbed remounts the volume
+// while its endpoint is being probed: the command must carry the endpoint that
+// was checked, not one read from the requests afterwards.
+func TestStartQEMU_DirectBoot_LaunchesTheEndpointItProbed(t *testing.T) {
+	m := directBootManager(t, &fakeNetworkPlumber{})
+	launches := withFakeQEMU(t)
+	probedEP := nbdListener(t)
+	remountedEP := nbdListener(t)
+
+	instance := directBootInstance("i-db-snap", EBSRequest{Name: "vol-data-a", NBDURI: probedEP})
+	var probed []string
+	orig := waitForNBDReady
+	t.Cleanup(func() { waitForNBDReady = orig })
+	waitForNBDReady = func(uri string, timeout time.Duration) error {
+		probed = append(probed, uri)
+		instance.EBSRequests.Mu.Lock()
+		instance.EBSRequests.Requests[0].NBDURI = remountedEP
+		instance.EBSRequests.Mu.Unlock()
+		return orig(uri, timeout)
+	}
+	require.NoError(t, m.startQEMU(instance))
+
+	assert.Equal(t, []string{probedEP}, probed)
+	got := launches()
+	require.Len(t, got, 1)
+	port := func(ep string) string { return ep[strings.LastIndex(ep, ":")+1:] }
+	assert.NotEmpty(t, argsContaining(got[0], port(probedEP)), "the probed endpoint is launched")
+	assert.Empty(t, argsContaining(got[0], port(remountedEP)), "an endpoint nobody probed is not launched")
+}
+
+func TestStartQEMU_DirectBoot_NoVolumesLaunchesTheBase(t *testing.T) {
+	m := directBootManager(t, &fakeNetworkPlumber{})
+	launches := withFakeQEMU(t)
+
+	instance := directBootInstance("i-db-novol")
+	require.NoError(t, m.startQEMU(instance))
+
+	got := launches()
+	require.Len(t, got, 1)
+	assert.Empty(t, argsContaining(got[0], "virtio-blk"))
+	assert.Len(t, argsContaining(got[0], "virtio-net-device"), 1)
+	assertCleanBase(t, instance.Config)
+}
+
+func TestStartQEMU_DirectBoot_InvalidDataRequestIsNotLaunched(t *testing.T) {
+	tests := []struct {
+		name    string
+		req     EBSRequest
+		wantErr string
+	}{
+		{name: "no export", req: EBSRequest{Name: "vol-none"}, wantErr: "vol-none"},
+		{name: "boot volume", req: EBSRequest{Name: "vol-boot", NBDURI: "nbd:unix:/tmp/boot.sock", Boot: true}, wantErr: "data volumes only"},
+		{name: "efi volume", req: EBSRequest{Name: "vol-efi", NBDURI: "nbd:unix:/tmp/efi.sock", EFI: true}, wantErr: "data volumes only"},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			m := directBootManager(t, &fakeNetworkPlumber{})
+			launches := withFakeQEMU(t)
+
+			instance := directBootInstance("i-db-bad", tt.req)
+			err := m.startQEMU(instance)
+
+			require.Error(t, err)
+			assert.Contains(t, err.Error(), tt.wantErr)
+			assert.Empty(t, launches(), "QEMU must not start")
+			assertCleanBase(t, instance.Config)
+		})
+	}
+}
+
+func TestStartQEMU_DiskBoot_DataVolumeStaysOnThePCIPath(t *testing.T) {
+	m := directBootManager(t, &fakeNetworkPlumber{})
+	launches := withFakeQEMU(t)
+	ep := nbdListener(t)
+
+	instance := &VM{ID: "i-disk-vol", InstanceType: "t3.nano"}
+	instance.EBSRequests.Requests = []EBSRequest{
+		{Name: "vol-root", NBDURI: "nbd:unix:" + filepath.Join(t.TempDir(), "root.sock"), Boot: true},
+		{Name: "vol-data-a", NBDURI: ep},
+	}
+	require.NoError(t, os.WriteFile(strings.TrimPrefix(instance.EBSRequests.Requests[0].NBDURI, "nbd:unix:"), nil, 0o600))
+	require.NoError(t, m.startQEMU(instance))
+
+	got := launches()
+	require.Len(t, got, 1)
+	data := argsContaining(got[0], "id=vdisk-vol-data-a")
+	require.Len(t, data, 1)
+	assert.True(t, strings.HasPrefix(data[0], "virtio-blk-pci,"), data[0])
+	assert.Empty(t, argsContaining(got[0], "virtio-blk-device"))
+	// The disk-boot path keeps building the volume into the instance Config.
+	assert.Len(t, argsContaining(deviceValues(instance.Config.Devices), "id=vdisk-vol-data-a"), 1)
+	assert.Equal(t, 1, instance.EBSRequests.Requests[1].HotplugPort)
+}
+
+func deviceValues(devs []Device) []string {
+	out := make([]string, len(devs))
+	for i, d := range devs {
+		out[i] = d.Value
+	}
+	return out
+}

spinifex/runtime/compute/vm/volumes.go

@@ -29,6 +29,10 @@ const blockdevDelMaxAttempts = 20
 // margin; the ebs.unmount seal keeps its separate unmountSealTimeout.
 const detachAggregateTimeout = 3 * time.Minute
 
+// ErrHotAttachUnsupported is returned by AttachVolume for a direct-boot microvm,
+// whose MMIO bus has no PCIe root port to hot-plug a volume into.
+var ErrHotAttachUnsupported = errors.New("volume hot-attach is not supported on a direct-boot microvm")
+
 // rollbackUnmount unmounts a volume while unwinding a failed AttachVolume.
 // Callers keep the volume non-available when the seal fails so a later attach
 // cannot discard local state that may still be owned by the backend.
@@ -82,6 +86,12 @@ func (m *Manager) AttachVolume(ctx context.Context, id, volumeID, device string)
 		return "", ErrInstanceNotFound
 	}
 
+	// Refused before anything is mounted or recorded: the hot-plug below needs
+	// a PCIe root port that a microvm's MMIO bus does not have.
+	if instance.DirectBoot {
+		return "", fmt.Errorf("%w: instance %s", ErrHotAttachUnsupported, id)
+	}
+
 	if status := m.Status(instance); status != StateRunning {
 		return "", fmt.Errorf("%w: cannot attach to instance %s in state %s",
 			ErrInvalidTransition, id, status)

spinifex/runtime/compute/vm/volumes_test.go

@@ -1919,3 +1919,38 @@ func TestCmdlineHasPauseAction(t *testing.T) {
 		})
 	}
 }
+
+// TestAttachVolume_DirectBootRejectedWithoutSideEffects covers a microvm guest:
+// hot-attach plugs into a PCIe root port it does not have, so the request is
+// refused before the volume is mounted or any instance or volume state moves.
+func TestAttachVolume_DirectBootRejectedWithoutSideEffects(t *testing.T) {
+	for _, device := range []string{"", "/dev/sdf"} {
+		t.Run("device="+device, func(t *testing.T) {
+			mounter := &fakeVolumeMounter{mountOneURI: "nbd://127.0.0.1:10809"}
+			states := &fakeVolumeStateUpdater{}
+			m := NewManagerWithDeps(Deps{VolumeMounter: mounter, VolumeStateUpdater: states})
+			existing := EBSRequest{Name: "vol-data-a", NBDURI: "nbd://127.0.0.1:10810", DeviceName: "/dev/sdf"}
+			v := &VM{
+				ID:         "i-db",
+				Status:     StateRunning,
+				DirectBoot: true,
+				Instance:   &ec2.Instance{},
+				Config:     Config{MachineType: "microvm,acpi=on"},
+			}
+			v.EBSRequests.Requests = []EBSRequest{existing}
+			m.Insert(v)
+
+			_, err := m.AttachVolume(t.Context(), "i-db", "vol-new", device)
+
+			require.Error(t, err)
+			assert.ErrorIs(t, err, ErrHotAttachUnsupported)
+			assert.Empty(t, mounter.mountedOne, "volume must not be mounted")
+			assert.Empty(t, mounter.unmountedOne)
+			assert.Empty(t, states.calls, "volume metadata must not change")
+			assert.Equal(t, []EBSRequest{existing}, v.EBSRequests.Requests)
+			assert.Empty(t, v.Instance.BlockDeviceMappings)
+			assert.Equal(t, StateRunning, v.Status)
+			assert.Equal(t, Config{MachineType: "microvm,acpi=on"}, v.Config)
+		})
+	}
+}

spinifex/runtime/daemon/daemon.go

@@ -1289,6 +1289,7 @@ func (d *Daemon) subscribeAll() error {
 			natsSub{ecr.SubjectManifestDescribe, handleNATSRequest(d.node, d.ecrMetaService.ManifestDescribe), "spinifex-workers"},
 			natsSub{ecr.SubjectManifestList, handleNATSRequest(d.node, d.ecrMetaService.ManifestList), "spinifex-workers"},
 			natsSub{ecr.SubjectManifestDelete, handleNATSRequest(d.node, d.ecrMetaService.ManifestDelete), "spinifex-workers"},
+			natsSub{ecr.SubjectReferrerList, handleNATSRequest(d.node, d.ecrMetaService.ReferrerList), "spinifex-workers"},
 			natsSub{ecr.SubjectUploadCreate, handleNATSRequest(d.node, d.ecrMetaService.UploadCreate), "spinifex-workers"},
 			natsSub{ecr.SubjectUploadGet, handleNATSRequest(d.node, d.ecrMetaService.UploadGet), "spinifex-workers"},
 			natsSub{ecr.SubjectUploadUpdate, handleNATSRequest(d.node, d.ecrMetaService.UploadUpdate), "spinifex-workers"},

spinifex/runtime/daemon/daemon_handlers_volume.go

@@ -191,6 +191,8 @@ func attachDetachErrorCode(err error) string {
 		return awserrors.ErrorInvalidInstanceIDNotFound
 	case errors.Is(err, vm.ErrInvalidTransition):
 		return awserrors.ErrorIncorrectInstanceState
+	case errors.Is(err, vm.ErrHotAttachUnsupported):
+		return awserrors.ErrorUnsupportedOperation
 	case errors.Is(err, vm.ErrAttachmentLimitExceeded):
 		return awserrors.ErrorAttachmentLimitExceeded
 	case errors.Is(err, vm.ErrVolumeNotAttached):

spinifex/runtime/daemon/daemon_handlers_volume_test.go

@@ -57,6 +57,16 @@ func TestAttachDetachErrorCode(t *testing.T) {
 			err:  fmt.Errorf("cannot attach in state stopped: %w", vm.ErrInvalidTransition),
 			want: awserrors.ErrorIncorrectInstanceState,
 		},
+		{
+			name: "ErrHotAttachUnsupported maps to UnsupportedOperation",
+			err:  vm.ErrHotAttachUnsupported,
+			want: awserrors.ErrorUnsupportedOperation,
+		},
+		{
+			name: "wrapped ErrHotAttachUnsupported still matches",
+			err:  fmt.Errorf("%w: instance i-db", vm.ErrHotAttachUnsupported),
+			want: awserrors.ErrorUnsupportedOperation,
+		},
 		{
 			name: "ErrAttachmentLimitExceeded maps to AttachmentLimitExceeded",
 			err:  vm.ErrAttachmentLimitExceeded,

tests/e2e/lb/lb_health_test.go

@@ -0,0 +1,296 @@
+//go:build e2e
+
+package lb
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"testing"
+	"time"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/service/elbv2"
+	"github.com/mulgadc/spinifex/tests/e2e/harness"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+const (
+	// healthIntervalSeconds and healthThreshold keep detection near 10s; the
+	// routing bound adds two lb-agent ticks and a config reload on top.
+	healthIntervalSeconds = 5
+	healthThreshold       = 2
+	healthRoutingBound    = 2 * time.Minute
+	healthProbeCount      = 20
+)
+
+// runNLBUnhealthyTargetsSuite checks that an NLB stops sending traffic to a
+// target whose health check fails while it still accepts TCP, sends traffic
+// again once it recovers, and fails open when every target is unhealthy.
+func runNLBUnhealthyTargetsSuite(t *testing.T, c *harness.AWSClient, f *sharedFixture) {
+	t.Helper()
+	const label = "NLB unhealthy targets"
+
+	tgArn := createHealthCheckedTCPTargetGroup(t, c, f, "lb-e2e-nlb-health-tg")
+	t.Cleanup(func() { deleteTargetGroup(t, c, tgArn) })
+	registerTargets(t, c, tgArn, f.AppInstanceIDs)
+	t.Cleanup(func() { deregisterTargets(t, c, tgArn, f.AppInstanceIDs) })
+
+	lb, _ := createActiveLB(t, c, f, "lb-e2e-nlb-health", "network", "internal", "TCP", tcpPort, tgArn, label, nil)
+	captureLBConsoleOnFailure(t, c, "net", lb)
+	harness.WaitForTargetsHealthy(t, c, tgArn, 2, label, 2*time.Minute)
+
+	lbIP := privateIP(lbENI(t, c, "net", lb))
+	require.NotEmpty(t, lbIP, label+" needs private IP")
+
+	appIPs := make([]string, len(f.AppInstanceIDs))
+	responders := make([]string, len(f.AppInstanceIDs))
+	for i, id := range f.AppInstanceIDs {
+		appIPs[i] = privateIP(harness.InstanceENI(t, c, id))
+		require.NotEmptyf(t, appIPs[i], "app instance %s needs a private IP", id)
+		responders[i] = requireSingleTCPResponder(t, f, appIPs[i])
+	}
+	require.NotEqual(t, responders[0], responders[1], "the two app instances must answer differently")
+	t.Cleanup(func() {
+		for _, ip := range appIPs {
+			_, _ = probeTrigger(f, "http", ip, healthPort, "/ok", 1)
+		}
+	})
+
+	harness.Phase(t, "%s: both healthy, both receive traffic", label)
+	r := requireTCPDistribution(t, f, lbIP, healthRoutingBound, func(r harness.TrafficResult) bool {
+		return len(r.Distribution) == 2 && r.Successful == healthProbeCount
+	}, "both targets serve")
+	t.Logf("%s: distribution %v", label, r.Distribution)
+
+	harness.Phase(t, "%s: target %s fails its health check but still accepts TCP", label, f.AppInstanceIDs[0])
+	setHealth(t, f, appIPs[0], false)
+	waitTargetState(t, c, tgArn, f.AppInstanceIDs[0], "unhealthy", healthRoutingBound)
+	r = requireTCPDistribution(t, f, lbIP, healthRoutingBound, func(r harness.TrafficResult) bool {
+		return r.Successful == healthProbeCount && r.Distribution[responders[0]] == 0
+	}, "no request reaches the unhealthy target")
+	assert.Positive(t, r.Distribution[responders[1]], "the healthy target serves all traffic")
+	// Still accepting TCP directly proves it was removed by health, not by being down.
+	assert.Equal(t, responders[0], requireSingleTCPResponder(t, f, appIPs[0]),
+		"the unhealthy target keeps accepting TCP")
+
+	harness.Phase(t, "%s: target recovers and receives traffic again", label)
+	setHealth(t, f, appIPs[0], true)
+	waitTargetState(t, c, tgArn, f.AppInstanceIDs[0], "healthy", healthRoutingBound)
+	requireTCPDistribution(t, f, lbIP, healthRoutingBound, func(r harness.TrafficResult) bool {
+		return len(r.Distribution) == 2 && r.Successful == healthProbeCount
+	}, "recovered target serves again")
+
+	harness.Phase(t, "%s: every target unhealthy, load balancer fails open", label)
+	for _, ip := range appIPs {
+		setHealth(t, f, ip, false)
+	}
+	for _, id := range f.AppInstanceIDs {
+		waitTargetState(t, c, tgArn, id, "unhealthy", healthRoutingBound)
+	}
+	requireTCPDistribution(t, f, lbIP, healthRoutingBound, func(r harness.TrafficResult) bool {
+		return len(r.Distribution) == 2 && r.Successful == healthProbeCount
+	}, "fail open sends traffic to every target")
+}
+
+// runALBFailOpenSuite checks that an ALB whose targets are all failing their
+// health check still serves traffic from them instead of answering 503.
+func runALBFailOpenSuite(t *testing.T, c *harness.AWSClient, f *sharedFixture) {
+	t.Helper()
+	const label = "ALB fail open"
+
+	tgArn := createTargetGroup(t, c, f, "lb-e2e-alb-failopen-tg", "HTTP", httpPort, "/hc.html")
+	t.Cleanup(func() { deleteTargetGroup(t, c, tgArn) })
+	registerTargets(t, c, tgArn, f.AppInstanceIDs)
+	t.Cleanup(func() { deregisterTargets(t, c, tgArn, f.AppInstanceIDs) })
+
+	sgID := createScenarioSG(t, c, f, "lb-e2e-alb-failopen-sg", httpPort)
+	lb, _ := createActiveLB(t, c, f, "lb-e2e-alb-failopen", "application", "internal", "HTTP", httpPort, tgArn, label, []string{sgID})
+	harness.WaitForTargetsHealthy(t, c, tgArn, 2, label, 2*time.Minute)
+
+	lbIP := privateIP(lbENI(t, c, "app", lb))
+	require.NotEmpty(t, lbIP, label+" needs private IP")
+
+	harness.Phase(t, "%s: a health-check path change reaches the load balancer", label)
+	setHealthPath(t, c, tgArn, "/health-check-missing")
+	for _, id := range f.AppInstanceIDs {
+		waitTargetState(t, c, tgArn, id, "unhealthy", healthRoutingBound)
+	}
+	setHealthPath(t, c, tgArn, "/hc.html")
+	harness.WaitForTargetsHealthy(t, c, tgArn, 2, label, healthRoutingBound)
+
+	appIPs := make([]string, len(f.AppInstanceIDs))
+	for i, id := range f.AppInstanceIDs {
+		appIPs[i] = privateIP(harness.InstanceENI(t, c, id))
+		require.NotEmptyf(t, appIPs[i], "app instance %s needs a private IP", id)
+	}
+	t.Cleanup(func() {
+		for _, ip := range appIPs {
+			_, _ = probeTrigger(f, "http", ip, healthPort, "/ok", 1)
+		}
+	})
+
+	harness.Phase(t, "%s: health path stops answering 200 on every target", label)
+	for _, ip := range appIPs {
+		setHealth(t, f, ip, false)
+	}
+	for _, id := range f.AppInstanceIDs {
+		waitTargetState(t, c, tgArn, id, "unhealthy", healthRoutingBound)
+	}
+
+	harness.EventuallyErr(t, func() error {
+		r, err := probeTrigger(f, "http", lbIP, httpPort, "/index.html", healthProbeCount)
+		if err != nil {
+			return err
+		}
+		if r.Successful != healthProbeCount {
+			return fmt.Errorf("%d/%d requests served while every target is unhealthy", r.Successful, healthProbeCount)
+		}
+		return nil
+	}, healthRoutingBound, 5*time.Second)
+}
+
+// createHealthCheckedTCPTargetGroup creates a TCP target group whose health
+// check is an HTTP GET to the toggle endpoint, separate from the data port.
+func createHealthCheckedTCPTargetGroup(t *testing.T, c *harness.AWSClient, f *sharedFixture, name string) string {
+	t.Helper()
+	// e2e:allow-create — the health-check settings are the subject under test.
+	out, err := c.ELBv2.CreateTargetGroup(&elbv2.CreateTargetGroupInput{
+		Name:                       aws.String(name),
+		Protocol:                   aws.String("TCP"),
+		Port:                       aws.Int64(tcpPort),
+		VpcId:                      aws.String(f.VPCID),
+		HealthCheckProtocol:        aws.String("HTTP"),
+		HealthCheckPort:            aws.String(fmt.Sprint(healthPort)),
+		HealthCheckPath:            aws.String("/health"),
+		HealthCheckIntervalSeconds: aws.Int64(healthIntervalSeconds),
+		HealthyThresholdCount:      aws.Int64(healthThreshold),
+		UnhealthyThresholdCount:    aws.Int64(healthThreshold),
+	})
+	require.NoErrorf(t, err, "create-target-group %s", name)
+	arn := aws.StringValue(out.TargetGroups[0].TargetGroupArn)
+	t.Logf("TG %s: %s", name, arn)
+	return arn
+}
+
+// setHealth flips an app instance's health endpoint through the probe client.
+func setHealth(t *testing.T, f *sharedFixture, appIP string, healthy bool) {
+	t.Helper()
+	path := "/fail"
+	if healthy {
+		path = "/ok"
+	}
+	r, err := probeTrigger(f, "http", appIP, healthPort, path, 1)
+	require.NoErrorf(t, err, "toggle health %s on %s", path, appIP)
+	t.Logf("health %s on %s: %d responses", path, appIP, r.Total)
+}
+
+// setHealthPath changes a target group's health-check path.
+func setHealthPath(t *testing.T, c *harness.AWSClient, tgArn, path string) {
+	t.Helper()
+	_, err := c.ELBv2.ModifyTargetGroup(&elbv2.ModifyTargetGroupInput{
+		TargetGroupArn:  aws.String(tgArn),
+		HealthCheckPath: aws.String(path),
+	})
+	require.NoErrorf(t, err, "modify-target-group health path %s", path)
+}
+
+// waitTargetState polls DescribeTargetHealth until one target reports want.
+func waitTargetState(t *testing.T, c *harness.AWSClient, tgArn, targetID, want string, timeout time.Duration) {
+	t.Helper()
+	start := time.Now()
+	defer func() { t.Logf("target %s reached %s after %s", targetID, want, time.Since(start).Round(time.Second)) }()
+	harness.EventuallyErr(t, func() error {
+		out, err := c.ELBv2.DescribeTargetHealth(&elbv2.DescribeTargetHealthInput{TargetGroupArn: aws.String(tgArn)})
+		if err != nil {
+			return err
+		}
+		for _, th := range out.TargetHealthDescriptions {
+			if aws.StringValue(th.Target.Id) == targetID {
+				got := aws.StringValue(th.TargetHealth.State)
+				if got == want {
+					return nil
+				}
+				return fmt.Errorf("target %s is %s, want %s", targetID, got, want)
+			}
+		}
+		return fmt.Errorf("target %s not listed", targetID)
+	}, timeout, 3*time.Second)
+}
+
+// requireSingleTCPResponder returns the hostname one backend answers with on
+// the TCP echo port, probing it directly rather than through a load balancer.
+func requireSingleTCPResponder(t *testing.T, f *sharedFixture, appIP string) string {
+	t.Helper()
+	var name string
+	harness.EventuallyErr(t, func() error {
+		r, err := probeTrigger(f, "tcp", appIP, tcpPort, "", 1)
+		if err != nil {
+			return err
+		}
+		if len(r.Distribution) != 1 {
+			return fmt.Errorf("no answer from %s:%d", appIP, tcpPort)
+		}
+		for k := range r.Distribution {
+			name = k
+		}
+		return nil
+	}, time.Minute, 3*time.Second)
+	return name
+}
+
+// requireTCPDistribution probes the load balancer until ok accepts a run.
+func requireTCPDistribution(t *testing.T, f *sharedFixture, lbIP string, timeout time.Duration, ok func(harness.TrafficResult) bool, what string) harness.TrafficResult {
+	t.Helper()
+	var last harness.TrafficResult
+	start := time.Now()
+	defer func() { t.Logf("%s after %s", what, time.Since(start).Round(time.Second)) }()
+	harness.EventuallyErr(t, func() error {
+		r, err := probeTrigger(f, "tcp", lbIP, tcpPort, "", healthProbeCount)
+		if err != nil {
+			return err
+		}
+		last = r
+		if !ok(r) {
+			return fmt.Errorf("%s: distribution %v, %d/%d answered", what, r.Distribution, r.Successful, r.Total)
+		}
+		return nil
+	}, timeout, 5*time.Second)
+	return last
+}
+
+// probeTrigger asks the shared probe client to send count requests to ip and
+// returns what each one saw. proto "http" uses port as the HTTP port; "tcp"
+// uses it as the echo port.
+func probeTrigger(f *sharedFixture, proto, ip string, port int64, path string, count int) (harness.TrafficResult, error) {
+	resultsFile := fmt.Sprintf("hlt-%s-%d-%d.txt", proto, port, time.Now().UnixNano())
+	order := map[string]any{
+		"proto":     proto,
+		"ip":        ip,
+		"count":     count,
+		"outfile":   resultsFile,
+		"http_port": port,
+		"http_path": path,
+		"tcp_port":  port,
+	}
+	body, err := json.Marshal(order)
+	if err != nil {
+		return harness.TrafficResult{}, err
+	}
+	client := &http.Client{Timeout: 3 * time.Minute}
+	resp, err := client.Post(fmt.Sprintf("http://%s:%d/", f.ClientPublicIP, triggerPort), "application/json", bytes.NewReader(body))
+	if err != nil {
+		return harness.TrafficResult{}, fmt.Errorf("trigger: %w", err)
+	}
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return harness.TrafficResult{}, fmt.Errorf("trigger status %d", resp.StatusCode)
+	}
+	results, err := plainHTTPGet(fmt.Sprintf("http://%s:%d/%s", f.ClientPublicIP, httpPort, resultsFile), 10*time.Second)
+	if err != nil {
+		return harness.TrafficResult{}, fmt.Errorf("fetch %s: %w", resultsFile, err)
+	}
+	return harness.VerifyResultsLines(results, proto), nil
+}

tests/e2e/lb/lb_test.go

@@ -39,6 +39,7 @@ const (
 	tcpPort      = 9000
 	udpPort      = 9001
 	triggerPort  = 9090
+	healthPort   = 8081
 	probesPerRun = 20
 )
 
@@ -159,6 +160,14 @@ func TestLoadBalancer(t *testing.T) {
 			t.Parallel()
 			runListenerRulesSuite(t, client, fixture)
 		})
+		t.Run("Internal_NLB_UnhealthyTargets", func(t *testing.T) {
+			t.Parallel()
+			runNLBUnhealthyTargetsSuite(t, client, fixture)
+		})
+		t.Run("Internal_ALB_FailOpen", func(t *testing.T) {
+			t.Parallel()
+			runALBFailOpenSuite(t, client, fixture)
+		})
 		return
 	}
 
@@ -190,6 +199,14 @@ func TestLoadBalancer(t *testing.T) {
 		skipIfInternalBroken(t)
 		runListenerRulesSuite(t, client, fixture)
 	})
+	t.Run("Internal_NLB_UnhealthyTargets", func(t *testing.T) {
+		skipIfInternalBroken(t)
+		runNLBUnhealthyTargetsSuite(t, client, fixture)
+	})
+	t.Run("Internal_ALB_FailOpen", func(t *testing.T) {
+		skipIfInternalBroken(t)
+		runALBFailOpenSuite(t, client, fixture)
+	})
 }
 
 // --- Fixture: shared VPC, subnet, IGW, SG, app instances ----------------
@@ -454,7 +471,7 @@ func configureDefaultSG(t *testing.T, c *harness.AWSClient, f *sharedFixture) {
 
 	// Use structured IpPermissions form — vpcd ignores the top-level shorthand fields
 	// so the OVN ACL would never be installed otherwise.
-	for _, port := range []int64{httpPort, tcpPort, triggerPort} {
+	for _, port := range []int64{httpPort, tcpPort, triggerPort, healthPort} {
 		_, err := c.EC2.AuthorizeSecurityGroupIngress(&ec2.AuthorizeSecurityGroupIngressInput{
 			GroupId: aws.String(f.SecurityGroup),
 			IpPermissions: []*ec2.IpPermission{{

tests/e2e/lb/testdata/app-userdata.sh

@@ -1,5 +1,5 @@
 #!/bin/bash
-# App instance user-data: triple-purpose responder for LB E2E tests.
+# App instance user-data: multi-purpose responder for LB E2E tests.
 #   HTTP:80   serves JSON {"instance_id": "<hostname>"} for ALB suites
 #   TCP:9000  echoes <hostname> for NLB TCP suites (also UDP-TG TCP health check)
 #   UDP:9001  echoes <hostname> for NLB UDP suites
@@ -9,6 +9,7 @@ mkdir -p /tmp/httpd/alpha /tmp/httpd/beta && cd /tmp/httpd
 echo "{\"instance_id\": \"${INSTANCE_ID}\"}" > index.html
 echo "{\"instance_id\": \"${INSTANCE_ID}\"}" > alpha/index.html
 echo "{\"instance_id\": \"${INSTANCE_ID}\"}" > beta/index.html
+echo ok > hc.html
 nohup python3 -m http.server 80 --bind 0.0.0.0 > /dev/null 2>&1 &
 
 cat > /tmp/tcp_echo.py << 'PYEOF'
@@ -30,3 +31,29 @@ socketserver.UDPServer.allow_reuse_address = True
 socketserver.UDPServer(("0.0.0.0", 9001), Handler).serve_forever()
 PYEOF
 nohup python3 /tmp/udp_echo.py > /dev/null 2>&1 &
+
+# TCP:8081 health switch the e2e driver flips: /health is 200 until /fail is
+# requested, then 500 until /ok, and /fail also removes :80/hc.html (404) until
+# /ok restores it. Data ports keep accepting.
+cat > /tmp/health.py << 'PYEOF'
+import http.server, os
+FLAG = "/tmp/health-fail"
+HC = "/tmp/httpd/hc.html"
+class Handler(http.server.BaseHTTPRequestHandler):
+    def do_GET(self):
+        if self.path == "/fail":
+            open(FLAG, "w").close()
+            if os.path.exists(HC):
+                os.remove(HC)
+        elif self.path == "/ok":
+            if os.path.exists(FLAG):
+                os.remove(FLAG)
+            open(HC, "w").write("ok\n")
+        code = 500 if self.path == "/health" and os.path.exists(FLAG) else 200
+        self.send_response(code); self.end_headers()
+        self.wfile.write(b"ok\n")
+    def log_message(self, *a, **kw):
+        pass
+http.server.ThreadingHTTPServer(("0.0.0.0", 8081), Handler).serve_forever()
+PYEOF
+nohup python3 /tmp/health.py > /dev/null 2>&1 &