fix: keep surviving flows on a failed IMDS repair and repair guests concurrently

Clear a tap's flows only once its endpoint is set up, so a repair that fails
after a kernel-module reload no longer takes down a guest that still had
connectivity. Repair up to 8 datapaths at once, since an ovs-vswitchd restart
drops every guest together. Test that teardown holds the ENI lock across
detach and tap cleanup, and count only the test guest's flows in the e2e.

docs/identity/imds/README.md

@@ -286,6 +286,7 @@ Useful background for host-side troubleshooting:
 - The IMDS HTTP server runs inside the `spinifex-vpcd` service on each host, listening on `169.254.169.254:80` with one listener per instance's primary interface — attribution is structural, with no source-IP trust.
 - A freshly launched guest's metadata service comes up within a few seconds of launch; cloud-init's built-in retries absorb this window.
 - Session tokens and cached role credentials are not persisted. A vpcd restart drops them; SDKs and cloud-init transparently reissue. Datapath state on the host survives service restarts.
+- An Open vSwitch restart, upgrade or crash drops every `br-imds` flow, which takes the guest's whole network down, not only IMDS. `spinifex-daemon` checks each running guest's datapath every 10 seconds and re-installs what was lost.
 
 ## Limits and Defaults
 
@@ -362,6 +363,15 @@ ovs-vsctl list-ports br-imds          # per-instance endpoint + patch ports
 ovs-vsctl list-ports br-int | grep imi-
 ```
 
+3. If the guest has lost all networking, not just IMDS, check whether `br-imds` still has its ports but no flows:
+
+```bash
+sudo ovs-ofctl dump-flows br-imds | grep -c cookie=   # 7 per running instance
+journalctl -u spinifex-daemon | grep 'IMDS: per-tap datapath'
+```
+
+That is the state an Open vSwitch restart leaves behind. The daemon repairs it within 10 seconds and logs `IMDS: per-tap datapath was lost; re-installed`. A count that stays at 0 with `IMDS: per-tap datapath repair failed` in the log names the step that failed.
+
 ### cloud-init Did Not Apply SSH Key or User Data
 
 cloud-init sources both from IMDS at first boot. Check `public-keys/0/openssh-key` and `user-data` respond from inside the guest (see [Using IMDS from a Guest](#using-imds-from-a-guest)), and review `/var/log/cloud-init.log` in the guest. A 404 on `public-keys/` means the instance was launched without `--key-name` or the key pair was since deleted.

docs/testing/e2e-coverage.md

@@ -19,7 +19,7 @@ The nightly permutation sets are deliberately narrower than the full ones: those
 
 ### `single` — the broad single-node EC2/EBS/VPC surface
 
-The largest suite, 25 top-level tests. Notable entry points:
+The largest suite, 27 top-level tests. Notable entry points:
 
 - **`TestVPCEgressPaths`** — **the authority on subnet egress semantics.** One VPC with a public and a private subnet, sharing two guests across four stages: `PublicSubnetEgress` (IGW egress works), `RouteBeforeSubnet` (a regression guard for an IGW route installed on the main route table before the subnet exists), the NAT Gateway rounds, and `EIPFlip` (associating an Elastic IP onto a running instance).
   - **`NATBaseline` hard-fails if a private-subnet guest reaches the internet with no NAT Gateway.** `NATGatewayUp` then proves egress appears and `NATGatewayDown` that it disappears. `SPINIFEX_VPCEGRESS_NAT_ROUNDS` repeats the up/down cycle for a soak run.
@@ -27,6 +27,7 @@ The largest suite, 25 top-level tests. Notable entry points:
 - `TestSGPolicyDatapath`, `TestSGReachabilityPolicy` — security groups as an actual datapath, not just API state.
 - `TestWANEgressSMTPBlock` — egress port blocking.
 - `TestInstanceMetadata` — IMDS from inside a guest.
+- **`TestIMDSDatapathSurvivesVswitchdRestart`** restarts `ovs-vswitchd` under a guest with an Elastic IP, once through systemd and once with SIGKILL. After each, the guest must answer over its EIP and fetch its instance ID through IMDSv2 within 60 seconds, and `br-imds` must hold all 7 of the guest's own flows. It is sequential because it takes the whole node's datapath down.
 - `TestStopStart`, `TestEarlyRebootLiveness`, `TestGuestChurnDurability`, `TestSnapshotLifecycle`, `TestSnapshotBackedLaunch`, `TestCreateImage`, `TestVolumeLifecycle`, `TestSpotInstanceLifecycle`, `TestLaunchTemplateBoot`.
 - `TestNegativeErrorPaths` — the `8a`–`8t` AWS error-shape cases (malformed AMI ID, invalid instance type, volume in use, detach-root forbidden, unknown or malformed instance IDs on the lifecycle verbs, and so on).
 - **`TestKVReplication`** — the single-node half of the KV replication policy, and it is not the multi-node assertion with a smaller number in it. On one node one replica per bucket is the correct answer rather than the defect, so what it proves is the opposite: that nothing declared a replica count this deployment cannot place, that `spx admin kv replicas` exits 0 and reports `WANT 1`, and that `--repair` run twice against a cluster with nothing to repair changes no bucket's replica count and leaves every bucket serving. The multi-node suites cannot cover any of that, because on three nodes the unplaceable count is the right one. It lives here rather than in its own suite so it runs in every single-node nightly cell, across all three host and network permutations.

spinifex/daemon/daemon.go

@@ -2307,6 +2307,9 @@ func (d *Daemon) startCluster() error {
 	d.startRecordRepair()
 	d.startInstanceRecovery()
 	d.vmMgr.StartPendingWatchdog(d.ctx)
+	// ovs-vswitchd keeps br-imds flows only in memory and nothing else restores
+	// them, so a restart would leave every guest on this node without a network.
+	d.vmMgr.StartIMDSDatapathReconcile(d.ctx)
 
 	// Reality→desired GC backstop (ADR-0003 §3): finish teardown interrupted by
 	// a node-down mid-cascade and purge completed terminated records. The volume

spinifex/daemon/daemon_mgmt_ip_test.go

@@ -639,6 +639,10 @@ func (p *recordingPlumber) CleanupTap(name string) error {
 func (p *recordingPlumber) EnsureIMDSDatapathBridge() error         { return nil }
 func (p *recordingPlumber) AttachIMDSDatapath(_, _, _ string) error { return nil }
 func (p *recordingPlumber) DetachIMDSDatapath(string) error         { return nil }
+func (p *recordingPlumber) IMDSDatapathStates([]string) (map[string]vm.IMDSDatapathState, error) {
+	return nil, nil
+}
+func (p *recordingPlumber) RepairIMDSDatapath(_, _, _ string) error { return nil }
 func (p *recordingPlumber) EnsureVPCHostPort(_, _, _ string) error  { return nil }
 func (p *recordingPlumber) RemoveVPCHostPort(string) error          { return nil }
 

spinifex/domains/network/host/imds_attach.go

@@ -23,6 +23,19 @@ func (p *OVSPlumber) DetachIMDSDatapath(eniID string) error {
 	return removeIMDSDatapath(context.Background(), NewExecRunner(), imdsDetachSpec(eniID))
 }
 
+// IMDSDatapathStates reports what survives of each primary ENI's datapath, so
+// the daemon can re-install one that an ovs-vswitchd restart or a kernel-module
+// reload took down.
+func (p *OVSPlumber) IMDSDatapathStates(eniIDs []string) (map[string]vm.IMDSDatapathState, error) {
+	return IMDSDatapathStates(context.Background(), NewExecRunner(), eniIDs)
+}
+
+// RepairIMDSDatapath re-installs a running guest's datapath. It returns
+// vm.ErrIMDSTapDetached, changing nothing, once the tap has left br-imds.
+func (p *OVSPlumber) RepairIMDSDatapath(eniID, mac, subnetID string) error {
+	return repairIMDSDatapath(context.Background(), NewExecRunner(), imdsDatapathSpec(eniID, mac, subnetID))
+}
+
 // EnsureIMDSDatapathBridge idempotently creates the dedicated IMDS bridge. The
 // daemon calls it before SetupTap places a primary tap on br-imds, since OVS
 // cannot add a port to a bridge that does not exist yet.
@@ -61,7 +74,7 @@ func imdsDetachSpec(eniID string) IMDSTapDatapath {
 // endpoint, patch, demux/egress flows, and reply routing. Stale flows are cleared
 // once here (all installers share the tap's cookie) before any flow is added.
 //
-// Bridge, cookie clear, endpoint, and patch are connectivity-critical (error bare):
+// Bridge, endpoint, cookie clear, and patch are connectivity-critical (error bare):
 // ListIMDSTaps advertises the tap off the patch and vpcd binds the endpoint, so the
 // endpoint must exist whenever the patch does, else vpcd binds a missing device every
 // reconcile forever. The flows and reply routing only serve, so theirs wrap
@@ -77,10 +90,12 @@ func installIMDSDatapath(ctx context.Context, r Runner, d IMDSTapDatapath) error
 	// The INPUT accept and the remap DNAT are wildcarded over every endpoint and
 	// installed once by vpcd at start. They are not attempted here: the daemon
 	// has no iptables grant and no CAP_NET_ADMIN, so trying aborts every launch.
-	if err := clearIMDSFlowsByCookie(ctx, r, imdsFlowCookie(d.Endpoint)); err != nil {
+	if err := ensureIMDSEndpoint(ctx, r, d); err != nil {
 		return err
 	}
-	if err := ensureIMDSEndpoint(ctx, r, d); err != nil {
+	// Cleared after the endpoint so a failed endpoint step leaves a repaired
+	// guest's surviving flows, and with them its connectivity, in place.
+	if err := clearIMDSFlowsByCookie(ctx, r, imdsFlowCookie(d.Endpoint)); err != nil {
 		return err
 	}
 	if err := installTapPatch(ctx, r, d); err != nil {

spinifex/domains/network/host/imds_bridge.go

@@ -7,8 +7,8 @@ import (
 )
 
 // IMDSBridge is the dedicated OVS bridge carrying the per-tap IMDS / VPC-DNS flows.
-// ovn-controller flushes foreign br-int flows wholesale on restart; a bridge absent
-// from ovn-bridge-mappings and not named br-int is never touched, so flows survive.
+// ovn-controller flushes foreign br-int flows on restart and never touches this
+// bridge. An ovs-vswitchd restart still drops its flows; the daemon re-installs them.
 const IMDSBridge = "br-imds"
 
 // imdsCookiePrefix tags every flow the IMDS datapath installs on IMDSBridge.

spinifex/domains/network/host/imds_drift.go

@@ -0,0 +1,98 @@
+package host
+
+import (
+	"bufio"
+	"bytes"
+	"context"
+	"fmt"
+	"os"
+	"path/filepath"
+	"slices"
+	"strconv"
+	"strings"
+
+	"github.com/mulgadc/spinifex/spinifex/runtime/compute/vm"
+)
+
+// IMDSTapFlowCount is how many flows installIMDSDatapath installs under a tap's
+// cookie: an ARP responder and an ingress demux per captured address, the
+// egress flow, and the two tap<->patch forward flows.
+const IMDSTapFlowCount = 7
+
+// IMDSFlowCookie is the cookie every br-imds flow of eniID's datapath carries.
+func IMDSFlowCookie(eniID string) string {
+	return imdsFlowCookie(IMDSEndpointName(eniID))
+}
+
+// endpointSysctlDir is where the kernel exposes per-device IPv4 settings.
+var endpointSysctlDir = "/proc/sys/net/ipv4/conf"
+
+// IMDSDatapathStates reports what survives of each primary ENI's datapath. An
+// ovs-vswitchd restart drops every flow. A kernel-module reload re-creates the
+// endpoint, which resets its sysctls and drops its reply rule and route.
+func IMDSDatapathStates(ctx context.Context, r Runner, eniIDs []string) (map[string]vm.IMDSDatapathState, error) {
+	flows, err := r.Run(ctx, "ovs-ofctl", "--no-stats", "dump-flows", IMDSBridge)
+	if err != nil {
+		return nil, fmt.Errorf("dump %s flows: %w", IMDSBridge, err)
+	}
+	byCookie, err := countFlowsByCookie(flows)
+	if err != nil {
+		return nil, err
+	}
+	states := make(map[string]vm.IMDSDatapathState, len(eniIDs))
+	for _, eniID := range eniIDs {
+		endpoint := IMDSEndpointName(eniID)
+		cookie, err := strconv.ParseUint(imdsFlowCookie(endpoint), 0, 64)
+		if err != nil {
+			return nil, fmt.Errorf("parse IMDS cookie for %s: %w", eniID, err)
+		}
+		states[eniID] = vm.IMDSDatapathState{
+			FlowsFound:    byCookie[cookie],
+			FlowsExpected: IMDSTapFlowCount,
+			EndpointReady: endpointConfigured(endpoint),
+		}
+	}
+	return states, nil
+}
+
+// countFlowsByCookie tallies `ovs-ofctl dump-flows` lines by cookie. A flow
+// with a zero cookie is printed without the field and is not counted.
+func countFlowsByCookie(out []byte) (map[uint64]int, error) {
+	counts := map[uint64]int{}
+	sc := bufio.NewScanner(bytes.NewReader(out))
+	for sc.Scan() {
+		rest, ok := strings.CutPrefix(strings.TrimSpace(sc.Text()), "cookie=")
+		if !ok {
+			continue
+		}
+		val, _, _ := strings.Cut(rest, ",")
+		cookie, err := strconv.ParseUint(val, 0, 64)
+		if err != nil {
+			return nil, fmt.Errorf("parse flow cookie %q: %w", val, err)
+		}
+		counts[cookie]++
+	}
+	return counts, sc.Err()
+}
+
+// endpointConfigured reports whether endpoint carries the accept_local the
+// install sets. Read from /proc rather than through ip, which the daemon can
+// only run under sudo; the device's reply rule and route go with the setting.
+func endpointConfigured(endpoint string) bool {
+	val, err := os.ReadFile(filepath.Join(endpointSysctlDir, endpoint, "accept_local"))
+	return err == nil && strings.TrimSpace(string(val)) == "1"
+}
+
+// repairIMDSDatapath re-installs a tap's datapath after part of it was lost.
+// It refuses a tap that is no longer on br-imds: installing would re-create the
+// patch and endpoint of a guest being torn down, which vpcd and OVN then claim.
+func repairIMDSDatapath(ctx context.Context, r Runner, d IMDSTapDatapath) error {
+	out, err := r.Run(ctx, "ovs-vsctl", "list-ports", IMDSBridge)
+	if err != nil {
+		return fmt.Errorf("list %s ports: %w", IMDSBridge, err)
+	}
+	if !slices.Contains(strings.Fields(string(out)), d.Tap) {
+		return vm.ErrIMDSTapDetached
+	}
+	return installIMDSDatapath(ctx, r, d)
+}

spinifex/domains/network/host/imds_drift_test.go

@@ -0,0 +1,207 @@
+package host
+
+import (
+	"context"
+	"errors"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"github.com/mulgadc/spinifex/spinifex/runtime/compute/vm"
+	hostcommand "github.com/mulgadc/spinifex/spinifex/runtime/host/command"
+)
+
+// dumpFlowLines renders n flows under eniID's cookie as `ovs-ofctl --no-stats
+// dump-flows` prints them.
+func dumpFlowLines(eniID string, n int) string {
+	var b strings.Builder
+	for range n {
+		b.WriteString(" cookie=" + imdsFlowCookie(IMDSEndpointName(eniID)) + ", priority=100,in_port=1 actions=output:3\n")
+	}
+	return b.String()
+}
+
+// withEndpointSysctls points endpointSysctlDir at a temp tree in which each
+// listed endpoint has accept_local set to the given value.
+func withEndpointSysctls(t *testing.T, acceptLocal map[string]string) {
+	t.Helper()
+	dir := t.TempDir()
+	for endpoint, val := range acceptLocal {
+		if err := os.MkdirAll(filepath.Join(dir, endpoint), 0o755); err != nil {
+			t.Fatal(err)
+		}
+		if err := os.WriteFile(filepath.Join(dir, endpoint, "accept_local"), []byte(val+"\n"), 0o644); err != nil {
+			t.Fatal(err)
+		}
+	}
+	orig := endpointSysctlDir
+	endpointSysctlDir = dir
+	t.Cleanup(func() { endpointSysctlDir = orig })
+}
+
+func TestIMDSDatapathStates(t *testing.T) {
+	const (
+		intact     = "eni-0intact"
+		partial    = "eni-0partial"
+		flushed    = "eni-0flushed"
+		reloaded   = "eni-0reloaded"
+		noEndpoint = "eni-0noendpoint"
+		departed   = "eni-0departed"
+	)
+	dump := "NXST_FLOW reply (xid=0x4):\n" +
+		dumpFlowLines(intact, IMDSTapFlowCount) +
+		dumpFlowLines(partial, 3) +
+		dumpFlowLines(reloaded, IMDSTapFlowCount) +
+		dumpFlowLines(noEndpoint, IMDSTapFlowCount) +
+		// A guest torn down on this node leaves no running ENI to claim these.
+		dumpFlowLines(departed, IMDSTapFlowCount) +
+		// A zero-cookie flow is printed without the field.
+		" priority=0 actions=drop\n"
+
+	s := newStubRunner()
+	s.expect("ovs-ofctl --no-stats dump-flows "+IMDSBridge, []byte(dump), nil)
+	withEndpointSysctls(t, map[string]string{
+		IMDSEndpointName(intact):   "1",
+		IMDSEndpointName(partial):  "1",
+		IMDSEndpointName(flushed):  "1",
+		IMDSEndpointName(reloaded): "0",
+	})
+
+	got, err := IMDSDatapathStates(context.Background(), s, []string{intact, partial, flushed, reloaded, noEndpoint})
+	if err != nil {
+		t.Fatalf("IMDSDatapathStates: %v", err)
+	}
+	want := map[string]vm.IMDSDatapathState{
+		intact:     {FlowsFound: IMDSTapFlowCount, FlowsExpected: IMDSTapFlowCount, EndpointReady: true},
+		partial:    {FlowsFound: 3, FlowsExpected: IMDSTapFlowCount, EndpointReady: true},
+		flushed:    {FlowsFound: 0, FlowsExpected: IMDSTapFlowCount, EndpointReady: true},
+		reloaded:   {FlowsFound: IMDSTapFlowCount, FlowsExpected: IMDSTapFlowCount, EndpointReady: false},
+		noEndpoint: {FlowsFound: IMDSTapFlowCount, FlowsExpected: IMDSTapFlowCount, EndpointReady: false},
+	}
+	if len(got) != len(want) {
+		t.Errorf("got %d states, want %d: %v", len(got), len(want), got)
+	}
+	for eni, w := range want {
+		if got[eni] != w {
+			t.Errorf("%s: got %+v, want %+v", eni, got[eni], w)
+		}
+	}
+	if !got[intact].Intact() {
+		t.Errorf("a complete datapath must read as intact: %+v", got[intact])
+	}
+	for _, eni := range []string{partial, flushed, reloaded, noEndpoint} {
+		if got[eni].Intact() {
+			t.Errorf("%s must not read as intact: %+v", eni, got[eni])
+		}
+	}
+}
+
+// A failed dump must surface, never read as "every ENI lost its flows" and
+// trigger a repair of every guest on the node.
+func TestIMDSDatapathStates_DumpFailure(t *testing.T) {
+	s := newStubRunner()
+	s.expect("ovs-ofctl", nil, errors.New("br-imds.mgmt: connection refused"))
+	if got, err := IMDSDatapathStates(context.Background(), s, []string{"eni-0abc"}); err == nil {
+		t.Fatalf("expected the dump failure to surface, got %v", got)
+	}
+}
+
+func TestCountFlowsByCookie_MalformedCookie(t *testing.T) {
+	if _, err := countFlowsByCookie([]byte(" cookie=0xzz, priority=1 actions=drop\n")); err == nil {
+		t.Fatal("expected a malformed cookie to be an error")
+	}
+}
+
+// IMDSTapFlowCount is what the drift check compares against, so it must match
+// what an install actually adds. A new flow without a bumped count would make
+// every healthy guest read as drifted and get re-installed every pass.
+func TestIMDSTapFlowCount_MatchesInstall(t *testing.T) {
+	s := newStubRunner()
+	s.expect("ovs-vsctl", nil, nil)
+	s.expect("ip", nil, nil)
+	s.expect(hostcommand.EndpointSysctlHelper, nil, nil)
+	s.expect("ovs-ofctl", nil, nil)
+
+	d := imdsDatapathSpec("eni-0abc1234", "02:00:00:00:01:05", "subnet-0fedcba9")
+	if err := installIMDSDatapath(context.Background(), s, d); err != nil {
+		t.Fatalf("installIMDSDatapath: %v", err)
+	}
+	cookie := imdsFlowCookie(d.Endpoint)
+	if n := countCalls(s.calls, "ovs-ofctl add-flow "+IMDSBridge+" cookie="+cookie+","); n != IMDSTapFlowCount {
+		t.Errorf("install added %d flows under the tap's cookie, IMDSTapFlowCount is %d", n, IMDSTapFlowCount)
+	}
+}
+
+func TestRepairIMDSDatapath(t *testing.T) {
+	const eniID = "eni-0abc1234"
+	d := imdsDatapathSpec(eniID, "02:00:00:00:01:05", "subnet-0fedcba9")
+	cookie := imdsFlowCookie(d.Endpoint)
+
+	t.Run("tap on br-imds is re-installed", func(t *testing.T) {
+		s := newStubRunner()
+		s.expect("ovs-vsctl list-ports "+IMDSBridge, []byte(d.Endpoint+"\n"+d.PatchIMDS+"\n"+d.Tap+"\n"), nil)
+		s.expect("ovs-vsctl --may-exist", nil, nil)
+		s.expect("ovs-vsctl set", nil, nil)
+		s.expect("ip", nil, nil)
+		s.expect(hostcommand.EndpointSysctlHelper, nil, nil)
+		s.expect("ovs-ofctl", nil, nil)
+
+		if err := repairIMDSDatapath(context.Background(), s, d); err != nil {
+			t.Fatalf("repairIMDSDatapath: %v", err)
+		}
+		if n := countCalls(s.calls, "ovs-ofctl add-flow "+IMDSBridge+" cookie="+cookie+","); n != IMDSTapFlowCount {
+			t.Errorf("repair added %d flows, want %d", n, IMDSTapFlowCount)
+		}
+		if !s.called("ip rule add oif " + d.Endpoint) {
+			t.Errorf("repair must restore the reply rule a kmod reload drops; calls: %v", s.calls)
+		}
+	})
+
+	// A tap gone from br-imds belongs to a guest being torn down. Re-creating
+	// its patch would leave an imi- port that vpcd binds and OVN claims.
+	t.Run("tap gone from br-imds is left alone", func(t *testing.T) {
+		s := newStubRunner()
+		s.expect("ovs-vsctl list-ports "+IMDSBridge, []byte(d.Endpoint+"\n"), nil)
+
+		err := repairIMDSDatapath(context.Background(), s, d)
+		if !errors.Is(err, vm.ErrIMDSTapDetached) {
+			t.Fatalf("want ErrIMDSTapDetached, got %v", err)
+		}
+		if len(s.calls) != 1 {
+			t.Errorf("a detached tap must not be touched; calls: %v", s.calls)
+		}
+	})
+
+	// After a kmod reload the flows survive and only the endpoint is lost; a
+	// repair that fails on the endpoint must not take the flows with it.
+	t.Run("endpoint failure keeps the surviving flows", func(t *testing.T) {
+		s := newStubRunner()
+		s.expect("ovs-vsctl list-ports "+IMDSBridge, []byte(d.Tap+"\n"), nil)
+		s.expect("ovs-vsctl --may-exist add-br", nil, nil)
+		s.expect("ovs-vsctl set", nil, nil)
+		s.expect("ip link set "+IMDSBridge, nil, nil)
+		s.expect("ovs-vsctl --may-exist add-port "+IMDSBridge+" "+d.Endpoint, nil, errors.New("ovsdb busy"))
+		s.expect("ovs-ofctl", nil, nil)
+
+		if err := repairIMDSDatapath(context.Background(), s, d); err == nil {
+			t.Fatal("want the endpoint failure to surface")
+		}
+		if s.called("ovs-ofctl del-flows") {
+			t.Errorf("flows cleared before the endpoint was ready; calls: %v", s.calls)
+		}
+	})
+
+	t.Run("port list failure installs nothing", func(t *testing.T) {
+		s := newStubRunner()
+		s.expect("ovs-vsctl list-ports", nil, errors.New("ovsdb unreachable"))
+
+		err := repairIMDSDatapath(context.Background(), s, d)
+		if err == nil || errors.Is(err, vm.ErrIMDSTapDetached) {
+			t.Fatalf("want a plain error, got %v", err)
+		}
+		if len(s.calls) != 1 {
+			t.Errorf("nothing may be installed without knowing the tap is attached; calls: %v", s.calls)
+		}
+	})
+}

spinifex/runtime/compute/vm/imds_reconcile.go

@@ -0,0 +1,198 @@
+package vm
+
+import (
+	"context"
+	"errors"
+	"log/slog"
+	"sync"
+	"time"
+
+	"github.com/aws/aws-sdk-go/aws"
+)
+
+// IMDSDatapathReconcileInterval is how often running guests' IMDS datapaths are
+// checked against what was installed. It bounds how long a guest stays dark
+// after ovs-vswitchd restarts. A var so tests can shorten it.
+var IMDSDatapathReconcileInterval = 10 * time.Second
+
+// imdsRepairConcurrency bounds how many datapaths one pass re-installs at once.
+// An ovs-vswitchd restart drops every guest's flows together, so serial repairs
+// would keep the last guest dark for the sum of everyone else's install.
+const imdsRepairConcurrency = 8
+
+// imdsDatapathTarget is a running guest whose primary ENI has an IMDS datapath,
+// copied out of the manager lock with everything a re-install needs.
+type imdsDatapathTarget struct {
+	instanceID string
+	eniID      string
+	mac        string
+	subnetID   string
+}
+
+// StartIMDSDatapathReconcile re-installs the IMDS datapath of any running guest
+// that has lost part of it. br-imds is fail-mode=secure, so a guest whose flows
+// are gone has no network at all. Runs once now, then every interval.
+func (m *Manager) StartIMDSDatapathReconcile(ctx context.Context) {
+	if m.deps.NetworkPlumber == nil {
+		return
+	}
+	go func() {
+		m.reconcileIMDSDatapaths()
+		ticker := time.NewTicker(IMDSDatapathReconcileInterval)
+		defer ticker.Stop()
+		for {
+			select {
+			case <-ctx.Done():
+				return
+			case <-ticker.C:
+				m.reconcileIMDSDatapaths()
+			}
+		}
+	}()
+}
+
+// reconcileIMDSDatapaths runs one pass: read every running guest's datapath in
+// one query, then repair each one that is not intact.
+func (m *Manager) reconcileIMDSDatapaths() {
+	targets := m.imdsDatapathTargets()
+	if len(targets) == 0 {
+		return
+	}
+	eniIDs := make([]string, len(targets))
+	for i, t := range targets {
+		eniIDs[i] = t.eniID
+	}
+	states, err := m.deps.NetworkPlumber.IMDSDatapathStates(eniIDs)
+	if err != nil {
+		slog.Warn("IMDS: datapath state read failed; skipping reconcile pass", "err", err)
+		return
+	}
+	sem := make(chan struct{}, imdsRepairConcurrency)
+	var wg sync.WaitGroup
+	for _, t := range imdsDatapathsToRepair(targets, states) {
+		sem <- struct{}{}
+		wg.Go(func() {
+			defer func() { <-sem }()
+			m.repairIMDSDatapath(t, states[t.eniID])
+		})
+	}
+	wg.Wait()
+}
+
+// imdsDatapathTargets returns the running guests attachPrimaryIMDSDatapath
+// gave a datapath: those with a primary ENI and a subnet.
+func (m *Manager) imdsDatapathTargets() []imdsDatapathTarget {
+	var targets []imdsDatapathTarget
+	m.View(func(vms map[string]*VM) {
+		for _, v := range vms {
+			if t, ok := imdsTargetOf(v); ok {
+				targets = append(targets, t)
+			}
+		}
+	})
+	return targets
+}
+
+// imdsTargetOf returns v as a repair target, or false if it should have no
+// datapath right now. The caller holds the manager lock.
+func imdsTargetOf(v *VM) (imdsDatapathTarget, bool) {
+	if v.Status != StateRunning || v.ENIId == "" || v.Instance == nil {
+		return imdsDatapathTarget{}, false
+	}
+	subnetID := aws.StringValue(v.Instance.SubnetId)
+	if subnetID == "" {
+		return imdsDatapathTarget{}, false
+	}
+	return imdsDatapathTarget{instanceID: v.ID, eniID: v.ENIId, mac: v.ENIMac, subnetID: subnetID}, true
+}
+
+// imdsDatapathsToRepair picks the targets whose datapath is not intact. An ENI
+// missing from states was not read, so it is left alone rather than repaired.
+func imdsDatapathsToRepair(targets []imdsDatapathTarget, states map[string]IMDSDatapathState) []imdsDatapathTarget {
+	var out []imdsDatapathTarget
+	for _, t := range targets {
+		if s, ok := states[t.eniID]; ok && !s.Intact() {
+			out = append(out, t)
+		}
+	}
+	return out
+}
+
+// repairIMDSDatapath re-installs one guest's datapath under its ENI lock, so it
+// cannot interleave with an attach or a teardown. The guest is re-checked under
+// the lock because a teardown may have finished since the pass read it.
+func (m *Manager) repairIMDSDatapath(t imdsDatapathTarget, s IMDSDatapathState) {
+	unlock := m.imdsLocks.lock(t.eniID)
+	defer unlock()
+
+	if !m.stillIMDSTarget(t) {
+		return
+	}
+	err := m.deps.NetworkPlumber.RepairIMDSDatapath(t.eniID, t.mac, t.subnetID)
+	switch {
+	case errors.Is(err, ErrIMDSTapDetached):
+		slog.Debug("IMDS: tap left br-imds before repair; skipping",
+			"instance", t.instanceID, "eni", t.eniID)
+	case err != nil:
+		slog.Error("IMDS: per-tap datapath repair failed; guest has no network until it succeeds",
+			"instance", t.instanceID, "eni", t.eniID, "flows_found", s.FlowsFound,
+			"flows_expected", s.FlowsExpected, "endpoint_ready", s.EndpointReady, "err", err)
+	default:
+		slog.Warn("IMDS: per-tap datapath was lost; re-installed",
+			"instance", t.instanceID, "eni", t.eniID, "flows_found", s.FlowsFound,
+			"flows_expected", s.FlowsExpected, "endpoint_ready", s.EndpointReady)
+	}
+}
+
+// stillIMDSTarget reports whether t is still the same running guest with the
+// same ENI, MAC and subnet the pass read.
+func (m *Manager) stillIMDSTarget(t imdsDatapathTarget) bool {
+	var ok bool
+	m.View(func(vms map[string]*VM) {
+		v, found := vms[t.instanceID]
+		if !found {
+			return
+		}
+		cur, isTarget := imdsTargetOf(v)
+		ok = isTarget && cur == t
+	})
+	return ok
+}
+
+// keyedMutex serialises work per key. Entries are dropped once unheld, so it
+// does not grow with every ENI the node has ever seen. The zero value is ready.
+type keyedMutex struct {
+	mu    sync.Mutex
+	locks map[string]*keyedLock
+}
+
+type keyedLock struct {
+	mu   sync.Mutex
+	refs int
+}
+
+// lock blocks until key is free and returns the function that releases it.
+func (k *keyedMutex) lock(key string) (unlock func()) {
+	k.mu.Lock()
+	if k.locks == nil {
+		k.locks = map[string]*keyedLock{}
+	}
+	l, ok := k.locks[key]
+	if !ok {
+		l = &keyedLock{}
+		k.locks[key] = l
+	}
+	l.refs++
+	k.mu.Unlock()
+
+	l.mu.Lock()
+	return func() {
+		l.mu.Unlock()
+		k.mu.Lock()
+		l.refs--
+		if l.refs == 0 {
+			delete(k.locks, key)
+		}
+		k.mu.Unlock()
+	}
+}

spinifex/runtime/compute/vm/imds_reconcile_test.go

@@ -0,0 +1,309 @@
+package vm
+
+import (
+	"errors"
+	"fmt"
+	"slices"
+	"sync"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/service/ec2"
+)
+
+func imdsTestVM(id, eni string, status InstanceState) *VM {
+	return &VM{
+		ID:       id,
+		Status:   status,
+		ENIId:    eni,
+		ENIMac:   "02:00:00:00:00:01",
+		Instance: &ec2.Instance{SubnetId: aws.String("subnet-0abc")},
+	}
+}
+
+var (
+	imdsIntact  = IMDSDatapathState{FlowsFound: 7, FlowsExpected: 7, EndpointReady: true}
+	imdsFlushed = IMDSDatapathState{FlowsFound: 0, FlowsExpected: 7, EndpointReady: true}
+)
+
+func TestIMDSTargetOf(t *testing.T) {
+	noSubnet := imdsTestVM("i-nosubnet", "eni-1", StateRunning)
+	noSubnet.Instance.SubnetId = nil
+	noInstance := imdsTestVM("i-noinstance", "eni-2", StateRunning)
+	noInstance.Instance = nil
+
+	cases := []struct {
+		name string
+		v    *VM
+		want bool
+	}{
+		{"running with ENI and subnet", imdsTestVM("i-run", "eni-0", StateRunning), true},
+		{"stopping", imdsTestVM("i-stop", "eni-0", StateStopping), false},
+		{"error after a crash", imdsTestVM("i-err", "eni-0", StateError), false},
+		{"pending mid-launch", imdsTestVM("i-pend", "eni-0", StatePending), false},
+		{"no primary ENI", imdsTestVM("i-noeni", "", StateRunning), false},
+		{"no subnet", noSubnet, false},
+		{"no instance record", noInstance, false},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			if _, got := imdsTargetOf(tc.v); got != tc.want {
+				t.Errorf("imdsTargetOf = %v, want %v", got, tc.want)
+			}
+		})
+	}
+}
+
+func TestIMDSDatapathsToRepair(t *testing.T) {
+	targets := []imdsDatapathTarget{
+		{instanceID: "i-intact", eniID: "eni-intact"},
+		{instanceID: "i-partial", eniID: "eni-partial"},
+		{instanceID: "i-flushed", eniID: "eni-flushed"},
+		{instanceID: "i-reloaded", eniID: "eni-reloaded"},
+		{instanceID: "i-unread", eniID: "eni-unread"},
+	}
+	states := map[string]IMDSDatapathState{
+		"eni-intact":   imdsIntact,
+		"eni-partial":  {FlowsFound: 3, FlowsExpected: 7, EndpointReady: true},
+		"eni-flushed":  imdsFlushed,
+		"eni-reloaded": {FlowsFound: 7, FlowsExpected: 7, EndpointReady: false},
+	}
+	got := imdsDatapathsToRepair(targets, states)
+	var ids []string
+	for _, t := range got {
+		ids = append(ids, t.eniID)
+	}
+	want := []string{"eni-partial", "eni-flushed", "eni-reloaded"}
+	if len(ids) != len(want) {
+		t.Fatalf("repair set = %v, want %v", ids, want)
+	}
+	for i := range want {
+		if ids[i] != want[i] {
+			t.Errorf("repair set = %v, want %v", ids, want)
+		}
+	}
+}
+
+// A re-install starts by clearing the tap's flows, so repairing an intact
+// datapath would cut a working guest's network every pass.
+func TestReconcileIMDSDatapaths_LeavesIntactGuestAlone(t *testing.T) {
+	plumber := &fakeNetworkPlumber{imdsStates: map[string]IMDSDatapathState{"eni-ok": imdsIntact}}
+	m := NewManagerWithDeps(Deps{NetworkPlumber: plumber})
+	m.Insert(imdsTestVM("i-ok", "eni-ok", StateRunning))
+
+	m.reconcileIMDSDatapaths()
+
+	if len(plumber.imdsRepairCalls) != 0 {
+		t.Errorf("intact datapath was repaired: %+v", plumber.imdsRepairCalls)
+	}
+}
+
+func TestReconcileIMDSDatapaths_RepairsLostDatapath(t *testing.T) {
+	plumber := &fakeNetworkPlumber{imdsStates: map[string]IMDSDatapathState{
+		"eni-ok":   imdsIntact,
+		"eni-lost": imdsFlushed,
+		"eni-stop": imdsFlushed,
+	}}
+	m := NewManagerWithDeps(Deps{NetworkPlumber: plumber})
+	m.Insert(imdsTestVM("i-ok", "eni-ok", StateRunning))
+	lost := imdsTestVM("i-lost", "eni-lost", StateRunning)
+	lost.ENIMac = "02:00:00:00:00:99"
+	lost.Instance.SubnetId = aws.String("subnet-0lost")
+	m.Insert(lost)
+	m.Insert(imdsTestVM("i-stop", "eni-stop", StateStopped))
+
+	m.reconcileIMDSDatapaths()
+
+	want := []imdsAttachCall{{eniID: "eni-lost", mac: "02:00:00:00:00:99", subnetID: "subnet-0lost"}}
+	if len(plumber.imdsRepairCalls) != 1 || plumber.imdsRepairCalls[0] != want[0] {
+		t.Errorf("repair calls = %+v, want %+v", plumber.imdsRepairCalls, want)
+	}
+}
+
+// A failed read says nothing about the datapaths, so it must not be taken as
+// every guest having lost its flows.
+func TestReconcileIMDSDatapaths_ReadFailureRepairsNothing(t *testing.T) {
+	plumber := &fakeNetworkPlumber{imdsStatesErr: errors.New("br-imds.mgmt: connection refused")}
+	m := NewManagerWithDeps(Deps{NetworkPlumber: plumber})
+	m.Insert(imdsTestVM("i-a", "eni-a", StateRunning))
+
+	m.reconcileIMDSDatapaths()
+
+	if len(plumber.imdsRepairCalls) != 0 {
+		t.Errorf("read failure triggered repairs: %+v", plumber.imdsRepairCalls)
+	}
+}
+
+// The pass reads its targets before it repairs them; a guest stopped or gone
+// in between must not have its datapath re-created.
+func TestRepairIMDSDatapath_RechecksGuestUnderLock(t *testing.T) {
+	cases := []struct {
+		name   string
+		mutate func(m *Manager)
+	}{
+		{"stopped", func(m *Manager) { m.UpdateState("i-a", func(v *VM) { v.Status = StateStopping }) }},
+		{"removed", func(m *Manager) { m.Delete("i-a") }},
+		{"ENI replaced", func(m *Manager) { m.UpdateState("i-a", func(v *VM) { v.ENIId = "eni-b" }) }},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			plumber := &fakeNetworkPlumber{}
+			m := NewManagerWithDeps(Deps{NetworkPlumber: plumber})
+			v := imdsTestVM("i-a", "eni-a", StateRunning)
+			m.Insert(v)
+			target, _ := imdsTargetOf(v)
+
+			tc.mutate(m)
+			m.repairIMDSDatapath(target, imdsFlushed)
+
+			if len(plumber.imdsRepairCalls) != 0 {
+				t.Errorf("repaired a guest no longer running with that ENI: %+v", plumber.imdsRepairCalls)
+			}
+		})
+	}
+}
+
+// Teardown holds the ENI lock across detach and tap cleanup; a repair must
+// wait for it rather than re-create what the teardown is removing.
+func TestRepairIMDSDatapath_WaitsForTeardownLock(t *testing.T) {
+	plumber := &fakeNetworkPlumber{}
+	m := NewManagerWithDeps(Deps{NetworkPlumber: plumber})
+	v := imdsTestVM("i-a", "eni-a", StateRunning)
+	m.Insert(v)
+	target, _ := imdsTargetOf(v)
+
+	unlock := m.imdsLocks.lock("eni-a")
+	done := make(chan struct{})
+	go func() {
+		m.repairIMDSDatapath(target, imdsFlushed)
+		close(done)
+	}()
+
+	select {
+	case <-done:
+		t.Fatal("repair ran while the ENI lock was held")
+	case <-time.After(50 * time.Millisecond):
+	}
+	unlock()
+	<-done
+	if len(plumber.imdsRepairCalls) != 1 {
+		t.Errorf("repair calls after unlock = %d, want 1", len(plumber.imdsRepairCalls))
+	}
+}
+
+// A repair landing between detach and tap cleanup would find the tap still on
+// br-imds and re-create the endpoint and patch of a guest being torn down.
+func TestCleanupTapDevices_HoldsENILockAcrossDetachAndCleanup(t *testing.T) {
+	inDetach, releaseDetach := make(chan struct{}), make(chan struct{})
+	inCleanup, releaseCleanup := make(chan struct{}), make(chan struct{})
+	plumber := &fakeNetworkPlumber{
+		detachHook:  func() { close(inDetach); <-releaseDetach },
+		cleanupHook: func() { close(inCleanup); <-releaseCleanup },
+	}
+	m := NewManagerWithDeps(Deps{NetworkPlumber: plumber})
+	v := imdsTestVM("i-a", "eni-a", StateRunning)
+	m.Insert(v)
+	target, _ := imdsTargetOf(v)
+
+	var wg sync.WaitGroup
+	wg.Go(func() { m.cleanupTapDevices(v) })
+	<-inDetach
+	wg.Go(func() { m.repairIMDSDatapath(target, imdsFlushed) })
+	for _, step := range []struct {
+		name    string
+		release chan struct{}
+		next    chan struct{}
+	}{{"detach", releaseDetach, inCleanup}, {"tap cleanup", releaseCleanup, nil}} {
+		time.Sleep(50 * time.Millisecond)
+		if n := plumber.repairCount(); n != 0 {
+			t.Fatalf("repair ran during %s: %d calls", step.name, n)
+		}
+		close(step.release)
+		if step.next != nil {
+			<-step.next
+		}
+	}
+	wg.Wait()
+
+	want := []string{"detach:eni-a", "cleanup:" + TapDeviceName("eni-a"), "repair:eni-a"}
+	if !slices.Equal(plumber.order, want) {
+		t.Errorf("call order = %v, want %v", plumber.order, want)
+	}
+}
+
+// An ovs-vswitchd restart drifts every guest at once; repairs must overlap so
+// the last guest is not dark for the sum of everyone else's install.
+func TestReconcileIMDSDatapaths_RepairsConcurrentlyUpToLimit(t *testing.T) {
+	const guests = imdsRepairConcurrency + 4
+	var inFlight atomic.Int32
+	var overLimit atomic.Bool
+	full, release := make(chan struct{}), make(chan struct{})
+	var fullOnce sync.Once
+	plumber := &fakeNetworkPlumber{
+		imdsStates: map[string]IMDSDatapathState{},
+		repairHook: func() {
+			n := inFlight.Add(1)
+			if n > imdsRepairConcurrency {
+				overLimit.Store(true)
+			}
+			if n == imdsRepairConcurrency {
+				fullOnce.Do(func() { close(full) })
+			}
+			<-release
+			inFlight.Add(-1)
+		},
+	}
+	m := NewManagerWithDeps(Deps{NetworkPlumber: plumber})
+	for i := range guests {
+		eni := fmt.Sprintf("eni-%d", i)
+		plumber.imdsStates[eni] = imdsFlushed
+		m.Insert(imdsTestVM(fmt.Sprintf("i-%d", i), eni, StateRunning))
+	}
+
+	done := make(chan struct{})
+	go func() {
+		m.reconcileIMDSDatapaths()
+		close(done)
+	}()
+	select {
+	case <-full:
+	case <-time.After(5 * time.Second):
+		t.Fatalf("repairs did not reach %d in flight: %d", imdsRepairConcurrency, inFlight.Load())
+	}
+	time.Sleep(50 * time.Millisecond)
+	if overLimit.Load() {
+		t.Errorf("more than %d repairs ran at once", imdsRepairConcurrency)
+	}
+	close(release)
+	<-done
+	if n := plumber.repairCount(); n != guests {
+		t.Errorf("repairs = %d, want %d", n, guests)
+	}
+}
+
+func TestRepairIMDSDatapath_DetachedTapIsNotAnError(t *testing.T) {
+	plumber := &fakeNetworkPlumber{imdsRepairErr: ErrIMDSTapDetached}
+	m := NewManagerWithDeps(Deps{NetworkPlumber: plumber})
+	v := imdsTestVM("i-a", "eni-a", StateRunning)
+	m.Insert(v)
+	target, _ := imdsTargetOf(v)
+
+	m.repairIMDSDatapath(target, imdsFlushed)
+
+	if len(plumber.imdsRepairCalls) != 1 {
+		t.Errorf("repair calls = %d, want 1", len(plumber.imdsRepairCalls))
+	}
+}
+
+func TestKeyedMutex_DropsUnheldKeys(t *testing.T) {
+	var k keyedMutex
+	unlockA := k.lock("a")
+	unlockB := k.lock("b")
+	unlockA()
+	unlockB()
+	if len(k.locks) != 0 {
+		t.Errorf("unheld keys retained: %d", len(k.locks))
+	}
+}

spinifex/runtime/compute/vm/lifecycle_test.go

@@ -1031,6 +1031,12 @@ func (p *scriptedNetworkPlumber) AttachIMDSDatapath(_, _, _ string) error { retu
 
 func (p *scriptedNetworkPlumber) DetachIMDSDatapath(_ string) error { return nil }
 
+func (p *scriptedNetworkPlumber) IMDSDatapathStates(_ []string) (map[string]IMDSDatapathState, error) {
+	return nil, nil
+}
+
+func (p *scriptedNetworkPlumber) RepairIMDSDatapath(_, _, _ string) error { return nil }
+
 func (p *scriptedNetworkPlumber) EnsureIMDSDatapathBridge() error { return nil }
 
 func (p *scriptedNetworkPlumber) EnsureVPCHostPort(_, _, _ string) error { return nil }

spinifex/runtime/compute/vm/manager.go

@@ -96,6 +96,10 @@ type Manager struct {
 	vms         map[string]*VM
 	deps        Deps
 	goroutineWg sync.WaitGroup
+
+	// imdsLocks serialises each primary ENI's IMDS datapath attach, repair and
+	// teardown, which run outside mu.
+	imdsLocks keyedMutex
 }
 
 // NewManager returns a Manager with no collaborators wired. Tests that only

spinifex/runtime/compute/vm/network_helpers.go

@@ -126,6 +126,8 @@ func (m *Manager) attachPrimaryIMDSDatapath(instance *VM) error {
 			"instance", instance.ID, "eni", instance.ENIId)
 		return nil
 	}
+	unlock := m.imdsLocks.lock(instance.ENIId)
+	defer unlock()
 	if err := m.deps.NetworkPlumber.AttachIMDSDatapath(instance.ENIId, instance.ENIMac, subnetID); err != nil {
 		slog.Error("IMDS: per-tap datapath install failed; aborting launch",
 			"instance", instance.ID, "eni", instance.ENIId, "err", err)

spinifex/runtime/compute/vm/network_helpers_test.go

@@ -5,6 +5,7 @@ import (
 	"fmt"
 	"reflect"
 	"strings"
+	"sync"
 	"testing"
 
 	"github.com/aws/aws-sdk-go/aws"
@@ -13,10 +14,19 @@ import (
 
 // fakeNetworkPlumber records calls so tests can assert per-spec behaviour.
 type fakeNetworkPlumber struct {
+	mu                  sync.Mutex
+	order               []string // cleanup, detach and repair calls, in order
+	detachHook          func()
+	cleanupHook         func()
+	repairHook          func()
 	setupCalls          []TapSpec
 	cleanupCalls        []string
 	imdsAttachCalls     []imdsAttachCall
 	imdsDetachCalls     []string
+	imdsRepairCalls     []imdsAttachCall
+	imdsStates          map[string]IMDSDatapathState
+	imdsStatesErr       error
+	imdsRepairErr       error
 	hostPortCalls       []hostPortCall
 	hostPortRemoveCalls []string
 	ensureBridgeCalls   int
@@ -49,7 +59,13 @@ func (p *fakeNetworkPlumber) SetupTap(spec TapSpec) error {
 }
 
 func (p *fakeNetworkPlumber) CleanupTap(name string) error {
+	if p.cleanupHook != nil {
+		p.cleanupHook()
+	}
+	p.mu.Lock()
+	defer p.mu.Unlock()
 	p.cleanupCalls = append(p.cleanupCalls, name)
+	p.order = append(p.order, "cleanup:"+name)
 	return p.cleanupErr
 }
 
@@ -59,10 +75,46 @@ func (p *fakeNetworkPlumber) AttachIMDSDatapath(eniID, mac, subnetID string) err
 }
 
 func (p *fakeNetworkPlumber) DetachIMDSDatapath(eniID string) error {
+	if p.detachHook != nil {
+		p.detachHook()
+	}
+	p.mu.Lock()
+	defer p.mu.Unlock()
 	p.imdsDetachCalls = append(p.imdsDetachCalls, eniID)
+	p.order = append(p.order, "detach:"+eniID)
 	return p.imdsDetachErr
 }
 
+func (p *fakeNetworkPlumber) IMDSDatapathStates(eniIDs []string) (map[string]IMDSDatapathState, error) {
+	if p.imdsStatesErr != nil {
+		return nil, p.imdsStatesErr
+	}
+	out := make(map[string]IMDSDatapathState, len(eniIDs))
+	for _, id := range eniIDs {
+		if s, ok := p.imdsStates[id]; ok {
+			out[id] = s
+		}
+	}
+	return out, nil
+}
+
+func (p *fakeNetworkPlumber) RepairIMDSDatapath(eniID, mac, subnetID string) error {
+	if p.repairHook != nil {
+		p.repairHook()
+	}
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	p.imdsRepairCalls = append(p.imdsRepairCalls, imdsAttachCall{eniID: eniID, mac: mac, subnetID: subnetID})
+	p.order = append(p.order, "repair:"+eniID)
+	return p.imdsRepairErr
+}
+
+func (p *fakeNetworkPlumber) repairCount() int {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	return len(p.imdsRepairCalls)
+}
+
 func (p *fakeNetworkPlumber) EnsureIMDSDatapathBridge() error {
 	p.ensureBridgeCalls++
 	return p.ensureBridgeErr

spinifex/runtime/compute/vm/network_plumber.go

@@ -11,6 +11,24 @@ import "errors"
 // was not set up and is equally fatal.
 var ErrIMDSServingDegraded = errors.New("imds: per-tap serving install failed (guest connectivity intact)")
 
+// ErrIMDSTapDetached is RepairIMDSDatapath declining a tap no longer on br-imds:
+// its guest is being torn down, and a repair would re-create ports it just lost.
+var ErrIMDSTapDetached = errors.New("imds: tap is no longer on the IMDS bridge")
+
+// IMDSDatapathState is what survives of a primary ENI's IMDS datapath: its
+// br-imds flows against the number installed, and whether its endpoint still
+// carries the settings the install gave it.
+type IMDSDatapathState struct {
+	FlowsFound    int
+	FlowsExpected int
+	EndpointReady bool
+}
+
+// Intact reports whether nothing the install put in place has been lost.
+func (s IMDSDatapathState) Intact() bool {
+	return s.FlowsFound >= s.FlowsExpected && s.EndpointReady
+}
+
 // NetworkPlumber handles tap device and OVS bridge operations. Defined in vm
 // so the manager avoids importing network/host. All methods must be idempotent.
 type NetworkPlumber interface {
@@ -32,6 +50,15 @@ type NetworkPlumber interface {
 	// so safe for an ENI that never had a datapath installed.
 	DetachIMDSDatapath(eniID string) error
 
+	// IMDSDatapathStates reports, per primary ENI, what survives of its IMDS
+	// datapath. One pass covers every ENI; an error means nothing was read.
+	IMDSDatapathStates(eniIDs []string) (map[string]IMDSDatapathState, error)
+
+	// RepairIMDSDatapath re-installs the IMDS datapath of a running guest whose
+	// state was lost. Returns ErrIMDSTapDetached, changing nothing, when the tap
+	// is no longer on br-imds.
+	RepairIMDSDatapath(eniID, mac, subnetID string) error
+
 	// EnsureVPCHostPort gives the host a routed presence in a managed VPC subnet
 	// by binding an ENI the daemon itself owns to an internal port on br-int.
 	// addr is CIDR notation carrying the ENI address at the SUBNET's prefix

spinifex/runtime/compute/vm/shutdown.go

@@ -730,11 +730,14 @@ func (m *Manager) shutdownQEMU(instance *VM) {
 func (m *Manager) cleanupTapDevices(instance *VM) {
 	if instance.ENIId != "" && m.deps.NetworkPlumber != nil {
 		// Detach the primary ENI's IMDS datapath before removing its tap, the
-		// inverse of the launch-time attach-after-SetupTap order.
+		// inverse of the launch-time attach-after-SetupTap order. Both run under
+		// the ENI's lock so a datapath repair cannot re-create what this removes.
+		unlock := m.imdsLocks.lock(instance.ENIId)
 		m.detachPrimaryIMDSDatapath(instance)
 		if err := m.deps.NetworkPlumber.CleanupTap(TapDeviceName(instance.ENIId)); err != nil {
 			slog.Warn("Failed to clean up tap device", "eni", instance.ENIId, "err", err)
 		}
+		unlock()
 		m.cleanupExtraENITaps(instance)
 	}
 

tests/e2e/single/tests_test.go

@@ -109,6 +109,10 @@ func TestGuestChurnDurability(t *testing.T) {
 	runGuestChurnDurability(t, requireSingleNodeFixture(t))
 }
 
+func TestIMDSDatapathSurvivesVswitchdRestart(t *testing.T) {
+	runIMDSDatapathSurvivesVswitchdRestart(t, requireSingleNodeFixture(t))
+}
+
 func TestSnapshotLifecycle(t *testing.T) {
 	runSnapshotLifecycle(t, requireSingleNodeFixture(t))
 }

tests/e2e/single/vswitchdrestart_test.go

@@ -0,0 +1,151 @@
+//go:build e2e
+
+package single
+
+import (
+	"context"
+	"fmt"
+	"os/exec"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/service/ec2"
+	"github.com/mulgadc/spinifex/spinifex/domains/network/host"
+	"github.com/mulgadc/spinifex/tests/e2e/harness"
+	"github.com/stretchr/testify/require"
+)
+
+// vswitchdRecoveryBudget bounds how long a guest may stay dark after
+// ovs-vswitchd restarts: one daemon reconcile interval, the re-install, and
+// ovn-controller re-pushing br-int.
+const vswitchdRecoveryBudget = 60 * time.Second
+
+// openvswitchUnits are the systemd units that run ovs-vswitchd, by distro
+// family: Debian/Ubuntu first, then RHEL/Oracle Linux.
+var openvswitchUnits = []string{"openvswitch-switch.service", "openvswitch.service"}
+
+// runIMDSDatapathSurvivesVswitchdRestart restarts ovs-vswitchd under a running
+// guest and requires the guest's EIP, its IMDS and its br-imds flows to come
+// back. ovs-vswitchd holds flows only in memory, and br-imds is fail-mode=secure,
+// so before the daemon re-installed them every guest on the node went dark.
+//
+// It takes the whole node's datapath down, so it must never run in parallel.
+func runIMDSDatapathSurvivesVswitchdRestart(t *testing.T, fix *Fixture) {
+	if !fix.PublicPool {
+		t.Skip("needs a pool of public addresses to allocate an Elastic IP from")
+	}
+	harness.SkipIfNoOVN(t)
+	harness.Phase(t, "Single — a guest keeps its network and IMDS across an ovs-vswitchd restart")
+
+	c := fix.AWS
+	unit := openvswitchUnit(t)
+	instType, _ := needInstanceTypeArch(t, fix)
+	keyName, keyPath := needKeyPair(t, fix)
+	amiID := needAMI(t, fix)
+	vpc := harness.EnsureDefaultVPC(t, fix.Harness)
+	require.NotEmpty(t, vpc.SGID, "default SG ID required")
+	harness.AuthorizeSSHIngress(t, c, vpc.SGID)
+
+	// Whatever happens below, the node must not be left without ovs-vswitchd.
+	t.Cleanup(func() {
+		_ = exec.Command("sudo", "-n", "systemctl", "start", unit).Run()
+	})
+
+	harness.Step(t, "launch guest")
+	instanceID := launchBaselineInstance(t, fix, amiID, instType, keyName, vpc.SubnetID, []string{vpc.SGID})
+
+	harness.Step(t, "allocate-address and associate it with %s", instanceID)
+	eipOut, err := c.EC2.AllocateAddress(&ec2.AllocateAddressInput{Domain: aws.String("vpc")}) // e2e:allow-create — the Elastic IP is the ingress path that must survive the restart.
+	require.NoError(t, err, "allocate-address")
+	allocID, eipIP := aws.StringValue(eipOut.AllocationId), aws.StringValue(eipOut.PublicIp)
+	require.NotEmpty(t, eipIP, "EIP PublicIp empty")
+	t.Cleanup(func() {
+		_, _ = c.EC2.ReleaseAddress(&ec2.ReleaseAddressInput{AllocationId: aws.String(allocID)})
+	})
+	assocOut, err := c.EC2.AssociateAddress(&ec2.AssociateAddressInput{
+		AllocationId: aws.String(allocID),
+		InstanceId:   aws.String(instanceID),
+	})
+	require.NoError(t, err, "associate-address")
+	t.Cleanup(func() {
+		_, _ = c.EC2.DisassociateAddress(&ec2.DisassociateAddressInput{AssociationId: assocOut.AssociationId})
+	})
+	if !sshHealth.TryReady(eipIP, 22, keyPath, harness.SSHReadyBudget) {
+		t.Fatalf("guest unreachable via EIP %s within %s", eipIP, harness.SSHReadyBudget)
+	}
+	tgt := harness.SSHTarget{User: "ubuntu", Host: eipIP, Port: 22, KeyPath: keyPath}
+
+	cookie := host.IMDSFlowCookie(primaryENI(t, harness.WaitForInstanceState(t, c, instanceID, "running")))
+	require.NoError(t, guestIMDSAnswers(t, tgt, instanceID), "IMDS before any restart")
+	require.Equal(t, host.IMDSTapFlowCount, guestIMDSFlowCount(t, cookie), "guest's br-imds flows before any restart")
+	harness.Detail(t, "eip", eipIP, "unit", unit, "cookie", cookie)
+
+	for _, disruption := range []struct {
+		name string
+		argv []string
+	}{
+		{"systemctl restart " + unit, []string{"systemctl", "restart", unit}},
+		// systemd restarts it; nothing runs the scripted flow save/restore.
+		{"SIGKILL ovs-vswitchd", []string{"pkill", "-9", "-x", "ovs-vswitchd"}},
+	} {
+		harness.Step(t, "%s", disruption.name)
+		start := time.Now()
+		out, err := exec.Command("sudo", append([]string{"-n"}, disruption.argv...)...).CombinedOutput()
+		require.NoErrorf(t, err, "%s: %s", disruption.name, strings.TrimSpace(string(out)))
+
+		// One condition, so the time it reports is when everything was back.
+		harness.EventuallyErr(t, func() error {
+			if err := guestIMDSAnswers(t, tgt, instanceID); err != nil {
+				return err
+			}
+			if n := guestIMDSFlowCount(t, cookie); n != host.IMDSTapFlowCount {
+				return fmt.Errorf("guest has %d br-imds flows, want %d", n, host.IMDSTapFlowCount)
+			}
+			return nil
+		}, vswitchdRecoveryBudget, 2*time.Second)
+		harness.Detail(t, "disruption", disruption.name, "recovered_ms", time.Since(start).Milliseconds())
+	}
+}
+
+// guestIMDSAnswers reaches the guest over its EIP and has it fetch its own
+// instance ID through IMDSv2, which crosses both halves of the br-imds datapath.
+func guestIMDSAnswers(t *testing.T, tgt harness.SSHTarget, instanceID string) error {
+	t.Helper()
+	ctx, cancel := context.WithTimeout(t.Context(), 15*time.Second)
+	defer cancel()
+	const cmd = `T=$(curl -sf -m3 -X PUT -H "X-aws-ec2-metadata-token-ttl-seconds: 60" http://169.254.169.254/latest/api/token) && ` +
+		`curl -sf -m3 -H "X-aws-ec2-metadata-token: $T" http://169.254.169.254/latest/meta-data/instance-id`
+	out, err := harness.RunGuestSSH(ctx, tgt, cmd)
+	if err != nil {
+		return err
+	}
+	if got := strings.TrimSpace(string(out)); got != instanceID {
+		return fmt.Errorf("IMDS instance-id = %q, want %q", got, instanceID)
+	}
+	return nil
+}
+
+// guestIMDSFlowCount returns how many br-imds flows carry the guest's cookie, or
+// -1 while ovs-vswitchd is not answering. Other guests' teardowns cannot skew it.
+func guestIMDSFlowCount(t *testing.T, cookie string) int {
+	t.Helper()
+	out, err := exec.Command("sudo", "-n", "ovs-ofctl", "--no-stats", "dump-flows", host.IMDSBridge, "cookie="+cookie+"/-1").Output()
+	if err != nil {
+		return -1
+	}
+	return strings.Count(string(out), "cookie=")
+}
+
+// openvswitchUnit returns the unit that runs ovs-vswitchd on this host.
+func openvswitchUnit(t *testing.T) string {
+	t.Helper()
+	for _, unit := range openvswitchUnits {
+		if exec.Command("systemctl", "cat", unit).Run() == nil {
+			return unit
+		}
+	}
+	t.Fatalf("none of %v is installed", openvswitchUnits)
+	return ""
+}