feat(daemon): DDIL Tier 1 — daemon-local-autonomy phase 1 (#204)

* refactor(daemon): extract Describe instance handlers to service (1b-pre phase 1)

Collapse the Describe family in daemon_handlers_instance.go to thin Pattern A
wrappers over a transport-neutral InstanceServiceImpl. This brings the read
path in line with the rest of the codebase and is the prerequisite for 1b's
local HTTP API (mulga-siv-14, daemon-local-autonomy plan).

Extracted: DescribeInstances, DescribeInstanceTypes, DescribeInstanceAttribute,
DescribeStoppedInstances, DescribeTerminatedInstances. Filter helpers
(instanceMatchesFilters, matchTagKey/Value, DescribeInstancesValidFilters,
IsInstanceVisible) move to the instance package alongside the service methods.

InstanceService gains the new methods and two dep interfaces
(ResourceCapacityProvider, StoppedInstanceStore) so the daemon's ResourceManager
and JetStreamManager can plug in without leaking concrete types. NATSInstanceService
returns a clear error for the fan-out describes (DescribeInstances,
DescribeInstanceTypes) since cluster-wide aggregation lives in the gateway's
scatter-gather helpers, not a single-node NATS request.

Behaviour changes: malformed JSON on DescribeInstanceAttribute now responds
with ValidationError (matching every other Pattern A handler) instead of
ServerInternal. Test updated.

Mutators (RunInstances, ModifyInstanceAttribute, StartStoppedInstance,
TerminateStoppedInstance, sub-handlers) remain Pattern B for now; follow-up
phases of mulga-siv-14 will collapse them.

* feat(daemon): local-first instance state persistence (1a)

Make /var/lib/spinifex/state/instance-state.json the source of truth for a
node's VMs; JetStream KV becomes a best-effort cluster cache. The daemon now
survives KV unavailability without losing instance state.

- New local_state.go: atomic write (tmp+fsync+rename), schema_version gate,
  missing-file = fresh install, corrupt = refuse start.
- Daemon.WriteState writes the local file under Instances.Mu, then calls
  jsManager.WriteStateBestEffort with a 1s timeout. KV failures log and
  continue; clients never see "KV down" during partition.
- Daemon.LoadState reads from the local file only — no KV fallback (a fresh
  node has empty KV anyway, so silent fallback would mask bugs).
- restoreInstances now returns error and Start() aborts on corrupt local
  state rather than silently starting empty.
- jsManager.WriteState no longer self-locks (caller holds Mu); added
  WriteStateBestEffort and shared marshalInstanceState helper.

Reconnect-driven KV resync deferred to siv-16 (1c owns the callback wiring).

Bead: mulga-siv-15
Plan: docs/development/feature/daemon-local-state-persistence.md

* fix(daemon): root local state at DataDir not BaseDir

LocalStatePath was falling back to BaseDir when DataDir was empty, putting
the file at /var/lib/spinifex/spinifex/state/instance-state.json (under the
daemon's own subdir) instead of the documented
/var/lib/spinifex/state/instance-state.json (the platform-shared root).

Drop the BaseDir fallback. Empty DataDir resolves to DefaultDataDir
(/var/lib/spinifex). Tests now set DataDir alongside BaseDir on their config
fixtures.

* feat(daemon): graceful NATS disconnect/reconnect handling (1c)

Daemon now flips a `mode` field (cluster|standalone) on NATS
disconnect/reconnect via optional callback hooks added to
utils.ConnectNATS. Reconnect bumps natsRetryCount and triggers 1a's
full local-state push to KV. NATSRequest/NATSScatterGather fail fast
with ErrClusterUnavailable when the connection is down. Gateway
short-circuits NATS-bound requests with a 503 ServiceUnavailable
carrying "cluster unavailable: NATS disconnected — check daemon
/local/status" so AWS clients see the failure mode immediately
instead of hanging on per-call timeouts.

Plan: docs/development/feature/daemon-autonomy-1c-disconnect.md

* fix(gateway): SigV4 middleware short-circuits on NATS disconnect

The 1c gateway.Request() IsConnected check (gateway.go:234) was
unreachable for any authenticated path because SigV4 auth middleware
runs first and calls IAMService.LookupAccessKey over JetStream KV. With
NATS down the lookup hangs ~5s on context.DeadlineExceeded and the
gateway returns 500 InternalError. AWS SDK retries it twice for ~18s of
user-visible latency instead of the <100ms 503 promised by the 1c plan.

Mirror the IsConnected check at the top of SigV4AuthMiddleware (after
credential parse so malformed auth headers still get the right SigV4
error code, before the IAM lookup). The catch-all in gateway.Request()
stays as defence-in-depth for any future unauthed paths.

Coverage debt picked up alongside: 1c utils.NATSRequest /
NATSScatterGather fail-fast paths, daemon Mode/onNATSDisconnect/
onNATSReconnect/localStatePath/Write/LoadState branches, and 1b-pre
InstanceServiceImpl Describe* methods (partial siv-22).

Bead: mulga-siv-23

* feat(daemon): read-only /local/* HTTP API (1b)

Wires GET /local/instances, GET /local/instances/{id}, GET /local/status
onto the existing ClusterManager TLS listener so observers can read this
node's instance state and connectivity mode while NATS is unreachable.

Also fixes a pre-existing race in WriteState: json.Marshal walked shared
*vm.VM pointers from SnapshotMap while concurrent TransitionState calls
mutated VM fields under the manager lock. Marshal now runs inside
vmMgr.View; WriteLocalStateBytes / WriteStateBytesBestEffort accept the
pre-marshalled JSON and commit lock-free. TestTransitionState_Concurrent
Transitions is race-clean across 30 iterations.

Adds Daemon.stateRevision (atomic.Uint64) bumped on every successful
local-state write, surfaced as /local/status.revision so observers can
detect changes without diffing payloads.

* feat(daemon): two-phase startup — survive NATS down (1d)

Split daemon Start() into startLocal() (HTTPS, mgmt bridge, OVS plumber,
OOM, local state load — no NATS) and startCluster() (background, infinite
retry exp backoff cap 60s for NATS connect, then JetStream + services +
restoreInstances + subscribeAll + heartbeat). Daemon never process-exits
on NATS failure; instead it stays up in standalone mode and reconnects in
the background.

Drop ExecStartPre=wait-for-nats.sh from spinifex-daemon.service — local
bootstrap no longer requires NATS to be reachable.

ConnectNATSWithRetry now supports infinite retry (WithMaxWait(0)),
context cancellation (WithContext), per-attempt error callback for the
nats_retry_count operator signal on /local/status, and a configurable
backoff cap (WithMaxRetryDelay).

bead: mulga-siv-17 (closes)

* feat(daemon): Tier 1 invariant guard for cluster-scoped service init (1e-audit)

Make the DDIL §1e-audit invariant explicit: startLocal must not initialise any
NATS-dependent or JetStream-KV-backed handle. Adds assertNoClusterServicesInitialised
as a runtime guard at the end of startLocal that fails fast if a future edit
hoists a cluster-scoped service into Phase 1, plus comment blocks naming the
invariant on both startLocal and startCluster.

Code audit confirms the existing 1d split already keeps every cluster-scoped
bucket out of Phase 1; this change pins the property so it cannot regress
silently. Tier table for the 25 daemon-process KV buckets is finalised in
docs/development/feature/daemon-local-autonomy.md §1e-audit.

bead: mulga-siv-19 (closes)

* feat(daemon): NATS observability hardening (1d-obs)

Closes the residual visibility gap left by the 1d non-blocking startup:

- Extend /local/status with kv_sync_failures, last_kv_sync_at,
  last_kv_sync_error so an operator polling the local API sees prolonged
  divergence between the local file and JetStream KV.
- Add a KVSyncObserver interface on JetStreamManager so the daemon can
  count best-effort write outcomes without coupling the manager to
  daemon state.
- Escalate utils.ConnectNATSWithRetry per-attempt log to slog.Error
  (rate-limited to once per minute, with disconnected_for context) once
  attempt count crosses the natsRetryEscalateAttempt threshold (~30
  attempts ≈ ~30 min disconnected at the 60s backoff cap), so a stuck
  NATS config surfaces in error logs instead of routine warn noise.

Prometheus metrics export deferred until a system-wide telemetry plan
(likely OTel) lands; the structured /local/status fields plus the
escalated retry log cover the Tier 1 visibility gap.

* feat(daemon): opt-in SPINIFEX_REQUIRE_NATS strict-startup mode (1d-strict)

When SPINIFEX_REQUIRE_NATS=1, startCluster's first NATS connect uses a
bounded ~30s retry and the daemon aborts on timeout. Default behaviour
(env unset) keeps the Tier 1 infinite-retry path so prod / edge nodes
still survive a missing NATS at boot. Restores pre-DDIL fail-fast UX
for dev / test / single-node deploys without flipping the prod default.

mulga-siv-44.

* test(scripts): systemd-level smoke for DDIL phase 1

Bash harness exercising daemon-local-autonomy on a real `make deploy` node:
baseline cluster mode → kill spinifex-nats → assert standalone + retry
counter advance + /local/instances served → restart NATS → assert mode flip
+ KV resync stamp → SPINIFEX_REQUIRE_NATS=1 bounded-exit. Cleanup trap
restores both services. Complements existing unit coverage by validating
real systemd wiring on port 8443.

* fix(test-ddil): daemon ClusterManager binds 4432 not 8443

8443 is predastore. Daemon's HTTPS ClusterManager (where /local/* lives)
binds [nodes.node1.daemon].host = 0.0.0.0:4432 in the deployed config.
Plan doc daemon-local-autonomy.md:112 incorrectly said 8443; actual code
uses d.config.Daemon.Host.

* fix(test-ddil): correct retry-counter site + bypass NATS Wants= for strict test

Two bugs in the v1 script:

1) nats_retry_count assertion ran during disconnect — but the counter
   only bumps on disconnect→reconnect *cycles* (daemon.go:321,
   onNATSReconnect). Moved assertion to step 3 (restart NATS) where
   the reconnect callback fires.

2) Step 4 stopped spinifex-nats then ran 'systemctl restart
   spinifex-daemon' — but the daemon unit's Wants=spinifex-nats.service
   re-pulled NATS up via the Wants= chain, so the strict-mode 30s
   timeout never fired. systemctl mask refuses on /etc-installed
   units, and --runtime mask is shadowed by /etc. Block via a runtime
   drop-in with ConditionPathExists=! pointing at a sentinel file.

Verified: all 5 checks pass on a single-node make deploy box; daemon
exits 32s after restart with SPINIFEX_REQUIRE_NATS=1 + NATS blocked.

* fix(admin): render data_dir into spinifex.toml so non-root nodes write local state

Pseudo-multinode E2E (rootless tf-user, ~/nodeN data roots) hit
"mkdir /var/lib/spinifex/state: permission denied" on every RunInstances:
Config.DataDir was empty because the template never emitted a data_dir
key, and LocalStatePath fell back to DefaultDataDir (/var/lib/spinifex).

Surfaced after 8eb9c5fa dropped the BaseDir fallback in LocalStatePath —
previously SPINIFEX_BASE_DIR=$DATA_DIR/spinifex/ (set by start-dev.sh)
masked the missing data_dir.

ConfigSettings.DataDir is already populated to spxRoot at admin
init/join; render it under [nodes.X] so it actually reaches the daemon.

build/systemd/spinifex-daemon.service

@@ -8,7 +8,15 @@ Wants=spinifex-nats.service
 Type=simple
 User=spinifex-daemon
 Group=spinifex
-ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
+# DDIL Tier 1 (siv-17): no ExecStartPre wait-for-nats. The daemon starts in
+# standalone mode and reconnects to NATS in the background — process-exiting
+# on a missing NATS would kill the local VM management plane.
+#
+# Opt-in strict-startup (§1d-strict, siv-44): set SPINIFEX_REQUIRE_NATS=1 to
+# restore pre-DDIL fail-fast UX. First NATS connect uses a ~30s bounded retry
+# and the daemon aborts on timeout. Intended for dev/test/single-node deploys
+# where silent NATS misconfig is more confusing than helpful. Do NOT set this
+# in the shipped unit — it re-introduces the SPOF that Tier 1 removed.
 ExecStart=/usr/local/bin/spx service spinifex start
 Restart=on-failure
 RestartSec=5

cmd/spinifex/cmd/templates/spinifex.toml

@@ -52,6 +52,7 @@ advertise = "{{.AdvertiseIP}}"
 {{- end}}
 region = "{{.Region}}"
 az = "{{.Az}}"
+data_dir = "{{.DataDir}}"
 {{- if .Services}}
 services = [{{range $i, $s := .Services}}{{if $i}}, {{end}}"{{$s}}"{{end}}]
 {{- end}}

scripts/test-ddil-phase1.sh

@@ -0,0 +1,121 @@
+#!/bin/bash
+# test-ddil-phase1.sh — systemd-level smoke test for daemon-local-autonomy
+# Tier 1 (1a / 1b / 1c / 1d / 1d-strict). Run after `make deploy` on a single
+# node where spinifex-daemon and spinifex-nats are managed by systemd.
+#
+# Usage: scripts/test-ddil-phase1.sh
+# Requires: sudo, curl, jq
+#
+# Destructive on the local node — stops/starts spinifex-nats and restarts
+# spinifex-daemon. Do NOT run against a host carrying workloads. The cleanup
+# trap restores both services on exit.
+
+set -euo pipefail
+
+DAEMON_URL="${DAEMON_URL:-https://127.0.0.1:4432}"
+DAEMON_SVC=spinifex-daemon
+NATS_SVC=spinifex-nats
+OVERRIDE=/etc/systemd/system/${DAEMON_SVC}.service.d/99-test-ddil.conf
+NATS_BLOCK=/run/spinifex-nats-blocked
+NATS_BLOCK_DROPIN=/run/systemd/system/${NATS_SVC}.service.d/99-test-block.conf
+CURL=(curl -fsS -k --max-time 5)
+
+ok()   { printf '\033[32mPASS\033[0m %s\n' "$1"; }
+fail() { printf '\033[31mFAIL\033[0m %s\n' "$1"; exit 1; }
+info() { printf '\033[36m==>\033[0m %s\n' "$1"; }
+
+cleanup() {
+    info "cleanup: removing nats block, override, restoring services"
+    sudo rm -f "$NATS_BLOCK" "$NATS_BLOCK_DROPIN" "$OVERRIDE"
+    sudo rmdir "$(dirname "$NATS_BLOCK_DROPIN")" 2>/dev/null || true
+    sudo systemctl daemon-reload
+    sudo systemctl start "$NATS_SVC"     2>/dev/null || true
+    sudo systemctl restart "$DAEMON_SVC" 2>/dev/null || true
+}
+trap cleanup EXIT
+
+field() { "${CURL[@]}" "${DAEMON_URL}/local/status" | jq -r ".$1 // empty"; }
+
+wait_field() {
+    local f=$1 want=$2 timeout=${3:-30} elapsed=0
+    while [ "$elapsed" -lt "$timeout" ]; do
+        [ "$(field "$f" 2>/dev/null || true)" = "$want" ] && return 0
+        sleep 1; elapsed=$((elapsed + 1))
+    done
+    return 1
+}
+
+# 0. baseline
+info "baseline: daemon active + cluster mode"
+sudo systemctl is-active --quiet "$DAEMON_SVC" || fail "daemon not running — run \`make deploy\` first"
+sudo systemctl start "$NATS_SVC"
+wait_field mode cluster 30 || fail "never reached mode=cluster (got: $(field mode))"
+[ "$(field nats)" = connected ] || fail "nats not connected (got: $(field nats))"
+ok "mode=cluster nats=connected"
+
+# 1. kill NATS → standalone, daemon survives
+info "1a/1c: kill NATS"
+retry_before=$(field nats_retry_count)
+sudo systemctl stop "$NATS_SVC"
+wait_field mode standalone 30   || fail "never went standalone (got: $(field mode))"
+wait_field nats disconnected 5  || fail "nats not disconnected (got: $(field nats))"
+sudo systemctl is-active --quiet "$DAEMON_SVC" || fail "daemon exited — should stay up"
+ok "standalone, nats=disconnected, daemon alive"
+
+# 2. /local/instances reachable during partition
+info "1b: /local/instances during partition"
+"${CURL[@]}" "${DAEMON_URL}/local/instances" | jq -e . >/dev/null \
+    || fail "/local/instances did not return JSON"
+ok "/local/instances served while NATS down"
+
+# 3. restart NATS → cluster mode + retry count advances + KV push observed
+# nats_retry_count counts disconnect→reconnect *cycles* (bumped from
+# onNATSReconnect, daemon.go:321). It stays flat during disconnect and
+# advances by 1 here when the reconnect callback fires.
+info "1c reconnect: restart NATS"
+before_sync=$(field last_kv_sync_at)
+sudo systemctl start "$NATS_SVC"
+wait_field mode cluster 60   || fail "did not return to cluster mode"
+wait_field nats connected 10 || fail "nats not reconnected"
+sleep 5
+retry_after=$(field nats_retry_count)
+[ "$retry_after" -gt "$retry_before" ] \
+    || fail "nats_retry_count did not advance on reconnect ($retry_before → $retry_after)"
+after_sync=$(field last_kv_sync_at)
+[ -n "$after_sync" ] && [ "$after_sync" != "$before_sync" ] \
+    || fail "last_kv_sync_at did not advance ($before_sync → $after_sync)"
+ok "cluster restored, retry $retry_before → $retry_after, KV resync at $after_sync"
+
+# 4. SPINIFEX_REQUIRE_NATS=1 → daemon exits within ~30s when NATS down.
+# Block spinifex-nats via a runtime drop-in with ConditionPathExists=!. This
+# defeats the daemon unit's Wants=spinifex-nats.service auto-start, which
+# would otherwise pull NATS back up on `systemctl restart spinifex-daemon`
+# and let the daemon connect immediately. mask refuses (real unit lives in
+# /etc) and --runtime mask is shadowed by /etc — the condition drop-in is
+# the only mechanism that works against an /etc-installed unit.
+info "1d-strict: SPINIFEX_REQUIRE_NATS=1 exits when NATS down"
+sudo mkdir -p "$(dirname "$NATS_BLOCK_DROPIN")"
+sudo tee "$NATS_BLOCK_DROPIN" >/dev/null <<EOF
+[Unit]
+ConditionPathExists=!$NATS_BLOCK
+EOF
+sudo touch "$NATS_BLOCK"
+sudo systemctl daemon-reload
+sudo systemctl stop "$NATS_SVC"
+sudo mkdir -p "$(dirname "$OVERRIDE")"
+sudo tee "$OVERRIDE" >/dev/null <<EOF
+[Service]
+Environment=SPINIFEX_REQUIRE_NATS=1
+Restart=no
+EOF
+sudo systemctl daemon-reload
+sudo systemctl restart "$DAEMON_SVC" || true
+start=$(date +%s)
+while sudo systemctl is-active --quiet "$DAEMON_SVC"; do
+    [ $(( $(date +%s) - start )) -gt 45 ] && fail "did not exit within 45s"
+    sleep 1
+done
+ok "exited after $(( $(date +%s) - start ))s"
+
+echo
+echo "DDIL Phase 1 systemd integration: ALL PASS"

spinifex/daemon/daemon.go

@@ -159,6 +159,17 @@ type Daemon struct {
 	// NATS connect retry options (nil uses defaults: 5min max, 500ms initial delay)
 	natsRetryOpts []utils.RetryOption
 
+	// requireNATSTimeout caps the first connectNATS attempt when the
+	// SPINIFEX_REQUIRE_NATS=1 strict-startup env var is set (§1d-strict).
+	// Default 30s; tests override to a shorter value to keep the strict-mode
+	// abort path fast.
+	requireNATSTimeout time.Duration
+
+	// exitFunc is invoked when SPINIFEX_REQUIRE_NATS=1 strict-startup is
+	// requested and the bounded first connect fails. Defaults to os.Exit;
+	// tests override to observe the abort without killing the test process.
+	exitFunc func(int)
+
 	// networkPlumber handles tap device lifecycle for VPC and management networking
 	networkPlumber vm.NetworkPlumber
 
@@ -187,9 +198,138 @@ type Daemon struct {
 	// are fully initialized. The health endpoint reports "starting" until ready.
 	ready atomic.Bool
 
+	// mode is the daemon's connectivity state — "cluster" when NATS is
+	// reachable, "standalone" when disconnected. Read-only via Mode(); flipped
+	// from utils/nats.go disconnect/reconnect callbacks (see onNATSDisconnect /
+	// onNATSReconnect). Initialised "standalone" so callers reading before
+	// connectNATS() returns get a sensible answer.
+	mode atomic.Value
+
+	// natsRetryCount counts disconnect→reconnect cycles since process start.
+	// Bumped from onNATSReconnect; surfaced via NATSRetryCount() for the
+	// /local/status endpoint added in 1b.
+	natsRetryCount atomic.Int64
+
+	// stateRevision is bumped on every successful local-state write. Surfaced
+	// via /local/status so observers can detect changes without diffing payloads.
+	stateRevision atomic.Uint64
+
+	// kvSyncFailures counts best-effort JetStream KV sync failures (timeout or
+	// put error) since process start. Bumped from RecordKVSyncFailure; surfaced
+	// via /local/status and the spinifex_daemon_kv_sync_failures_total metric.
+	kvSyncFailures atomic.Int64
+	// lastKVSyncAt holds the unix-nano timestamp of the most recent successful
+	// best-effort KV sync. Zero means "never synced since process start".
+	lastKVSyncAt atomic.Int64
+	// lastKVSyncError holds the most recent best-effort KV sync error message
+	// as a string. Cleared back to "" on the next successful sync.
+	lastKVSyncError atomic.Value
+
 	mu sync.Mutex
 }
 
+// Daemon connectivity modes stored in Daemon.mode.
+const (
+	DaemonModeStandalone = "standalone"
+	DaemonModeCluster    = "cluster"
+)
+
+// Mode returns the daemon's current connectivity mode ("cluster" or
+// "standalone"). Safe to call from any goroutine.
+func (d *Daemon) Mode() string {
+	v := d.mode.Load()
+	if v == nil {
+		return DaemonModeStandalone
+	}
+	s, ok := v.(string)
+	if !ok {
+		return DaemonModeStandalone
+	}
+	return s
+}
+
+// NATSRetryCount returns the number of disconnect→reconnect cycles observed
+// since process start.
+func (d *Daemon) NATSRetryCount() int64 {
+	return d.natsRetryCount.Load()
+}
+
+// Revision returns the local-state revision counter. Bumped on every successful
+// WriteState; observers can detect changes without diffing the full payload.
+func (d *Daemon) Revision() uint64 {
+	return d.stateRevision.Load()
+}
+
+// KVSyncFailures returns the number of best-effort JetStream KV sync failures
+// (timeout or put error) observed since process start.
+func (d *Daemon) KVSyncFailures() int64 {
+	return d.kvSyncFailures.Load()
+}
+
+// LastKVSyncAt returns the timestamp of the most recent successful best-effort
+// KV sync. Zero time means "never synced since process start".
+func (d *Daemon) LastKVSyncAt() time.Time {
+	n := d.lastKVSyncAt.Load()
+	if n == 0 {
+		return time.Time{}
+	}
+	return time.Unix(0, n)
+}
+
+// LastKVSyncError returns the most recent best-effort KV sync error message.
+// Empty string means the last attempt succeeded (or no attempt has been made).
+func (d *Daemon) LastKVSyncError() string {
+	v := d.lastKVSyncError.Load()
+	if v == nil {
+		return ""
+	}
+	s, _ := v.(string)
+	return s
+}
+
+// RecordKVSyncSuccess implements KVSyncObserver. The JetStream manager calls
+// this from its best-effort write path on a successful Put.
+func (d *Daemon) RecordKVSyncSuccess(_ string) {
+	d.lastKVSyncAt.Store(time.Now().UnixNano())
+	d.lastKVSyncError.Store("")
+}
+
+// RecordKVSyncFailure implements KVSyncObserver. Bumps the failure counter and
+// records the error message for /local/status. The bucket arg is reserved for
+// future per-bucket labelling; the current call site is the instance-state
+// bucket and the value is logged via the manager's slog.Warn line.
+func (d *Daemon) RecordKVSyncFailure(_ string, err error) {
+	d.kvSyncFailures.Add(1)
+	if err != nil {
+		d.lastKVSyncError.Store(err.Error())
+	}
+}
+
+// onNATSDisconnect runs when the NATS client loses its connection. Flips the
+// daemon to standalone mode so /local/status callers and scatter-gather
+// bailouts react immediately. Must not block — runs on a NATS client goroutine.
+func (d *Daemon) onNATSDisconnect(_ *nats.Conn, _ error) {
+	d.mode.Store(DaemonModeStandalone)
+}
+
+// onNATSReconnect runs when the NATS client reattaches to a server. Flips back
+// to cluster mode, bumps the retry counter, and pushes the full local instance
+// state to KV so the cluster view re-converges. The KV push runs in a goroutine
+// to keep the NATS client callback non-blocking.
+func (d *Daemon) onNATSReconnect(_ *nats.Conn) {
+	d.mode.Store(DaemonModeCluster)
+	d.natsRetryCount.Add(1)
+
+	if d.jsManager == nil {
+		return
+	}
+	go func() {
+		if err := d.WriteState(); err != nil {
+			slog.Warn("Reconnect KV resync failed", "error", err)
+		}
+	}()
+}
+
 // getSystemMemory returns the total system memory in GB
 func getSystemMemory() (float64, error) {
 	switch runtime.GOOS {
@@ -449,20 +589,24 @@ func NewDaemon(cfg *config.ClusterConfig) (*Daemon, error) {
 		return nil, fmt.Errorf("initialize resource manager: %w", err)
 	}
 
-	return &Daemon{
-		node:              cfg.Node,
-		clusterConfig:     cfg,
-		config:            &nodeCfg,
-		resourceMgr:       rm,
-		gpuProbe:          gpuProbe,
-		gpuManager:        gpuMgr,
-		ctx:               ctx,
-		cancel:            cancel,
-		vmMgr:             vm.NewManager(),
-		natsSubscriptions: make(map[string]*nats.Subscription),
-		startTime:         time.Now(),
-		detachDelay:       1 * time.Second,
-	}, nil
+	d := &Daemon{
+		node:               cfg.Node,
+		clusterConfig:      cfg,
+		config:             &nodeCfg,
+		resourceMgr:        rm,
+		gpuProbe:           gpuProbe,
+		gpuManager:         gpuMgr,
+		ctx:                ctx,
+		cancel:             cancel,
+		vmMgr:              vm.NewManager(),
+		natsSubscriptions:  make(map[string]*nats.Subscription),
+		startTime:          time.Now(),
+		detachDelay:        1 * time.Second,
+		requireNATSTimeout: 30 * time.Second,
+		exitFunc:           os.Exit,
+	}
+	d.mode.Store(DaemonModeStandalone)
+	return d, nil
 }
 
 // natsSub defines a single NATS subscription entry for the table-driven setup.
@@ -631,20 +775,181 @@ func (d *Daemon) subscribeAll() error {
 	return nil
 }
 
-// Start initializes and starts the daemon
+// Start brings the daemon up in two phases (DDIL Tier 1, §1d):
+//
+//  1. startLocal — bootstraps everything that does not need NATS (cluster
+//     manager HTTPS, mgmt bridge + IP allocator, OVS plumber, OOM score,
+//     local instance state via 1a). The daemon is reachable on /local/* and
+//     /health as soon as this returns.
+//  2. startCluster — runs in the background, retries NATS forever, then
+//     initialises JetStream + cluster-scoped services, restores instances,
+//     and subscribes to NATS topics. Mode flips to "cluster" once connected.
+//
+// Process-exit on NATS failure is no longer possible; staying up degraded is
+// always better than killing the local VM management plane.
 func (d *Daemon) Start() error {
-	if err := d.connectNATS(); err != nil {
-		return fmt.Errorf("failed to connect to NATS: %w", err)
+	if err := d.startLocal(); err != nil {
+		return err
 	}
 
-	// ClusterManager must start before JetStream init so peers can reach
-	// /health during bootstrap.
+	d.setupShutdown()
+
+	d.shutdownWg.Go(func() {
+		if err := d.startCluster(); err != nil {
+			slog.Warn("Cluster bootstrap aborted", "err", err)
+		}
+	})
+
+	d.awaitShutdown()
+	return nil
+}
+
+// startLocal performs the no-NATS bootstrap: HTTPS cluster manager,
+// management bridge detection, network plumber, OOM protection, and local
+// instance-state recovery. Failures here are fatal — these are local
+// configuration errors (TLS misconfig, bad config path) that retry would not
+// fix. The daemon is reachable via /local/* and /health once this returns.
+//
+// Invariant (DDIL §1e-audit): no code in startLocal may touch JetStream KV.
+// The 25 cluster-scoped / read-cache / expendable buckets enumerated in
+// daemon-local-autonomy.md §1e-audit are initialised exclusively from
+// startCluster() — touching any of them here would defeat Tier 1 autonomy by
+// blocking on NATS at boot. The only KV bucket the daemon owns at Tier 1 is
+// spinifex-instance-state, and even that is read from the local file (see 1a),
+// not JetStream, in this phase. assertNoClusterServicesInitialised below
+// enforces the invariant at runtime.
+func (d *Daemon) startLocal() error {
+	// ClusterManager serves /health and /local/* over HTTPS. NATS-independent.
 	if err := d.ClusterManager(); err != nil {
 		return fmt.Errorf("failed to start cluster manager: %w", err)
 	}
 
+	// Detect management bridge for system instance control plane NICs.
+	mgmtBridge := "br-mgmt"
+	if d.config.Daemon.MgmtBridge != "" {
+		mgmtBridge = d.config.Daemon.MgmtBridge
+	}
+	bridgeIP, bridgeErr := GetBridgeIPv4(mgmtBridge)
+	if bridgeErr != nil {
+		slog.Warn("Management bridge not detected, system instances will not get mgmt NIC", "bridge", mgmtBridge, "err", bridgeErr)
+	} else if bridgeIP == "" {
+		slog.Warn("Management bridge not found, system instances will not get mgmt NIC", "bridge", mgmtBridge)
+	} else {
+		d.mgmtBridgeIP = bridgeIP
+		alloc, allocErr := NewMgmtIPAllocator(bridgeIP)
+		if allocErr != nil {
+			slog.Error("Failed to create mgmt IP allocator", "bridgeIP", bridgeIP, "err", allocErr)
+		} else {
+			d.mgmtIPAllocator = alloc
+			slog.Info("Management bridge detected", "bridge", mgmtBridge, "ip", bridgeIP)
+		}
+	}
+
+	// Initialise OVS network plumber (no NATS dep).
+	if d.networkPlumber == nil {
+		d.networkPlumber = &OVSNetworkPlumber{}
+	}
+
+	// Protect daemon from OOM killer (prefer killing QEMU VMs instead).
+	if err := utils.SetOOMScore(os.Getpid(), -500); err != nil {
+		slog.Warn("Failed to set daemon OOM score", "err", err)
+	}
+
+	// Recover local instance state from disk (1a). Read-only — KV migrations,
+	// QMP reconnects, NATS subscriptions and crashed-VM relaunches happen in
+	// startCluster() once NATS is reachable. Fatal if the file is corrupt:
+	// silently dropping instance state would orphan running VMs.
+	if err := d.LoadState(); err != nil {
+		return fmt.Errorf("load local instance state: %w", err)
+	}
+	slog.Info("Loaded local instance state", "instance count", d.vmMgr.Count())
+
+	// Rebuild mgmt IP allocator from restored VMs.
+	if d.mgmtIPAllocator != nil {
+		d.mgmtIPAllocator.Rebuild(d.vmMgr.SnapshotMap())
+		slog.Info("Rebuilt mgmt IP allocator from restored instances", "allocated", d.mgmtIPAllocator.AllocatedCount())
+	}
+
+	if err := d.assertNoClusterServicesInitialised(); err != nil {
+		return fmt.Errorf("startLocal Tier 1 invariant violated: %w", err)
+	}
+
+	d.ready.Store(true)
+	slog.Info("Daemon local-bootstrap complete", "node", d.node, "elapsed", time.Since(d.startTime).Round(time.Second))
+	return nil
+}
+
+// assertNoClusterServicesInitialised guards the DDIL §1e-audit Tier 1
+// invariant: at the end of startLocal, no NATS-dependent or KV-backed handle
+// may exist. A non-nil field here means a future edit accidentally hoisted a
+// cluster-scoped initialiser into the no-NATS phase, which would re-introduce
+// the boot-time NATS dependency that 1d removed. Cheap nil sweep — runs once
+// per process, on the bootstrap path only.
+func (d *Daemon) assertNoClusterServicesInitialised() error {
+	switch {
+	case d.natsConn != nil:
+		return errors.New("d.natsConn must be nil before startCluster")
+	case d.jsManager != nil:
+		return errors.New("d.jsManager must be nil before startCluster")
+	case d.instanceService != nil:
+		return errors.New("d.instanceService must be nil before startCluster")
+	case d.imageService != nil:
+		return errors.New("d.imageService must be nil before startCluster")
+	case d.snapshotService != nil:
+		return errors.New("d.snapshotService must be nil before startCluster")
+	case d.volumeService != nil:
+		return errors.New("d.volumeService must be nil before startCluster")
+	case d.eigwService != nil:
+		return errors.New("d.eigwService must be nil before startCluster")
+	case d.igwService != nil:
+		return errors.New("d.igwService must be nil before startCluster")
+	case d.placementGroupService != nil:
+		return errors.New("d.placementGroupService must be nil before startCluster")
+	case d.vpcService != nil:
+		return errors.New("d.vpcService must be nil before startCluster")
+	case d.routeTableService != nil:
+		return errors.New("d.routeTableService must be nil before startCluster")
+	case d.natGatewayService != nil:
+		return errors.New("d.natGatewayService must be nil before startCluster")
+	case d.externalIPAM != nil:
+		return errors.New("d.externalIPAM must be nil before startCluster")
+	case d.eipService != nil:
+		return errors.New("d.eipService must be nil before startCluster")
+	case d.accountService != nil:
+		return errors.New("d.accountService must be nil before startCluster")
+	case d.elbv2Service != nil:
+		return errors.New("d.elbv2Service must be nil before startCluster")
+	}
+	return nil
+}
+
+// startCluster performs the cluster-integration phase asynchronously. It
+// retries NATS indefinitely (cap 60s backoff) and only returns once the node
+// is fully participating in the cluster or d.ctx is cancelled. Errors here
+// are logged, never propagated as a process-exit.
+//
+// Invariant (DDIL §1e-audit): every JetStream KV bucket — the 18 cluster-scoped
+// buckets, 4 read-cache (IAM) buckets, and 2 expendable buckets enumerated in
+// daemon-local-autonomy.md §1e-audit — is initialised here, never in
+// startLocal. Adding a new cluster-scoped service belongs in this function;
+// hoisting one into startLocal trips assertNoClusterServicesInitialised.
+func (d *Daemon) startCluster() error {
+	if os.Getenv("SPINIFEX_REQUIRE_NATS") == "1" {
+		// §1d-strict opt-in: bounded first connect, abort on timeout. Restores
+		// the pre-DDIL fail-fast UX for dev/test/single-node deploys without
+		// flipping the prod default (which would re-introduce the SPOF that 1d
+		// removed).
+		if err := d.connectNATS(utils.WithMaxWait(d.requireNATSTimeout)); err != nil {
+			slog.Error("SPINIFEX_REQUIRE_NATS=1 set, NATS connect failed within 30s, aborting", "err", err, "timeout", d.requireNATSTimeout)
+			d.exitFunc(1)
+			return fmt.Errorf("connect NATS (strict): %w", err)
+		}
+	} else if err := d.connectNATS(); err != nil {
+		return fmt.Errorf("connect NATS: %w", err)
+	}
+
 	if err := d.initJetStream(); err != nil {
-		return fmt.Errorf("failed to initialize JetStream: %w", err)
+		return fmt.Errorf("initialize JetStream: %w", err)
 	}
 
 	// Write service manifest so other nodes know what this node runs
@@ -661,7 +966,7 @@ func (d *Daemon) Start() error {
 
 	// Create services before loading/launching instances, since LaunchInstance depends on them
 	store := objectstore.NewS3ObjectStoreFromConfig(admin.DialTarget(d.config.Predastore.Host), d.config.Predastore.Region, d.config.Predastore.AccessKey, d.config.Predastore.SecretKey)
-	d.instanceService = handlers_ec2_instance.NewInstanceServiceImpl(d.config, d.resourceMgr.instanceTypes, d.natsConn, store)
+	d.instanceService = handlers_ec2_instance.NewInstanceServiceImpl(d.config, d.resourceMgr.instanceTypes, d.natsConn, store, d.vmMgr, d.resourceMgr, d.jsManager)
 	d.keyService = handlers_ec2_key.NewKeyServiceImpl(d.config)
 	d.imageService = handlers_ec2_image.NewImageServiceImpl(d.config, d.natsConn)
 
@@ -811,28 +1116,6 @@ func (d *Daemon) Start() error {
 	// Wire LB VM lifecycle: instance launcher for system VMs.
 	d.elbv2Service.InstanceLauncher = d
 
-	// Detect management bridge for system instance control plane NICs.
-	// Must run before wireLBAgentConfig so the gateway URL uses br-mgmt IP.
-	mgmtBridge := "br-mgmt"
-	if d.config.Daemon.MgmtBridge != "" {
-		mgmtBridge = d.config.Daemon.MgmtBridge
-	}
-	bridgeIP, bridgeErr := GetBridgeIPv4(mgmtBridge)
-	if bridgeErr != nil {
-		slog.Warn("Management bridge not detected, system instances will not get mgmt NIC", "bridge", mgmtBridge, "err", bridgeErr)
-	} else if bridgeIP == "" {
-		slog.Warn("Management bridge not found, system instances will not get mgmt NIC", "bridge", mgmtBridge)
-	} else {
-		d.mgmtBridgeIP = bridgeIP
-		alloc, allocErr := NewMgmtIPAllocator(bridgeIP)
-		if allocErr != nil {
-			slog.Error("Failed to create mgmt IP allocator", "bridgeIP", bridgeIP, "err", allocErr)
-		} else {
-			d.mgmtIPAllocator = alloc
-			slog.Info("Management bridge detected", "bridge", mgmtBridge, "ip", bridgeIP)
-		}
-	}
-
 	// Wire system credentials + gateway URL for LB agent SigV4 auth.
 	d.wireLBAgentConfig()
 
@@ -885,23 +1168,15 @@ func (d *Daemon) Start() error {
 		d.ensureDefaultVPCInfrastructure()
 	}
 
-	// Initialize network plumber for VPC tap device management
-	if d.networkPlumber == nil {
-		d.networkPlumber = &OVSNetworkPlumber{}
-	}
-
 	// Wire vm.Manager collaborators now that NATS, JetStream, network plumber,
 	// volume service, and resource manager are all ready.
 	d.vmMgr.SetDeps(d.buildVMManagerDeps())
 
-	// Protect daemon from OOM killer (prefer killing QEMU VMs instead)
-	if err := utils.SetOOMScore(os.Getpid(), -500); err != nil {
-		slog.Warn("Failed to set daemon OOM score", "err", err)
-	}
-
 	d.waitForClusterReady()
 	d.upgradeJetStreamReplicas()
-	d.vmMgr.Restore()
+	if err := d.restoreInstances(); err != nil {
+		return fmt.Errorf("restore instances: %w", err)
+	}
 
 	// Rebuild mgmt IP allocator from restored VMs so we don't re-allocate IPs
 	// that are already in use by running system instances.
@@ -932,16 +1207,30 @@ func (d *Daemon) Start() error {
 	return nil
 }
 
-// connectNATS establishes a connection to the NATS server with retry and
-// exponential backoff. On multi-node clusters, the local NATS server may not
-// be ready immediately after daemon start (e.g. if start-dev.sh is still
-// launching services). This retries for up to 5 minutes before giving up.
-func (d *Daemon) connectNATS() error {
-	nc, err := utils.ConnectNATSWithRetry(admin.DialTarget(d.config.NATS.Host), d.config.NATS.ACL.Token, d.config.NATS.CACert, d.natsRetryOpts...)
+// connectNATS establishes a connection to the NATS server. Defaults to
+// infinite retry with exponential backoff (cap 60s) so the daemon stays up
+// in standalone mode through extended NATS outages instead of process-exiting
+// (DDIL Tier 1). Tests override d.natsRetryOpts to bound the wait. Callers
+// (e.g. §1d-strict) may pass extraOpts that are applied after d.natsRetryOpts
+// so they win on conflicting fields like WithMaxWait.
+func (d *Daemon) connectNATS(extraOpts ...utils.RetryOption) error {
+	opts := append([]utils.RetryOption{
+		utils.WithMaxWait(0), // infinite retry; cancelled via d.ctx
+		utils.WithMaxRetryDelay(60 * time.Second),
+		utils.WithContext(d.ctx),
+		utils.WithDisconnectHandler(d.onNATSDisconnect),
+		utils.WithReconnectHandler(d.onNATSReconnect),
+		utils.WithAttemptErrHandler(func(_ error, _ int) {
+			d.natsRetryCount.Add(1)
+		}),
+	}, d.natsRetryOpts...)
+	opts = append(opts, extraOpts...)
+	nc, err := utils.ConnectNATSWithRetry(admin.DialTarget(d.config.NATS.Host), d.config.NATS.ACL.Token, d.config.NATS.CACert, opts...)
 	if err != nil {
 		return err
 	}
 	d.natsConn = nc
+	d.mode.Store(DaemonModeCluster)
 	return nil
 }
 
@@ -972,6 +1261,7 @@ func (d *Daemon) initJetStream() error {
 		}
 
 		if err == nil {
+			d.jsManager.SetSyncObserver(d)
 			slog.Info("JetStream KV stores initialized successfully", "replicas", 1, "attempts", attempt, "elapsed", time.Since(start).Round(time.Second))
 			break
 		}
@@ -1100,6 +1390,43 @@ func (d *Daemon) checkPredastoreReady() bool {
 	return true
 }
 
+// LoadState loads the instance state from the local file. Missing file is the
+// fresh-install signal (start with empty map). Corrupt or unknown-schema files
+// are fatal — caller refuses start rather than silently losing data. There is
+// no KV fallback: a fresh node has empty KV anyway, so silent KV-fallback on
+// file loss would just mask bugs.
+func (d *Daemon) LoadState() error {
+	path := d.localStatePath()
+	state, err := ReadLocalState(path)
+	if err != nil {
+		slog.Error("Local state load failed", "path", path, "error", err)
+		return fmt.Errorf("read local state: %w", err)
+	}
+
+	if state == nil {
+		d.vmMgr.Replace(map[string]*vm.VM{})
+		slog.Info("No local state file, starting with empty instance map", "path", path)
+		return nil
+	}
+
+	d.vmMgr.Replace(state.VMS)
+	slog.Info("Loaded local state", "path", path, "instances", len(state.VMS))
+	return nil
+}
+
+// restoreInstances delegates to vm.Manager.Restore. vm.Manager only
+// persists to the cluster StateStore (JetStream); the daemon's local
+// crash-recovery file is owned by daemon.WriteState, so we sync it here
+// to preserve the pre-2b invariant that local state == in-memory state
+// after restore. Phase 2f folds this back into vm/.
+func (d *Daemon) restoreInstances() error {
+	d.vmMgr.Restore()
+	if err := d.WriteState(); err != nil {
+		slog.Error("Failed to persist local state after restore", "error", err)
+	}
+	return nil
+}
+
 // awaitShutdown blocks until the daemon's shutdown wait group completes.
 func (d *Daemon) awaitShutdown() {
 	done := make(chan struct{})
@@ -1194,6 +1521,8 @@ func (d *Daemon) ClusterManager() error {
 		}
 	})
 
+	d.registerLocalRoutes(r)
+
 	// Load TLS certificate.
 	// Resolve relative cert paths against config directory (cert lives alongside spinifex.toml).
 	// For binary installs, systemd sets absolute paths via env vars; for dev, the config
@@ -1244,22 +1573,54 @@ func (d *Daemon) ClusterManager() error {
 	return nil
 }
 
-// WriteState writes the instance state to JetStream KV store (required).
-// The marshal+put runs under the manager lock so VM fields can't change
-// mid-encode. Lock-across-Put is a known limitation; splitting marshal from
-// put requires a JetStreamManager API change and is deferred.
+// kvSyncTimeout bounds the best-effort cluster sync so a degraded NATS does
+// not stall every state transition. 1s is well above healthy KV.Put latency
+// and well below a user-visible delay.
+const kvSyncTimeout = time.Second
+
+// localStatePath returns the on-disk path to this daemon's instance state file.
+func (d *Daemon) localStatePath() string {
+	if d.config == nil {
+		return LocalStatePath("")
+	}
+	return LocalStatePath(d.config.DataDir)
+}
+
+// WriteState persists the instance state. Local file is the source of truth;
+// JetStream KV is best-effort cluster cache. The local write is fatal on
+// failure; KV failures are logged and swallowed so partition-time clients
+// never see "KV down" errors.
+//
+// Both wire forms are marshalled inside vmMgr.View so json.Marshal sees a
+// stable VM-field snapshot. Marshaling outside the lock would race against
+// concurrent TransitionState writers under the data race detector.
 func (d *Daemon) WriteState() error {
-	if d.jsManager == nil {
-		return fmt.Errorf("JetStream manager not initialized - cannot write state")
-	}
-	var writeErr error
+	var (
+		localData, kvData []byte
+		marshalErr        error
+	)
 	d.vmMgr.View(func(vms map[string]*vm.VM) {
-		writeErr = d.jsManager.WriteState(d.node, vms)
+		localData, marshalErr = MarshalLocalState(vms)
+		if marshalErr != nil {
+			return
+		}
+		kvData, marshalErr = marshalInstanceState(vms)
 	})
-	if writeErr != nil {
-		slog.Error("JetStream write failed", "error", writeErr)
-		return fmt.Errorf("failed to write state to JetStream: %w", writeErr)
+	if marshalErr != nil {
+		return fmt.Errorf("marshal state: %w", marshalErr)
 	}
+
+	path := d.localStatePath()
+	if err := WriteLocalStateBytes(path, localData); err != nil {
+		slog.Error("Local state write failed", "path", path, "error", err)
+		return fmt.Errorf("write local state: %w", err)
+	}
+	d.stateRevision.Add(1)
+
+	if d.jsManager != nil {
+		d.jsManager.WriteStateBytesBestEffort(d.node, kvData, kvSyncTimeout)
+	}
+
 	return nil
 }
 

spinifex/daemon/daemon_handlers_instance.go

@@ -638,8 +638,7 @@ func matchTagValue(tags []*ec2.Tag, values []string) bool {
 	return false
 }
 
-// handleEC2DescribeInstances processes incoming EC2 DescribeInstances requests
-// This handler responds with instances running on this node owned by the caller's account
+// handleEC2DescribeInstances responds with instances running on this node visible to the caller.
 func (d *Daemon) handleEC2DescribeInstances(msg *nats.Msg) {
 	slog.Debug("Received message", "subject", msg.Subject, "data", string(msg.Data))
 
@@ -774,48 +773,9 @@ func (d *Daemon) handleEC2DescribeInstances(msg *nats.Msg) {
 	slog.Info("handleEC2DescribeInstances completed", "count", len(reservations))
 }
 
-// handleEC2DescribeInstanceTypes processes incoming EC2 DescribeInstanceTypes requests
-// This handler responds with instance types that can currently be provisioned on this node
-// based on available resources (CPU and memory not already allocated to running instances)
+// handleEC2DescribeInstanceTypes responds with instance types provisionable on this node.
 func (d *Daemon) handleEC2DescribeInstanceTypes(msg *nats.Msg) {
-	slog.Debug("Received message", "subject", msg.Subject)
-
-	// Initialize input
-	describeInput := &ec2.DescribeInstanceTypesInput{}
-	errResp := utils.UnmarshalJsonPayload(describeInput, msg.Data)
-	if errResp != nil {
-		if err := msg.Respond(errResp); err != nil {
-			slog.Error("Failed to respond to NATS request", "err", err)
-		}
-		slog.Error("Request does not match DescribeInstanceTypesInput")
-		return
-	}
-
-	slog.Info("Processing DescribeInstanceTypes request from this node")
-
-	// Check if "capacity" filter is set to "true"
-	showCapacity := false
-	for _, f := range describeInput.Filters {
-		if f.Name != nil && *f.Name == "capacity" {
-			for _, v := range f.Values {
-				if v != nil && *v == "true" {
-					showCapacity = true
-					break
-				}
-			}
-		}
-	}
-
-	// Get instance types based on capacity and the showCapacity flag
-	filteredTypes := d.resourceMgr.GetAvailableInstanceTypeInfos(showCapacity)
-
-	// Create the response
-	output := &ec2.DescribeInstanceTypesOutput{
-		InstanceTypes: filteredTypes,
-	}
-
-	respondWithJSON(msg, output)
-	slog.Info("handleEC2DescribeInstanceTypes completed", "count", len(filteredTypes))
+	handleNATSRequest(msg, d.instanceService.DescribeInstanceTypes)
 }
 
 // startStoppedInstanceRequest is the payload for ec2.start topic
@@ -1309,7 +1269,6 @@ func (d *Daemon) handleEC2ModifyInstanceAttribute(msg *nats.Msg) {
 }
 
 // handleEC2DescribeInstanceAttribute returns a single requested attribute for an instance.
-// It checks running instances first (in-memory), then falls back to stopped instances in KV.
 func (d *Daemon) handleEC2DescribeInstanceAttribute(msg *nats.Msg) {
 	var input ec2.DescribeInstanceAttributeInput
 	if err := json.Unmarshal(msg.Data, &input); err != nil {

spinifex/daemon/daemon_handlers_test.go

@@ -89,6 +89,14 @@ func createFullTestDaemonWithJetStream(t *testing.T, natsURL string) *Daemon {
 	require.NoError(t, err)
 	daemon.stateStore = newStateStoreAdapter(daemon.jsManager)
 
+	// Re-bind the instance service so describe-stopped/terminated handlers see
+	// the KV that was just initialised.
+	daemon.instanceService = handlers_ec2_instance.NewInstanceServiceImpl(
+		daemon.config, daemon.resourceMgr.instanceTypes, daemon.natsConn,
+		objectstore.NewMemoryObjectStore(),
+		daemon.vmMgr, daemon.resourceMgr, daemon.jsManager,
+	)
+
 	return daemon
 }
 
@@ -516,6 +524,7 @@ func TestHandleEC2RunInstances_ServiceErrorPropagated(t *testing.T) {
 	daemon.instanceService = handlers_ec2_instance.NewInstanceServiceImpl(
 		daemon.config, emptyTypes, daemon.natsConn,
 		objectstore.NewMemoryObjectStore(),
+		daemon.vmMgr, daemon.resourceMgr, nil,
 	)
 
 	sub, err := daemon.natsConn.QueueSubscribe("ec2.RunInstances", "spinifex-workers", daemon.handleEC2RunInstances)

spinifex/daemon/daemon_test.go

@@ -73,6 +73,7 @@ func createTestDaemon(t *testing.T, natsURL string) *Daemon {
 
 	cfg := &config.Config{
 		BaseDir: tmpDir,
+		DataDir: tmpDir,
 		WalDir:  tmpDir,
 		NATS: config.NATSConfig{
 			Host: natsURL,
@@ -102,8 +103,11 @@ func createTestDaemon(t *testing.T, natsURL string) *Daemon {
 	daemon.natsConn = nc
 	daemon.detachDelay = 0 // Skip sleep in tests
 
-	// Initialize services (needed for handler tests)
-	daemon.instanceService = handlers_ec2_instance.NewInstanceServiceImpl(cfg, daemon.resourceMgr.instanceTypes, nc, objectstore.NewMemoryObjectStore())
+	// Initialize services (needed for handler tests).
+	// jsManager is nil here; pass a nil literal to keep the StoppedInstanceStore
+	// interface itself nil (rather than a typed-nil pointer) so the service can
+	// short-circuit cleanly when no KV is available.
+	daemon.instanceService = handlers_ec2_instance.NewInstanceServiceImpl(cfg, daemon.resourceMgr.instanceTypes, nc, objectstore.NewMemoryObjectStore(), daemon.vmMgr, daemon.resourceMgr, nil)
 	daemon.volumeService = handlers_ec2_volume.NewVolumeServiceImplWithStore(cfg, objectstore.NewMemoryObjectStore(), nc)
 
 	// Wire the minimum vm.Deps that handler tests rely on. Lifecycle (Run/Start/
@@ -393,6 +397,7 @@ func TestDaemon_Initialization(t *testing.T) {
 
 	cfg := &config.Config{
 		BaseDir: tmpDir,
+		DataDir: tmpDir,
 		WalDir:  tmpDir,
 		NATS: config.NATSConfig{
 			Host: "nats://localhost:4222",
@@ -939,11 +944,11 @@ func TestDaemon_BootAllocation(t *testing.T) {
 	// Create daemon with NATS connection
 	clusterCfg := &config.ClusterConfig{
 		Node:  "node-1",
-		Nodes: map[string]config.Config{"node-1": {BaseDir: tmpDir}},
+		Nodes: map[string]config.Config{"node-1": {BaseDir: tmpDir, DataDir: tmpDir}},
 	}
 	daemon, err := NewDaemon(clusterCfg)
 	require.NoError(t, err)
-	daemon.config = &config.Config{BaseDir: tmpDir}
+	daemon.config = &config.Config{BaseDir: tmpDir, DataDir: tmpDir}
 
 	// Connect to NATS and initialize JetStream
 	nc, err := nats.Connect(natsURL)
@@ -956,14 +961,14 @@ func TestDaemon_BootAllocation(t *testing.T) {
 	err = daemon.jsManager.InitKVBucket()
 	require.NoError(t, err)
 
-	// Pre-populate JetStream with test state
-	err = daemon.jsManager.WriteState("node-1", vms)
+	// Pre-populate the local state file with test state. Post-1a, LoadState
+	// reads from the local file (KV is best-effort cache only).
+	err = WriteLocalState(daemon.localStatePath(), vms)
 	require.NoError(t, err)
 
-	// Manually trigger the load and allocation logic normally found in Start()
-	loaded, err := daemon.jsManager.LoadState(daemon.node)
+	// Manually trigger the LoadState and allocation logic normally found in Start()
+	err = daemon.LoadState()
 	require.NoError(t, err)
-	daemon.vmMgr.Replace(loaded)
 
 	// Simulate the allocation loop in Start()
 	for _, instance := range daemon.vmMgr.Snapshot() {
@@ -3020,13 +3025,68 @@ func TestInstanceTypeMemoryMiB_NilSafety(t *testing.T) {
 	}))
 }
 
-// --- Daemon.WriteState nil jsManager ---
+// --- Daemon.WriteState / Daemon.LoadState nil jsManager ---
+//
+// Post-1a: local file is the source of truth, KV is best-effort. A nil
+// jsManager is a valid configuration (e.g. standalone daemon, fresh install
+// before NATS is up) and must not block local persistence.
 
 func TestDaemon_WriteState_NilJSManager(t *testing.T) {
-	d := &Daemon{jsManager: nil}
-	err := d.WriteState()
-	assert.Error(t, err)
-	assert.Contains(t, err.Error(), "JetStream manager not initialized")
+	tmpDir := t.TempDir()
+	d := &Daemon{
+		jsManager: nil,
+		config:    &config.Config{DataDir: tmpDir},
+		vmMgr:     vm.NewManager(),
+	}
+	d.vmMgr.Insert(&vm.VM{ID: "i-1"})
+	require.NoError(t, d.WriteState())
+
+	state, err := ReadLocalState(LocalStatePath(tmpDir))
+	require.NoError(t, err)
+	require.NotNil(t, state)
+	assert.Contains(t, state.VMS, "i-1")
+}
+
+func TestDaemon_LoadState_NilJSManager(t *testing.T) {
+	tmpDir := t.TempDir()
+	require.NoError(t, WriteLocalState(LocalStatePath(tmpDir), map[string]*vm.VM{
+		"i-seed": {ID: "i-seed"},
+	}))
+
+	d := &Daemon{
+		jsManager: nil,
+		config:    &config.Config{DataDir: tmpDir},
+		vmMgr:     vm.NewManager(),
+	}
+	require.NoError(t, d.LoadState())
+	_, ok := d.vmMgr.Get("i-seed")
+	assert.True(t, ok)
+}
+
+func TestDaemon_LoadState_MissingFileIsFreshInstall(t *testing.T) {
+	d := &Daemon{
+		jsManager: nil,
+		config:    &config.Config{DataDir: t.TempDir()},
+		vmMgr:     vm.NewManager(),
+	}
+	require.NoError(t, d.LoadState())
+	assert.Equal(t, 0, d.vmMgr.Count())
+}
+
+func TestDaemon_LoadState_CorruptFileFatal(t *testing.T) {
+	tmpDir := t.TempDir()
+	path := LocalStatePath(tmpDir)
+	require.NoError(t, os.MkdirAll(filepath.Dir(path), 0o750))
+	require.NoError(t, os.WriteFile(path, []byte("{not json"), 0o600))
+
+	d := &Daemon{
+		jsManager: nil,
+		config:    &config.Config{DataDir: tmpDir},
+		vmMgr:     vm.NewManager(),
+	}
+	err := d.LoadState()
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "read local state")
 }
 
 // --- GetAvailableInstanceTypeInfos edge cases ---
@@ -3688,6 +3748,7 @@ func TestClusterManager_TLSServesHTTPS(t *testing.T) {
 		Nodes: map[string]config.Config{
 			"node-1": {
 				BaseDir: tmpDir,
+				DataDir: tmpDir,
 				Daemon: config.DaemonConfig{
 					Host:    addr,
 					TLSCert: certPath,

spinifex/daemon/disconnect_test.go

@@ -0,0 +1,439 @@
+package daemon
+
+import (
+	"crypto/tls"
+	"encoding/json"
+	"fmt"
+	"net"
+	"net/http"
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+
+	"github.com/mulgadc/spinifex/spinifex/config"
+	"github.com/mulgadc/spinifex/spinifex/utils"
+	"github.com/mulgadc/spinifex/spinifex/vm"
+	"github.com/nats-io/nats-server/v2/server"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// TestDaemonModeDefaultStandalone — Mode() returns standalone before Start().
+func TestDaemonModeDefaultStandalone(t *testing.T) {
+	clusterCfg := &config.ClusterConfig{
+		Node:  "node-1",
+		Nodes: map[string]config.Config{"node-1": {}},
+	}
+	d, err := NewDaemon(clusterCfg)
+	require.NoError(t, err)
+
+	assert.Equal(t, DaemonModeStandalone, d.Mode())
+	assert.Equal(t, int64(0), d.NATSRetryCount())
+}
+
+// TestDaemonModeFlipsOnConnectAndDisconnect — connecting flips to cluster,
+// killing NATS flips back to standalone.
+func TestDaemonModeFlipsOnConnectAndDisconnect(t *testing.T) {
+	port := freePortForTest(t)
+	ns := startTestNATSOnPortForTest(t, port)
+
+	clusterCfg := &config.ClusterConfig{
+		Node:  "node-1",
+		Nodes: map[string]config.Config{"node-1": {NATS: config.NATSConfig{Host: ns.ClientURL()}}},
+	}
+	d, err := NewDaemon(clusterCfg)
+	require.NoError(t, err)
+
+	require.NoError(t, d.connectNATS())
+	defer d.natsConn.Close()
+	assert.Equal(t, DaemonModeCluster, d.Mode())
+
+	ns.Shutdown()
+	require.Eventually(t, func() bool { return d.Mode() == DaemonModeStandalone }, 3*time.Second, 50*time.Millisecond,
+		"mode should flip to standalone after NATS shutdown")
+}
+
+// TestDaemonReconnectBumpsRetryCount — reconnect callback fires, count++.
+func TestDaemonReconnectBumpsRetryCount(t *testing.T) {
+	port := freePortForTest(t)
+	ns := startTestNATSOnPortForTest(t, port)
+
+	clusterCfg := &config.ClusterConfig{
+		Node:  "node-1",
+		Nodes: map[string]config.Config{"node-1": {NATS: config.NATSConfig{Host: ns.ClientURL()}}},
+	}
+	d, err := NewDaemon(clusterCfg)
+	require.NoError(t, err)
+
+	require.NoError(t, d.connectNATS())
+	defer d.natsConn.Close()
+	require.Equal(t, int64(0), d.NATSRetryCount())
+
+	ns.Shutdown()
+	require.Eventually(t, func() bool { return !d.natsConn.IsConnected() }, 3*time.Second, 50*time.Millisecond)
+
+	startTestNATSOnPortForTest(t, port)
+	require.Eventually(t, func() bool { return d.Mode() == DaemonModeCluster }, 5*time.Second, 50*time.Millisecond,
+		"mode should flip back to cluster on reconnect")
+	assert.GreaterOrEqual(t, d.NATSRetryCount(), int64(1))
+}
+
+func freePortForTest(t *testing.T) int {
+	t.Helper()
+	l, err := net.Listen("tcp", "127.0.0.1:0")
+	require.NoError(t, err)
+	addr, ok := l.Addr().(*net.TCPAddr)
+	require.True(t, ok)
+	port := addr.Port
+	require.NoError(t, l.Close())
+	return port
+}
+
+func startTestNATSOnPortForTest(t *testing.T, port int) *server.Server {
+	t.Helper()
+	opts := &server.Options{Host: "127.0.0.1", Port: port, NoLog: true, NoSigs: true}
+	ns, err := server.NewServer(opts)
+	require.NoError(t, err)
+	go ns.Start()
+	require.True(t, ns.ReadyForConnections(5*time.Second))
+	t.Cleanup(func() { ns.Shutdown() })
+	return ns
+}
+
+// TestMode_NilStored_ReturnsStandalone — bare Daemon with no mode stored returns standalone.
+// Covers the d.mode.Load() == nil branch in Mode().
+func TestMode_NilStored_ReturnsStandalone(t *testing.T) {
+	d := &Daemon{}
+	assert.Equal(t, DaemonModeStandalone, d.Mode())
+}
+
+// TestMode_WrongTypeStored_ReturnsStandalone — non-string in atomic.Value falls
+// back to standalone instead of panicking. Covers the type-assertion !ok branch.
+func TestMode_WrongTypeStored_ReturnsStandalone(t *testing.T) {
+	d := &Daemon{}
+	d.mode.Store(struct{ x string }{x: "not a string"})
+	assert.Equal(t, DaemonModeStandalone, d.Mode())
+}
+
+// TestOnNATSDisconnect_FlipsMode — direct unit test of the disconnect callback
+// without needing an actual NATS disconnect roundtrip.
+func TestOnNATSDisconnect_FlipsMode(t *testing.T) {
+	d := &Daemon{}
+	d.mode.Store(DaemonModeCluster)
+	d.onNATSDisconnect(nil, nil)
+	assert.Equal(t, DaemonModeStandalone, d.Mode())
+}
+
+// TestOnNATSReconnect_NoJetStreamManager_DoesNotPanic — reconnect with
+// jsManager nil flips mode + bumps counter but skips the goroutine WriteState.
+func TestOnNATSReconnect_NoJetStreamManager_DoesNotPanic(t *testing.T) {
+	d := &Daemon{}
+	d.mode.Store(DaemonModeStandalone)
+
+	d.onNATSReconnect(nil)
+
+	assert.Equal(t, DaemonModeCluster, d.Mode())
+	assert.Equal(t, int64(1), d.NATSRetryCount())
+}
+
+// TestLocalStatePath_NilConfig_UsesDefault — defensive path used by callers
+// that build a Daemon without populating Config (older test fixtures, etc).
+func TestLocalStatePath_NilConfig_UsesDefault(t *testing.T) {
+	d := &Daemon{}
+	assert.Equal(t, "/var/lib/spinifex/state/instance-state.json", d.localStatePath())
+}
+
+// TestLocalStatePath_RootedAtDataDir — verifies 1a's DataDir-rooted layout.
+func TestLocalStatePath_RootedAtDataDir(t *testing.T) {
+	d := &Daemon{config: &config.Config{DataDir: "/var/lib/spinifex/n1"}}
+	assert.Equal(t, "/var/lib/spinifex/n1/state/instance-state.json", d.localStatePath())
+}
+
+// TestDaemonWriteState_LocalFile — WriteState persists to the configured DataDir
+// even with a nil jsManager (best-effort KV is the second half; local file
+// must succeed standalone).
+func TestDaemonWriteState_LocalFile(t *testing.T) {
+	dataDir := t.TempDir()
+	d := &Daemon{
+		config: &config.Config{DataDir: dataDir},
+		vmMgr:  vm.NewManager(),
+	}
+	d.vmMgr.Insert(&vm.VM{ID: "i-w1", InstanceType: "t3.micro"})
+
+	require.NoError(t, d.WriteState())
+
+	state, err := ReadLocalState(filepath.Join(dataDir, "state", "instance-state.json"))
+	require.NoError(t, err)
+	require.NotNil(t, state)
+	assert.Equal(t, "t3.micro", state.VMS["i-w1"].InstanceType)
+}
+
+// TestDaemonLoadState_MissingFile — fresh-install path: empty map, no error.
+func TestDaemonLoadState_MissingFile(t *testing.T) {
+	d := &Daemon{
+		config: &config.Config{DataDir: t.TempDir()},
+		vmMgr:  vm.NewManager(),
+	}
+	require.NoError(t, d.LoadState())
+	assert.Equal(t, 0, d.vmMgr.Count())
+}
+
+// TestDaemonLoadState_RoundTrip — write, then read back into a fresh daemon
+// rooted at the same DataDir.
+func TestDaemonLoadState_RoundTrip(t *testing.T) {
+	dataDir := t.TempDir()
+	writer := &Daemon{
+		config: &config.Config{DataDir: dataDir},
+		vmMgr:  vm.NewManager(),
+	}
+	writer.vmMgr.Insert(&vm.VM{ID: "i-rt1", InstanceType: "m5.large"})
+	require.NoError(t, writer.WriteState())
+
+	reader := &Daemon{
+		config: &config.Config{DataDir: dataDir},
+		vmMgr:  vm.NewManager(),
+	}
+	require.NoError(t, reader.LoadState())
+	assert.Equal(t, 1, reader.vmMgr.Count())
+	got, ok := reader.vmMgr.Get("i-rt1")
+	require.True(t, ok)
+	assert.Equal(t, "m5.large", got.InstanceType)
+}
+
+// TestDaemonLoadState_CorruptFile — corruption is fatal, daemon refuses start.
+func TestDaemonLoadState_CorruptFile(t *testing.T) {
+	dataDir := t.TempDir()
+	d := &Daemon{config: &config.Config{DataDir: dataDir}}
+
+	path := d.localStatePath()
+	require.NoError(t, writeCorruptStateFile(path))
+
+	err := d.LoadState()
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "read local state")
+}
+
+func writeCorruptStateFile(path string) error {
+	if err := os.MkdirAll(filepath.Dir(path), 0o750); err != nil {
+		return err
+	}
+	return os.WriteFile(path, []byte("{not json"), 0o600)
+}
+
+// TestStartLocal_SucceedsWithoutNATS — §1d: startLocal() brings the daemon up
+// (cluster manager HTTPS, local state load, ready flag) without contacting
+// NATS. /local/status reports standalone mode immediately.
+func TestStartLocal_SucceedsWithoutNATS(t *testing.T) {
+	tmpDir := t.TempDir()
+	certPEM, keyPEM := generateTestCert(t)
+	certPath := filepath.Join(tmpDir, "server.pem")
+	keyPath := filepath.Join(tmpDir, "server.key")
+	require.NoError(t, os.WriteFile(certPath, certPEM, 0o600))
+	require.NoError(t, os.WriteFile(keyPath, keyPEM, 0o600))
+
+	addr, err := freeAddrForTest()
+	require.NoError(t, err)
+
+	cfg := &config.ClusterConfig{
+		Node: "node-1",
+		Nodes: map[string]config.Config{
+			"node-1": {
+				BaseDir: tmpDir,
+				DataDir: tmpDir,
+				NATS:    config.NATSConfig{Host: "nats://127.0.0.1:1"}, // unreachable
+				Daemon: config.DaemonConfig{
+					Host:    addr,
+					TLSCert: certPath,
+					TLSKey:  keyPath,
+				},
+			},
+		},
+	}
+
+	d, err := NewDaemon(cfg)
+	require.NoError(t, err)
+	d.configPath = filepath.Join(tmpDir, "spinifex.toml")
+
+	require.NoError(t, d.startLocal())
+	defer func() {
+		if d.clusterServer != nil {
+			_ = d.clusterServer.Close()
+		}
+	}()
+
+	assert.True(t, d.ready.Load(), "ready flag should be set after startLocal")
+	assert.Equal(t, DaemonModeStandalone, d.Mode())
+
+	// /local/status served via the cluster manager TLS listener.
+	client := &http.Client{
+		Transport: &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}},
+		Timeout:   2 * time.Second,
+	}
+	resp, err := client.Get(fmt.Sprintf("https://%s/local/status", addr))
+	require.NoError(t, err)
+	defer resp.Body.Close()
+	assert.Equal(t, http.StatusOK, resp.StatusCode)
+
+	var body LocalStatus
+	require.NoError(t, json.NewDecoder(resp.Body).Decode(&body))
+	assert.Equal(t, DaemonModeStandalone, body.Mode)
+	assert.Equal(t, natsDisconnected, body.NATS)
+}
+
+// TestAssertNoClusterServicesInitialised_GuardsTier1Invariant — §1e-audit:
+// the invariant guard at the end of startLocal must reject any state where a
+// cluster-scoped handle has been initialised before startCluster runs. A
+// future edit that hoists JetStream/KV-touching init into startLocal would
+// re-introduce the boot-time NATS dependency that 1d removed; this test
+// pins the regression.
+func TestAssertNoClusterServicesInitialised_GuardsTier1Invariant(t *testing.T) {
+	clean := &Daemon{}
+	require.NoError(t, clean.assertNoClusterServicesInitialised(),
+		"freshly-constructed daemon must satisfy the Tier 1 invariant")
+
+	dirty := &Daemon{jsManager: &JetStreamManager{}}
+	err := dirty.assertNoClusterServicesInitialised()
+	require.Error(t, err, "non-nil cluster handle must trip the invariant")
+	assert.Contains(t, err.Error(), "jsManager")
+}
+
+// TestStartCluster_StrictRequireNATS_BoundedAndExits — §1d-strict:
+// SPINIFEX_REQUIRE_NATS=1 caps the first connect to requireNATSTimeout and
+// invokes exitFunc(1) when the bounded retry expires. Validates the dev/test
+// fail-fast path without flipping the prod default.
+func TestStartCluster_StrictRequireNATS_BoundedAndExits(t *testing.T) {
+	t.Setenv("SPINIFEX_REQUIRE_NATS", "1")
+
+	tmpDir := t.TempDir()
+	cfg := &config.ClusterConfig{
+		Node: "node-1",
+		Nodes: map[string]config.Config{
+			"node-1": {
+				BaseDir: tmpDir,
+				DataDir: tmpDir,
+				NATS:    config.NATSConfig{Host: "nats://127.0.0.1:1"}, // unreachable
+			},
+		},
+	}
+
+	d, err := NewDaemon(cfg)
+	require.NoError(t, err)
+	d.requireNATSTimeout = 200 * time.Millisecond
+	d.natsRetryOpts = []utils.RetryOption{utils.WithRetryDelay(20 * time.Millisecond)}
+
+	exitCh := make(chan int, 1)
+	d.exitFunc = func(code int) { exitCh <- code }
+
+	done := make(chan error, 1)
+	go func() { done <- d.startCluster() }()
+
+	select {
+	case code := <-exitCh:
+		assert.Equal(t, 1, code, "strict-mode timeout must request exit(1)")
+	case <-time.After(3 * time.Second):
+		t.Fatal("strict-mode startCluster did not invoke exitFunc within deadline")
+	}
+
+	select {
+	case err := <-done:
+		require.Error(t, err)
+		assert.Contains(t, err.Error(), "connect NATS (strict)")
+	case <-time.After(3 * time.Second):
+		t.Fatal("startCluster did not return after exitFunc was called")
+	}
+}
+
+// TestStartCluster_NoStrictEnv_UsesInfiniteRetry — §1d-strict:
+// without SPINIFEX_REQUIRE_NATS, startCluster keeps the Tier 1 infinite-retry
+// path. exitFunc must not fire, even when NATS is unreachable. Cancellation
+// via d.ctx is the only way out.
+func TestStartCluster_NoStrictEnv_UsesInfiniteRetry(t *testing.T) {
+	t.Setenv("SPINIFEX_REQUIRE_NATS", "")
+
+	tmpDir := t.TempDir()
+	cfg := &config.ClusterConfig{
+		Node: "node-1",
+		Nodes: map[string]config.Config{
+			"node-1": {
+				BaseDir: tmpDir,
+				DataDir: tmpDir,
+				NATS:    config.NATSConfig{Host: "nats://127.0.0.1:1"}, // unreachable
+			},
+		},
+	}
+
+	d, err := NewDaemon(cfg)
+	require.NoError(t, err)
+	d.natsRetryOpts = []utils.RetryOption{utils.WithRetryDelay(20 * time.Millisecond)}
+
+	exitCalled := make(chan int, 1)
+	d.exitFunc = func(code int) { exitCalled <- code }
+
+	done := make(chan error, 1)
+	go func() { done <- d.startCluster() }()
+
+	// Give the loop time to attempt several retries; it must not exit.
+	select {
+	case code := <-exitCalled:
+		t.Fatalf("default mode must not invoke exitFunc; got code=%d", code)
+	case <-time.After(300 * time.Millisecond):
+	}
+
+	d.cancel()
+
+	select {
+	case err := <-done:
+		require.Error(t, err)
+		assert.Contains(t, err.Error(), "connect NATS")
+		assert.NotContains(t, err.Error(), "strict")
+	case <-time.After(3 * time.Second):
+		t.Fatal("startCluster did not return after ctx cancellation")
+	}
+}
+
+// TestStartCluster_RetriesUntilContextCancelled — §1d: startCluster's NATS
+// connect loop honours d.ctx. Cancelling the daemon context unblocks an
+// otherwise-infinite retry.
+func TestStartCluster_RetriesUntilContextCancelled(t *testing.T) {
+	tmpDir := t.TempDir()
+	cfg := &config.ClusterConfig{
+		Node: "node-1",
+		Nodes: map[string]config.Config{
+			"node-1": {
+				BaseDir: tmpDir,
+				DataDir: tmpDir,
+				NATS:    config.NATSConfig{Host: "nats://127.0.0.1:1"}, // unreachable
+			},
+		},
+	}
+
+	d, err := NewDaemon(cfg)
+	require.NoError(t, err)
+	// Bound retry delay so ctx-cancel unblocks promptly.
+	d.natsRetryOpts = []utils.RetryOption{utils.WithRetryDelay(50 * time.Millisecond)}
+
+	done := make(chan error, 1)
+	go func() { done <- d.startCluster() }()
+
+	time.Sleep(200 * time.Millisecond)
+	d.cancel()
+
+	select {
+	case err := <-done:
+		require.Error(t, err)
+		assert.Contains(t, err.Error(), "connect NATS")
+	case <-time.After(3 * time.Second):
+		t.Fatal("startCluster did not return after ctx cancellation")
+	}
+}
+
+func freeAddrForTest() (string, error) {
+	l, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		return "", err
+	}
+	addr := l.Addr().String()
+	_ = l.Close()
+	return addr, nil
+}

spinifex/daemon/jetstream.go

@@ -35,6 +35,15 @@ const (
 	TerminatedInstanceBucketVersion = 1
 )
 
+// KVSyncObserver receives best-effort KV sync outcomes from
+// WriteStateBytesBestEffort. Implementations must be safe for concurrent use
+// and must not block — callbacks run in the same goroutine that performed the
+// Put. nil observer is allowed.
+type KVSyncObserver interface {
+	RecordKVSyncSuccess(bucket string)
+	RecordKVSyncFailure(bucket string, err error)
+}
+
 // JetStreamManager manages JetStream KV store operations for instance state
 type JetStreamManager struct {
 	js           nats.JetStreamContext
@@ -43,6 +52,13 @@ type JetStreamManager struct {
 	terminatedKV nats.KeyValue // spinifex-terminated-instances
 	replicas     int
 	kvMu         sync.Mutex // protects kv during recovery
+	obs          KVSyncObserver
+}
+
+// SetSyncObserver registers obs to receive best-effort KV sync outcomes. Pass
+// nil to clear. Safe to call before or after Init*Bucket.
+func (m *JetStreamManager) SetSyncObserver(obs KVSyncObserver) {
+	m.obs = obs
 }
 
 // NewJetStreamManager creates a new JetStreamManager from a NATS connection.
@@ -396,14 +412,7 @@ func (m *JetStreamManager) WriteState(nodeID string, vms map[string]*vm.VM) erro
 		return errors.New("KV bucket not initialized")
 	}
 
-	// Create a struct without the mutex to avoid copying the lock
-	state := struct {
-		VMS map[string]*vm.VM `json:"vms"`
-	}{
-		VMS: vms,
-	}
-
-	jsonData, err := json.Marshal(state)
+	jsonData, err := marshalInstanceState(vms)
 	if err != nil {
 		return err
 	}
@@ -430,6 +439,78 @@ func (m *JetStreamManager) WriteState(nodeID string, vms map[string]*vm.VM) erro
 	return nil
 }
 
+// WriteStateBestEffort attempts to push instance state to KV with a deadline.
+// On timeout or error, it logs a warning and returns — never blocks the caller
+// past `timeout` and never returns an error. Used when the local state file is
+// the source of truth and KV is a best-effort cache. vms must be a snapshot
+// owned by the caller (e.g. from vm.Manager.SnapshotMap).
+//
+// Note: kv.Put has no context API. On timeout, the in-flight Put goroutine
+// continues and completes (or fails) on its own. This leaks at most one
+// goroutine per write; under sustained partition the leak is bounded by
+// WriteState call cadence (per-state-transition, not a tight loop).
+func (m *JetStreamManager) WriteStateBestEffort(nodeID string, vms map[string]*vm.VM, timeout time.Duration) {
+	if m.kv == nil {
+		slog.Debug("KV bucket not initialized, skipping cluster sync", "node", nodeID)
+		return
+	}
+
+	jsonData, err := marshalInstanceState(vms)
+	if err != nil {
+		slog.Warn("KV sync skipped: marshal failed", "node", nodeID, "err", err)
+		return
+	}
+
+	m.WriteStateBytesBestEffort(nodeID, jsonData, timeout)
+}
+
+// WriteStateBytesBestEffort behaves like WriteStateBestEffort but accepts
+// pre-marshalled JSON. Used by hot paths that marshal under a short-lived lock
+// and commit lock-free.
+func (m *JetStreamManager) WriteStateBytesBestEffort(nodeID string, jsonData []byte, timeout time.Duration) {
+	if m.kv == nil {
+		slog.Debug("KV bucket not initialized, skipping cluster sync", "node", nodeID)
+		return
+	}
+
+	key := InstanceStatePrefix + nodeID
+	done := make(chan error, 1)
+	go func() {
+		_, putErr := m.kv.Put(key, jsonData)
+		done <- putErr
+	}()
+
+	select {
+	case putErr := <-done:
+		if putErr != nil {
+			if m.obs != nil {
+				m.obs.RecordKVSyncFailure(InstanceStateBucket, putErr)
+			}
+			slog.Warn("KV sync failed (best-effort)", "key", key, "err", putErr)
+			return
+		}
+		if m.obs != nil {
+			m.obs.RecordKVSyncSuccess(InstanceStateBucket)
+		}
+		slog.Debug("Wrote state to KV (best-effort)", "key", key, "bytes", len(jsonData))
+	case <-time.After(timeout):
+		if m.obs != nil {
+			m.obs.RecordKVSyncFailure(InstanceStateBucket, fmt.Errorf("kv sync timeout after %s", timeout))
+		}
+		slog.Warn("KV sync timed out (best-effort)", "key", key, "timeout", timeout)
+	}
+}
+
+// marshalInstanceState produces the JSON wire form of vms.
+func marshalInstanceState(vms map[string]*vm.VM) ([]byte, error) {
+	state := struct {
+		VMS map[string]*vm.VM `json:"vms"`
+	}{
+		VMS: vms,
+	}
+	return json.Marshal(state)
+}
+
 // LoadState loads the instance state from the KV store for the given node.
 // Returns an empty (non-nil) map when no state exists for the node.
 func (m *JetStreamManager) LoadState(nodeID string) (map[string]*vm.VM, error) {

spinifex/daemon/jetstream_test.go

@@ -3,7 +3,9 @@ package daemon
 import (
 	"encoding/json"
 	"errors"
+	"sync"
 	"testing"
+	"time"
 
 	"github.com/mulgadc/spinifex/spinifex/utils"
 	"github.com/mulgadc/spinifex/spinifex/vm"
@@ -1246,3 +1248,91 @@ func TestJetStreamManager_ListTerminatedInstances_SkipsVersionKey(t *testing.T)
 	require.NoError(t, err)
 	assert.Empty(t, instances)
 }
+
+// fakeKVObserver records observer callbacks for tests.
+type fakeKVObserver struct {
+	mu        sync.Mutex
+	successes []string
+	failures  []fakeKVObserverFailure
+}
+
+type fakeKVObserverFailure struct {
+	bucket string
+	err    error
+}
+
+func (f *fakeKVObserver) RecordKVSyncSuccess(bucket string) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.successes = append(f.successes, bucket)
+}
+
+func (f *fakeKVObserver) RecordKVSyncFailure(bucket string, err error) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.failures = append(f.failures, fakeKVObserverFailure{bucket: bucket, err: err})
+}
+
+func (f *fakeKVObserver) snapshot() ([]string, []fakeKVObserverFailure) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	successes := append([]string(nil), f.successes...)
+	failures := append([]fakeKVObserverFailure(nil), f.failures...)
+	return successes, failures
+}
+
+func TestJetStreamManager_BestEffort_Success_NotifiesObserver(t *testing.T) {
+	nc, err := nats.Connect(sharedJSNATSURL)
+	require.NoError(t, err)
+	defer nc.Close()
+
+	jsm, err := NewJetStreamManager(nc, 1)
+	require.NoError(t, err)
+	require.NoError(t, jsm.InitKVBucket())
+
+	obs := &fakeKVObserver{}
+	jsm.SetSyncObserver(obs)
+
+	jsm.WriteStateBytesBestEffort("obs-success", []byte(`{"vms":{}}`), 5*time.Second)
+
+	successes, failures := obs.snapshot()
+	require.Len(t, successes, 1)
+	assert.Equal(t, InstanceStateBucket, successes[0])
+	assert.Empty(t, failures)
+}
+
+func TestJetStreamManager_BestEffort_PutError_NotifiesObserver(t *testing.T) {
+	nc, err := nats.Connect(sharedJSNATSURL)
+	require.NoError(t, err)
+
+	jsm, err := NewJetStreamManager(nc, 1)
+	require.NoError(t, err)
+	require.NoError(t, jsm.InitKVBucket())
+
+	obs := &fakeKVObserver{}
+	jsm.SetSyncObserver(obs)
+
+	// Closing the connection forces the inflight Put to fail without timing out.
+	nc.Close()
+
+	jsm.WriteStateBytesBestEffort("obs-fail", []byte(`{"vms":{}}`), 5*time.Second)
+
+	successes, failures := obs.snapshot()
+	assert.Empty(t, successes)
+	require.Len(t, failures, 1)
+	assert.Equal(t, InstanceStateBucket, failures[0].bucket)
+	assert.Error(t, failures[0].err)
+}
+
+func TestJetStreamManager_BestEffort_NilObserver_NoPanic(t *testing.T) {
+	nc, err := nats.Connect(sharedJSNATSURL)
+	require.NoError(t, err)
+	defer nc.Close()
+
+	jsm, err := NewJetStreamManager(nc, 1)
+	require.NoError(t, err)
+	require.NoError(t, jsm.InitKVBucket())
+
+	// No observer set — must not panic on success or failure paths.
+	jsm.WriteStateBytesBestEffort("obs-nil", []byte(`{"vms":{}}`), 5*time.Second)
+}

spinifex/daemon/local_api.go

@@ -0,0 +1,128 @@
+package daemon
+
+import (
+	"encoding/json"
+	"log/slog"
+	"net/http"
+	"sort"
+	"time"
+
+	"github.com/go-chi/chi/v5"
+	"github.com/mulgadc/spinifex/spinifex/utils"
+	"github.com/mulgadc/spinifex/spinifex/vm"
+)
+
+// NATS connectivity strings reported by /local/status.
+const (
+	natsConnected    = "connected"
+	natsDisconnected = "disconnected"
+)
+
+// LocalInstance is one entry of GET /local/instances. Mirrors the harness shape
+// in spinifex/tests/e2e/ddil/harness/daemon_client.go.
+type LocalInstance struct {
+	InstanceID string `json:"instance_id"`
+	State      string `json:"state"`
+	PID        int    `json:"pid,omitempty"`
+}
+
+// LocalStatus is the response shape for GET /local/status. Reports the
+// daemon's connectivity mode plus enough context for an operator to detect
+// flapping (nats_retry_count), stale state (revision), or prolonged divergence
+// between the local file and JetStream KV (kv_sync_failures, last_kv_sync_at,
+// last_kv_sync_error).
+type LocalStatus struct {
+	Node            string `json:"node"`
+	Mode            string `json:"mode"`
+	NATS            string `json:"nats"`
+	NATSRetryCount  int64  `json:"nats_retry_count"`
+	Revision        uint64 `json:"revision"`
+	KVSyncFailures  int64  `json:"kv_sync_failures"`
+	LastKVSyncAt    string `json:"last_kv_sync_at,omitempty"`
+	LastKVSyncError string `json:"last_kv_sync_error,omitempty"`
+}
+
+// registerLocalRoutes wires the read-only /local/* endpoints onto r. Called
+// from ClusterManager so the routes share the existing TLS listener and stay
+// reachable while NATS is down.
+func (d *Daemon) registerLocalRoutes(r chi.Router) {
+	r.Get("/local/instances", d.handleLocalInstances)
+	r.Get("/local/instances/{id}", d.handleLocalInstance)
+	r.Get("/local/status", d.handleLocalStatus)
+}
+
+func (d *Daemon) handleLocalInstances(w http.ResponseWriter, _ *http.Request) {
+	out := []LocalInstance{}
+	d.vmMgr.View(func(vms map[string]*vm.VM) {
+		for _, v := range vms {
+			out = append(out, vmToLocalInstance(v))
+		}
+	})
+	sort.Slice(out, func(i, j int) bool { return out[i].InstanceID < out[j].InstanceID })
+	writeJSON(w, http.StatusOK, out)
+}
+
+func (d *Daemon) handleLocalInstance(w http.ResponseWriter, r *http.Request) {
+	id := chi.URLParam(r, "id")
+	v, ok := d.vmMgr.Get(id)
+	if !ok {
+		writeJSONError(w, http.StatusNotFound, "instance not found")
+		return
+	}
+	var li LocalInstance
+	d.vmMgr.Inspect(v, func(v *vm.VM) {
+		li = vmToLocalInstance(v)
+	})
+	writeJSON(w, http.StatusOK, li)
+}
+
+func (d *Daemon) handleLocalStatus(w http.ResponseWriter, _ *http.Request) {
+	status := LocalStatus{
+		Node:            d.node,
+		Mode:            d.Mode(),
+		NATS:            d.natsConnectivity(),
+		NATSRetryCount:  d.NATSRetryCount(),
+		Revision:        d.Revision(),
+		KVSyncFailures:  d.KVSyncFailures(),
+		LastKVSyncError: d.LastKVSyncError(),
+	}
+	if t := d.LastKVSyncAt(); !t.IsZero() {
+		status.LastKVSyncAt = t.UTC().Format(time.RFC3339)
+	}
+	writeJSON(w, http.StatusOK, status)
+}
+
+// natsConnectivity reports the live NATS link state. Mirrors the check used by
+// the /health handler so /local/status agrees with /health on the same node.
+func (d *Daemon) natsConnectivity() string {
+	if d.natsConn != nil && d.natsConn.IsConnected() {
+		return natsConnected
+	}
+	return natsDisconnected
+}
+
+func vmToLocalInstance(v *vm.VM) LocalInstance {
+	li := LocalInstance{
+		InstanceID: v.ID,
+		State:      string(v.Status),
+	}
+	// PID is read from the QEMU pidfile each request; missing file or unreadable
+	// value just omits the field (json:"omitempty"). Read failures are not
+	// surfaced — /local/instances must keep working even when QEMU is gone.
+	if pid, err := utils.ReadPidFile(v.ID); err == nil {
+		li.PID = pid
+	}
+	return li
+}
+
+func writeJSON(w http.ResponseWriter, status int, body any) {
+	w.Header().Set("Content-Type", "application/json")
+	w.WriteHeader(status)
+	if err := json.NewEncoder(w).Encode(body); err != nil {
+		slog.Error("local API encode failed", "error", err)
+	}
+}
+
+func writeJSONError(w http.ResponseWriter, status int, msg string) {
+	writeJSON(w, status, map[string]string{"error": msg})
+}

spinifex/daemon/local_api_test.go

@@ -0,0 +1,188 @@
+package daemon
+
+import (
+	"encoding/json"
+	"errors"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+	"time"
+
+	"github.com/go-chi/chi/v5"
+	"github.com/mulgadc/spinifex/spinifex/config"
+	"github.com/mulgadc/spinifex/spinifex/utils"
+	"github.com/mulgadc/spinifex/spinifex/vm"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// newLocalAPITestDaemon returns a Daemon with just enough wiring to exercise
+// the read-only /local/* handlers: a vm.Manager, a node name, and a Config so
+// any path-resolving code (revision bump via WriteState) has a temp DataDir.
+// XDG_RUNTIME_DIR is pinned to a temp dir so utils.WritePidFile / ReadPidFile
+// (used by vmToLocalInstance to surface PIDs) stay scoped to the test.
+func newLocalAPITestDaemon(t *testing.T) *Daemon {
+	t.Helper()
+	t.Setenv("XDG_RUNTIME_DIR", t.TempDir())
+	d := &Daemon{
+		node:   "node-1",
+		vmMgr:  vm.NewManager(),
+		config: &config.Config{DataDir: t.TempDir()},
+	}
+	d.mode.Store(DaemonModeStandalone)
+	return d
+}
+
+func newLocalAPIRouter(d *Daemon) chi.Router {
+	r := chi.NewRouter()
+	d.registerLocalRoutes(r)
+	return r
+}
+
+func doGET(t *testing.T, r chi.Router, path string) *httptest.ResponseRecorder {
+	t.Helper()
+	req := httptest.NewRequest(http.MethodGet, path, nil)
+	rec := httptest.NewRecorder()
+	r.ServeHTTP(rec, req)
+	return rec
+}
+
+func TestLocalAPI_Instances_Empty(t *testing.T) {
+	d := newLocalAPITestDaemon(t)
+	rec := doGET(t, newLocalAPIRouter(d), "/local/instances")
+	require.Equal(t, http.StatusOK, rec.Code)
+
+	var got []LocalInstance
+	require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &got))
+	assert.Equal(t, []LocalInstance{}, got)
+}
+
+func TestLocalAPI_Instances_ListsVMs(t *testing.T) {
+	d := newLocalAPITestDaemon(t)
+	d.vmMgr.Insert(&vm.VM{ID: "i-b", Status: vm.StateRunning})
+	d.vmMgr.Insert(&vm.VM{ID: "i-a", Status: vm.StateStopped})
+	require.NoError(t, utils.WritePidFile("i-b", 4242))
+
+	rec := doGET(t, newLocalAPIRouter(d), "/local/instances")
+	require.Equal(t, http.StatusOK, rec.Code)
+
+	var got []LocalInstance
+	require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &got))
+	require.Len(t, got, 2)
+	// Sorted by InstanceID for stable output.
+	assert.Equal(t, "i-a", got[0].InstanceID)
+	assert.Equal(t, "stopped", got[0].State)
+	assert.Zero(t, got[0].PID)
+	assert.Equal(t, "i-b", got[1].InstanceID)
+	assert.Equal(t, "running", got[1].State)
+	assert.Equal(t, 4242, got[1].PID)
+}
+
+func TestLocalAPI_Instance_Found(t *testing.T) {
+	d := newLocalAPITestDaemon(t)
+	d.vmMgr.Insert(&vm.VM{ID: "i-find", Status: vm.StateRunning})
+	require.NoError(t, utils.WritePidFile("i-find", 7))
+
+	rec := doGET(t, newLocalAPIRouter(d), "/local/instances/i-find")
+	require.Equal(t, http.StatusOK, rec.Code)
+
+	var got LocalInstance
+	require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &got))
+	assert.Equal(t, LocalInstance{InstanceID: "i-find", State: "running", PID: 7}, got)
+}
+
+func TestLocalAPI_Instance_NotFound(t *testing.T) {
+	d := newLocalAPITestDaemon(t)
+	rec := doGET(t, newLocalAPIRouter(d), "/local/instances/i-missing")
+	assert.Equal(t, http.StatusNotFound, rec.Code)
+}
+
+func TestLocalAPI_Status_Standalone_NoNATS(t *testing.T) {
+	d := newLocalAPITestDaemon(t)
+
+	rec := doGET(t, newLocalAPIRouter(d), "/local/status")
+	require.Equal(t, http.StatusOK, rec.Code)
+
+	var got LocalStatus
+	require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &got))
+	assert.Equal(t, LocalStatus{
+		Node:           "node-1",
+		Mode:           DaemonModeStandalone,
+		NATS:           natsDisconnected,
+		NATSRetryCount: 0,
+		Revision:       0,
+	}, got)
+}
+
+func TestLocalAPI_Status_RetryCountAndRevisionPropagate(t *testing.T) {
+	d := newLocalAPITestDaemon(t)
+	d.mode.Store(DaemonModeCluster)
+	d.natsRetryCount.Store(3)
+	require.NoError(t, d.WriteState())
+	require.NoError(t, d.WriteState())
+
+	rec := doGET(t, newLocalAPIRouter(d), "/local/status")
+	require.Equal(t, http.StatusOK, rec.Code)
+
+	var got LocalStatus
+	require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &got))
+	assert.Equal(t, DaemonModeCluster, got.Mode)
+	assert.Equal(t, int64(3), got.NATSRetryCount)
+	assert.Equal(t, uint64(2), got.Revision)
+}
+
+func TestLocalAPI_Status_ContentType(t *testing.T) {
+	d := newLocalAPITestDaemon(t)
+	rec := doGET(t, newLocalAPIRouter(d), "/local/status")
+	assert.Equal(t, "application/json", rec.Header().Get("Content-Type"))
+}
+
+func TestLocalAPI_Status_KVSync_DefaultsAreZero(t *testing.T) {
+	d := newLocalAPITestDaemon(t)
+
+	rec := doGET(t, newLocalAPIRouter(d), "/local/status")
+	require.Equal(t, http.StatusOK, rec.Code)
+
+	var got LocalStatus
+	require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &got))
+	assert.Zero(t, got.KVSyncFailures)
+	assert.Empty(t, got.LastKVSyncAt)
+	assert.Empty(t, got.LastKVSyncError)
+}
+
+func TestLocalAPI_Status_KVSyncFailure_PropagatesFields(t *testing.T) {
+	d := newLocalAPITestDaemon(t)
+
+	d.RecordKVSyncFailure(InstanceStateBucket, errors.New("kv timeout"))
+	d.RecordKVSyncFailure(InstanceStateBucket, errors.New("kv put rejected"))
+
+	rec := doGET(t, newLocalAPIRouter(d), "/local/status")
+	require.Equal(t, http.StatusOK, rec.Code)
+
+	var got LocalStatus
+	require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &got))
+	assert.Equal(t, int64(2), got.KVSyncFailures)
+	assert.Equal(t, "kv put rejected", got.LastKVSyncError)
+	assert.Empty(t, got.LastKVSyncAt, "no successful sync yet")
+}
+
+func TestLocalAPI_Status_KVSyncSuccess_ClearsErrorAndStampsTime(t *testing.T) {
+	d := newLocalAPITestDaemon(t)
+
+	d.RecordKVSyncFailure(InstanceStateBucket, errors.New("transient"))
+	before := time.Now()
+	d.RecordKVSyncSuccess(InstanceStateBucket)
+
+	rec := doGET(t, newLocalAPIRouter(d), "/local/status")
+	require.Equal(t, http.StatusOK, rec.Code)
+
+	var got LocalStatus
+	require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &got))
+	assert.Equal(t, int64(1), got.KVSyncFailures, "counter is monotonic")
+	assert.Empty(t, got.LastKVSyncError, "success clears the error string")
+
+	require.NotEmpty(t, got.LastKVSyncAt)
+	parsed, err := time.Parse(time.RFC3339, got.LastKVSyncAt)
+	require.NoError(t, err)
+	assert.WithinDuration(t, before, parsed, 5*time.Second)
+}

spinifex/daemon/local_state.go

@@ -0,0 +1,155 @@
+package daemon
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"log/slog"
+	"os"
+	"path/filepath"
+
+	"github.com/mulgadc/spinifex/spinifex/vm"
+)
+
+const (
+	// LocalStateSchemaVersion is the on-disk schema version for instance-state.json.
+	// Bump on any breaking change to the LocalState shape; daemon refuses to start
+	// on an unknown version rather than silently losing data.
+	LocalStateSchemaVersion = 1
+
+	// DefaultLocalStateDir is the default directory under DataDir for local state files.
+	DefaultLocalStateDir = "state"
+
+	// LocalStateFileName is the on-disk filename for per-node instance state.
+	LocalStateFileName = "instance-state.json"
+
+	// DefaultDataDir is the fallback DataDir when config.DataDir is empty.
+	DefaultDataDir = "/var/lib/spinifex"
+)
+
+// LocalState is the on-disk representation of a node's instance state.
+// SchemaVersion gates compatibility; unknown versions are rejected.
+type LocalState struct {
+	SchemaVersion int               `json:"schema_version"`
+	VMS           map[string]*vm.VM `json:"vms"`
+}
+
+// LocalStatePath returns the absolute path to the per-node instance state file
+// rooted at dataDir. Empty dataDir falls back to DefaultDataDir
+// (/var/lib/spinifex), the platform-shared root. The state directory sits at
+// the platform root so it lives next to other shared state (predastore,
+// viperblock data dirs) rather than nested under the daemon's own subdir.
+func LocalStatePath(dataDir string) string {
+	if dataDir == "" {
+		dataDir = DefaultDataDir
+	}
+	return filepath.Join(dataDir, DefaultLocalStateDir, LocalStateFileName)
+}
+
+// MarshalLocalState produces the JSON wire form of vms wrapped with the schema
+// version. Callers that already hold a stable view of vms (e.g. inside a
+// vm.Manager.View callback) can marshal under the lock and pass the bytes to
+// WriteLocalStateBytes — that avoids racing json.Marshal against concurrent
+// VM-field mutations.
+func MarshalLocalState(vms map[string]*vm.VM) ([]byte, error) {
+	state := LocalState{
+		SchemaVersion: LocalStateSchemaVersion,
+		VMS:           vms,
+	}
+	data, err := json.Marshal(state)
+	if err != nil {
+		return nil, fmt.Errorf("marshal local state: %w", err)
+	}
+	return data, nil
+}
+
+// WriteLocalState atomically writes the instance state to path. vms must be a
+// snapshot owned by the caller (e.g. from vm.Manager.SnapshotMap) — this
+// function does not lock. Convenience wrapper around MarshalLocalState +
+// WriteLocalStateBytes for callers that own an exclusive snapshot.
+//
+// Atomicity: marshal → write to <path>.tmp → fsync → rename. The rename is
+// atomic on POSIX so a concurrent reader sees either the old file or the new
+// one, never a half-written file.
+func WriteLocalState(path string, vms map[string]*vm.VM) error {
+	data, err := MarshalLocalState(vms)
+	if err != nil {
+		return err
+	}
+	return WriteLocalStateBytes(path, data)
+}
+
+// WriteLocalStateBytes atomically writes pre-marshalled state JSON to path.
+// Used by hot paths that marshal under a short-lived lock and then commit to
+// disk lock-free.
+func WriteLocalStateBytes(path string, data []byte) error {
+	dir := filepath.Dir(path)
+	if err := os.MkdirAll(dir, 0o750); err != nil {
+		return fmt.Errorf("mkdir %s: %w", dir, err)
+	}
+
+	tmp := path + ".tmp"
+	f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
+	if err != nil {
+		return fmt.Errorf("open tmp %s: %w", tmp, err)
+	}
+
+	if _, err := f.Write(data); err != nil {
+		f.Close()
+		os.Remove(tmp)
+		return fmt.Errorf("write tmp %s: %w", tmp, err)
+	}
+
+	if err := f.Sync(); err != nil {
+		f.Close()
+		os.Remove(tmp)
+		return fmt.Errorf("fsync tmp %s: %w", tmp, err)
+	}
+
+	if err := f.Close(); err != nil {
+		os.Remove(tmp)
+		return fmt.Errorf("close tmp %s: %w", tmp, err)
+	}
+
+	if err := os.Rename(tmp, path); err != nil {
+		os.Remove(tmp)
+		return fmt.Errorf("rename %s -> %s: %w", tmp, path, err)
+	}
+
+	return nil
+}
+
+// ReadLocalState reads the per-node instance state from path.
+//
+// Returns (nil, nil) if the file does not exist — fresh-install signal,
+// caller should start with an empty instance map.
+//
+// Returns an error on:
+//   - JSON parse failure (corruption — caller refuses start)
+//   - Unknown SchemaVersion (caller refuses start; safer than silent migration)
+func ReadLocalState(path string) (*LocalState, error) {
+	data, err := os.ReadFile(path)
+	if err != nil {
+		if errors.Is(err, os.ErrNotExist) {
+			slog.Debug("Local state file missing, treating as fresh install", "path", path)
+			return nil, nil
+		}
+		return nil, fmt.Errorf("read local state %s: %w", path, err)
+	}
+
+	var state LocalState
+	if err := json.Unmarshal(data, &state); err != nil {
+		return nil, fmt.Errorf("parse local state %s: %w", path, err)
+	}
+
+	if state.SchemaVersion != LocalStateSchemaVersion {
+		return nil, fmt.Errorf("local state %s: unknown schema_version %d (expected %d)",
+			path, state.SchemaVersion, LocalStateSchemaVersion)
+	}
+
+	if state.VMS == nil {
+		state.VMS = make(map[string]*vm.VM)
+	}
+
+	return &state, nil
+}

spinifex/daemon/local_state_test.go

@@ -0,0 +1,115 @@
+package daemon
+
+import (
+	"encoding/json"
+	"os"
+	"path/filepath"
+	"testing"
+
+	"github.com/mulgadc/spinifex/spinifex/vm"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestLocalStatePath_UsesDataDir(t *testing.T) {
+	got := LocalStatePath("/data")
+	assert.Equal(t, "/data/state/instance-state.json", got)
+}
+
+func TestLocalStatePath_DefaultWhenEmpty(t *testing.T) {
+	got := LocalStatePath("")
+	assert.Equal(t, "/var/lib/spinifex/state/instance-state.json", got)
+}
+
+func TestWriteLocalState_RoundTrip(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "state", "instance-state.json")
+
+	vms := map[string]*vm.VM{
+		"i-aaa": {ID: "i-aaa", InstanceType: "t3.micro"},
+		"i-bbb": {ID: "i-bbb", InstanceType: "m5.large"},
+	}
+
+	require.NoError(t, WriteLocalState(path, vms))
+
+	loaded, err := ReadLocalState(path)
+	require.NoError(t, err)
+	require.NotNil(t, loaded)
+	assert.Equal(t, LocalStateSchemaVersion, loaded.SchemaVersion)
+	assert.Len(t, loaded.VMS, 2)
+	assert.Equal(t, "t3.micro", loaded.VMS["i-aaa"].InstanceType)
+	assert.Equal(t, "m5.large", loaded.VMS["i-bbb"].InstanceType)
+}
+
+func TestWriteLocalState_MkdirParent(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "deep", "nested", "state", "instance-state.json")
+	require.NoError(t, WriteLocalState(path, map[string]*vm.VM{}))
+	_, err := os.Stat(path)
+	require.NoError(t, err)
+}
+
+func TestWriteLocalState_NoTmpLeftBehind(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "instance-state.json")
+	require.NoError(t, WriteLocalState(path, map[string]*vm.VM{}))
+
+	entries, err := os.ReadDir(dir)
+	require.NoError(t, err)
+	for _, e := range entries {
+		assert.NotContains(t, e.Name(), ".tmp", "stale tmp file left behind")
+	}
+}
+
+func TestWriteLocalState_AtomicReplace(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "instance-state.json")
+
+	require.NoError(t, WriteLocalState(path, map[string]*vm.VM{
+		"i-1": {ID: "i-1"},
+	}))
+
+	require.NoError(t, WriteLocalState(path, map[string]*vm.VM{
+		"i-2": {ID: "i-2"},
+	}))
+
+	loaded, err := ReadLocalState(path)
+	require.NoError(t, err)
+	require.NotNil(t, loaded)
+	assert.Len(t, loaded.VMS, 1)
+	_, ok := loaded.VMS["i-2"]
+	assert.True(t, ok, "second write should replace, not merge")
+}
+
+func TestReadLocalState_Missing(t *testing.T) {
+	state, err := ReadLocalState(filepath.Join(t.TempDir(), "absent.json"))
+	require.NoError(t, err)
+	assert.Nil(t, state, "missing file is the fresh-install signal")
+}
+
+func TestReadLocalState_CorruptJSON(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "instance-state.json")
+	require.NoError(t, os.WriteFile(path, []byte("{not json"), 0o600))
+
+	_, err := ReadLocalState(path)
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "parse local state")
+}
+
+func TestReadLocalState_UnknownSchema(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "instance-state.json")
+	data, _ := json.Marshal(map[string]any{"schema_version": 99, "vms": map[string]any{}})
+	require.NoError(t, os.WriteFile(path, data, 0o600))
+
+	_, err := ReadLocalState(path)
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "unknown schema_version 99")
+}
+
+func TestReadLocalState_NilVMSMapInitialized(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "instance-state.json")
+	data, _ := json.Marshal(map[string]any{"schema_version": LocalStateSchemaVersion})
+	require.NoError(t, os.WriteFile(path, data, 0o600))
+
+	state, err := ReadLocalState(path)
+	require.NoError(t, err)
+	require.NotNil(t, state)
+	assert.NotNil(t, state.VMS)
+}

spinifex/daemon/state_test.go

@@ -28,11 +28,11 @@ func createDaemonWithJetStream(t *testing.T) *Daemon {
 
 	clusterCfg := &config.ClusterConfig{
 		Node:  "node-1",
-		Nodes: map[string]config.Config{"node-1": {BaseDir: tmpDir}},
+		Nodes: map[string]config.Config{"node-1": {BaseDir: tmpDir, DataDir: tmpDir}},
 	}
 	daemon, err := NewDaemon(clusterCfg)
 	require.NoError(t, err)
-	daemon.config = &config.Config{BaseDir: tmpDir}
+	daemon.config = &config.Config{BaseDir: tmpDir, DataDir: tmpDir}
 
 	daemon.natsConn = nc
 	daemon.jsManager, err = NewJetStreamManager(nc, 1)

spinifex/gateway/auth.go

@@ -105,6 +105,19 @@ func (gw *GatewayConfig) SigV4AuthMiddleware() func(http.Handler) http.Handler {
 				return
 			}
 
+			// Fail fast when NATS is down. IAMService.LookupAccessKey runs against
+			// JetStream KV and waits 5s per attempt before context.DeadlineExceeded;
+			// the catch-all IsConnected check in gateway.Request() is unreachable
+			// because this middleware runs first. Mirror it here so authenticated
+			// paths return the cluster-unavailable 503 promised by 1c. Only fires
+			// when NATSConn is set but disconnected — production always assigns
+			// the conn in awsgw.launchService; test helpers leaving it nil get
+			// the existing pre-1c behaviour.
+			if gw.NATSConn != nil && !gw.NATSConn.IsConnected() {
+				gw.writeClusterUnavailable(w, r, service)
+				return
+			}
+
 			// Lookup access key in IAM KV store
 			if gw.IAMService == nil {
 				slog.Error("SigV4 auth: IAM service not initialized")

spinifex/gateway/auth_test.go

@@ -15,7 +15,9 @@ import (
 	"github.com/mulgadc/predastore/auth"
 	"github.com/mulgadc/spinifex/spinifex/awserrors"
 	handlers_iam "github.com/mulgadc/spinifex/spinifex/handlers/iam"
+	"github.com/mulgadc/spinifex/spinifex/testutil"
 	"github.com/mulgadc/spinifex/spinifex/utils"
+	"github.com/nats-io/nats.go"
 )
 
 const (
@@ -2028,3 +2030,56 @@ func TestSignedHeadersIncludeHostAndDate(t *testing.T) {
 		})
 	}
 }
+
+// TestSigV4Auth_NATSDisconnectedShortCircuit verifies that the SigV4 middleware
+// returns the cluster-unavailable 503 promised by 1c without ever reaching the
+// IAM lookup when the NATS connection is disconnected. With IAMService nil the
+// post-lookup path would return 500 InternalError, so a 503 here proves the
+// short-circuit fired before the lookup.
+func TestSigV4Auth_NATSDisconnectedShortCircuit(t *testing.T) {
+	ns, _ := testutil.StartTestNATS(t)
+	nc, err := nats.Connect(ns.ClientURL())
+	if err != nil {
+		t.Fatalf("nats connect: %v", err)
+	}
+	nc.Close() // disconnected — IsConnected() returns false
+
+	gw := &GatewayConfig{
+		DisableLogging: true,
+		Region:         testRegion,
+		NATSConn:       nc,
+		IAMService:     nil,
+	}
+
+	r := chi.NewRouter()
+	r.Use(gw.SigV4AuthMiddleware())
+	r.HandleFunc("/*", func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write([]byte("OK"))
+	})
+
+	authHeader, timestamp := generateTestAuthHeader(
+		"GET", "/", "", "",
+		testAccessKey, testSecretKey, testRegion, testService,
+	)
+	req := httptest.NewRequest(http.MethodGet, "/", nil)
+	req.Host = "localhost:9999"
+	req.Header.Set("Authorization", authHeader)
+	req.Header.Set("X-Amz-Date", timestamp)
+
+	resp := doRequest(r, req)
+
+	if resp.StatusCode != http.StatusServiceUnavailable {
+		t.Errorf("Expected 503, got %d", resp.StatusCode)
+	}
+	body, _ := io.ReadAll(resp.Body)
+	xmlStr := string(body)
+	if !strings.Contains(xmlStr, "<Code>ServiceUnavailable</Code>") {
+		t.Errorf("Expected ServiceUnavailable code in body, got: %s", xmlStr)
+	}
+	if !strings.Contains(xmlStr, "cluster unavailable: NATS disconnected") {
+		t.Errorf("Expected cluster-unavailable message in body, got: %s", xmlStr)
+	}
+	if !strings.Contains(xmlStr, "/local/status") {
+		t.Errorf("Expected /local/status hint in body, got: %s", xmlStr)
+	}
+}

spinifex/gateway/gateway.go

@@ -149,6 +149,49 @@ func (gw *GatewayConfig) throttleKeyFuncs() []ratelimit.KeyFunc {
 	}
 }
 
+// clusterUnavailableMsg is the body returned when the gateway short-circuits
+// because the daemon's NATS connection is down. Points operators at the
+// daemon's /local/status (1b) for triage instead of letting them watch the
+// AWS CLI hang on per-call timeouts.
+const clusterUnavailableMsg = "cluster unavailable: NATS disconnected — check daemon /local/status"
+
+// writeClusterUnavailable returns a 503 ServiceUnavailable for the given
+// service-flavoured XML format. The body carries the literal
+// clusterUnavailableMsg in <Message> — the generic GenerateEC2ErrorResponse
+// path drops the message string (see ErrorHandler), so we emit XML directly
+// here to make sure the /local/status hint actually reaches the operator.
+func (gw *GatewayConfig) writeClusterUnavailable(w http.ResponseWriter, _ *http.Request, svc string) {
+	requestID := uuid.NewString()
+	var xmlBody string
+	if svc == "iam" {
+		iam := IAMErrorResponse{
+			Error: IAMErrorDetail{
+				Type:    "Sender",
+				Code:    awserrors.ErrorServiceUnavailable,
+				Message: clusterUnavailableMsg,
+			},
+			RequestID: requestID,
+		}
+		out, err := xml.MarshalIndent(iam, "", "  ")
+		if err != nil {
+			slog.Error("Failed to marshal IAM cluster-unavailable XML", "err", err)
+			out = []byte(`<ErrorResponse><Error><Type>Sender</Type><Code>ServiceUnavailable</Code><Message>` + clusterUnavailableMsg + `</Message></Error><RequestId>` + requestID + `</RequestId></ErrorResponse>`)
+		}
+		xmlBody = xml.Header + string(out)
+	} else {
+		// ec2, elasticloadbalancing, account, spinifex — all share the EC2 envelope.
+		xmlBody = xml.Header + `<Response><Errors><Error><Code>` + awserrors.ErrorServiceUnavailable +
+			`</Code><Message>` + clusterUnavailableMsg + `</Message></Error></Errors><RequestID>` +
+			requestID + `</RequestID></Response>`
+	}
+
+	w.Header().Set("Content-Type", "application/xml")
+	w.WriteHeader(http.StatusServiceUnavailable)
+	if _, err := w.Write([]byte(xmlBody)); err != nil {
+		slog.Error("Failed to write cluster-unavailable response", "err", err)
+	}
+}
+
 // writeThrottleError writes the service-appropriate throttle rejection response.
 func (gw *GatewayConfig) writeThrottleError(w http.ResponseWriter, r *http.Request) {
 	requestID := uuid.NewString()
@@ -190,6 +233,15 @@ func (gw *GatewayConfig) Request(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 
+	// Fail fast when NATS is down — every NATS-bound per-service handler would
+	// otherwise hang until per-call timeout. The body points operators at
+	// /local/status (1b) for diagnosis. Account is a no-op stub that never
+	// reaches NATS, so it is exempt.
+	if svc != "account" && (gw.NATSConn == nil || !gw.NATSConn.IsConnected()) {
+		gw.writeClusterUnavailable(w, r, svc)
+		return
+	}
+
 	switch svc {
 	case "ec2":
 		err = gw.EC2_Request(w, r)

spinifex/gateway/gateway_test.go

@@ -553,7 +553,9 @@ func TestRequest_MalformedQueryString_EndToEnd(t *testing.T) {
 	}
 	for _, tc := range tests {
 		t.Run(tc.name, func(t *testing.T) {
-			gw := &GatewayConfig{DisableLogging: true}
+			// Connected NATS satisfies the 1c cluster-available gate so the
+			// request reaches the per-service malformed-query parser.
+			gw := &GatewayConfig{DisableLogging: true, NATSConn: connectedNATS(t)}
 			req := httptest.NewRequest(http.MethodPost, "/", strings.NewReader(tc.body))
 			ctx := context.WithValue(req.Context(), ctxService, tc.service)
 			ctx = context.WithValue(ctx, ctxAccountID, "123456789012")
@@ -571,7 +573,7 @@ func TestRequest_MalformedQueryString_EndToEnd(t *testing.T) {
 }
 
 func TestRequest_EC2MissingAction(t *testing.T) {
-	gw := &GatewayConfig{DisableLogging: true}
+	gw := &GatewayConfig{DisableLogging: true, NATSConn: connectedNATS(t)}
 	req := httptest.NewRequest(http.MethodPost, "/", strings.NewReader(""))
 	ctx := context.WithValue(req.Context(), ctxService, "ec2")
 	ctx = context.WithValue(ctx, ctxAccountID, "123456789012")
@@ -587,7 +589,7 @@ func TestRequest_EC2MissingAction(t *testing.T) {
 }
 
 func TestRequest_IAMNilService(t *testing.T) {
-	gw := &GatewayConfig{DisableLogging: true, IAMService: nil}
+	gw := &GatewayConfig{DisableLogging: true, IAMService: nil, NATSConn: connectedNATS(t)}
 	req := httptest.NewRequest(http.MethodPost, "/", strings.NewReader("Action=CreateUser&UserName=test"))
 	ctx := context.WithValue(req.Context(), ctxService, "iam")
 	ctx = context.WithValue(ctx, ctxAccountID, "123456789012")
@@ -602,6 +604,17 @@ func TestRequest_IAMNilService(t *testing.T) {
 	assert.Contains(t, string(body), "InternalError")
 }
 
+// connectedNATS returns a live test NATS connection for short-circuit-bypass
+// tests that exercise per-service handlers without actually publishing.
+func connectedNATS(t *testing.T) *nats.Conn {
+	t.Helper()
+	ns, _ := testutil.StartTestNATS(t)
+	nc, err := nats.Connect(ns.ClientURL())
+	require.NoError(t, err)
+	t.Cleanup(nc.Close)
+	return nc
+}
+
 func TestRequest_AccountReturns200(t *testing.T) {
 	gw := &GatewayConfig{DisableLogging: true}
 	req := httptest.NewRequest(http.MethodPost, "/", nil)
@@ -1268,3 +1281,60 @@ func TestThrottleMiddleware_PerActionIsolation(t *testing.T) {
 	resp = makeReq("RunInstances")
 	assert.Equal(t, 200, resp.StatusCode)
 }
+
+func TestRequest_ClusterUnavailableNilConn_EC2(t *testing.T) {
+	gw := &GatewayConfig{DisableLogging: true}
+
+	req := httptest.NewRequest(http.MethodPost, "/", nil)
+	ctx := context.WithValue(req.Context(), ctxService, "ec2")
+	req = req.WithContext(ctx)
+
+	w := httptest.NewRecorder()
+	gw.Request(w, req)
+	resp := w.Result()
+
+	assert.Equal(t, http.StatusServiceUnavailable, resp.StatusCode)
+	body, _ := io.ReadAll(resp.Body)
+	xmlStr := string(body)
+	assert.Contains(t, xmlStr, "<Code>ServiceUnavailable</Code>")
+	assert.Contains(t, xmlStr, "cluster unavailable: NATS disconnected")
+	assert.Contains(t, xmlStr, "/local/status")
+	assert.Contains(t, xmlStr, "<Response>") // EC2 XML envelope
+}
+
+func TestRequest_ClusterUnavailableNilConn_IAM(t *testing.T) {
+	gw := &GatewayConfig{DisableLogging: true}
+
+	req := httptest.NewRequest(http.MethodPost, "/", nil)
+	ctx := context.WithValue(req.Context(), ctxService, "iam")
+	req = req.WithContext(ctx)
+
+	w := httptest.NewRecorder()
+	gw.Request(w, req)
+	resp := w.Result()
+
+	assert.Equal(t, http.StatusServiceUnavailable, resp.StatusCode)
+	body, _ := io.ReadAll(resp.Body)
+	xmlStr := string(body)
+	assert.Contains(t, xmlStr, "<Code>ServiceUnavailable</Code>")
+	assert.Contains(t, xmlStr, "<ErrorResponse>") // IAM XML envelope
+}
+
+func TestRequest_ClusterUnavailableClosedConn(t *testing.T) {
+	ns, _ := testutil.StartTestNATS(t)
+	nc, err := nats.Connect(ns.ClientURL())
+	require.NoError(t, err)
+	nc.Close()
+
+	gw := &GatewayConfig{DisableLogging: true, NATSConn: nc}
+
+	req := httptest.NewRequest(http.MethodPost, "/", nil)
+	ctx := context.WithValue(req.Context(), ctxService, "ec2")
+	req = req.WithContext(ctx)
+
+	w := httptest.NewRecorder()
+	gw.Request(w, req)
+	resp := w.Result()
+
+	assert.Equal(t, http.StatusServiceUnavailable, resp.StatusCode)
+}

spinifex/gateway/ratelimit_test.go

@@ -12,6 +12,9 @@ import (
 	"github.com/go-chi/chi/v5"
 	"github.com/mulgadc/spinifex/spinifex/awserrors"
 	handlers_iam "github.com/mulgadc/spinifex/spinifex/handlers/iam"
+	"github.com/mulgadc/spinifex/spinifex/testutil"
+	"github.com/nats-io/nats.go"
+	"github.com/stretchr/testify/require"
 )
 
 func TestCheckIP_AllowsUnknownIP(t *testing.T) {
@@ -332,8 +335,12 @@ func TestExtractClientIP(t *testing.T) {
 }
 
 // setupTestAppWithRateLimiter creates a test HTTP handler with SigV4 auth and
-// the given rate limiter attached.
-func setupTestAppWithRateLimiter(accessKey, secretKey string, rl *AuthRateLimiter) http.Handler {
+// the given rate limiter attached. A real NATS test connection is attached so
+// the SigV4 middleware's cluster-unavailable short-circuit (mulga-siv-23) does
+// not fire and mask rate-limit behaviour.
+func setupTestAppWithRateLimiter(t *testing.T, accessKey, secretKey string, rl *AuthRateLimiter) http.Handler {
+	t.Helper()
+
 	encryptedSecret, err := handlers_iam.EncryptSecret(secretKey, testMasterKey)
 	if err != nil {
 		panic("failed to encrypt test secret: " + err.Error())
@@ -351,11 +358,17 @@ func setupTestAppWithRateLimiter(accessKey, secretKey string, rl *AuthRateLimite
 		},
 	}
 
+	ns, _ := testutil.StartTestNATS(t)
+	nc, err := nats.Connect(ns.ClientURL())
+	require.NoError(t, err)
+	t.Cleanup(nc.Close)
+
 	gw := &GatewayConfig{
 		DisableLogging: true,
 		Region:         testRegion,
 		IAMService:     mockSvc,
 		RateLimiter:    rl,
+		NATSConn:       nc,
 	}
 
 	r := chi.NewRouter()
@@ -371,7 +384,7 @@ func TestRateLimitIntegration_LockedIPGets503(t *testing.T) {
 	rl := NewAuthRateLimiter()
 	defer rl.Stop()
 
-	handler := setupTestAppWithRateLimiter(testAccessKey, testSecretKey, rl)
+	handler := setupTestAppWithRateLimiter(t, testAccessKey, testSecretKey, rl)
 
 	// Send maxFailures requests with invalid signatures to trigger lockout.
 	for range maxFailures {
@@ -405,7 +418,7 @@ func TestRateLimitIntegration_SuccessResetsLockout(t *testing.T) {
 	rl := NewAuthRateLimiter()
 	defer rl.Stop()
 
-	handler := setupTestAppWithRateLimiter(testAccessKey, testSecretKey, rl)
+	handler := setupTestAppWithRateLimiter(t, testAccessKey, testSecretKey, rl)
 
 	// Accumulate failures below threshold.
 	for range maxFailures - 1 {

spinifex/handlers/ec2/instance/instance_filter_test.go

@@ -0,0 +1,215 @@
+package handlers_ec2_instance
+
+import (
+	"testing"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/service/ec2"
+	"github.com/mulgadc/spinifex/spinifex/filterutil"
+	"github.com/mulgadc/spinifex/spinifex/vm"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestInstanceMatchesFilters_NoFilters(t *testing.T) {
+	inst := &vm.VM{ID: "i-123", InstanceType: "t3.micro"}
+	ic := &ec2.Instance{
+		InstanceId:   aws.String("i-123"),
+		InstanceType: aws.String("t3.micro"),
+		State:        &ec2.InstanceState{Name: aws.String("running")},
+	}
+	assert.True(t, instanceMatchesFilters(inst, ic, nil))
+}
+
+func TestInstanceMatchesFilters_SingleFilter_Match(t *testing.T) {
+	inst := &vm.VM{ID: "i-abc", InstanceType: "t3.micro"}
+	ic := &ec2.Instance{
+		InstanceId:   aws.String("i-abc"),
+		InstanceType: aws.String("t3.micro"),
+		State:        &ec2.InstanceState{Name: aws.String("running")},
+	}
+	filters := map[string][]string{
+		"instance-state-name": {"running"},
+	}
+	assert.True(t, instanceMatchesFilters(inst, ic, filters))
+}
+
+func TestInstanceMatchesFilters_SingleFilter_NoMatch(t *testing.T) {
+	inst := &vm.VM{ID: "i-abc", InstanceType: "t3.micro"}
+	ic := &ec2.Instance{
+		InstanceId:   aws.String("i-abc"),
+		InstanceType: aws.String("t3.micro"),
+		State:        &ec2.InstanceState{Name: aws.String("running")},
+	}
+	filters := map[string][]string{
+		"instance-state-name": {"stopped"},
+	}
+	assert.False(t, instanceMatchesFilters(inst, ic, filters))
+}
+
+func TestInstanceMatchesFilters_MultipleValues_ORLogic(t *testing.T) {
+	inst := &vm.VM{ID: "i-abc", InstanceType: "t3.micro"}
+	ic := &ec2.Instance{
+		InstanceId:   aws.String("i-abc"),
+		InstanceType: aws.String("t3.micro"),
+		State:        &ec2.InstanceState{Name: aws.String("running")},
+	}
+	filters := map[string][]string{
+		"instance-state-name": {"running", "stopped"},
+	}
+	assert.True(t, instanceMatchesFilters(inst, ic, filters))
+}
+
+func TestInstanceMatchesFilters_MultipleFilters_ANDLogic(t *testing.T) {
+	inst := &vm.VM{ID: "i-abc", InstanceType: "t3.micro"}
+	ic := &ec2.Instance{
+		InstanceId:   aws.String("i-abc"),
+		InstanceType: aws.String("t3.micro"),
+		VpcId:        aws.String("vpc-111"),
+		State:        &ec2.InstanceState{Name: aws.String("running")},
+	}
+	filters := map[string][]string{
+		"instance-state-name": {"running"},
+		"vpc-id":              {"vpc-111"},
+	}
+	assert.True(t, instanceMatchesFilters(inst, ic, filters))
+
+	filters["vpc-id"] = []string{"vpc-999"}
+	assert.False(t, instanceMatchesFilters(inst, ic, filters))
+}
+
+func TestInstanceMatchesFilters_InstanceId(t *testing.T) {
+	inst := &vm.VM{ID: "i-abc"}
+	ic := &ec2.Instance{
+		InstanceId: aws.String("i-abc"),
+		State:      &ec2.InstanceState{Name: aws.String("running")},
+	}
+	filters := map[string][]string{"instance-id": {"i-abc"}}
+	assert.True(t, instanceMatchesFilters(inst, ic, filters))
+
+	filters["instance-id"] = []string{"i-other"}
+	assert.False(t, instanceMatchesFilters(inst, ic, filters))
+}
+
+func TestInstanceMatchesFilters_InstanceType(t *testing.T) {
+	inst := &vm.VM{ID: "i-abc", InstanceType: "m5.large"}
+	ic := &ec2.Instance{
+		InstanceId:   aws.String("i-abc"),
+		InstanceType: aws.String("m5.large"),
+		State:        &ec2.InstanceState{Name: aws.String("running")},
+	}
+	filters := map[string][]string{"instance-type": {"m5.large"}}
+	assert.True(t, instanceMatchesFilters(inst, ic, filters))
+}
+
+func TestInstanceMatchesFilters_SubnetId(t *testing.T) {
+	inst := &vm.VM{ID: "i-abc"}
+	ic := &ec2.Instance{
+		InstanceId: aws.String("i-abc"),
+		SubnetId:   aws.String("subnet-123"),
+		State:      &ec2.InstanceState{Name: aws.String("running")},
+	}
+	filters := map[string][]string{"subnet-id": {"subnet-123"}}
+	assert.True(t, instanceMatchesFilters(inst, ic, filters))
+
+	filters["subnet-id"] = []string{"subnet-999"}
+	assert.False(t, instanceMatchesFilters(inst, ic, filters))
+}
+
+func TestInstanceMatchesFilters_Wildcard(t *testing.T) {
+	inst := &vm.VM{ID: "i-abc", InstanceType: "t3.micro"}
+	ic := &ec2.Instance{
+		InstanceId:   aws.String("i-abc"),
+		InstanceType: aws.String("t3.micro"),
+		State:        &ec2.InstanceState{Name: aws.String("running")},
+	}
+	filters := map[string][]string{"instance-type": {"t3.*"}}
+	assert.True(t, instanceMatchesFilters(inst, ic, filters))
+
+	filters["instance-type"] = []string{"m5.*"}
+	assert.False(t, instanceMatchesFilters(inst, ic, filters))
+}
+
+func TestInstanceMatchesFilters_TagFilter(t *testing.T) {
+	inst := &vm.VM{ID: "i-abc"}
+	ic := &ec2.Instance{
+		InstanceId: aws.String("i-abc"),
+		State:      &ec2.InstanceState{Name: aws.String("running")},
+		Tags: []*ec2.Tag{
+			{Key: aws.String("Environment"), Value: aws.String("prod")},
+			{Key: aws.String("Team"), Value: aws.String("backend")},
+		},
+	}
+
+	filters := map[string][]string{"tag:Environment": {"prod"}}
+	assert.True(t, instanceMatchesFilters(inst, ic, filters))
+
+	filters["tag:Environment"] = []string{"staging"}
+	assert.False(t, instanceMatchesFilters(inst, ic, filters))
+}
+
+func TestInstanceMatchesFilters_TagKey(t *testing.T) {
+	inst := &vm.VM{ID: "i-abc"}
+	ic := &ec2.Instance{
+		InstanceId: aws.String("i-abc"),
+		State:      &ec2.InstanceState{Name: aws.String("running")},
+		Tags: []*ec2.Tag{
+			{Key: aws.String("Environment"), Value: aws.String("prod")},
+		},
+	}
+	filters := map[string][]string{"tag-key": {"Environment"}}
+	assert.True(t, instanceMatchesFilters(inst, ic, filters))
+
+	filters["tag-key"] = []string{"MissingKey"}
+	assert.False(t, instanceMatchesFilters(inst, ic, filters))
+}
+
+func TestInstanceMatchesFilters_TagValue(t *testing.T) {
+	inst := &vm.VM{ID: "i-abc"}
+	ic := &ec2.Instance{
+		InstanceId: aws.String("i-abc"),
+		State:      &ec2.InstanceState{Name: aws.String("running")},
+		Tags: []*ec2.Tag{
+			{Key: aws.String("Environment"), Value: aws.String("prod")},
+		},
+	}
+	filters := map[string][]string{"tag-value": {"prod"}}
+	assert.True(t, instanceMatchesFilters(inst, ic, filters))
+
+	filters["tag-value"] = []string{"staging"}
+	assert.False(t, instanceMatchesFilters(inst, ic, filters))
+}
+
+func TestInstanceMatchesFilters_EmptyResults(t *testing.T) {
+	inst := &vm.VM{ID: "i-abc", InstanceType: "t3.micro"}
+	ic := &ec2.Instance{
+		InstanceId:   aws.String("i-abc"),
+		InstanceType: aws.String("t3.micro"),
+		State:        &ec2.InstanceState{Name: aws.String("running")},
+	}
+	filters := map[string][]string{"instance-type": {"c5.4xlarge"}}
+	assert.False(t, instanceMatchesFilters(inst, ic, filters))
+}
+
+func TestParseFilters_UnknownFilterName(t *testing.T) {
+	filters := []*ec2.Filter{
+		{Name: aws.String("bogus-filter"), Values: []*string{aws.String("val")}},
+	}
+	_, err := filterutil.ParseFilters(filters, DescribeInstancesValidFilters)
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "InvalidParameterValue")
+}
+
+func TestEC2TagsToMap(t *testing.T) {
+	tags := []*ec2.Tag{
+		{Key: aws.String("a"), Value: aws.String("1")},
+		{Key: aws.String("b"), Value: aws.String("2")},
+	}
+	m := filterutil.EC2TagsToMap(tags)
+	assert.Equal(t, "1", m["a"])
+	assert.Equal(t, "2", m["b"])
+}
+
+func TestEC2TagsToMap_Nil(t *testing.T) {
+	assert.Nil(t, filterutil.EC2TagsToMap(nil))
+}

spinifex/handlers/ec2/instance/service.go

@@ -1,8 +1,30 @@
 package handlers_ec2_instance
 
-import "github.com/aws/aws-sdk-go/service/ec2"
+import (
+	"github.com/aws/aws-sdk-go/service/ec2"
+	"github.com/mulgadc/spinifex/spinifex/vm"
+)
 
 // InstanceService defines the interface for EC2 instance operations business logic
 type InstanceService interface {
 	RunInstances(input *ec2.RunInstancesInput, accountID string) (*ec2.Reservation, error)
+	DescribeInstances(input *ec2.DescribeInstancesInput, accountID string) (*ec2.DescribeInstancesOutput, error)
+	DescribeInstanceTypes(input *ec2.DescribeInstanceTypesInput, accountID string) (*ec2.DescribeInstanceTypesOutput, error)
+	DescribeInstanceAttribute(input *ec2.DescribeInstanceAttributeInput, accountID string) (*ec2.DescribeInstanceAttributeOutput, error)
+	DescribeStoppedInstances(input *ec2.DescribeInstancesInput, accountID string) (*ec2.DescribeInstancesOutput, error)
+	DescribeTerminatedInstances(input *ec2.DescribeInstancesInput, accountID string) (*ec2.DescribeInstancesOutput, error)
+}
+
+// ResourceCapacityProvider exposes per-node instance-type availability used by
+// DescribeInstanceTypes. Implemented by daemon.ResourceManager.
+type ResourceCapacityProvider interface {
+	GetAvailableInstanceTypeInfos(showCapacity bool) []*ec2.InstanceTypeInfo
+}
+
+// StoppedInstanceStore provides KV-backed read access to stopped/terminated
+// instances. Implemented by daemon.JetStreamManager.
+type StoppedInstanceStore interface {
+	LoadStoppedInstance(instanceID string) (*vm.VM, error)
+	ListStoppedInstances() ([]*vm.VM, error)
+	ListTerminatedInstances() ([]*vm.VM, error)
 }

spinifex/handlers/ec2/instance/service_impl.go

@@ -19,6 +19,7 @@ import (
 	"github.com/kdomanski/iso9660"
 	"github.com/mulgadc/spinifex/spinifex/awserrors"
 	"github.com/mulgadc/spinifex/spinifex/config"
+	"github.com/mulgadc/spinifex/spinifex/filterutil"
 	"github.com/mulgadc/spinifex/spinifex/objectstore"
 	spxtypes "github.com/mulgadc/spinifex/spinifex/types"
 	"github.com/mulgadc/spinifex/spinifex/utils"
@@ -232,15 +233,29 @@ type InstanceServiceImpl struct {
 	instanceTypes map[string]*ec2.InstanceTypeInfo
 	natsConn      *nats.Conn
 	objectStore   objectstore.ObjectStore
+	vmMgr         *vm.Manager
+	resourceMgr   ResourceCapacityProvider
+	stoppedStore  StoppedInstanceStore
 }
 
 // NewInstanceServiceImpl creates a new instance service implementation for daemon use
-func NewInstanceServiceImpl(cfg *config.Config, instanceTypes map[string]*ec2.InstanceTypeInfo, nc *nats.Conn, store objectstore.ObjectStore) *InstanceServiceImpl {
+func NewInstanceServiceImpl(
+	cfg *config.Config,
+	instanceTypes map[string]*ec2.InstanceTypeInfo,
+	nc *nats.Conn,
+	store objectstore.ObjectStore,
+	vmMgr *vm.Manager,
+	resourceMgr ResourceCapacityProvider,
+	stoppedStore StoppedInstanceStore,
+) *InstanceServiceImpl {
 	return &InstanceServiceImpl{
 		config:        cfg,
 		instanceTypes: instanceTypes,
 		natsConn:      nc,
 		objectStore:   store,
+		vmMgr:         vmMgr,
+		resourceMgr:   resourceMgr,
+		stoppedStore:  stoppedStore,
 	}
 }
 
@@ -842,3 +857,401 @@ func generateHostname(instanceID string) string {
 	}
 	return "spinifex-vm-unknown"
 }
+
+// DescribeInstancesValidFilters lists the filter names accepted by DescribeInstances
+// (and the stopped/terminated variants, which share the same filter shape).
+var DescribeInstancesValidFilters = map[string]bool{
+	"instance-state-name": true,
+	"instance-id":         true,
+	"instance-type":       true,
+	"vpc-id":              true,
+	"subnet-id":           true,
+	"tag-key":             true,
+	"tag-value":           true,
+}
+
+// IsInstanceVisible reports whether the caller can see this instance.
+// Pre-Phase4 instances (empty AccountID) are only visible to root (GlobalAccountID).
+func IsInstanceVisible(callerAccountID, ownerAccountID string) bool {
+	if ownerAccountID == "" {
+		return callerAccountID == utils.GlobalAccountID
+	}
+	return callerAccountID == ownerAccountID
+}
+
+// instanceMatchesFilters checks whether a VM + its built ec2.Instance copy satisfy all parsed filters.
+func instanceMatchesFilters(inst *vm.VM, ic *ec2.Instance, filters map[string][]string) bool {
+	for name, values := range filters {
+		if strings.HasPrefix(name, "tag:") {
+			continue
+		}
+
+		var field string
+		switch name {
+		case "instance-state-name":
+			if ic.State != nil && ic.State.Name != nil {
+				field = *ic.State.Name
+			}
+		case "instance-id":
+			field = inst.ID
+		case "instance-type":
+			field = inst.InstanceType
+		case "vpc-id":
+			if ic.VpcId != nil {
+				field = *ic.VpcId
+			}
+		case "subnet-id":
+			if ic.SubnetId != nil {
+				field = *ic.SubnetId
+			}
+		case "tag-key":
+			if !matchTagKey(ic.Tags, values) {
+				return false
+			}
+			continue
+		case "tag-value":
+			if !matchTagValue(ic.Tags, values) {
+				return false
+			}
+			continue
+		default:
+			return false
+		}
+
+		if !filterutil.MatchesAny(values, field) {
+			return false
+		}
+	}
+
+	tags := filterutil.EC2TagsToMap(ic.Tags)
+	return filterutil.MatchesTags(filters, tags)
+}
+
+func matchTagKey(tags []*ec2.Tag, values []string) bool {
+	for _, t := range tags {
+		if t.Key != nil && filterutil.MatchesAny(values, *t.Key) {
+			return true
+		}
+	}
+	return false
+}
+
+func matchTagValue(tags []*ec2.Tag, values []string) bool {
+	for _, t := range tags {
+		if t.Value != nil && filterutil.MatchesAny(values, *t.Value) {
+			return true
+		}
+	}
+	return false
+}
+
+// DescribeInstances returns instances on this node visible to the caller's account.
+func (s *InstanceServiceImpl) DescribeInstances(input *ec2.DescribeInstancesInput, accountID string) (*ec2.DescribeInstancesOutput, error) {
+	slog.Info("Processing DescribeInstances request from this node", "accountID", accountID)
+
+	instanceIDFilter := make(map[string]bool)
+	for _, id := range input.InstanceIds {
+		if id != nil && *id != "" {
+			if !strings.HasPrefix(*id, "i-") {
+				return nil, errors.New(awserrors.ErrorInvalidInstanceIDMalformed)
+			}
+			instanceIDFilter[*id] = true
+		}
+	}
+
+	parsedFilters, err := filterutil.ParseFilters(input.Filters, DescribeInstancesValidFilters)
+	if err != nil {
+		slog.Warn("DescribeInstances: invalid filter", "err", err)
+		return nil, errors.New(awserrors.ErrorInvalidParameterValue)
+	}
+
+	reservationMap := make(map[string]*ec2.Reservation)
+
+	// Iterate under the manager lock — VM fields (Status, Instance, Reservation,
+	// PublicIP, PlacementGroupName) are mutated through manager-locked
+	// Inspect/UpdateState elsewhere, so reading them lock-free would race.
+	s.vmMgr.View(func(vms map[string]*vm.VM) {
+		for _, instance := range vms {
+			if !IsInstanceVisible(accountID, instance.AccountID) {
+				continue
+			}
+			if len(instanceIDFilter) > 0 && !instanceIDFilter[instance.ID] {
+				continue
+			}
+			if instance.Reservation == nil || instance.Instance == nil {
+				continue
+			}
+
+			resID := ""
+			if instance.Reservation.ReservationId != nil {
+				resID = *instance.Reservation.ReservationId
+			}
+
+			if _, exists := reservationMap[resID]; !exists {
+				reservation := &ec2.Reservation{}
+				reservation.SetReservationId(resID)
+				if instance.Reservation.OwnerId != nil {
+					reservation.SetOwnerId(*instance.Reservation.OwnerId)
+				}
+				reservation.Instances = []*ec2.Instance{}
+				reservationMap[resID] = reservation
+			}
+
+			instanceCopy := *instance.Instance
+			instanceCopy.State = &ec2.InstanceState{}
+
+			if instance.PublicIP != "" && instanceCopy.PublicIpAddress == nil {
+				instanceCopy.PublicIpAddress = aws.String(instance.PublicIP)
+			}
+
+			if info, ok := vm.EC2StateCodes[instance.Status]; ok {
+				instanceCopy.State.SetCode(info.Code)
+				instanceCopy.State.SetName(info.Name)
+			} else {
+				slog.Warn("Instance has unmapped status, reporting as pending",
+					"instanceId", instance.ID, "status", string(instance.Status))
+				instanceCopy.State.SetCode(0)
+				instanceCopy.State.SetName("pending")
+			}
+
+			if instance.PlacementGroupName != "" {
+				instanceCopy.Placement = &ec2.Placement{
+					GroupName:        aws.String(instance.PlacementGroupName),
+					AvailabilityZone: aws.String(s.config.AZ),
+				}
+			}
+
+			if len(parsedFilters) > 0 && !instanceMatchesFilters(instance, &instanceCopy, parsedFilters) {
+				continue
+			}
+
+			reservationMap[resID].Instances = append(reservationMap[resID].Instances, &instanceCopy)
+		}
+	})
+
+	reservations := make([]*ec2.Reservation, 0, len(reservationMap))
+	for _, reservation := range reservationMap {
+		reservations = append(reservations, reservation)
+	}
+
+	slog.Info("DescribeInstances completed", "count", len(reservations))
+	return &ec2.DescribeInstancesOutput{Reservations: reservations}, nil
+}
+
+// DescribeInstanceTypes returns instance types provisionable on this node.
+// The "capacity" filter (when "true") expands each type to one entry per
+// available slot so callers can report cluster-wide capacity by aggregating.
+func (s *InstanceServiceImpl) DescribeInstanceTypes(input *ec2.DescribeInstanceTypesInput, _ string) (*ec2.DescribeInstanceTypesOutput, error) {
+	slog.Info("Processing DescribeInstanceTypes request from this node")
+
+	if s.resourceMgr == nil {
+		return nil, errors.New(awserrors.ErrorServerInternal)
+	}
+
+	showCapacity := false
+	for _, f := range input.Filters {
+		if f.Name != nil && *f.Name == "capacity" {
+			for _, v := range f.Values {
+				if v != nil && *v == "true" {
+					showCapacity = true
+					break
+				}
+			}
+		}
+	}
+
+	filteredTypes := s.resourceMgr.GetAvailableInstanceTypeInfos(showCapacity)
+	slog.Info("DescribeInstanceTypes completed", "count", len(filteredTypes))
+	return &ec2.DescribeInstanceTypesOutput{InstanceTypes: filteredTypes}, nil
+}
+
+// DescribeStoppedInstances returns stopped instances from shared KV.
+func (s *InstanceServiceImpl) DescribeStoppedInstances(input *ec2.DescribeInstancesInput, accountID string) (*ec2.DescribeInstancesOutput, error) {
+	if s.stoppedStore == nil {
+		return nil, errors.New(awserrors.ErrorServerInternal)
+	}
+	return s.describeInstancesFromKV(input, accountID, s.stoppedStore.ListStoppedInstances, 80, "stopped", "DescribeStoppedInstances")
+}
+
+// DescribeTerminatedInstances returns terminated instances from the terminated KV bucket.
+func (s *InstanceServiceImpl) DescribeTerminatedInstances(input *ec2.DescribeInstancesInput, accountID string) (*ec2.DescribeInstancesOutput, error) {
+	if s.stoppedStore == nil {
+		return nil, errors.New(awserrors.ErrorServerInternal)
+	}
+	return s.describeInstancesFromKV(input, accountID, s.stoppedStore.ListTerminatedInstances, 48, "terminated", "DescribeTerminatedInstances")
+}
+
+func (s *InstanceServiceImpl) describeInstancesFromKV(input *ec2.DescribeInstancesInput, accountID string, listFn func() ([]*vm.VM, error), fallbackCode int64, fallbackName, opName string) (*ec2.DescribeInstancesOutput, error) {
+	instanceIDFilter := make(map[string]bool)
+	for _, id := range input.InstanceIds {
+		if id != nil {
+			instanceIDFilter[*id] = true
+		}
+	}
+
+	parsedFilters, filterErr := filterutil.ParseFilters(input.Filters, DescribeInstancesValidFilters)
+	if filterErr != nil {
+		slog.Warn(opName+": invalid filter", "err", filterErr)
+		return nil, errors.New(awserrors.ErrorInvalidParameterValue)
+	}
+
+	instances, err := listFn()
+	if err != nil {
+		slog.Error(opName+": failed to list instances", "err", err)
+		return nil, errors.New(awserrors.ErrorServerInternal)
+	}
+
+	reservationMap := make(map[string]*ec2.Reservation)
+
+	for _, instance := range instances {
+		if !IsInstanceVisible(accountID, instance.AccountID) {
+			continue
+		}
+		if len(instanceIDFilter) > 0 && !instanceIDFilter[instance.ID] {
+			continue
+		}
+		if instance.Reservation == nil || instance.Instance == nil {
+			slog.Warn(opName+": skipping instance with nil Reservation/Instance (data integrity issue)",
+				"instanceId", instance.ID)
+			continue
+		}
+
+		resID := ""
+		if instance.Reservation.ReservationId != nil {
+			resID = *instance.Reservation.ReservationId
+		}
+
+		if _, exists := reservationMap[resID]; !exists {
+			reservation := &ec2.Reservation{}
+			reservation.SetReservationId(resID)
+			if instance.Reservation.OwnerId != nil {
+				reservation.SetOwnerId(*instance.Reservation.OwnerId)
+			}
+			reservation.Instances = []*ec2.Instance{}
+			reservationMap[resID] = reservation
+		}
+
+		instanceCopy := *instance.Instance
+		instanceCopy.State = &ec2.InstanceState{}
+		if info, ok := vm.EC2StateCodes[instance.Status]; ok {
+			instanceCopy.State.SetCode(info.Code)
+			instanceCopy.State.SetName(info.Name)
+		} else {
+			instanceCopy.State.SetCode(fallbackCode)
+			instanceCopy.State.SetName(fallbackName)
+		}
+
+		if len(parsedFilters) > 0 && !instanceMatchesFilters(instance, &instanceCopy, parsedFilters) {
+			continue
+		}
+
+		reservationMap[resID].Instances = append(reservationMap[resID].Instances, &instanceCopy)
+	}
+
+	reservations := make([]*ec2.Reservation, 0, len(reservationMap))
+	for _, reservation := range reservationMap {
+		reservations = append(reservations, reservation)
+	}
+
+	slog.Info(opName+" completed", "count", len(reservations))
+	return &ec2.DescribeInstancesOutput{Reservations: reservations}, nil
+}
+
+// DescribeInstanceAttribute returns a single requested attribute for an instance.
+// Checks running instances first (in-memory), then falls back to stopped instances in KV.
+func (s *InstanceServiceImpl) DescribeInstanceAttribute(input *ec2.DescribeInstanceAttributeInput, accountID string) (*ec2.DescribeInstanceAttributeOutput, error) {
+	if input.InstanceId == nil || *input.InstanceId == "" {
+		return nil, errors.New(awserrors.ErrorMissingParameter)
+	}
+	if input.Attribute == nil || *input.Attribute == "" {
+		return nil, errors.New(awserrors.ErrorMissingParameter)
+	}
+
+	instanceID := *input.InstanceId
+	attribute := *input.Attribute
+
+	var instance *vm.VM
+	if running, ok := s.vmMgr.Get(instanceID); ok {
+		instance = running
+	}
+
+	if instance == nil {
+		if s.stoppedStore == nil {
+			slog.Error("DescribeInstanceAttribute: stopped store not available")
+			return nil, errors.New(awserrors.ErrorServerInternal)
+		}
+		stopped, err := s.stoppedStore.LoadStoppedInstance(instanceID)
+		if err != nil {
+			slog.Error("DescribeInstanceAttribute: failed to load stopped instance",
+				"instanceId", instanceID, "err", err)
+			return nil, errors.New(awserrors.ErrorServerInternal)
+		}
+		instance = stopped
+	}
+
+	if instance == nil {
+		slog.Warn("DescribeInstanceAttribute: instance not found",
+			"instanceId", instanceID)
+		return nil, errors.New(awserrors.ErrorInvalidInstanceIDNotFound)
+	}
+
+	if !IsInstanceVisible(accountID, instance.AccountID) {
+		slog.Warn("DescribeInstanceAttribute: instance not visible",
+			"instanceId", instanceID, "callerAccount", accountID, "ownerAccount", instance.AccountID)
+		return nil, errors.New(awserrors.ErrorInvalidInstanceIDNotFound)
+	}
+
+	output := &ec2.DescribeInstanceAttributeOutput{
+		InstanceId: &instanceID,
+	}
+
+	switch attribute {
+	case ec2.InstanceAttributeNameInstanceType:
+		val := instance.InstanceType
+		output.InstanceType = &ec2.AttributeValue{Value: &val}
+
+	case ec2.InstanceAttributeNameUserData:
+		val := instance.UserData
+		output.UserData = &ec2.AttributeValue{Value: &val}
+
+	case ec2.InstanceAttributeNameDisableApiTermination:
+		val := false
+		output.DisableApiTermination = &ec2.AttributeBooleanValue{Value: &val}
+
+	case ec2.InstanceAttributeNameDisableApiStop:
+		val := false
+		output.DisableApiStop = &ec2.AttributeBooleanValue{Value: &val}
+
+	case ec2.InstanceAttributeNameInstanceInitiatedShutdownBehavior:
+		val := ec2.ShutdownBehaviorStop
+		output.InstanceInitiatedShutdownBehavior = &ec2.AttributeValue{Value: &val}
+
+	case ec2.InstanceAttributeNameEbsOptimized:
+		val := false
+		output.EbsOptimized = &ec2.AttributeBooleanValue{Value: &val}
+
+	case ec2.InstanceAttributeNameEnaSupport:
+		val := true
+		output.EnaSupport = &ec2.AttributeBooleanValue{Value: &val}
+
+	case ec2.InstanceAttributeNameSourceDestCheck:
+		val := true
+		output.SourceDestCheck = &ec2.AttributeBooleanValue{Value: &val}
+
+	case ec2.InstanceAttributeNameGroupSet:
+		if instance.Instance != nil && len(instance.Instance.SecurityGroups) > 0 {
+			output.Groups = instance.Instance.SecurityGroups
+		} else {
+			output.Groups = []*ec2.GroupIdentifier{}
+		}
+
+	default:
+		slog.Warn("DescribeInstanceAttribute: unsupported attribute",
+			"instanceId", instanceID, "attribute", attribute)
+		return nil, errors.New(awserrors.ErrorInvalidParameterValue)
+	}
+
+	slog.Info("DescribeInstanceAttribute: completed",
+		"instanceId", instanceID, "attribute", attribute, "accountID", accountID)
+	return output, nil
+}

spinifex/handlers/ec2/instance/service_impl_test.go

@@ -10,12 +10,43 @@ import (
 	"github.com/aws/aws-sdk-go/aws"
 	"github.com/aws/aws-sdk-go/service/ec2"
 	"github.com/mulgadc/spinifex/spinifex/awserrors"
+	"github.com/mulgadc/spinifex/spinifex/config"
+	"github.com/mulgadc/spinifex/spinifex/objectstore"
 	"github.com/mulgadc/spinifex/spinifex/utils"
 	"github.com/mulgadc/spinifex/spinifex/vm"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 )
 
+// mgrWith returns a vm.Manager pre-populated with vms. Test fixture for
+// services that previously took a *vm.Instances; post-vm.Manager refactor we
+// build a Manager and Replace its set rather than inlining a map.
+func mgrWith(vms map[string]*vm.VM) *vm.Manager {
+	m := vm.NewManager()
+	if len(vms) > 0 {
+		m.Replace(vms)
+	}
+	return m
+}
+
+func TestNewInstanceServiceImpl(t *testing.T) {
+	cfg := &config.Config{}
+	instanceTypes := map[string]*ec2.InstanceTypeInfo{
+		"t3.micro": {InstanceType: aws.String("t3.micro")},
+	}
+	store := objectstore.NewMemoryObjectStore()
+	mgr := vm.NewManager()
+
+	svc := NewInstanceServiceImpl(cfg, instanceTypes, nil, store, mgr, nil, nil)
+
+	require.NotNil(t, svc)
+	assert.Equal(t, cfg, svc.config)
+	assert.Equal(t, instanceTypes, svc.instanceTypes)
+	assert.Nil(t, svc.natsConn)
+	assert.Equal(t, mgr, svc.vmMgr)
+	assert.Equal(t, store, svc.objectStore)
+}
+
 func TestGenerateHostname(t *testing.T) {
 	tests := []struct {
 		name       string
@@ -606,3 +637,402 @@ func TestCloudInitVolumeNamePerInstance(t *testing.T) {
 		seen[cloudInitName] = true
 	}
 }
+
+// --- 1b-pre Describe* coverage (siv-22 parts) ------------------------------
+
+type fakeResourceCapacityProvider struct {
+	types      []*ec2.InstanceTypeInfo
+	gotShowCap bool
+	calls      int
+}
+
+func (f *fakeResourceCapacityProvider) GetAvailableInstanceTypeInfos(showCapacity bool) []*ec2.InstanceTypeInfo {
+	f.calls++
+	f.gotShowCap = showCapacity
+	return f.types
+}
+
+type fakeStoppedStore struct {
+	stopped     []*vm.VM
+	terminated  []*vm.VM
+	loadByID    map[string]*vm.VM
+	listErr     error
+	listTermErr error
+	loadErr     error
+}
+
+func (f *fakeStoppedStore) LoadStoppedInstance(id string) (*vm.VM, error) {
+	if f.loadErr != nil {
+		return nil, f.loadErr
+	}
+	if v, ok := f.loadByID[id]; ok {
+		return v, nil
+	}
+	return nil, nil
+}
+func (f *fakeStoppedStore) ListStoppedInstances() ([]*vm.VM, error) {
+	if f.listErr != nil {
+		return nil, f.listErr
+	}
+	return f.stopped, nil
+}
+func (f *fakeStoppedStore) ListTerminatedInstances() ([]*vm.VM, error) {
+	if f.listTermErr != nil {
+		return nil, f.listTermErr
+	}
+	return f.terminated, nil
+}
+
+func TestDescribeInstanceTypes_NilResourceMgr(t *testing.T) {
+	svc := &InstanceServiceImpl{}
+	_, err := svc.DescribeInstanceTypes(&ec2.DescribeInstanceTypesInput{}, "")
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
+}
+
+func TestDescribeInstanceTypes_ReturnsProviderTypes(t *testing.T) {
+	prov := &fakeResourceCapacityProvider{
+		types: []*ec2.InstanceTypeInfo{
+			{InstanceType: aws.String("t3.micro")},
+			{InstanceType: aws.String("t3.small")},
+		},
+	}
+	svc := &InstanceServiceImpl{resourceMgr: prov}
+
+	out, err := svc.DescribeInstanceTypes(&ec2.DescribeInstanceTypesInput{}, "")
+	require.NoError(t, err)
+	require.Len(t, out.InstanceTypes, 2)
+	assert.False(t, prov.gotShowCap)
+}
+
+func TestDescribeInstanceTypes_CapacityFilterPropagates(t *testing.T) {
+	prov := &fakeResourceCapacityProvider{}
+	svc := &InstanceServiceImpl{resourceMgr: prov}
+
+	input := &ec2.DescribeInstanceTypesInput{
+		Filters: []*ec2.Filter{
+			{Name: aws.String("capacity"), Values: []*string{aws.String("true")}},
+		},
+	}
+	_, err := svc.DescribeInstanceTypes(input, "")
+	require.NoError(t, err)
+	assert.True(t, prov.gotShowCap, "capacity=true filter must reach the provider")
+}
+
+func TestDescribeInstances_Empty(t *testing.T) {
+	svc := &InstanceServiceImpl{vmMgr: mgrWith(map[string]*vm.VM{})}
+	out, err := svc.DescribeInstances(&ec2.DescribeInstancesInput{}, utils.GlobalAccountID)
+	require.NoError(t, err)
+	assert.Empty(t, out.Reservations)
+}
+
+func TestDescribeInstances_OneVisibleInstance(t *testing.T) {
+	id := "i-aaa111"
+	resID := "r-1"
+	owner := "111122223333"
+	v := &vm.VM{
+		ID:           id,
+		InstanceType: "t3.micro",
+		Status:       vm.StateRunning,
+		AccountID:    owner,
+		Reservation: &ec2.Reservation{
+			ReservationId: aws.String(resID),
+			OwnerId:       aws.String(owner),
+		},
+		Instance: &ec2.Instance{InstanceId: aws.String(id), InstanceType: aws.String("t3.micro")},
+	}
+	svc := &InstanceServiceImpl{vmMgr: mgrWith(map[string]*vm.VM{id: v})}
+
+	out, err := svc.DescribeInstances(&ec2.DescribeInstancesInput{}, owner)
+	require.NoError(t, err)
+	require.Len(t, out.Reservations, 1)
+	assert.Equal(t, resID, *out.Reservations[0].ReservationId)
+	require.Len(t, out.Reservations[0].Instances, 1)
+	assert.Equal(t, id, *out.Reservations[0].Instances[0].InstanceId)
+}
+
+func TestDescribeInstances_AccountFilteringHidesOtherTenant(t *testing.T) {
+	v := &vm.VM{
+		ID:        "i-other",
+		AccountID: "999988887777",
+		Reservation: &ec2.Reservation{
+			ReservationId: aws.String("r-other"),
+			OwnerId:       aws.String("999988887777"),
+		},
+		Instance: &ec2.Instance{InstanceId: aws.String("i-other")},
+	}
+	svc := &InstanceServiceImpl{vmMgr: mgrWith(map[string]*vm.VM{v.ID: v})}
+
+	out, err := svc.DescribeInstances(&ec2.DescribeInstancesInput{}, "111122223333")
+	require.NoError(t, err)
+	assert.Empty(t, out.Reservations)
+}
+
+func TestDescribeInstances_MalformedID(t *testing.T) {
+	svc := &InstanceServiceImpl{vmMgr: mgrWith(map[string]*vm.VM{})}
+	input := &ec2.DescribeInstancesInput{
+		InstanceIds: []*string{aws.String("not-an-id")},
+	}
+	_, err := svc.DescribeInstances(input, utils.GlobalAccountID)
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorInvalidInstanceIDMalformed, err.Error())
+}
+
+func TestDescribeInstances_FilterByInstanceID(t *testing.T) {
+	keep := &vm.VM{
+		ID: "i-keep",
+		Reservation: &ec2.Reservation{
+			ReservationId: aws.String("r-keep"),
+		},
+		Instance: &ec2.Instance{InstanceId: aws.String("i-keep")},
+	}
+	drop := &vm.VM{
+		ID: "i-drop",
+		Reservation: &ec2.Reservation{
+			ReservationId: aws.String("r-drop"),
+		},
+		Instance: &ec2.Instance{InstanceId: aws.String("i-drop")},
+	}
+	svc := &InstanceServiceImpl{vmMgr: mgrWith(map[string]*vm.VM{
+		keep.ID: keep,
+		drop.ID: drop,
+	})}
+
+	input := &ec2.DescribeInstancesInput{InstanceIds: []*string{aws.String("i-keep")}}
+	out, err := svc.DescribeInstances(input, utils.GlobalAccountID)
+	require.NoError(t, err)
+	require.Len(t, out.Reservations, 1)
+	assert.Equal(t, "i-keep", *out.Reservations[0].Instances[0].InstanceId)
+}
+
+func TestDescribeInstanceAttribute_MissingInstanceID(t *testing.T) {
+	svc := &InstanceServiceImpl{vmMgr: mgrWith(map[string]*vm.VM{})}
+	_, err := svc.DescribeInstanceAttribute(&ec2.DescribeInstanceAttributeInput{
+		Attribute: aws.String(ec2.InstanceAttributeNameInstanceType),
+	}, "")
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorMissingParameter, err.Error())
+}
+
+func TestDescribeInstanceAttribute_MissingAttribute(t *testing.T) {
+	svc := &InstanceServiceImpl{vmMgr: mgrWith(map[string]*vm.VM{})}
+	_, err := svc.DescribeInstanceAttribute(&ec2.DescribeInstanceAttributeInput{
+		InstanceId: aws.String("i-aaa"),
+	}, "")
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorMissingParameter, err.Error())
+}
+
+func TestDescribeInstanceAttribute_RunningInstance(t *testing.T) {
+	id := "i-attr1"
+	owner := utils.GlobalAccountID
+	v := &vm.VM{
+		ID:           id,
+		InstanceType: "t3.large",
+		AccountID:    owner,
+		UserData:     "raw-user-data",
+	}
+	svc := &InstanceServiceImpl{vmMgr: mgrWith(map[string]*vm.VM{id: v})}
+
+	tests := []struct {
+		name       string
+		attribute  string
+		assertions func(t *testing.T, out *ec2.DescribeInstanceAttributeOutput)
+	}{
+		{
+			name:      "instanceType",
+			attribute: ec2.InstanceAttributeNameInstanceType,
+			assertions: func(t *testing.T, out *ec2.DescribeInstanceAttributeOutput) {
+				require.NotNil(t, out.InstanceType)
+				assert.Equal(t, "t3.large", *out.InstanceType.Value)
+			},
+		},
+		{
+			name:      "userData",
+			attribute: ec2.InstanceAttributeNameUserData,
+			assertions: func(t *testing.T, out *ec2.DescribeInstanceAttributeOutput) {
+				require.NotNil(t, out.UserData)
+				assert.Equal(t, "raw-user-data", *out.UserData.Value)
+			},
+		},
+		{
+			name:      "disableApiTermination",
+			attribute: ec2.InstanceAttributeNameDisableApiTermination,
+			assertions: func(t *testing.T, out *ec2.DescribeInstanceAttributeOutput) {
+				require.NotNil(t, out.DisableApiTermination)
+				assert.False(t, *out.DisableApiTermination.Value)
+			},
+		},
+		{
+			name:      "ebsOptimized",
+			attribute: ec2.InstanceAttributeNameEbsOptimized,
+			assertions: func(t *testing.T, out *ec2.DescribeInstanceAttributeOutput) {
+				require.NotNil(t, out.EbsOptimized)
+				assert.False(t, *out.EbsOptimized.Value)
+			},
+		},
+		{
+			name:      "enaSupport",
+			attribute: ec2.InstanceAttributeNameEnaSupport,
+			assertions: func(t *testing.T, out *ec2.DescribeInstanceAttributeOutput) {
+				require.NotNil(t, out.EnaSupport)
+				assert.True(t, *out.EnaSupport.Value)
+			},
+		},
+		{
+			name:      "sourceDestCheck",
+			attribute: ec2.InstanceAttributeNameSourceDestCheck,
+			assertions: func(t *testing.T, out *ec2.DescribeInstanceAttributeOutput) {
+				require.NotNil(t, out.SourceDestCheck)
+				assert.True(t, *out.SourceDestCheck.Value)
+			},
+		},
+	}
+
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			out, err := svc.DescribeInstanceAttribute(&ec2.DescribeInstanceAttributeInput{
+				InstanceId: aws.String(id),
+				Attribute:  aws.String(tc.attribute),
+			}, owner)
+			require.NoError(t, err)
+			require.NotNil(t, out)
+			assert.Equal(t, id, *out.InstanceId)
+			tc.assertions(t, out)
+		})
+	}
+}
+
+func TestDescribeInstanceAttribute_NotRunning_NoStore(t *testing.T) {
+	svc := &InstanceServiceImpl{vmMgr: mgrWith(map[string]*vm.VM{})}
+	_, err := svc.DescribeInstanceAttribute(&ec2.DescribeInstanceAttributeInput{
+		InstanceId: aws.String("i-missing"),
+		Attribute:  aws.String(ec2.InstanceAttributeNameInstanceType),
+	}, "")
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
+}
+
+func TestDescribeInstanceAttribute_FoundInStoppedStore(t *testing.T) {
+	id := "i-stopped1"
+	owner := utils.GlobalAccountID
+	store := &fakeStoppedStore{loadByID: map[string]*vm.VM{
+		id: {ID: id, InstanceType: "t3.medium", AccountID: owner},
+	}}
+	svc := &InstanceServiceImpl{
+		vmMgr:        mgrWith(map[string]*vm.VM{}),
+		stoppedStore: store,
+	}
+
+	out, err := svc.DescribeInstanceAttribute(&ec2.DescribeInstanceAttributeInput{
+		InstanceId: aws.String(id),
+		Attribute:  aws.String(ec2.InstanceAttributeNameInstanceType),
+	}, owner)
+	require.NoError(t, err)
+	assert.Equal(t, "t3.medium", *out.InstanceType.Value)
+}
+
+func TestDescribeInstanceAttribute_NotFound(t *testing.T) {
+	svc := &InstanceServiceImpl{
+		vmMgr:        mgrWith(map[string]*vm.VM{}),
+		stoppedStore: &fakeStoppedStore{loadByID: map[string]*vm.VM{}},
+	}
+	_, err := svc.DescribeInstanceAttribute(&ec2.DescribeInstanceAttributeInput{
+		InstanceId: aws.String("i-ghost"),
+		Attribute:  aws.String(ec2.InstanceAttributeNameInstanceType),
+	}, utils.GlobalAccountID)
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorInvalidInstanceIDNotFound, err.Error())
+}
+
+func TestDescribeInstanceAttribute_HiddenForOtherAccount(t *testing.T) {
+	id := "i-other-acct"
+	v := &vm.VM{ID: id, InstanceType: "t3.micro", AccountID: "999988887777"}
+	svc := &InstanceServiceImpl{vmMgr: mgrWith(map[string]*vm.VM{id: v})}
+
+	_, err := svc.DescribeInstanceAttribute(&ec2.DescribeInstanceAttributeInput{
+		InstanceId: aws.String(id),
+		Attribute:  aws.String(ec2.InstanceAttributeNameInstanceType),
+	}, "111122223333")
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorInvalidInstanceIDNotFound, err.Error())
+}
+
+func TestDescribeStoppedInstances_NilStore(t *testing.T) {
+	svc := &InstanceServiceImpl{}
+	_, err := svc.DescribeStoppedInstances(&ec2.DescribeInstancesInput{}, "")
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
+}
+
+func TestDescribeStoppedInstances_HappyPath(t *testing.T) {
+	owner := utils.GlobalAccountID
+	store := &fakeStoppedStore{
+		stopped: []*vm.VM{
+			{
+				ID:        "i-stop1",
+				AccountID: owner,
+				Reservation: &ec2.Reservation{
+					ReservationId: aws.String("r-stop1"),
+					OwnerId:       aws.String(owner),
+				},
+				Instance: &ec2.Instance{InstanceId: aws.String("i-stop1")},
+			},
+		},
+	}
+	svc := &InstanceServiceImpl{stoppedStore: store}
+
+	out, err := svc.DescribeStoppedInstances(&ec2.DescribeInstancesInput{}, owner)
+	require.NoError(t, err)
+	require.Len(t, out.Reservations, 1)
+	assert.Equal(t, "i-stop1", *out.Reservations[0].Instances[0].InstanceId)
+}
+
+func TestDescribeTerminatedInstances_NilStore(t *testing.T) {
+	svc := &InstanceServiceImpl{}
+	_, err := svc.DescribeTerminatedInstances(&ec2.DescribeInstancesInput{}, "")
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
+}
+
+func TestDescribeTerminatedInstances_HappyPath(t *testing.T) {
+	owner := utils.GlobalAccountID
+	store := &fakeStoppedStore{
+		terminated: []*vm.VM{
+			{
+				ID:        "i-term1",
+				AccountID: owner,
+				Reservation: &ec2.Reservation{
+					ReservationId: aws.String("r-term1"),
+					OwnerId:       aws.String(owner),
+				},
+				Instance: &ec2.Instance{InstanceId: aws.String("i-term1")},
+			},
+		},
+	}
+	svc := &InstanceServiceImpl{stoppedStore: store}
+
+	out, err := svc.DescribeTerminatedInstances(&ec2.DescribeInstancesInput{}, owner)
+	require.NoError(t, err)
+	require.Len(t, out.Reservations, 1)
+	assert.Equal(t, "i-term1", *out.Reservations[0].Instances[0].InstanceId)
+}
+
+func TestIsInstanceVisible(t *testing.T) {
+	tests := []struct {
+		name   string
+		caller string
+		owner  string
+		want   bool
+	}{
+		{"empty owner, global caller", utils.GlobalAccountID, "", true},
+		{"empty owner, non-global caller", "111122223333", "", false},
+		{"matching account", "111122223333", "111122223333", true},
+		{"different accounts", "111122223333", "999988887777", false},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			assert.Equal(t, tc.want, IsInstanceVisible(tc.caller, tc.owner))
+		})
+	}
+}

spinifex/handlers/ec2/instance/service_mock.go

@@ -35,3 +35,23 @@ func (s *MockInstanceService) RunInstances(input *ec2.RunInstancesInput, account
 
 	return reservation, nil
 }
+
+func (s *MockInstanceService) DescribeInstances(_ *ec2.DescribeInstancesInput, _ string) (*ec2.DescribeInstancesOutput, error) {
+	return &ec2.DescribeInstancesOutput{}, nil
+}
+
+func (s *MockInstanceService) DescribeInstanceTypes(_ *ec2.DescribeInstanceTypesInput, _ string) (*ec2.DescribeInstanceTypesOutput, error) {
+	return &ec2.DescribeInstanceTypesOutput{}, nil
+}
+
+func (s *MockInstanceService) DescribeInstanceAttribute(input *ec2.DescribeInstanceAttributeInput, _ string) (*ec2.DescribeInstanceAttributeOutput, error) {
+	return &ec2.DescribeInstanceAttributeOutput{InstanceId: input.InstanceId}, nil
+}
+
+func (s *MockInstanceService) DescribeStoppedInstances(_ *ec2.DescribeInstancesInput, _ string) (*ec2.DescribeInstancesOutput, error) {
+	return &ec2.DescribeInstancesOutput{}, nil
+}
+
+func (s *MockInstanceService) DescribeTerminatedInstances(_ *ec2.DescribeInstancesInput, _ string) (*ec2.DescribeInstancesOutput, error) {
+	return &ec2.DescribeInstancesOutput{}, nil
+}

spinifex/handlers/ec2/instance/service_nats.go

@@ -10,7 +10,14 @@ import (
 	"github.com/nats-io/nats.go"
 )
 
-// NATSInstanceService handles instance operations via NATS messaging
+// NATSInstanceService handles instance operations via NATS messaging.
+//
+// Note: DescribeInstances and DescribeInstanceTypes subscriptions are
+// fan-out (no queue group); a single NATSRequest only captures one
+// responder. Cluster-wide aggregation is the gateway's job (see
+// spinifex/gateway/ec2/instance). The methods below are kept for symmetry
+// with the InstanceService interface and intentionally fail rather than
+// silently returning a one-node view.
 type NATSInstanceService struct {
 	natsConn *nats.Conn
 }
@@ -29,3 +36,27 @@ func (s *NATSInstanceService) RunInstances(input *ec2.RunInstancesInput, account
 	topic := fmt.Sprintf("ec2.RunInstances.%s", aws.StringValue(input.InstanceType))
 	return utils.NATSRequest[ec2.Reservation](s.natsConn, topic, input, 5*time.Minute, accountID)
 }
+
+// DescribeInstances is a fan-out operation; use the gateway's scatter-gather
+// helper instead of this single-node client.
+func (s *NATSInstanceService) DescribeInstances(_ *ec2.DescribeInstancesInput, _ string) (*ec2.DescribeInstancesOutput, error) {
+	return nil, fmt.Errorf("DescribeInstances is fan-out; use gateway scatter-gather, not NATSInstanceService")
+}
+
+// DescribeInstanceTypes is a fan-out operation; use the gateway's scatter-gather
+// helper instead of this single-node client.
+func (s *NATSInstanceService) DescribeInstanceTypes(_ *ec2.DescribeInstanceTypesInput, _ string) (*ec2.DescribeInstanceTypesOutput, error) {
+	return nil, fmt.Errorf("DescribeInstanceTypes is fan-out; use gateway scatter-gather, not NATSInstanceService")
+}
+
+func (s *NATSInstanceService) DescribeInstanceAttribute(input *ec2.DescribeInstanceAttributeInput, accountID string) (*ec2.DescribeInstanceAttributeOutput, error) {
+	return utils.NATSRequest[ec2.DescribeInstanceAttributeOutput](s.natsConn, "ec2.DescribeInstanceAttribute", input, 30*time.Second, accountID)
+}
+
+func (s *NATSInstanceService) DescribeStoppedInstances(input *ec2.DescribeInstancesInput, accountID string) (*ec2.DescribeInstancesOutput, error) {
+	return utils.NATSRequest[ec2.DescribeInstancesOutput](s.natsConn, "ec2.DescribeStoppedInstances", input, 30*time.Second, accountID)
+}
+
+func (s *NATSInstanceService) DescribeTerminatedInstances(input *ec2.DescribeInstancesInput, accountID string) (*ec2.DescribeInstancesOutput, error) {
+	return utils.NATSRequest[ec2.DescribeInstancesOutput](s.natsConn, "ec2.DescribeTerminatedInstances", input, 30*time.Second, accountID)
+}

spinifex/utils/nats.go

@@ -1,6 +1,7 @@
 package utils
 
 import (
+	"context"
 	"crypto/tls"
 	"crypto/x509"
 	"encoding/json"
@@ -21,23 +22,51 @@ var (
 	ErrCACertParse = errors.New("failed to parse CA cert")
 )
 
+// ErrClusterUnavailable is returned by NATS request helpers when the underlying
+// connection is not currently connected. Callers (gateway, scatter-gather
+// fan-out) use this to fail fast instead of waiting for per-call timeouts.
+var ErrClusterUnavailable = errors.New("cluster unavailable: NATS disconnected")
+
+// natsRetryEscalateAttempt is the attempt count past which ConnectNATSWithRetry
+// promotes the per-attempt log line from slog.Warn to slog.Error (rate-limited
+// to once a minute). With the 60s backoff cap, ~30 attempts corresponds to
+// ~30 minutes of continuous disconnection — long enough to suspect a config
+// error rather than a routine NATS restart.
+const natsRetryEscalateAttempt = 30
+
 // ConnectNATS establishes a connection to a NATS server with standard reconnect
 // handling and logging. If token is non-empty, token authentication is used.
 // If caCertPath is non-empty, TLS is enabled using the given CA certificate.
-func ConnectNATS(host, token, caCertPath string) (*nats.Conn, error) {
-	opts := []nats.Option{
+//
+// Optional callback hooks (WithDisconnectHandler / WithReconnectHandler) wrap
+// the default log lines so callers can react to connectivity changes (e.g. the
+// daemon flips its cluster/standalone mode field) without losing the existing
+// disconnect/reconnect log output.
+func ConnectNATS(host, token, caCertPath string, opts ...RetryOption) (*nats.Conn, error) {
+	cfg := retryConfig{}
+	for _, o := range opts {
+		o(&cfg)
+	}
+
+	natsOpts := []nats.Option{
 		nats.ReconnectWait(time.Second),
 		nats.MaxReconnects(-1),
 		nats.DisconnectErrHandler(func(nc *nats.Conn, err error) {
 			slog.Warn("NATS disconnected", "err", err)
+			if cfg.onDisconnect != nil {
+				cfg.onDisconnect(nc, err)
+			}
 		}),
 		nats.ReconnectHandler(func(nc *nats.Conn) {
 			slog.Warn("NATS reconnected", "url", nc.ConnectedUrl())
+			if cfg.onReconnect != nil {
+				cfg.onReconnect(nc)
+			}
 		}),
 	}
 
 	if token != "" {
-		opts = append(opts, nats.Token(token))
+		natsOpts = append(natsOpts, nats.Token(token))
 	}
 
 	if caCertPath != "" {
@@ -49,12 +78,12 @@ func ConnectNATS(host, token, caCertPath string) (*nats.Conn, error) {
 		if !pool.AppendCertsFromPEM(caCert) {
 			return nil, fmt.Errorf("%w from %s", ErrCACertParse, caCertPath)
 		}
-		opts = append(opts, nats.Secure(&tls.Config{
+		natsOpts = append(natsOpts, nats.Secure(&tls.Config{
 			RootCAs: pool,
 		}))
 	}
 
-	nc, err := nats.Connect(host, opts...)
+	nc, err := nats.Connect(host, natsOpts...)
 	if err != nil {
 		return nil, fmt.Errorf("NATS connect failed: %w", err)
 	}
@@ -63,13 +92,22 @@ func ConnectNATS(host, token, caCertPath string) (*nats.Conn, error) {
 	return nc, nil
 }
 
-// retryConfig holds parameters for ConnectNATSWithRetry.
+// retryConfig holds parameters for ConnectNATS / ConnectNATSWithRetry.
+//
+// retry-related fields tune the outer reconnect-with-backoff loop;
+// callback fields wrap nats.go's connection-state handlers so callers can
+// react to disconnects/reconnects without losing the default log lines.
 type retryConfig struct {
-	maxWait    time.Duration
-	retryDelay time.Duration
+	maxWait       time.Duration
+	retryDelay    time.Duration
+	maxRetryDelay time.Duration
+	onDisconnect  func(*nats.Conn, error)
+	onReconnect   func(*nats.Conn)
+	onAttemptErr  func(err error, attempt int)
+	ctx           context.Context
 }
 
-// RetryOption configures ConnectNATSWithRetry behavior.
+// RetryOption configures ConnectNATS / ConnectNATSWithRetry behavior.
 type RetryOption func(*retryConfig)
 
 // WithMaxWait sets the maximum total time to keep retrying before giving up.
@@ -82,22 +120,62 @@ func WithRetryDelay(d time.Duration) RetryOption {
 	return func(c *retryConfig) { c.retryDelay = d }
 }
 
+// WithMaxRetryDelay overrides the upper bound on the exponential backoff
+// (default 10s).
+func WithMaxRetryDelay(d time.Duration) RetryOption {
+	return func(c *retryConfig) { c.maxRetryDelay = d }
+}
+
+// WithDisconnectHandler registers an optional callback invoked after the
+// default disconnect log line. Callback runs on a NATS client goroutine; keep
+// it non-blocking (atomic stores, channel sends, goroutine spawns).
+func WithDisconnectHandler(fn func(*nats.Conn, error)) RetryOption {
+	return func(c *retryConfig) { c.onDisconnect = fn }
+}
+
+// WithReconnectHandler registers an optional callback invoked after the
+// default reconnect log line. Same goroutine constraints as WithDisconnectHandler.
+func WithReconnectHandler(fn func(*nats.Conn)) RetryOption {
+	return func(c *retryConfig) { c.onReconnect = fn }
+}
+
+// WithAttemptErrHandler registers an optional callback invoked after each
+// failed connect attempt during ConnectNATSWithRetry's outer loop. Used by the
+// daemon to surface initial-connect retries as a counter on /local/status.
+func WithAttemptErrHandler(fn func(err error, attempt int)) RetryOption {
+	return func(c *retryConfig) { c.onAttemptErr = fn }
+}
+
+// WithContext lets callers cancel the retry loop. When ctx is done,
+// ConnectNATSWithRetry returns ctx.Err().
+func WithContext(ctx context.Context) RetryOption {
+	return func(c *retryConfig) { c.ctx = ctx }
+}
+
 // ConnectNATSWithRetry calls ConnectNATS in a retry loop with exponential
-// backoff. It retries for up to 5 minutes (default) before giving up. TLS
+// backoff. It retries for up to 5 minutes (default) before giving up; pass
+// WithMaxWait(0) to retry indefinitely (cancel via WithContext). TLS
 // configuration errors (ErrCACertRead, ErrCACertParse) are permanent and
 // cause an immediate return without retrying.
 func ConnectNATSWithRetry(host, token, caCertPath string, opts ...RetryOption) (*nats.Conn, error) {
 	cfg := retryConfig{
-		maxWait:    5 * time.Minute,
-		retryDelay: 500 * time.Millisecond,
+		maxWait:       5 * time.Minute,
+		retryDelay:    500 * time.Millisecond,
+		maxRetryDelay: 10 * time.Second,
 	}
 	for _, o := range opts {
 		o(&cfg)
 	}
+	if cfg.maxRetryDelay <= 0 {
+		cfg.maxRetryDelay = 10 * time.Second
+	}
 
 	start := time.Now()
+	attempt := 0
+	var lastEscalatedLog time.Time
 	for {
-		nc, err := ConnectNATS(host, token, caCertPath)
+		attempt++
+		nc, err := ConnectNATS(host, token, caCertPath, opts...)
 		if err == nil {
 			if time.Since(start) > time.Second {
 				slog.Info("NATS connection established", "elapsed", time.Since(start).Round(time.Second))
@@ -110,14 +188,38 @@ func ConnectNATSWithRetry(host, token, caCertPath string, opts ...RetryOption) (
 			return nil, fmt.Errorf("NATS TLS configuration error: %w", err)
 		}
 
+		if cfg.onAttemptErr != nil {
+			cfg.onAttemptErr(err, attempt)
+		}
+
 		elapsed := time.Since(start)
-		if elapsed >= cfg.maxWait {
+		if cfg.maxWait > 0 && elapsed >= cfg.maxWait {
 			return nil, fmt.Errorf("NATS connect failed after %s: %w", elapsed.Round(time.Second), err)
 		}
 
-		slog.Warn("NATS not ready, retrying...", "error", err, "elapsed", elapsed.Round(time.Second), "retryIn", cfg.retryDelay)
-		time.Sleep(cfg.retryDelay)
-		cfg.retryDelay = min(cfg.retryDelay*2, 10*time.Second)
+		// Past the retry-escalation threshold (~30 attempts ≈ ~30 min disconnected
+		// at the 60s backoff cap) the warn-on-every-retry pattern can hide a
+		// stuck NATS config in routine warning noise. Escalate to slog.Error,
+		// rate-limited to once a minute, so the operator's error log surfaces it.
+		if attempt > natsRetryEscalateAttempt {
+			if lastEscalatedLog.IsZero() || time.Since(lastEscalatedLog) >= time.Minute {
+				slog.Error("NATS still disconnected", "error", err, "disconnected_for", elapsed.Round(time.Second), "attempt", attempt)
+				lastEscalatedLog = time.Now()
+			}
+		} else {
+			slog.Warn("NATS not ready, retrying...", "error", err, "elapsed", elapsed.Round(time.Second), "retryIn", cfg.retryDelay, "attempt", attempt)
+		}
+
+		if cfg.ctx != nil {
+			select {
+			case <-cfg.ctx.Done():
+				return nil, fmt.Errorf("NATS connect cancelled after %s: %w", elapsed.Round(time.Second), cfg.ctx.Err())
+			case <-time.After(cfg.retryDelay):
+			}
+		} else {
+			time.Sleep(cfg.retryDelay)
+		}
+		cfg.retryDelay = min(cfg.retryDelay*2, cfg.maxRetryDelay)
 	}
 }
 
@@ -131,6 +233,10 @@ const AccountIDHeader = "X-Account-ID"
 // successful response into Out. Handlers can ignore the account ID if the
 // operation is unscoped (e.g. DescribeInstanceTypes).
 func NATSRequest[Out any](conn *nats.Conn, subject string, input any, timeout time.Duration, accountID string) (*Out, error) {
+	if conn == nil || !conn.IsConnected() {
+		return nil, ErrClusterUnavailable
+	}
+
 	jsonData, err := json.Marshal(input)
 	if err != nil {
 		return nil, fmt.Errorf("failed to marshal input: %w", err)
@@ -179,6 +285,10 @@ const (
 // error is returned. When expectedNodes > 0, collection exits early once that
 // many responses have been received.
 func NATSScatterGather[Out any](conn *nats.Conn, subject string, input any, timeout time.Duration, expectedNodes int, accountID string) (*Out, error) {
+	if conn == nil || !conn.IsConnected() {
+		return nil, ErrClusterUnavailable
+	}
+
 	jsonData, err := json.Marshal(input)
 	if err != nil {
 		return nil, fmt.Errorf("failed to marshal input: %w", err)

spinifex/utils/nats_test.go

@@ -1,6 +1,7 @@
 package utils
 
 import (
+	"bytes"
 	"crypto/ecdsa"
 	"crypto/elliptic"
 	"crypto/rand"
@@ -9,10 +10,13 @@ import (
 	"crypto/x509/pkix"
 	"encoding/json"
 	"encoding/pem"
+	"log/slog"
 	"math/big"
 	"net"
 	"os"
 	"path/filepath"
+	"strconv"
+	"strings"
 	"testing"
 	"time"
 
@@ -573,3 +577,247 @@ func TestConnectNATSWithRetry_TLSErrorNoRetry(t *testing.T) {
 	assert.Contains(t, err.Error(), "NATS TLS configuration error")
 	assert.Less(t, elapsed, time.Second, "should fail immediately without retrying")
 }
+
+// --- Disconnect/Reconnect callback tests ---
+
+func TestConnectNATS_DisconnectCallbackFires(t *testing.T) {
+	ns := startTestNATSServer(t)
+
+	disconnected := make(chan struct{}, 1)
+	nc, err := ConnectNATS(ns.ClientURL(), "", "",
+		WithDisconnectHandler(func(_ *nats.Conn, _ error) {
+			select {
+			case disconnected <- struct{}{}:
+			default:
+			}
+		}),
+	)
+	require.NoError(t, err)
+	defer nc.Close()
+	require.True(t, nc.IsConnected())
+
+	ns.Shutdown()
+
+	select {
+	case <-disconnected:
+	case <-time.After(3 * time.Second):
+		t.Fatal("disconnect callback did not fire")
+	}
+}
+
+func TestConnectNATS_ReconnectCallbackFires(t *testing.T) {
+	// Pin the test NATS to a specific port so we can restart it on the same URL.
+	port := freePort(t)
+	ns := startTestNATSOnPort(t, port)
+
+	reconnected := make(chan struct{}, 1)
+	nc, err := ConnectNATS(ns.ClientURL(), "", "",
+		WithReconnectHandler(func(_ *nats.Conn) {
+			select {
+			case reconnected <- struct{}{}:
+			default:
+			}
+		}),
+	)
+	require.NoError(t, err)
+	defer nc.Close()
+	require.True(t, nc.IsConnected())
+
+	ns.Shutdown()
+	// Wait until the client noticed the drop so the reconnect path runs.
+	require.Eventually(t, func() bool { return !nc.IsConnected() }, 3*time.Second, 50*time.Millisecond)
+
+	startTestNATSOnPort(t, port)
+
+	select {
+	case <-reconnected:
+	case <-time.After(5 * time.Second):
+		t.Fatal("reconnect callback did not fire")
+	}
+}
+
+// --- Fast-fail when disconnected ---
+
+func TestNATSRequest_DisconnectedFastFail(t *testing.T) {
+	ns := startTestNATSServer(t)
+	nc, err := ConnectNATS(ns.ClientURL(), "", "")
+	require.NoError(t, err)
+	defer nc.Close()
+
+	ns.Shutdown()
+	require.Eventually(t, func() bool { return !nc.IsConnected() }, 3*time.Second, 50*time.Millisecond)
+
+	start := time.Now()
+	_, err = NATSRequest[map[string]any](nc, "ec2.Describe", struct{}{}, 5*time.Second, "")
+	elapsed := time.Since(start)
+
+	require.ErrorIs(t, err, ErrClusterUnavailable)
+	assert.Less(t, elapsed, 500*time.Millisecond, "should bail before per-call timeout")
+}
+
+func TestNATSScatterGather_DisconnectedFastFail(t *testing.T) {
+	ns := startTestNATSServer(t)
+	nc, err := ConnectNATS(ns.ClientURL(), "", "")
+	require.NoError(t, err)
+	defer nc.Close()
+
+	ns.Shutdown()
+	require.Eventually(t, func() bool { return !nc.IsConnected() }, 3*time.Second, 50*time.Millisecond)
+
+	start := time.Now()
+	_, err = NATSScatterGather[map[string]any](nc, "ec2.Describe", struct{}{}, 5*time.Second, 0, "")
+	elapsed := time.Since(start)
+
+	require.ErrorIs(t, err, ErrClusterUnavailable)
+	assert.Less(t, elapsed, 500*time.Millisecond, "should bail before fan-out timeout")
+}
+
+// --- 1c fail-fast tests: NATS request helpers reject when conn is down ---
+
+func TestNATSRequest_NilConn_ReturnsClusterUnavailable(t *testing.T) {
+	type Resp struct{}
+	_, err := NATSRequest[Resp](nil, "test.never", struct{}{}, 50*time.Millisecond, "")
+	require.Error(t, err)
+	assert.ErrorIs(t, err, ErrClusterUnavailable)
+}
+
+func TestNATSRequest_ClosedConn_ReturnsClusterUnavailable(t *testing.T) {
+	ns := startTestNATSServer(t)
+	nc, err := nats.Connect(ns.ClientURL())
+	require.NoError(t, err)
+	nc.Close()
+
+	type Resp struct{}
+	_, err = NATSRequest[Resp](nc, "test.never", struct{}{}, 50*time.Millisecond, "")
+	require.Error(t, err)
+	assert.ErrorIs(t, err, ErrClusterUnavailable)
+}
+
+func TestNATSScatterGather_NilConn_ReturnsClusterUnavailable(t *testing.T) {
+	type Resp struct{}
+	_, err := NATSScatterGather[Resp](nil, "test.never", struct{}{}, 50*time.Millisecond, 1, "")
+	require.Error(t, err)
+	assert.ErrorIs(t, err, ErrClusterUnavailable)
+}
+
+func TestNATSScatterGather_ClosedConn_ReturnsClusterUnavailable(t *testing.T) {
+	ns := startTestNATSServer(t)
+	nc, err := nats.Connect(ns.ClientURL())
+	require.NoError(t, err)
+	nc.Close()
+
+	type Resp struct{}
+	_, err = NATSScatterGather[Resp](nc, "test.never", struct{}{}, 50*time.Millisecond, 1, "")
+	require.Error(t, err)
+	assert.ErrorIs(t, err, ErrClusterUnavailable)
+}
+
+// --- 1c callback hook plumbing: WithDisconnectHandler / WithReconnectHandler ---
+
+func TestConnectNATS_DisconnectReconnectCallbacks(t *testing.T) {
+	port := freePort(t)
+	ns := startTestNATSOnPort(t, port)
+
+	disconnects := make(chan struct{}, 4)
+	reconnects := make(chan struct{}, 4)
+
+	nc, err := ConnectNATS("nats://127.0.0.1:"+strconv.Itoa(port), "", "",
+		WithDisconnectHandler(func(_ *nats.Conn, _ error) { disconnects <- struct{}{} }),
+		WithReconnectHandler(func(_ *nats.Conn) { reconnects <- struct{}{} }),
+	)
+	require.NoError(t, err)
+	defer nc.Close()
+
+	ns.Shutdown()
+	select {
+	case <-disconnects:
+	case <-time.After(3 * time.Second):
+		t.Fatal("disconnect callback never fired")
+	}
+
+	startTestNATSOnPort(t, port)
+	select {
+	case <-reconnects:
+	case <-time.After(5 * time.Second):
+		t.Fatal("reconnect callback never fired")
+	}
+}
+
+// --- helpers for restartable NATS server ---
+
+func freePort(t *testing.T) int {
+	t.Helper()
+	l, err := net.Listen("tcp", "127.0.0.1:0")
+	require.NoError(t, err)
+	addr, ok := l.Addr().(*net.TCPAddr)
+	require.True(t, ok)
+	port := addr.Port
+	require.NoError(t, l.Close())
+	return port
+}
+
+func startTestNATSOnPort(t *testing.T, port int) *server.Server {
+	t.Helper()
+	opts := &server.Options{
+		Host:   "127.0.0.1",
+		Port:   port,
+		NoLog:  true,
+		NoSigs: true,
+	}
+	ns, err := server.NewServer(opts)
+	require.NoError(t, err)
+	go ns.Start()
+	require.True(t, ns.ReadyForConnections(5*time.Second))
+	t.Cleanup(func() { ns.Shutdown() })
+	return ns
+}
+
+// TestConnectNATSWithRetry_LogEscalatesPastThreshold drives ConnectNATSWithRetry
+// against a closed port with a tight retry loop until the attempt count exceeds
+// natsRetryEscalateAttempt, then asserts that at least one escalated slog.Error
+// line was emitted with disconnected_for context (rate-limited to once per
+// minute) while earlier attempts stayed at slog.Warn.
+func TestConnectNATSWithRetry_LogEscalatesPastThreshold(t *testing.T) {
+	var buf bytes.Buffer
+	prev := slog.Default()
+	t.Cleanup(func() { slog.SetDefault(prev) })
+	slog.SetDefault(slog.New(slog.NewTextHandler(&buf, &slog.HandlerOptions{Level: slog.LevelDebug})))
+
+	_, err := ConnectNATSWithRetry("nats://127.0.0.1:1", "", "",
+		WithRetryDelay(1*time.Millisecond),
+		WithMaxRetryDelay(1*time.Millisecond),
+		WithMaxWait(300*time.Millisecond),
+	)
+	require.Error(t, err)
+
+	logs := buf.String()
+	warnCount := strings.Count(logs, "level=WARN msg=\"NATS not ready, retrying...\"")
+	errCount := strings.Count(logs, "level=ERROR msg=\"NATS still disconnected\"")
+
+	assert.GreaterOrEqual(t, warnCount, 1, "expect warn logs for first 30 attempts")
+	assert.GreaterOrEqual(t, errCount, 1, "expect at least one escalated error log past the threshold")
+	assert.LessOrEqual(t, errCount, 2, "rate-limited to once per minute, so a sub-second test should see at most one or two")
+	assert.Contains(t, logs, "disconnected_for=", "escalated error should include disconnected_for")
+}
+
+// TestConnectNATSWithRetry_NoEscalation_BelowThreshold keeps the attempt count
+// under natsRetryEscalateAttempt and checks that no escalated slog.Error line
+// is produced.
+func TestConnectNATSWithRetry_NoEscalation_BelowThreshold(t *testing.T) {
+	var buf bytes.Buffer
+	prev := slog.Default()
+	t.Cleanup(func() { slog.SetDefault(prev) })
+	slog.SetDefault(slog.New(slog.NewTextHandler(&buf, &slog.HandlerOptions{Level: slog.LevelDebug})))
+
+	// Exponential backoff capped at 5ms × maxWait 50ms = ≲15 attempts < 30.
+	_, err := ConnectNATSWithRetry("nats://127.0.0.1:1", "", "",
+		WithRetryDelay(5*time.Millisecond),
+		WithMaxRetryDelay(5*time.Millisecond),
+		WithMaxWait(50*time.Millisecond),
+	)
+	require.Error(t, err)
+
+	logs := buf.String()
+	assert.NotContains(t, logs, "NATS still disconnected", "should not escalate before threshold")
+	assert.Contains(t, logs, "NATS not ready, retrying...", "should still log warn lines")
+}