fix: prod review of 14 September — IPsec follower reads, stopped-instance EIP probes, audit client IP and JetStream formation divergence (#1029)

* fix: read NB_Global.ipsec through the NB remote list so followers and compute nodes answer quietly

* fix: stop probing and nudging the EIP datapath of a stopped instance's port

* fix: log the real client IP, request ID and duration_ms in the awsgw access line

* test: mark the UI proxy header test as in-package

* fix: discard pre-formation JetStream data when forming a multi-node cluster

A node that ran as its own single-node cluster carries KV streams into
formation. nats-server adopts a local stream with the same name as the
clustered one, and raft catch-up only fetches sequences beyond a
follower's last, so the conflicting prefix is never repaired and replicas
diverge permanently.

- spx admin join removes the local JetStream store once formation
  succeeds. A store holding streams now needs --force, and the
  identity-discard warning names it.
- spx admin init --nodes >= 2 refuses a store that holds streams unless
  --discard-jetstream is given, and removes it once every node has joined.
  Keeping it as the seed is unsafe: stream leadership ignores who holds
  data, and unassigned local streams are deleted as orphans.
- Neither removes a store while `spx service nats` runs, --force included.
- install-node.sh passes --discard-jetstream and waits for every
  `spx service` process to exit before formation.

* feat: add spx admin kv digest for per-replica JetStream consistency checks

Replicas of a stream can hold different content while raft reports them
current, and nothing in the cluster detects it. A direct get over NATS is
no way to inspect one replica: a client's request goes to a random member
of the direct-get queue group, routed replicas included.

kv digest copies this node's $G streams tree (without consumer state) to a
temporary directory, opens the copy in a private in-process server with no
listener, and prints first/last sequence, message count and a SHA-256 over
every stored message per stream. Running it on each node and diffing the
output shows which streams diverge; --seqs locates the first differing
sequence. The live store is only ever read.

* fix: let spx admin kv digest read replicated streams

A replicated stream's meta.inf carries num_replicas > 1, which a standalone
server refuses to recover, so the digest could not read any production store.
Rewrite the copy's config to one replica with no expiry and a recomputed
checksum, and test it against a three-node cluster and an expired bucket.

* test: use assert.ErrorIs for the removed JetStream store

* test: mark the JetStream store and digest tests as in-package

* fix: stop the process-group test misreading a reaped child as alive

The test polled until the child was gone, then checked once more. A task
being reaped shows state X for an instant, which processRunning counted as
running, so that second look failed about 1 run in 50 under -race.

* fix: wait for the recovery retry loop before removing a test's data dir

restoreInstances starts the retry loop untracked, and it rewrites the local
state file when it ends. A write landing during t.TempDir cleanup failed
TestRestoreInstances_* with RemoveAll: directory not empty.

scripts/install-node.sh

@@ -368,6 +368,22 @@ for host in "${HOSTS[@]}"; do
     on "$host" "sudo systemctl stop spinifex.target" || fail "$host: could not stop spinifex.target"
 done
 
+# Init and join remove each node's JetStream store and refuse while NATS still
+# runs. A stopped target is not proof of that: its storage units can outlive it.
+# Matched on the service argv so the ssh shell running this cannot match itself.
+for host in "${HOSTS[@]}"; do
+    $DRY_RUN && { echo "  [dry-run] $host: wait for every 'spx service' process to exit"; continue; }
+    on "$host" "for u in spinifex-predastore spinifex-viperblock spinifex-nats; do
+            systemctl is-active --quiet \$u && sudo systemctl stop \$u
+        done
+        for _ in \$(seq 1 45); do
+            pgrep -f '^(/usr/local/bin/)?spx service ' >/dev/null || exit 0
+            sleep 2
+        done
+        pgrep -af '^(/usr/local/bin/)?spx service ' >&2
+        exit 1" || fail "$host: spx services still running 90s after stopping spinifex.target"
+done
+
 # --- Disarm the host firewall ----------------------------------------------
 #
 # The policy lives in the kernel and outlives the services, so stopping the
@@ -529,6 +545,7 @@ done
 # target.
 init_args=(
     --force
+    --discard-jetstream
     --node "${NODE_NAMES[0]}"
     --nodes "$N"
     --bind "${LAN_IPS[0]}"
@@ -577,8 +594,8 @@ else
 fi
 
 # --force: each joining node arrived from the ISO already initialized, with its
-# own CA and master key. Joining replaces them with the init node's, and
-# `spx admin join` refuses to do that silently.
+# own CA, master key and JetStream store. Joining replaces the keys with the init
+# node's and removes the store, and `spx admin join` refuses to do that silently.
 #
 # Every join must run concurrently. `spx admin join` registers and then blocks
 # until the formation server has all N nodes, so joining one at a time deadlocks