fix: prod review of 14 September — IPsec follower reads, stopped-instance EIP probes, audit client IP and JetStream formation divergence (#1029)

* fix: read NB_Global.ipsec through the NB remote list so followers and compute nodes answer quietly

* fix: stop probing and nudging the EIP datapath of a stopped instance's port

* fix: log the real client IP, request ID and duration_ms in the awsgw access line

* test: mark the UI proxy header test as in-package

* fix: discard pre-formation JetStream data when forming a multi-node cluster

A node that ran as its own single-node cluster carries KV streams into
formation. nats-server adopts a local stream with the same name as the
clustered one, and raft catch-up only fetches sequences beyond a
follower's last, so the conflicting prefix is never repaired and replicas
diverge permanently.

- spx admin join removes the local JetStream store once formation
  succeeds. A store holding streams now needs --force, and the
  identity-discard warning names it.
- spx admin init --nodes >= 2 refuses a store that holds streams unless
  --discard-jetstream is given, and removes it once every node has joined.
  Keeping it as the seed is unsafe: stream leadership ignores who holds
  data, and unassigned local streams are deleted as orphans.
- Neither removes a store while `spx service nats` runs, --force included.
- install-node.sh passes --discard-jetstream and waits for every
  `spx service` process to exit before formation.

* feat: add spx admin kv digest for per-replica JetStream consistency checks

Replicas of a stream can hold different content while raft reports them
current, and nothing in the cluster detects it. A direct get over NATS is
no way to inspect one replica: a client's request goes to a random member
of the direct-get queue group, routed replicas included.

kv digest copies this node's $G streams tree (without consumer state) to a
temporary directory, opens the copy in a private in-process server with no
listener, and prints first/last sequence, message count and a SHA-256 over
every stored message per stream. Running it on each node and diffing the
output shows which streams diverge; --seqs locates the first differing
sequence. The live store is only ever read.

* fix: let spx admin kv digest read replicated streams

A replicated stream's meta.inf carries num_replicas > 1, which a standalone
server refuses to recover, so the digest could not read any production store.
Rewrite the copy's config to one replica with no expiry and a recomputed
checksum, and test it against a three-node cluster and an expired bucket.

* test: use assert.ErrorIs for the removed JetStream store

* test: mark the JetStream store and digest tests as in-package

* fix: stop the process-group test misreading a reaped child as alive

The test polled until the child was gone, then checked once more. A task
being reaped shows state X for an instant, which processRunning counted as
running, so that second look failed about 1 run in 50 under -race.

* fix: wait for the recovery retry loop before removing a test's data dir

restoreInstances starts the retry loop untracked, and it rewrites the local
state file when it ends. A write landing during t.TempDir cleanup failed
TestRestoreInstances_* with RemoveAll: directory not empty.
This commit is contained in:
Ben Duncan
2026-09-14 15:39:03 -05:00
committed by GitHub
parent d7c2a202fc
commit 455c97a2e7
26 changed files with 1745 additions and 78 deletions
+4 -2
View File
@@ -235,7 +235,9 @@ sudo ovn-sbctl show
Run init and join **concurrently** — init blocks until all nodes have joined.
`--force` is in every command below so the sequence is identical whichever way you installed. It does the work on ISO-installed servers, which arrive as their own single-node cluster: joining replaces that server's CA and master key with server 1's, and `--force` is the confirmation. On `spx admin init` it is idempotent — existing keys, credentials and CA are preserved, and only the config files and server certificate are refreshed. On a freshly installed server there is nothing to lose either way.
`--force` is in every command below so the sequence is identical whichever way you installed. It does the work on ISO-installed servers, which arrive as their own single-node cluster: joining replaces that server's CA and master key with server 1's and removes its JetStream store, and `--force` is the confirmation. On `spx admin init` it is idempotent — existing keys, credentials and CA are preserved, and only the config files and server certificate are refreshed. On a freshly installed server there is nothing to lose either way.
`--discard-jetstream` on server 1 removes the streams its single-node cluster wrote, once every server has joined. Without it init refuses to form a cluster over them, because NATS would adopt each server's own copy as a replica and those copies never converge. Stop `spinifex.target` on every server first and confirm `pgrep -af 'spx service'` prints nothing: init and join both refuse while NATS is still running.
> [!WARNING]
> Do not point these commands at a server that has already been in service. Joining discards its master key, orphaning every volume and fragment sealed under it. That is what `--force` overrides, and it is unrecoverable.
@@ -243,7 +245,7 @@ Run init and join **concurrently** — init blocks until all nodes have joined.
**Server 1 — initialize:**
```bash
sudo spx admin init --force \
sudo spx admin init --force --discard-jetstream \
--node node1 --nodes 3 \
--bind $SPINIFEX_NODE1 --cluster-bind $SPINIFEX_NODE1 \
--port 4432 --region $AWS_REGION --az $AWS_AZ