* fix(iam): persist policy tags and count attachments across all principal kinds
CreatePolicy built its record with an empty tag list and never read input.Tags, and GetPolicy, ListPolicies and CreatePolicy's own response projected no Tags field at all. Both halves are fixed, matching the role path.
buildAttachmentCounts scanned only the users bucket, so a policy attached to a role or group counted as unattached. That is not just a misleading number: DeletePolicy gates its conflict check on the same count, so an attached policy could be deleted, leaving the principal holding a dangling ARN.
The count and the reverse lookup are the same traversal with the names kept or discarded, so both now derive from one buildPolicyAttachments over users, roles and groups. Nothing is stored, so no counter can drift from the attachments it counts.
ListEntitiesForPolicy is implemented on that traversal, honouring EntityFilter and PathPrefix.
Closes: mulga-fvhlc
Closes: mulga-9i6uh
Closes: mulga-ryu30
Closes: mulga-8ranb
* test(iam): account for ListEntitiesForPolicy in the cross-tenant and policy-ARN gates
The action resolves PolicyArn through the same branch as GetPolicy, so it
belongs with that helper's siblings; it was in neither gate, which is what
the accounting test exists to catch.
* fix(ecr): store and project repository encryption, scanning and tags
CreateRepository parsed none of encryptionConfiguration, imageScanningConfiguration or tags, so none were stored and none came back. Both blocks are now parsed, persisted on RepoMeta and projected, defaulting to AES256 and scanOnPush false as AWS does.
encryptionType KMS is rejected rather than echoed. Predastore seals every fragment with AES-256-GCM under a required master key, so reporting AES256 states something true, but no customer-managed key material is plumbed anywhere and accepting a KMS key ARN would assert a protection that does not exist.
Repository tags are stored on the same record, and TagResource and UntagResource now work instead of returning NotImplemented.
PutImageScanningConfiguration gets its own handler: asking for scanning off succeeds as a no-op, which is the already-true state, while asking for it on still returns OperationNotSupportedException because no scanner exists. Reporting scanning as enabled when nothing scans would read as a false all-clear.
The AWS Repository shape was built in three places with no shared helper, so every new field would have been added three times. They now share one buildRepository projection.
Closes: mulga-ywvtu
Closes: mulga-j2y54
Closes: mulga-3whvb
* fix: accept and store repository tags on ECR CreateRepository
The AWS API takes a tags parameter on CreateRepository; the gateway accepted
the request and discarded them, leaving the repository untagged with no error.
Decode them into the RepoMeta tag map at create time, rejecting an empty key
the same way TagResource does.
DescribeTaskDefinition now echoes portMappings[].name and runtimePlatform,
DescribeServices projects the stored networkConfiguration and
enableECSManagedTags, DescribeClusters returns the four instance, task and
service counts, and the five list operations return their ARN key as a
present empty list instead of omitting it.
UpdateService now honours networkConfiguration and enableECSManagedTags so a
changed value converges instead of producing a permanent diff.
The eleven container runtime fields the task definition also drops are
deliberately left absent: nothing between the daemon and the on-host agent
carries them, and reporting a container as privileged or read-only-root when
nothing enforces it is worse than reporting nothing.
Closes: mulga-0t9q2
Closes: mulga-7thn7
Closes: mulga-d8kif
Closes: mulga-5z8z9
Three defects at the gateway's HTTP error boundary, none of which change an authorisation outcome.
xmlErrorBody defaulted to the EC2 query envelope, so any scope it did not enumerate, including every service the gateway does not serve, received a shape a REST-XML client cannot deserialize and the real code never surfaced. The generic ErrorResponse is now the default and the query-protocol services are named explicitly. An unparsed signature keeps the EC2 shape, since no service is known there and that path is not what this changes.
The supportedServices gate logged the truthful reason and then emitted the canned credentials text, because writeSigV4Error had no parameter to carry a message. It takes one now, and the gate names the service it refused. The signature is never verified at that point, so blaming the caller's credentials was actively misleading.
ctxAction was set only for the query protocol, so every JSON-1.1 request shared the throttle bucket "<account>:unknown" and no per-action limit could ever match. It now resolves from X-Amz-Target, and for the path-routed rest-json services from the route's own lookup function.
Also sets X-Amzn-Errortype on the four JSON emitters, which JSON-protocol SDKs read to resolve the modelled exception type.
Recovery tested whether the QEMU process was still running before it looked at the instance's state, so a live process short-circuited the transitional-state handling and reconnectInstance then set the instance back to running. A stop or terminate the caller had already been told was under way was silently discarded.
Check for StateStopping and StateShuttingDown first, kill a QEMU that outlived the transition, and finalize. reconnectInstance now refuses a transitional instance outright rather than promoting it.
Whether br-mgmt is a per-host island or one L2 segment spanning the cluster
decides whether a system instance can reach the gateway address it was baked
with. No artifact recorded which one a run built, and the environment is
destroyed before it can be asked, so the decisive evidence for an LB agent
heartbeat failure was gone by the time anyone looked.
Records ports, address and fail_mode per node, then probes every node's
br-mgmt address from every node. Islands reach only themselves and spanned
segments reach everything; a mixed result is the failure this catches.
* fix: carry a handler's own refusal message to the client without leaking wrapper context
A daemon that explains a refusal had that explanation discarded one step short of the client, so every NATS-backed error surfaced as the generic ErrorLookup blurb for its code. The message crossed NATS intact; resolveErrorDetail then dropped it because decodedNATSError was not the shape it recognises.
Rebuild the decoded response through awserrors.Errorf so the walk finds the message, and send only ResolveErrorDetail's curated message from the two encode sites instead of err.Error(). The latter rendered every %w layer, including context a call site added purely for its own logs, so restoring the message without also fixing the encode side would have turned a fidelity bug into an information disclosure.
Two existing tests pinned that wrapper context onto the wire, directly contradicting the gateway's own no-leak assertion; both are rewritten to pin the sanitized contract instead.
Also name the rejected filter in the twenty-four ParseFilters sites that logged the detail and returned a bare code, and give the ACM force-renew refusals a registered code so their reason still reaches the caller.
* fix: stop the filter rejection repeating its own error code to the client
ParseFilters spelled the code into its message text, so once the code began travelling separately the client saw it twice: "InvalidParameterValue: InvalidParameterValue: The filter 'x' is invalid".
Give ParseFilters the code directly via awserrors.Errorf and have its twenty-five callers return that error unwrapped rather than re-wrapping it. All of them already mapped it to InvalidParameterValue, so nothing changes but the duplication.
* fix: surface why a security group rule was rejected
ipPermissionsToSGRules already names the reason, "IPv6 rules are not supported" or "IpPermission must specify at least one IpRange or UserIdGroupPair", and both Authorize call sites logged it and returned a bare InvalidParameterValue, so the caller saw only the canned blurb.
This is the case mulga-e9qqx names specifically. It does not make IPv6 rules work; it makes the refusal legible.
CreateCluster and CreateNodegroup accepted fields that DescribeCluster and DescribeNodegroup never returned. The unmodified terraform-aws-modules/eks/aws module reads those fields back, finds them empty, and either aborts the apply or proposes a destroy of a cluster that is already correct.
Two of the omissions aborted the apply outright. A module precondition on data.aws_eks_cluster failed with cluster_service_cidr is "" because kubernetesNetworkConfig was absent, and the tag resource address built from clusterSecurityGroupId came out as (,suite) because that field was empty.
Store what the caller asked for on ClusterMeta and NodegroupRecord at create time, and project it back. Both records are encoding/json with omitempty, so records already in KV stay readable and project exactly as they did before.
The read-backs echo the request rather than synthesising an AWS default, because UpdateClusterConfig is not implemented — a value the caller did not ask for would be a diff they could never converge. Where a field does report a default it is the value genuinely in force: 10.43.0.0/16 is k3s's own service CIDR, API is the only authentication mode implemented, and logging reports every type disabled because no control-plane log export exists.
resourcesVpcConfig.securityGroupIds previously overwrote the caller's own groups with the platform's pair, which live in a VPC the caller cannot see. It now echoes the caller. The customer-facing cluster security group is created in the caller's account and VPC at cluster-create time, so the id reported by clusterSecurityGroupId resolves for the tenant that owns it. That creation is idempotent with the one launchNodegroupInfra already makes, and matches AWS, which creates the group with the cluster.
updateNodegroupConfig accepted UpdateConfig and Taints, ignored both, and returned SUCCESSFUL. It now applies them under the existing compare-and-swap, so a plan converges instead of asking forever. Caller-supplied labels and taints still never reach kubelet; that gap predates this change and is tracked separately.
Verified on env19: the module applies clean and a re-plan reports no changes.
Six defects in the EC2 describe path, found by running unmodified Terraform Registry modules against a live cluster and planning immediately after a clean apply. They share one shape: the write succeeds, the read-back is incomplete, and the caller cannot tell.
Populate Association on the primary network interface. The public IP was projected onto the instance but never onto the NIC, so the AWS provider read associate_public_ip_address as false for an instance that had a working public address. That attribute is force-new, so a no-op plan proposed destroying and recreating the instance.
Stamp EnclaveOptions. AWS always returns it, and its absence rendered as a permanent enclave_options block in every plan.
Stop emitting empty Reservations. The reservation map entry was created before the filter predicate ran, so a filter that excluded every instance in a reservation still emitted the reservation with no instances. Projection and filtering now run before the insert, in both the live and the KV path.
Merge reservations by ID in the gateway. Response frames were concatenated rather than merged, so one reservation could appear twice in a single response and which copy landed first varied between calls.
Mint the reservation ID once per RunInstances call. Each node minted its own, so a launch spread across nodes came back as several reservations. The gateway now allocates the ID and passes it to every node in a NATS header; a node that does not read the header mints its own as before, so the change is additive in both directions.
Answer an incomplete DescribeInstances sweep with a retryable ServiceUnavailable. Suppressing a false InvalidInstanceID.NotFound when a node had not answered was correct, but no branch was ever written for that case, so the caller received an empty 200 that read as authoritative. A filters-only query with no matches is still a legitimate empty result.
Accept the AWS spellings of the DescribeVpcs filters. The filters were implemented but named cidr-block and is-default where AWS uses cidr and isDefault, so Terraform's aws_vpc data source was rejected outright. Both spellings now work, since an internal caller depends on the old one. The parse error also names the offending filter to the client instead of only the log.
Resolve image owner aliases before filtering on them. describe-images --owners self compared an alias against a numeric account ID and matched nothing; the correct resolution already existed but ran after the filter.
Also builds the segscan test oracle with -buildvcs=false. That is unrelated to the above and fails on current toolchains regardless of this change, but it blocks preflight.
Host reservations, ECR image scanning and the flow-logs integration template
were declared not applicable, and the reserved instance family with them. None
of those are ruled out: reservations and scheduled instances can be mocked over
capacity reservations the way spot instances already are, scanning needs only a
scanner behind the API, and the template needs CloudFormation.
What stays is what the platform cannot have: retired AWS features, operations
whose content is an AWS price or counterparty, and AWS-hosted services with
nothing behind them here.
The roadmap carries IPv4 pools, local zones as an installation option, and an
S3 gap closure naming inventory and intelligent tiering. All three were
published as operations this platform will never serve.
Twenty-nine EC2 operations — local gateways, CoIP pools, BYOIP and the zone
group opt-in — and nine S3 operations are no longer declared. Carrier gateways
stay, as a route into a telco's 5G network in a Wavelength Zone.
The internal report listed every unimplemented operation together, so the 87
EC2 operations declared not applicable sat in the same list as the work, where
anything reading it for candidates would pick them up.
They are now listed apart, grouped under the reason they were declared with,
and the counts line reports them as not applicable rather than as gaps.
EC2 published one not-applicable operation against 625 in the model, so the
page was silent about a large surface that is absent for a structural reason
rather than an unfinished one.
Eighty-six more are declared across nine families: EC2-Classic, the 2016
resource ID migration, FPGA images, Elastic Graphics, Outposts and the edge
zone gateways, capacity purchasing, AWS backbone telemetry, Marketplace
product codes and BYOIP. S3 loses its replication and analytics declarations,
which named work that is not ruled out.
Multi-AZ clusters, read replicas and point-in-time restore were declared not
applicable, which says the platform will never serve them. All three are
planned, so the pages claimed something the roadmap contradicts.
Only a declaration now publishes that claim. A handler that refuses an
operation today establishes nothing about tomorrow, so it reads as the gap it
is rather than a permanent one.
Splitting them into a second table repeated a reason on every row that shared
it and separated an operation from the rest of the surface it belongs to. One
table carries every operation the platform answers, served or refused, and the
reasons return to numbered footnotes below it.
S3 was the one served surface with no coverage page. The gateway has no
operation-name dispatch table for it, so there was nothing to compare against
the model.
Predastore now declares the operations each of its S3 routes answers, so the
generator compares that inventory the same way it compares the other nine
services. The page credits Predastore rather than Spinifex, and its intro
separates what is routed from what conforms, pointing at the ceph/s3-tests
report for the second.
The index listed a count per service and left the reader to add them up. It
now opens with the total and closes the table with it, counting only the
services it links.
The coverage pages led with a percentage and an "x of y" count, and every
table listed the modelled operations Spinifex does not implement, which reads
as a scorecard against AWS rather than a statement of what the platform serves.
A published page now opens with the operation count alone, tables the
operations the gateway serves, and carries a separate not-applicable table with
each reason inline instead of footnotes. The unimplemented operations and the
counts move to docs/coverage/internal-report.md, which no docs-site slug points
at. A service with no implemented operations has no publishable page.
Every node's startup pass, the account.created event and the gateway's
EnsureDefaultVpc request can each build an account's defaults on a different
node. Each checked for an existing record, then wrote a fresh random ID with
Put, so racing nodes built a VPC, security group, route table and internet
gateway each. A per-account claim, written with Create, now fixes the IDs
before anything is built; every record is created under those IDs with
Create, and only the creator announces it to vpcd. nextVNI retries its CAS
instead of failing the loser.
spx admin init now discards the node's pre-formation JetStream store by default for --nodes >= 2; --discard-jetstream=false restores the refusal. spx admin kv digest gains --json with stream replica, retention and limit settings and per-sequence subject, time and TTL. The new spx admin kv compare reads every node's digest and fails only on real divergence: different content at one sequence, a missing message nothing explains, or wrong replica placement. install-node.sh gates formation on it and gains --ipsec on|off, verified on every node. The multi-node guide documents both.
A weekend gap is two runs of evidence lost on a suite whose remaining
failures are a rate rather than a bug. GitHub queues scheduled runs behind
every other top-of-hour cron and sheds them under load, which is why
2026-09-14 never fired and why every run started 3h15m-3h45m late.
* fix: reboot a guest by asking it, not by pulling its reset line
RebootInstances was QMP system_reset alone. The guest kernel is never told,
so it does not sync, does not unmount and loses every dirty page - which is
how a reboot issued seconds after a write loses that write, and how a guest
rebooted during cloud-init can fail to come back at all.
Guests now launch with -action shutdown=pause, so a clean guest poweroff
leaves QEMU paused rather than gone. Reboot presses the power button, waits
for the guest to reach that state, then resets and resumes the same process:
no volume, tap or state churn, and no window where the guest is missing. A
guest that will not go quietly is hard-reset, which is what AWS documents.
The button is pressed again every 10s while the guest ignores it. One signal
is not a shutdown request on a guest whose acpid has not started, which is
the same boot race that produced the failure.
Two consequences handled here: a guest that powers itself off now leaves
QEMU paused, so the heartbeat quits it; and a guest launched before the flag
exits on powerdown rather than pausing, so it is reset without being asked.
* fix: answer a reboot when it is accepted, not when the guest is back
Asking a guest to shut down takes as long as the guest takes. RebootInstances
gives the daemon five seconds to reply, so a reboot that waited for the guest
reported a reboot that had in fact succeeded as InvalidInstanceID.NotFound.
Manager.Reboot now admits the reboot — the instance exists and is running — and
runs the powerdown and reset in the background, which is the contract EC2
documents. The two admission failures are still reported; nothing after them
can be, so a reset that fails is logged and left to the heartbeat.
The gateway also stops calling a NATS timeout NotFound. Only the absence of a
subscriber says anything about where the instance is, which is how the sibling
start and terminate paths already read it.
* test: prove the churn reboot by its boot ID, not by how fast it happened
The subtest waited ten seconds and then read /proc/uptime. A reboot is
asynchronous, so the guest answering at that point may be the one that has
not shut down yet, and its uptime is whatever it was before — which the test
reads as "reboot may not have occurred".
Waiting for the boot ID to change is positive proof the guest rebooted and
does not depend on how long the guest takes to go down. The uptime assertion
stays, now measured against a boot that is known to be new.
* test: prove the graceful reboot against real guests and a real engine
Two tests the reboot design asked for and the unit tests cannot give.
Cell 18 launches two guests that differ only in how they answer the power
button. One installs a shutdown hook writing an fsynced line a second, so the
file that survives measures how much of its shutdown it was allowed to run —
a guest reset instead of asked leaves none of it. The other ignores the power
key entirely and must be reset once its budget expires, with the daemon saying
so. Each variant asserts the evidence only it can produce, and the slow one
asserts the absence of the line the wedged one requires.
Cell 27 loads a PostgreSQL instance past its shared buffers with pgbench and
reboots it dirty. The probe is an UNLOGGED table: PostgreSQL writes no WAL for
one, so a clean shutdown checkpoints it and crash recovery truncates it
unconditionally. Rows committed normally survive a hard reset too, so the
obvious assertion would pass against the bug.
The rds suite and cell 27 both gain budget for the load phase.
* fix: stop the process-group test misreading a reaped child as alive
The test polled until the child was gone, then checked once more. A task
being reaped shows state X for an instant, which processRunning counted as
running, so that second look failed about 1 run in 50 under -race.
* fix: wait for the recovery retry loop before removing a test's data dir
restoreInstances starts the retry loop untracked, and it rewrites the local
state file when it ends. A write landing during t.TempDir cleanup failed
TestRestoreInstances_* with RemoveAll: directory not empty.
* fix: read NB_Global.ipsec through the NB remote list so followers and compute nodes answer quietly
* fix: stop probing and nudging the EIP datapath of a stopped instance's port
* fix: log the real client IP, request ID and duration_ms in the awsgw access line
* test: mark the UI proxy header test as in-package
* fix: discard pre-formation JetStream data when forming a multi-node cluster
A node that ran as its own single-node cluster carries KV streams into
formation. nats-server adopts a local stream with the same name as the
clustered one, and raft catch-up only fetches sequences beyond a
follower's last, so the conflicting prefix is never repaired and replicas
diverge permanently.
- spx admin join removes the local JetStream store once formation
succeeds. A store holding streams now needs --force, and the
identity-discard warning names it.
- spx admin init --nodes >= 2 refuses a store that holds streams unless
--discard-jetstream is given, and removes it once every node has joined.
Keeping it as the seed is unsafe: stream leadership ignores who holds
data, and unassigned local streams are deleted as orphans.
- Neither removes a store while `spx service nats` runs, --force included.
- install-node.sh passes --discard-jetstream and waits for every
`spx service` process to exit before formation.
* feat: add spx admin kv digest for per-replica JetStream consistency checks
Replicas of a stream can hold different content while raft reports them
current, and nothing in the cluster detects it. A direct get over NATS is
no way to inspect one replica: a client's request goes to a random member
of the direct-get queue group, routed replicas included.
kv digest copies this node's $G streams tree (without consumer state) to a
temporary directory, opens the copy in a private in-process server with no
listener, and prints first/last sequence, message count and a SHA-256 over
every stored message per stream. Running it on each node and diffing the
output shows which streams diverge; --seqs locates the first differing
sequence. The live store is only ever read.
* fix: let spx admin kv digest read replicated streams
A replicated stream's meta.inf carries num_replicas > 1, which a standalone
server refuses to recover, so the digest could not read any production store.
Rewrite the copy's config to one replica with no expiry and a recomputed
checksum, and test it against a three-node cluster and an expired bucket.
* test: use assert.ErrorIs for the removed JetStream store
* test: mark the JetStream store and digest tests as in-package
* fix: stop the process-group test misreading a reaped child as alive
The test polled until the child was gone, then checked once more. A task
being reaped shows state X for an instant, which processRunning counted as
running, so that second look failed about 1 run in 50 under -race.
* fix: wait for the recovery retry loop before removing a test's data dir
restoreInstances starts the retry loop untracked, and it rewrites the local
state file when it ends. A write landing during t.TempDir cleanup failed
TestRestoreInstances_* with RemoveAll: directory not empty.
* fix: stop the process-group test misreading a reaped child as alive
The test polled until the child was gone, then checked once more. A task
being reaped shows state X for an instant, which processRunning counted as
running, so that second look failed about 1 run in 50 under -race.
* fix: wait for the recovery retry loop before removing a test's data dir
restoreInstances starts the retry loop untracked, and it rewrites the local
state file when it ends. A write landing during t.TempDir cleanup failed
TestRestoreInstances_* with RemoveAll: directory not empty.
* fix: retry a recovery relaunch that lost a race with a starting dependency
A guest drain-stopped by a coordinated cluster shutdown could be lost for good.
The relaunch ran before JetStream was serving, the volume lease claim timed out,
and the failure was recorded as permanent.
The backoff this needed already existed. relaunchWithRetry retries five times,
but only on ErrMountRetryable, and mountErrRetryable recognised exactly two
viperblock state-load sentinels. A lease claim returning context deadline
exceeded matched neither, so the mount was reported as permanently failed and
the backoff never engaged once.
- Classify a lease store that cannot answer as ErrLeaseStoreUnavailable, and
include it in mountErrRetryable. A lease held by another node is deliberately
excluded: that is an answer, not an outage.
- Probe readiness rather than connectedness. checkViperblockReady tested
natsConn.IsConnected(), which is true against a server whose JetStream is
still starting; it now calls AccountInfo with a timeout.
- Reconsider a latched failure. RetryRecoveryFailed relaunches instances parked
in error by a dependency-unavailable reason, once the backing store is
confirmed ready, for a bounded number of passes, then leaves them alone.
Only mount-state and launch failures qualify; a failed reconnect or
pre-relaunch hook can mean the guest itself needs looking at.
Closes the e2e gap that made this feel covered. The node-failure tests stop the
service units directly so guests keep running and recovery is a QMP re-attach to
a live process. Nothing stopped a guest, so nothing relaunched one from a sealed
volume against dependencies that were still starting.
TestMultinodeClusterRestartResumesGuests does, including the wait for every spx
process to exit that makes it a real restart rather than a stop that returned
early.
Beads mulga-v7kp4, mulga-kyb1d.
* fix: stop every spx process by unit in the cluster-restart helper
The first CI run of TestMultinodeClusterRestartResumesGuests failed at the
process wait with "node1 still has 2 spx process(es)", and the same thing
happened by hand on a prod node.
Every service is the same binary, and the stop was driven from a hard-coded
list. StopNode names seven units and omits spinifex-northstar and
spinifex-qmp-collector, which is exactly the two survivors. The unit name now
comes from the surviving process's cgroup, so the set that stops is whoever is
still running the binary and the list cannot go stale when a service is added.
StopNode itself is left alone: its list is deliberate for the hard outage it
simulates, where the target's drain must not fire.
Also marks recovery_retry_test.go as //test:in-package — the retry policy is
unexported and the tests drive it through the seams restore_test.go uses.
* fix: stop spinifex.target in the cluster-restart helper, not the units
The second CI run reformed NATS to 2 peers of 3: node3's NATS was stopped and
never started again.
StopNode stops the service units directly and leaves spinifex.target active.
That is right for the hard outage it simulates and wrong for a real shutdown —
the target then reads as satisfied while its units are dead, and the later start
raced a stop job that was still queued.
The helper now runs what the operator procedure and update-nodes.sh run: stop
spinifex.target, wait for every spx process to exit, and additionally wait for
systemd to have no spinifex job left queued before starting. That last wait is
the one that closes this race.
* fix: wait for the right NATS peer count in the cluster-restart helper
WaitNATSPeers counts remote peers, so a healthy three-node cluster reports two.
Asking for three could never be satisfied, and the run that hit it had in fact
restarted all three NATS correctly.
The QEMU guard never fired. pgrep -x and pkill -x match comm, which the kernel
truncates to 15 characters, so qemu-system-x86_64 is stored as qemu-system-x86
and neither the -x name nor the -x alternation could ever match. The script
therefore never signalled a guest and never waited, and wiped viperblock state
out from under live guests. Match the 11-character prefix against comm, the same
rule qemuProcessPrefix already applies in vm/orphan_scan.go, and scope it to the
user the daemon runs guests as so a host carrying libvirt guests alongside
Spinifex ones does not lose the wrong set. Not -f: that matches the whole
command line, where argv[0] may carry a path.
Nothing waited for spx either. Stopping spinifex.target returns once the target
is inactive, but spinifex-shutdown.service is ordered After= the storage
services, so their stop jobs queue behind its ExecStop and NATS can still be
running. JetStream then rewrites its index after the rm and before the
verification, failing the reset on a file the script had already deleted. Add
the wait update-nodes.sh has carried since d9bc6e35.
The firewall restore was gated on a mode file the failed run had already
deleted, so a retry read the absence as "never armed", skipped the restore, and
install-node.sh honoured that and formed the cluster around an open node. Stash
the mode outside every wiped tree, and fail rather than warn when a node that
arrived armed cannot be put back.
install-node.sh had the other half of that cascade: it disarms every host to let
the OVN raft ports through and only re-arms at the end, so any abort in between
left them all open. Re-arm from an EXIT trap, and say so loudly when a host
arrives with no policy.
The wipe loop deletes files but keeps every directory, so one empty
$G/streams/KV_<name> directory per stream survived the reset. NATS enumerates
those on start and reports them as HA assets it must restore, leaving the meta
group at size=4 with no leader over hundreds of assets that have no data behind
them. Every service then fails to reach JetStream and the daemon aborts its
bootstrap.
Remove the tree outright, mirroring the predastore/cluster removal above it.
Josh Siviour