Filling the filesystem behind the JetStream store to zero free bytes leaves
nats-server with nowhere to write while it recovers a stream. Every error
return inside the locked region of newFileStoreWithCreated assigns nil to the
named return value fs, and the deferred unlock then dereferences it, so a
recovery write failure surfaces as a nil-pointer panic instead of an error.
That structure is unchanged in v2.14.3, in the pinned v2.14.5 and in the
current v2.14.6, so no available version bump addresses it.
The nats service now keeps a preallocated reserve file beside the jetstream
directory and hands it back at startup when the filesystem can no longer
supply that headroom on its own, re-arming once space returns. Recovery
therefore always has somewhere to write, whichever process filled the disk.
* feat: add the e2e guest disk-performance gate
Nothing measured guest disk performance in CI, which is why a quadratic
index scan shipped in viperblock 35dd9f2 and survived three subsequent
fixes to adjacent code. It is also why the two availability clauses at
the end of the ebs-performance plan's Stage 0 have never been evaluated:
a sustained random-write workload took the guest, and then the host,
down, and nothing since has demonstrated that it no longer does.
tests/e2e/diskperf/ launches an instance, attaches a dedicated blank
volume per repetition, and drives a pinned fio profile over guest SSH.
Two profiles: randwrite-4k at 4 jobs x 4 GiB, the shape the exit
criterion names, and randrw-70-30-4k, the mixed workload the headline
IOPS came from. The target is a raw device with direct=1 rather than a
file on the boot disk, so guest page cache, filesystem allocation and
journal traffic are out of the path.
Two classes of assertion, kept apart because folding them together lets
a run that left the host unreachable pass on a respectable IOPS figure.
Availability is hard pass/fail and absolute, so it gates from the first
run: the job completes inside its budget, a guest responsiveness probe
answers throughout on a second SSH channel, and the host's own sshd
answers throughout. The guest probe reads the root volume while the
workload runs on a different one, so a pass is evidence of isolation
between volumes rather than of the job merely being slow. The host probe
is a TCP connect plus an SSH banner read, which needs no credential and
still proves the kernel scheduled sshd.
Throughput and latency are banded against a committed baseline: fail
above 25% regression, warn above 10%, per-metric median across
repetitions. baseline.json is committed and embedded, with every figure
zeroed, and a zero reference records and warns rather than failing. Only
the availability class gates until numbers are captured on the bare-metal
cell; a throughput band nobody measured would be a fabricated gate.
A fresh volume per repetition is the reset. Extent count grows
monotonically, so reusing a volume makes the gate order-dependent and
drifts its own numbers upward over time, slowly normalising the
regression it exists to catch. Each job is its own guest invocation and
fio's JSON is fetched by a separate command, so a wedged job loses only
itself and reports as a budget breach rather than a parse failure.
e2e-baremetal.yml gains suites, suite-env and cell inputs so one reusable
workflow serves both the correctness cell and nightly cell 28, which
queues behind cell 20 on the shared concurrency group. Cells 25-27 are
the dedicated eks, ecs and rds cells. The per-suite Go test timeout is
resolved from e2e_suite_timeout in suite-sets.sh, where diskperf declares
its 180-minute budget beside rds's rather than in a second place.
The gate's own arithmetic runs in preflight via test-harness: band edges,
the unmeasured cases, per-stream metric selection, fio JSON parsing, and
the probe accounting including a slow-but-successful answer counting as a
breach. A gate whose comparison logic is untested is the failure mode the
plan is otherwise full of.
The extent-count ceiling named as a fourth availability assertion is not
implemented: it needs viperblock.index.entries from S1.5, which does not
exist.
* feat: let a manual bare-metal run choose its e2e suites
A dispatch run only sees inputs declared under workflow_dispatch, which had
none, so suites and suite-env resolved empty and fell back to the correctness
set. Capturing a diskperf baseline by hand was therefore unreachable without a
workflow_call caller.
* fix: build the bare-metal suites the run step will execute
The build step packs pre-built binaries into the tarball and did not see the
suite selection, so a selected suite was absent from the node unless it
happened to be one of the three the build script hardcoded.
* feat: commit the measured disk-performance baseline
Medians of three repetitions on the bare-metal cell. The throughput and
latency bands now gate instead of warning, so a 2x regression in either
job fails the suite.
randwrite-4k's write tail is two orders above the mixed job's. It is
recorded as measured rather than zeroed: an ungated metric would miss the
regression class the gate exists to catch.
* feat: stamp the owning node's AZ on the instance record
RunInstance never persists a node's AZ onto the stored instance, so
today only the launching node's own ProjectInstance call can produce
a correct AZ for a describe. Add VM.AZ, carry it on InstanceStatus
since it is placement observed by the running node, and stamp it in
WriteRunningSet alongside Status.LastNode from the node's own
config.AZ.
* docs: trim the AZ field comment to the project's three-line limit
* fix: identify Gather responders by node, not by frame count
utils.Gather previously counted reply frames toward completeness, so
two subscriptions replying from one host (a daemon mid-restart) could
satisfy a fan-out while a genuinely configured node stayed silent.
Every daemon reply now carries an X-Node-ID header set by the shared
response helpers via nats.Msg.RespondMsg, so Gather can attribute each
frame to the node that sent it. Gather gains a CollectionMode
(CollectServeData preserves today's early-exit behavior;
CollectUntilDeadline collects to the full deadline for absence-proof
callers) and returns node-identified Frames plus Responders/
SuccessResponders/ErrorResponders sets. Identity mode retains only the
first payload seen per node; a disagreeing later frame from the same
node flags a conflict but never overwrites the retained data.
Node identity uses one namespace across the reply header,
config.ClusterConfig.Nodes keys, daemon.Heartbeat.Node, and
vm.VM.LastNode, all traced back to Daemon.node - confirmed to already
agree, with a new test asserting it at runtime.
This is a mechanical caller-side follow-on: every existing
utils.Gather caller now reads frame.Data instead of frame directly.
* feat: thread configured node set to DescribeInstances
GatewayConfig gains NodeIDs, the keys of config.ClusterConfig.Nodes,
distinct from the existing ExpectedNodes count. awsgw builds it from
cluster config and wires it into GatewayConfig. The DescribeInstances
action passes gw.NodeIDs through instead of relying on
gw.DiscoverActiveNodes(ctx) at that call site, so completeness can be
judged against the nodes that are actually configured rather than the
nodes that happened to answer a discovery probe.
* fix: judge DescribeInstances completeness by node identity
DescribeInstancesChecked now judges completeness on decoded frames
against the configured node set (ValidResponders) rather than on
Gather's frame/byte counts, which cannot know the describe schema. A
node that answers with a nil Reservations list is a valid responder,
not an invalid one - the existing merge guard on Reservations != nil
governs how frames combine, not whether the node counted as having
answered.
The explicit-instance-ID 404 path now collects with
CollectUntilDeadline so a node that hasn't replied yet cannot be
mistaken for a node that has nothing to report; general listing stays
on CollectServeData. Any ambiguity - an unidentified frame, a node
appearing in both the success and error sets, a node replying twice
with disagreeing frames, a Gather cap hit, or an empty/unset node set
- suppresses the NotFound rather than risking a false one.
* fix: let a describe by instance id stop once every id is found
CollectUntilDeadline made every DescribeInstances with explicit ids wait the
full 3s fan-out deadline, measured at 4650ms against 1655ms before, on a
healthy cluster where all three nodes answer in about a millisecond.
Gather gains a Settled predicate, consulted only under CollectUntilDeadline,
that ends collection once the replies already answer the question. Absence
still pays the whole deadline, and an instance the caller cannot see is
filtered out of every frame so it takes the absence path too, which keeps the
two NotFound answers indistinguishable in time.
* refactor: resolve IMDS instance metadata local-first, drop the fan-out
IMDS only ever serves a guest on its own node, so a DescribeInstances
fan-out to the whole cluster is unnecessary latency and load. Resolve
from the node's own on-disk VM state first, falling back to a direct
record-space read for an instance that has moved off this node.
Neither path issues a NATS request.
* perf: cache the local instance state parse for IMDS lookups
IMDS request rate is guest-driven and unbounded, so re-reading and
re-parsing the whole on-disk VM state on every request does not
scale with VM count on a node. Cache the parsed state and only
re-read when the file's mtime or size changes; a stat or parse
failure clears the cache and falls back to the record space rather
than serving a stale hit.
* fix: share ManagedBy visibility rule between DescribeInstances and IMDS
Extracts IsInstanceVisibleToCaller so account ownership and the
platform-managed hide rule live in one place instead of two
independently maintained copies.
* refactor: decouple IMDS instance lookup from the daemon package
handlers/imds no longer imports daemon: NewIMDSServiceImpl now takes a
localStateReader and a recordLoader interface instead of a dataDir
string, and vpcd (the composition root for both planes) constructs
both adapters, including the stat-guarded local-state cache that used
to live inline in handlers/imds. Either adapter may be nil, degrading
that source to a miss rather than failing IMDS to start.
* test: move the local state reader test to an external package
* chore: exempt export_test.go from the external test package check
The check's own remedy text tells you to move unexported hooks into an
export_test.go, then flags that file for being in-package — which it is by
definition, since exporting internals is its whole purpose.
* fix: bound how many describes may hold a full fan-out deadline
A describe naming an instance the caller cannot see is filtered out of every
frame, so it never settles early and holds a gateway goroutine and inbox
subscription for the whole 3s deadline. Issuing those is free for the caller,
so nothing stopped a burst of them pinning unbounded gateway state.
Absence proofs now draw on a fixed budget. Past it, callers get a retryable
RequestLimitExceeded rather than being queued, since queueing would let the
backlog grow. The decision is taken before any reply is read, so a nonexistent
id and another account's id throttle identically and the throttle cannot become
the timing oracle the deadline exists to close.
* perf: settle describe-by-id on responder coverage, not the full deadline
Every describe-by-id was paying the whole 3s fan-out deadline whenever the
requested instance was not in a running-instance frame — which includes a
stopped instance, a terminated one, an id belonging to another account, and
an id that does not exist.
Gather gains ResponderGrace: once every ExpectedResponders node has
answered, collection closes after a short grace window instead of running to
the deadline. Core NATS gives a request only to the subscribers present when
it was published, so an overlapping subscription replies alongside the first
and one that arrives later never receives the request at all. Waiting past
full coverage cannot find a reply that full coverage missed. The window is
armed once and never extended, so a node replying repeatedly cannot hold the
fan-out open. The full deadline is now paid only when a node stays silent,
which is the case it was always meant to cover.
The stopped and terminated bucket queries now run alongside the fan-out
rather than after it, so their latency overlaps it instead of adding to it.
They feed a shared instanceSettler that ends collection as soon as every
requested id has been seen, skipping even the grace window.
The timing oracle is unchanged. An instance the caller cannot see is
filtered out of every source before the reply is sent, so it takes the same
path as an id that does not exist: both answer NotFound after full responder
coverage, and remain indistinguishable in time.
* fix: stop parallel tests racing the terminate backoff seam
terminateRetrySleep is a package global, and every test that needed it
neutered swapped it and restored the old value from t.Cleanup. Those tests
run in parallel, so two cleanups could write the seam at once — a real data
race the detector caught once enough parallel tests shared the binary.
Installed once in TestMain instead. Nothing in this package wants the real
backoff, so there is no value in restoring it per test.
The timeout was written out separately in each workflow and drifted: rds
ran 50m on the PR gate and 30m nightly, where it panicked and took six
in-flight subtests with it. e2e-suites.yml had the same hardcoded 30m on
its single path, which also runs rds.
A new e2e_suite_timeout helper sits beside the eligibility lists it
already guards, and all four call sites resolve through it. e2e.yml loops
over suites on the remote VM, which has no checkout, so it resolves the
pairs on the runner and looks them up in the loop.
Also lift the nightly cell ceiling to 50 minutes so the rds cell is not
cut short by the job before its own timeout applies.
checkSGDependencies knew which ENI or referencing security group blocked the
delete and discarded it, returning a bare DependencyViolation with no log line,
so the blocking id had to be recovered by cross-referencing the ENI table.
Return it via awserrors.Errorf and log it at the point of refusal.
shutdownAndUnmount logged the volume unmount error and returned nothing, so
StopAll returned nil and the DRAIN phase ACKed success with vms_stopped set.
The coordinator then advanced to STORAGE and stopped viperblock while the seal
was still in flight, tearing the VBState envelope and leaving the guest unable
to mount its root volume on the next boot. The deploy reported success.
shutdownAndUnmount now returns ErrVolumeSealFailed when the unmount did not
seal; QMP, PID-file, fw_cfg and telemetry cleanup stay best-effort. stopCleanup
and stopOne carry it without aborting the rest of the teardown, since QEMU is
already down by then, and StopAll aggregates the per-VM failures so
handleShutdownDrain ACKs with Error set and never reports vms_stopped.
volumeMounterAdapter.Unmount was the second swallow site: it logged each failed
seal and returned nil unconditionally. It now attempts every volume and returns
the aggregate, which is what the VolumeMounter contract already described.
Terminate still tolerates a failed seal — the volumes are deleted next, so an
unsealed block map loses nothing a caller can ask for.
This does not fix the seal itself. It converts silent data loss into a refused
upgrade that an operator can act on.
The policy gate had the cluster and nodegroup name but not the trailing
UUID, so it evaluated nodegroup/<cluster>/<name>/*. That value matches the
AWS-documented wildcard policy spelling but not one naming the exact stored
ARN, which is what a tenant gets by pasting DescribeNodegroup output, so
that Deny stayed inert.
The discriminator is now derived from (account, cluster, name) as an RFC
9562 version-8 UUID, so the gateway and the handler compute the same value
independently and the gate names the stored ARN exactly with no lookup.
Nodegroups created before this keep their random discriminator and their
existing wildcard-only behaviour.
Drives every vpc.* topology topic end to end against the mock OVN client and
asserts the resulting rows: router, subnet switch and router port, logical
switch port addresses and port-group membership, gateway router port, NAT
rules and egress policies. Invalid CIDR, IP and MAC events are pinned to a
failed reply envelope with no partial write.
Moves the elbv2 dispatch table to an external test package.
subscribers 53.5% -> 73.4%.
Adds a 35-row dispatch table covering every NATS-backed elbv2 gateway
wrapper. Each row subscribes a stub only to the expected subject, so a
mis-wired subject fails as no-responders, and asserts the returned value,
the forwarded request body and the account header. A second table covers
the 28 validation guards that had no test, each pinned to its AWS error
code.
Package coverage 49.5% -> 93.8%.
Tables the twelve subjects Serve subscribes through the malformed-body,
version-skew and provider-failure arms, and round-trips every verb through
NATSProvider against a served MemoryProvider so client encoding and server
decoding are asserted against each other.
Covers MemoryProvider.ListVolumes, ListSnapshots and CopySnapshot, all of
which were at zero, plus NATSProvider's nil-payload and subject-token guards.
natsserve 28.8% -> 91.3%, ebsprovider 67.5% -> 86.3%, nullprovider 0% -> 100%.
Drives the model through Update with real key messages: a full walkthrough
asserting the config it produces, every validation gate keeping the operator on
the screen with the cursor on the field at fault, and the rendered output for
facts that appear nowhere else (role layout, pool geometry, folded planes).
The cross-host ICMP probe fired once, with no convergence budget, the moment
sshd first answered on the source guest. The default SG self-ingress rule
matches on the northd-derived $<pg>_ip4 address set, which only reaches the
remote chassis after northd regenerates it from port-group membership and
ovn-controller installs the flows there, so a first-packet burst can be
swallowed by the priority-900 default drop with no log line anywhere.
Give the probe the same 45s/2s budget the single-node suite already uses, and
snapshot NB/SB address sets, port groups, ACLs, port bindings and br-int flows
from every node plus the source guest on the first failed burst rather than at
teardown, so the state is captured while the drop is still observable. Emit
attempt count and convergence time so a pass that needed most of the budget
still surfaces as a propagation delay instead of a silent green.
One behavioural suite runs against LiveClient over the in-memory NB server and
against mock.Client, so the semantics reconcile, policy, external, topology,
subscribers and vpcd rely on are pinned in both. Covers NAT router scoping and
duplicate deletion, the not-found sentinels, gateway chassis idempotence,
ReplaceACLs no-churn, port group membership and DHCP lookups.
The readiness loop terminates on wall clock, so a stubbed sleep alone still
costs two minutes; clusterReadyMaxWait becomes a var alongside clusterReadySleep
so the timeout branch is reachable. Adds the in-package marker to the EIP
association tests, which drive setupTestEIP and read svc.natsConn.
setupTestEIP built the EIP service with a nil VPCService, so every ENI
lookup AssociateAddress performs was structurally unreachable: lookupENI,
lookupENIByInstance and AssociatedPublicIPForInstance had no test at all
and any test reaching them would have panicked. Wire a real VPC service
over the same embedded NATS and cover association by ENI id and by
instance id, the private-IP override, both not-found codes, and the
add-nat/delete-nat events the NAT rule is built from.
sgRulesToPolicyRules was never carried a tenant security-group rule: the
golden scenarios build IntentState by hand and bypass the adapter, so a
transposed FromPort/ToPort or a dropped SourceSG would have shipped
silently. Load real SG records through KV and pin the field mapping.
ECS and ECR take the existing both-directions shape against their Actions
tables. RDS carries its scopes on the dispatch table itself, so the gate is the
account-level half instead: every served action is either scoped or carries a
written reason, and one in both sets or neither fails. HasScope and
ScopedActions are added to drive it.
The condition-key gate pins the door's key set and asserts the IAM write path
accepts exactly the operator/key pairs the evaluator enforces.
The seven sourcePolicyARN actions resolved to the PolicyArn the caller sent,
so the gate decided against an ARN whose account the caller chose. The scope
now takes the ARN's path and name and rebuilds it on the caller's account, as
sourceOIDCARN beside it already does, and widens to account-wide when the ARN
carries no policy component. Lands with the cross-tenant integration suite that
found it.
snapshotVolumeEngine's unmounted branch built its engine by hand instead of
going through openVolumeVB, so it was the last control-plane path that opened
an engine over a volume without holding the cluster-wide lease. A snapshot
request landing on a node that does not have the volume mounted would open a
second engine over storage another node was writing.
Two engines on one encrypted volume read the same SeqNumHighWater and issue the
same AES-GCM nonces, which discloses the XOR of the plaintexts and permits
recovery of the authentication subkey, so this is not only a consistency
question.
Route the branch through openVolumeVB and give the lease the engine's lifetime.
The tail both branches shared moves into createSnapshotWithVB, mirroring
copySnapshotOnVolume, which was converted earlier and is what this one was
measured against.
A refusal is reported as unavailable rather than internal: the holder gives the
volume up on detach, so the caller should retry instead of treating a live
mount elsewhere as permanent.
The gateway does not serve S3, so an S3-scoped credential is rejected on
scope by the SigV4 middleware. That rejection was written through
GenerateEC2ErrorResponse, giving the client the EC2 query envelope
<Response><Errors><Error>. SDKs look for a flat top-level <Error><Code>
on an S3 response, find nothing, and report an empty code and an empty
message, which reads as a credential fault rather than a wrong endpoint.
Add the flat S3 envelope and resolve the envelope from the service the
client actually signed with: GetService reports an unserved scope as
empty, which is exactly the case that has to render as S3.
The service-to-envelope decision was copied into four emitters and the
copies had already drifted - writeClusterUnavailable put ELBv2 on the
EC2 envelope while ErrorHandler and writeThrottleError put it on the IAM
one. Collapse them onto a single xmlErrorBody so a new service cannot be
wired into some emitters and missed by others.
s3 is deliberately not added to supportedServices: only the shape of the
rejection changes, not the rejection.
* fix: ECS RunTask and CreateLaunchTemplateVersion honour ClientToken
A retried RunTask placed the tasks again: it mints a fresh task ID per task,
reserves CPU/memory/GPU, hot-plugs an ENI in awsvpc mode and starts the
containers, so a resend with Count 5 left ten running tasks, double the reserved
capacity and five leaked ENIs. CreateLaunchTemplateVersion added a second
identical version and moved $Latest.
RunTask gets a per-action wrapper over idempotency.Do rather than a dispatch
decorator: it is the only ECS action that needs one. A partial placement
finalizes rather than aborts, since re-running would add tasks on top of the
ones already started.
CreateLaunchTemplateVersion reuses the EC2 dispatch decorator, so it is an
opt-in rather than new machinery.
CreateLaunchTemplate, ModifyLaunchTemplate and ECS CreateService accept a token
they do not store; each is name-keyed or a CAS and cannot duplicate, which is
now recorded at the handler.
* test: cover the RunTask client-token binding layer
Split the token layer out of RunTask's NATS binding as runTaskIdempotent, the
same seam runTask already has, so the no-token, store-failure and replay paths
are reachable without a live connection.
CreateVolume, CreateNetworkInterface, CreateNatGateway, CreateRouteTable,
CreateEgressOnlyInternetGateway, CopyImage and CreateCapacityReservation parsed
a ClientToken and never read it, so a retried request created a second resource.
Adds ec2IdempotentHandler at the EC2 dispatch table rather than seven
hand-written wrappers, keyed <action>.<account>.<token> in the bucket
RunInstances already uses. RunInstances keeps its unprefixed keys: re-keying
live records would make a retry re-launch instead of replay.
Promotes the shared orchestration to idempotency.Do, now that eight callers
share it, and wraps store failures in ErrUnavailable so callers map every
non-mismatch token failure onto one API error.
ServeNATSRequestCtx ends its consumer span in a defer that runs after the
reply is published, so a test returning from NATSRequest could read the
span recorder before the consumer span had ended and see no account
attribute. The subscriber callbacks now signal completion and both tests
wait on that signal instead of on the reply.
The cache was a union that only grew: each run restored everything, added
what it compiled and saved the result, so objects superseded by later
commits stayed for Go's full five-day window. Deleting every entry and
letting it rebuild took Build and Test from 3434 MB to 478 MB, restore
from 75s to 13s and save from 64s to 7s.
Dropping entries nothing has touched in four hours holds it near that.
Go bumps an entry's mtime when a run uses it, so the working set
survives; the step runs after the tests, so a pruned entry costs a later
run a partial recompile at worst and never the current one.
Restore is what makes CI fast; save is what got expensive. A same-era A/B
across the switch to this cache shows what restore buys: wall time went
6:50 to 3:52, Build Spinifex 63s to 10s, test-race 385s to 188s. Since
then the cache has grown to its steady state and the work has not moved,
so Build and Test now spends 75s restoring and 64s saving where it once
spent 13s and 8s.
A PR's save is read by nothing else. Splitting restore from save and
gating the save on dev and main keeps every branch reading a cache a dev
merge refreshes minutes later, and takes the upload off the 56% of runs
that are pull requests.
pass and keyPass logged every failure at WARN as "retrying next cycle" and
returned the callee's deadline, which for a failed callee is zero. Zero means
no deadline, so the next attempt was the resync — five minutes by default.
Two things were wrong with that. A fault that clears in seconds left the world
unconverged for minutes, when the loop already had the mechanism to ask for
sooner. And a terminal failure was announced as recoverable: "retrying next
cycle" is false for a 400, which will be a 400 every cycle, forever, at WARN.
Both now go through failed(), which classifies with awserrors.IsTerminal — the
predicate added in #923 for exactly this decision and until now called by
nothing. A terminal error is logged at ERROR and asks for no early retry; any
other error asks to be revisited after DefaultErrorRetry. A deadline the callee
named survives when it is sooner, and clampRevisit still caps every request at
the resync, so a loop can only become more eager than it is today.
It answered its question: of 18 GB on disk, 1.45 GB is untouched for a
day and nothing at all survives three. Go's own trim and the rate CI
runs already keep the cache live, so pruning before the save would shed
under 4% of the bytes at a two-day cut and risk recompiles for it.
The lifecycle handlers published record changes onto the dns.recordset.change
queue group, which load-balances rather than serialises, so any daemon could be
read-modify-writing a zone object at any moment. zonelock.go existed to make
that safe. With the reconcile pass now applying its own batch on the node that
won the election, the desired set is the only path into a zone and the elected
node is its only writer, so both the publish path and the lock go.
Deleted: the four lifecycle publish call sites and their helpers, publish.go,
zonelock.go, SubjectRecordsetChange and QueueGroup, and the lock from
applyZone. Every record they published is already in the desired set the
reconcile rebuilds each pass, and every record they withdrew is already pruned
by it, so the classes DNS carries are unchanged.
The build cache key rolls per run, so each job restores the previous
run's cache, adds to it and uploads the result. Nothing removes entries,
and Go only trims at 5 days, so every restore and save carries whatever
dead weight has accumulated. Build and Test moves 3.2 GB per run: 59s
down, 56s up.
Whether pruning that is worth doing depends on how much of the 3.2 GB is
still live, which is not visible from outside the runner. This reports
the age distribution and deletes nothing. It comes out once the number
is known.
The reconnect tests paid the client's one-second default sleep between
attempts, which ConnectNATS already lets a caller override; the retry test
scales its floor with the delay it sets, so it still proves a retry happened.
7.9s to 5.0s.
The reconcile elected a leader, computed a converging batch, then published it
to the queue group, which load-balances rather than serialises. The batch was
therefore applied by an arbitrary peer, not by the node that won the election.
That split the read that decides which records are stale from the write that
removes them. A lifecycle upsert landing between the two was invisible to the
second, so a record re-added after the reconcile read it was deleted by a
decision that predated it. Applying locally closes the window, and makes the
elected node the only writer the reconcile puts on a zone.
Deleted with their tests: the batch splitter, the payload limits and the
partial-acknowledgement check, none of which have anything left to bound now
that no batch crosses a NATS connection. The lifecycle publishers and the zone
lock are untouched: they are still a second writer, and retiring them is
separate.
Every test already owns its embedded JetStream server on an ephemeral port
under its own temp dir, so nothing here has to run serially. The recovery
failure test also left its doomed Set on a deadline-free context, which made
it wait out the jetstream client's own five-second default.
9.5s to 0.8s, and 11.8s to 2.0s under -race.
Bryn Mailer