* fix: stop the process-group test misreading a reaped child as alive
The test polled until the child was gone, then checked once more. A task
being reaped shows state X for an instant, which processRunning counted as
running, so that second look failed about 1 run in 50 under -race.
* fix: wait for the recovery retry loop before removing a test's data dir
restoreInstances starts the retry loop untracked, and it rewrites the local
state file when it ends. A write landing during t.TempDir cleanup failed
TestRestoreInstances_* with RemoveAll: directory not empty.
* fix: retry a recovery relaunch that lost a race with a starting dependency
A guest drain-stopped by a coordinated cluster shutdown could be lost for good.
The relaunch ran before JetStream was serving, the volume lease claim timed out,
and the failure was recorded as permanent.
The backoff this needed already existed. relaunchWithRetry retries five times,
but only on ErrMountRetryable, and mountErrRetryable recognised exactly two
viperblock state-load sentinels. A lease claim returning context deadline
exceeded matched neither, so the mount was reported as permanently failed and
the backoff never engaged once.
- Classify a lease store that cannot answer as ErrLeaseStoreUnavailable, and
include it in mountErrRetryable. A lease held by another node is deliberately
excluded: that is an answer, not an outage.
- Probe readiness rather than connectedness. checkViperblockReady tested
natsConn.IsConnected(), which is true against a server whose JetStream is
still starting; it now calls AccountInfo with a timeout.
- Reconsider a latched failure. RetryRecoveryFailed relaunches instances parked
in error by a dependency-unavailable reason, once the backing store is
confirmed ready, for a bounded number of passes, then leaves them alone.
Only mount-state and launch failures qualify; a failed reconnect or
pre-relaunch hook can mean the guest itself needs looking at.
Closes the e2e gap that made this feel covered. The node-failure tests stop the
service units directly so guests keep running and recovery is a QMP re-attach to
a live process. Nothing stopped a guest, so nothing relaunched one from a sealed
volume against dependencies that were still starting.
TestMultinodeClusterRestartResumesGuests does, including the wait for every spx
process to exit that makes it a real restart rather than a stop that returned
early.
Beads mulga-v7kp4, mulga-kyb1d.
* fix: stop every spx process by unit in the cluster-restart helper
The first CI run of TestMultinodeClusterRestartResumesGuests failed at the
process wait with "node1 still has 2 spx process(es)", and the same thing
happened by hand on a prod node.
Every service is the same binary, and the stop was driven from a hard-coded
list. StopNode names seven units and omits spinifex-northstar and
spinifex-qmp-collector, which is exactly the two survivors. The unit name now
comes from the surviving process's cgroup, so the set that stops is whoever is
still running the binary and the list cannot go stale when a service is added.
StopNode itself is left alone: its list is deliberate for the hard outage it
simulates, where the target's drain must not fire.
Also marks recovery_retry_test.go as //test:in-package — the retry policy is
unexported and the tests drive it through the seams restore_test.go uses.
* fix: stop spinifex.target in the cluster-restart helper, not the units
The second CI run reformed NATS to 2 peers of 3: node3's NATS was stopped and
never started again.
StopNode stops the service units directly and leaves spinifex.target active.
That is right for the hard outage it simulates and wrong for a real shutdown —
the target then reads as satisfied while its units are dead, and the later start
raced a stop job that was still queued.
The helper now runs what the operator procedure and update-nodes.sh run: stop
spinifex.target, wait for every spx process to exit, and additionally wait for
systemd to have no spinifex job left queued before starting. That last wait is
the one that closes this race.
* fix: wait for the right NATS peer count in the cluster-restart helper
WaitNATSPeers counts remote peers, so a healthy three-node cluster reports two.
Asking for three could never be satisfied, and the run that hit it had in fact
restarted all three NATS correctly.
The QEMU guard never fired. pgrep -x and pkill -x match comm, which the kernel
truncates to 15 characters, so qemu-system-x86_64 is stored as qemu-system-x86
and neither the -x name nor the -x alternation could ever match. The script
therefore never signalled a guest and never waited, and wiped viperblock state
out from under live guests. Match the 11-character prefix against comm, the same
rule qemuProcessPrefix already applies in vm/orphan_scan.go, and scope it to the
user the daemon runs guests as so a host carrying libvirt guests alongside
Spinifex ones does not lose the wrong set. Not -f: that matches the whole
command line, where argv[0] may carry a path.
Nothing waited for spx either. Stopping spinifex.target returns once the target
is inactive, but spinifex-shutdown.service is ordered After= the storage
services, so their stop jobs queue behind its ExecStop and NATS can still be
running. JetStream then rewrites its index after the rm and before the
verification, failing the reset on a file the script had already deleted. Add
the wait update-nodes.sh has carried since d9bc6e35.
The firewall restore was gated on a mode file the failed run had already
deleted, so a retry read the absence as "never armed", skipped the restore, and
install-node.sh honoured that and formed the cluster around an open node. Stash
the mode outside every wiped tree, and fail rather than warn when a node that
arrived armed cannot be put back.
install-node.sh had the other half of that cascade: it disarms every host to let
the OVN raft ports through and only re-arms at the end, so any abort in between
left them all open. Re-arm from an EXIT trap, and say so loudly when a host
arrives with no policy.
The wipe loop deletes files but keeps every directory, so one empty
$G/streams/KV_<name> directory per stream survived the reset. NATS enumerates
those on start and reports them as HA assets it must restore, leaving the meta
group at size=4 with no leader over hundreds of assets that have no data behind
them. Every service then fails to reach JetStream and the daemon aborts its
bootstrap.
Remove the tree outright, mirroring the predastore/cluster removal above it.
The existing docker push e2e test pushes a single 4KB scratch layer, too
small to exercise predastore's storage path for a real multi-hundred-MB
blob. Add a 3-layer (100+80+60MiB) push/pull case that verifies re-pulled
layer digests against the pushed image and directly checks Content-Length
on a raw HTTP GET of the largest blob, targeting a reported defect where
large blob GETs returned 200 with content-length: 0. Re-pull uses
`docker builder prune -af`, not just `rmi`, since buildx's build cache can
otherwise satisfy the pull without touching the registry.
* ci: benchmark Predastore across a real 3-node Spinifex env
The nightly throughput job runs three s3d processes on loopback aliases on a
single box against local NVMe, so nothing measures Predastore in the shape it
ships in: three machines, with reads crossing a real NIC between hosts.
This provisions a 3-node env the way the multi-node e2e job does, then hands
its three gates to the existing harness through PERF_EXTERNAL_HOSTS. The
harness skips its own start and stop and measures them directly, so the
workloads and analysis are unchanged and the two numbers stay comparable.
It runs on a systemd LAN runner rather than the ephemeral pool because a pool
VM sits behind the geneve overlay at MTU 1408, which would put the runner's
own network path inside the measurement.
* ci: trigger the 3-node benchmark on its own branch for a first run
A workflow_dispatch only resolves against the default branch, so this cannot
be dispatched until it is merged. The push trigger is scoped to this branch
and comes back out once the workflow is on dev.
* fix: measure as the identity the provisioned cluster actually mints
spx admin init generates a random access key pair at bootstrap and writes it
to node1's ~/.aws/credentials, so the benchmark's built-in key is one this
cluster has never heard of and every run died on the first CreateBucket.
The pair is read and used inside a single step, the way e2e-ddil.yml does it:
a step-level env: block is echoed verbatim into the run log, a shell variable
is not.
* ci: default the 3-node benchmark to the compare preset
smoke was the right default while the pipeline was unproven. It is green now,
and compare is what the nightly loopback run uses, so it is the preset whose
numbers can be set beside it.
* ci: make the cluster's ref a knob, pinned to v1.18.0 for one run
The harness stays on this branch while the cluster is built from whatever ref
is asked for, so an old release is measured with the same ruler as the current
one. run-info.txt records the deployed ref rather than the harness checkout.
The v1.18.0 default is temporary: a push event carries no dispatch inputs, and
this cannot be dispatched until it is on dev. It comes back out after the run.
* test: probe ranged GETs, which no workload here exercises
Warp reads whole objects, so the throughput figures stay healthy while ranged
GETs read the whole object per request. The probe reads the same boundary
crossing range from a 64 MiB and a 256 MiB object: under read amplification
the cost tracks object size, and otherwise the two match.
Cluster ref is pinned to the fix branch for this run and comes back out after.
* test: run the ranged GET probe against dev to prove it fails there
A probe that has only ever been green proves nothing about the fix. Pointing
the cluster at dev, which still has the defect, is the control that shows the
probe detects it. The pin comes back out after.
* ci: drop the branch push trigger and the cluster ref pin
Both were scaffolding to run this before it existed on dev. The workflow is
dispatch only again, and cluster_ref defaults to the branch under test.
All three need unexported access and cannot move to an external test
package: the daemon suite drives shared package fixtures and reads vmMgr,
resourceMgr and localStatePath, while the other two assert on buildArgs
and volumeVBConfig directly.
They were added on dev without the marker, so check-test-package.sh
failed for anyone running preflight on dev, where the base ref is
origin/main rather than origin/dev.
The controlplane egress selector also carries the authentication
token-webhook dial, which targets 127.0.0.1:8443 on the control-plane VM.
Tunnelling it through konnectivity hands that dial to a randomly chosen
agent, where nothing is listening, so the authenticator returns an error
rather than a decision and every bearer-token request fails closed with a
bare 401.
It only surfaced once a second agent existed. The control-plane node runs
an agent itself, so while it is the only backend every tunnelled dial
lands back home and authentication works by luck. The apiserver's webhook
client also holds its connection for the 90s IdleConnTimeout, so there is
one backend selection per connection rather than per request.
TestEKS/IRSAPod hit both conditions every run: it builds a kubeconfig and
then waits ~105s for its nodegroup to go ACTIVE before issuing any
request, which both expires the pooled connection and brings up the
cluster's second agent.
Only the cluster egress needs the tunnel, for kubelet exec/logs/
portforward. This matches the upstream reference configuration.
The three ci-single runners map to env3/bottlebrush, env10/banksia and
env11/wattle, and the cell landed on whichever was free. Only bottlebrush
has NVMe; banksia and wattle are Dells with none, which is where every
etcd flush measurement to date was taken.
The eks suite boots its control plane from the eks-node AMI, which only
the nightly built. Bare metal had no EKS or mkosi step at all, so naming
the suite there produced a run that could never pass.
dev did not build: the WALBaseDir call sites landed while go.mod still
pinned viperblock v1.18.0, which predates the field.
constructMountedVB passed WALBaseDir to the nbdkit plugin but not to its
own engine, so the daemon would look for WALs under BaseDir while the
plugin wrote them under WALBaseDir.
A WAL fsync shares the device queue with predastore, the chunk cache and
every other volume on the node. Measured on the e2e stack, competing
filesystem traffic alone moves fsync p99 from 4.7ms to 22ms with no
change to viperblock, which is the tail an etcd guest reports as its
commit latency.
wal_base_dir under [nodes.<n>.viperblock] routes the WAL elsewhere. It
reaches both the short-lived VBs the service opens and, through the
nbdkit argv, the plugin backing each mounted volume. Unset keeps the WAL
under base_dir.
* fix: report CPU and memory pressure in the EKS control-plane health diagnostic
An unhealthy control plane reported readyz, etcd reachability and etcd-disk
free space. When etcd goes unreachable with disk:ok and no OOM kill, that set
cannot distinguish a starved guest from a genuine etcd fault, so add the
loadavg and MemAvailable that separate them.
An empty /readyz body also rendered as readyz:[none], which reads as "no
subcheck failed" when it means the probe never answered. Report that as
unreachable instead.
* fix: sample the node's load, not just the runner host's
Every runner agent runs on one host while their nodes sit on several, so the
existing sample measured the runner's host and was labelled as the hypervisor.
It happens to coincide when the node is on that same host and misleads
otherwise. Relabel it and add a node-side sample, which is where the guests are
actually scheduled and so where a starved guest shows up.
* feat: build the mkosi EKS image in CI and give its control plane telemetry
The Ubuntu/systemd EKS image is what customers run, and nothing built it: the
composition existed in mkosi-build.sh's image_profiles table but had no caller,
so a change that broke it was found by hand or not at all. Cell-25 builds the
Alpine image, which leaves the published one with no gate. Add a Makefile target
beside the Alpine one and a job in the existing path-filtered image workflow.
The guest also collected nothing. k3s has served embedded-etcd metrics on
127.0.0.1:2381 since the daemon started setting etcd-expose-metrics, and no
collector ever read them, so an etcd stall was only ever visible as
"etcd:unreachable" in the state report — which cannot separate a slow disk from
a dead one. Ship otelcol-contrib pinned to the version the cluster hosts already
run, scraping etcd and guest host metrics.
The unit is gated on ConditionPathExists for its endpoint file rather than
started unconditionally: whether the control-plane guest can reach the sink from
the isolated CP VPC is unverified, and a collector with nowhere to export is a
restart loop, not telemetry.
* fix: report etcd fsync latency in the control-plane health diagnostic
disk:ok measures free space, which says nothing about how long a write takes.
etcd stalls on fsync latency long before it runs out of room, and this guest's
disk is an EBS volume served by viperblock, so a fsync is a network round trip.
An unhealthy control plane reporting healthy memory and disk was uninterpretable
without it.
Derived from the histogram's sum and count rather than a bucket percentile: two
greps against an interpolation this script has no business doing, and the
difference between a healthy single-digit reading and a stalled one is an order
of magnitude, not a tail. Rides the state report's existing gateway path, so it
needs no route the CP VPC does not already grant, and lands on both images
because the script is shared.
The scrape's failure is guarded explicitly. Under set -e a bare assignment from
a curl against a port nothing answers aborts the whole diagnosis, dropping the
reason field exactly when it is wanted.
* feat: publish etcd fsync on every control-plane report, not only failing ones
A figure that appears only once the apiserver is already unhealthy has no
baseline to be read against, which is the same gap a passing cell that uploads
no journal leaves: the number is unanchored when it finally arrives. Move the
scrape into its own helper, emit it on healthy reports too, and carry it as a
typed field rather than only inside the reason string.
Slowness is logged, never gated. observe() decides cluster health, and a working
cluster on a slow disk is working — returning an issue there would fail
customers' clusters on a latency threshold. The warning is latched on the
transition instead, because the control plane republishes on its own timer and a
line every interval is how a real signal gets tuned out.
An absent sample is not a fast one. Too few fsyncs to average, an etcd serving
no metrics and an older AMI all arrive as nil, and the field is omitted rather
than defaulted so none of them can be read as healthy. The minimum sample count
is the same lesson as the slow-shard floor: a mean over three fsyncs on a
just-started etcd is a number with no meaning.
* fix: log an etcd fsync baseline and name the two no-sample cases apart
A healthy cluster logs no health lines, so the fsync figure reached the
daemon on every report and was never surfaced — a passing cell left nothing
for a later slow reading to be compared against. Log the first known sample.
Split the "unknown" sentinel into nometrics (etcd served none) and warmup
(too few fsyncs to average): different faults, different fixes. The payload
guard now tests for a number rather than the sentinel word, so a future
sentinel cannot leak into the JSON as a bare value.
* feat: measure fdatasync latency in the diskperf gate
The two pinned profiles are buffered-bypassing throughput jobs at queue
depth 32. Neither measures the cost of making a write durable, which is what
a consensus datastore pays per raft entry and stalls on long before it runs
short of bandwidth. A 5ms barrier on bare metal went unseen as a result.
syncwrite-4k is the etcd-shaped profile: sequential 4k, queue depth 1,
fdatasync every write, buffered psync so the sync has a page cache to flush.
fio reports the sync distribution apart from the write stream, so the parser
reads it separately — gating on write clat would score the time to reach the
cache and call it durability.
No figures captured on this hardware yet, so the baseline entry is zero and
the job records without gating.
* fix: use the real otelcol checksums asset name in the EKS server image
The checksums file is named for the release repo and the distribution and
carries no version, unlike the tarball beside it. The guessed name 404'd and
took the image build with it.
* fix: extract otelcol as root in the EKS server image
The upstream tarball records its build uid (runner/docker, 1001:118), which
does not exist in the image. tar running as root tries to restore it and
fails outright rather than degrading, taking the build with it.
---------
Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>
Ben Duncan