The wipe loop deletes files but keeps every directory, so one empty
$G/streams/KV_<name> directory per stream survived the reset. NATS enumerates
those on start and reports them as HA assets it must restore, leaving the meta
group at size=4 with no leader over hundreds of assets that have no data behind
them. Every service then fails to reach JetStream and the daemon aborts its
bootstrap.
Remove the tree outright, mirroring the predastore/cluster removal above it.
The existing docker push e2e test pushes a single 4KB scratch layer, too
small to exercise predastore's storage path for a real multi-hundred-MB
blob. Add a 3-layer (100+80+60MiB) push/pull case that verifies re-pulled
layer digests against the pushed image and directly checks Content-Length
on a raw HTTP GET of the largest blob, targeting a reported defect where
large blob GETs returned 200 with content-length: 0. Re-pull uses
`docker builder prune -af`, not just `rmi`, since buildx's build cache can
otherwise satisfy the pull without touching the registry.
* ci: benchmark Predastore across a real 3-node Spinifex env
The nightly throughput job runs three s3d processes on loopback aliases on a
single box against local NVMe, so nothing measures Predastore in the shape it
ships in: three machines, with reads crossing a real NIC between hosts.
This provisions a 3-node env the way the multi-node e2e job does, then hands
its three gates to the existing harness through PERF_EXTERNAL_HOSTS. The
harness skips its own start and stop and measures them directly, so the
workloads and analysis are unchanged and the two numbers stay comparable.
It runs on a systemd LAN runner rather than the ephemeral pool because a pool
VM sits behind the geneve overlay at MTU 1408, which would put the runner's
own network path inside the measurement.
* ci: trigger the 3-node benchmark on its own branch for a first run
A workflow_dispatch only resolves against the default branch, so this cannot
be dispatched until it is merged. The push trigger is scoped to this branch
and comes back out once the workflow is on dev.
* fix: measure as the identity the provisioned cluster actually mints
spx admin init generates a random access key pair at bootstrap and writes it
to node1's ~/.aws/credentials, so the benchmark's built-in key is one this
cluster has never heard of and every run died on the first CreateBucket.
The pair is read and used inside a single step, the way e2e-ddil.yml does it:
a step-level env: block is echoed verbatim into the run log, a shell variable
is not.
* ci: default the 3-node benchmark to the compare preset
smoke was the right default while the pipeline was unproven. It is green now,
and compare is what the nightly loopback run uses, so it is the preset whose
numbers can be set beside it.
* ci: make the cluster's ref a knob, pinned to v1.18.0 for one run
The harness stays on this branch while the cluster is built from whatever ref
is asked for, so an old release is measured with the same ruler as the current
one. run-info.txt records the deployed ref rather than the harness checkout.
The v1.18.0 default is temporary: a push event carries no dispatch inputs, and
this cannot be dispatched until it is on dev. It comes back out after the run.
* test: probe ranged GETs, which no workload here exercises
Warp reads whole objects, so the throughput figures stay healthy while ranged
GETs read the whole object per request. The probe reads the same boundary
crossing range from a 64 MiB and a 256 MiB object: under read amplification
the cost tracks object size, and otherwise the two match.
Cluster ref is pinned to the fix branch for this run and comes back out after.
* test: run the ranged GET probe against dev to prove it fails there
A probe that has only ever been green proves nothing about the fix. Pointing
the cluster at dev, which still has the defect, is the control that shows the
probe detects it. The pin comes back out after.
* ci: drop the branch push trigger and the cluster ref pin
Both were scaffolding to run this before it existed on dev. The workflow is
dispatch only again, and cluster_ref defaults to the branch under test.
All three need unexported access and cannot move to an external test
package: the daemon suite drives shared package fixtures and reads vmMgr,
resourceMgr and localStatePath, while the other two assert on buildArgs
and volumeVBConfig directly.
They were added on dev without the marker, so check-test-package.sh
failed for anyone running preflight on dev, where the base ref is
origin/main rather than origin/dev.
The controlplane egress selector also carries the authentication
token-webhook dial, which targets 127.0.0.1:8443 on the control-plane VM.
Tunnelling it through konnectivity hands that dial to a randomly chosen
agent, where nothing is listening, so the authenticator returns an error
rather than a decision and every bearer-token request fails closed with a
bare 401.
It only surfaced once a second agent existed. The control-plane node runs
an agent itself, so while it is the only backend every tunnelled dial
lands back home and authentication works by luck. The apiserver's webhook
client also holds its connection for the 90s IdleConnTimeout, so there is
one backend selection per connection rather than per request.
TestEKS/IRSAPod hit both conditions every run: it builds a kubeconfig and
then waits ~105s for its nodegroup to go ACTIVE before issuing any
request, which both expires the pooled connection and brings up the
cluster's second agent.
Only the cluster egress needs the tunnel, for kubelet exec/logs/
portforward. This matches the upstream reference configuration.
The three ci-single runners map to env3/bottlebrush, env10/banksia and
env11/wattle, and the cell landed on whichever was free. Only bottlebrush
has NVMe; banksia and wattle are Dells with none, which is where every
etcd flush measurement to date was taken.
The eks suite boots its control plane from the eks-node AMI, which only
the nightly built. Bare metal had no EKS or mkosi step at all, so naming
the suite there produced a run that could never pass.
dev did not build: the WALBaseDir call sites landed while go.mod still
pinned viperblock v1.18.0, which predates the field.
constructMountedVB passed WALBaseDir to the nbdkit plugin but not to its
own engine, so the daemon would look for WALs under BaseDir while the
plugin wrote them under WALBaseDir.
A WAL fsync shares the device queue with predastore, the chunk cache and
every other volume on the node. Measured on the e2e stack, competing
filesystem traffic alone moves fsync p99 from 4.7ms to 22ms with no
change to viperblock, which is the tail an etcd guest reports as its
commit latency.
wal_base_dir under [nodes.<n>.viperblock] routes the WAL elsewhere. It
reaches both the short-lived VBs the service opens and, through the
nbdkit argv, the plugin backing each mounted volume. Unset keeps the WAL
under base_dir.
* fix: report CPU and memory pressure in the EKS control-plane health diagnostic
An unhealthy control plane reported readyz, etcd reachability and etcd-disk
free space. When etcd goes unreachable with disk:ok and no OOM kill, that set
cannot distinguish a starved guest from a genuine etcd fault, so add the
loadavg and MemAvailable that separate them.
An empty /readyz body also rendered as readyz:[none], which reads as "no
subcheck failed" when it means the probe never answered. Report that as
unreachable instead.
* fix: sample the node's load, not just the runner host's
Every runner agent runs on one host while their nodes sit on several, so the
existing sample measured the runner's host and was labelled as the hypervisor.
It happens to coincide when the node is on that same host and misleads
otherwise. Relabel it and add a node-side sample, which is where the guests are
actually scheduled and so where a starved guest shows up.
* feat: build the mkosi EKS image in CI and give its control plane telemetry
The Ubuntu/systemd EKS image is what customers run, and nothing built it: the
composition existed in mkosi-build.sh's image_profiles table but had no caller,
so a change that broke it was found by hand or not at all. Cell-25 builds the
Alpine image, which leaves the published one with no gate. Add a Makefile target
beside the Alpine one and a job in the existing path-filtered image workflow.
The guest also collected nothing. k3s has served embedded-etcd metrics on
127.0.0.1:2381 since the daemon started setting etcd-expose-metrics, and no
collector ever read them, so an etcd stall was only ever visible as
"etcd:unreachable" in the state report — which cannot separate a slow disk from
a dead one. Ship otelcol-contrib pinned to the version the cluster hosts already
run, scraping etcd and guest host metrics.
The unit is gated on ConditionPathExists for its endpoint file rather than
started unconditionally: whether the control-plane guest can reach the sink from
the isolated CP VPC is unverified, and a collector with nowhere to export is a
restart loop, not telemetry.
* fix: report etcd fsync latency in the control-plane health diagnostic
disk:ok measures free space, which says nothing about how long a write takes.
etcd stalls on fsync latency long before it runs out of room, and this guest's
disk is an EBS volume served by viperblock, so a fsync is a network round trip.
An unhealthy control plane reporting healthy memory and disk was uninterpretable
without it.
Derived from the histogram's sum and count rather than a bucket percentile: two
greps against an interpolation this script has no business doing, and the
difference between a healthy single-digit reading and a stalled one is an order
of magnitude, not a tail. Rides the state report's existing gateway path, so it
needs no route the CP VPC does not already grant, and lands on both images
because the script is shared.
The scrape's failure is guarded explicitly. Under set -e a bare assignment from
a curl against a port nothing answers aborts the whole diagnosis, dropping the
reason field exactly when it is wanted.
* feat: publish etcd fsync on every control-plane report, not only failing ones
A figure that appears only once the apiserver is already unhealthy has no
baseline to be read against, which is the same gap a passing cell that uploads
no journal leaves: the number is unanchored when it finally arrives. Move the
scrape into its own helper, emit it on healthy reports too, and carry it as a
typed field rather than only inside the reason string.
Slowness is logged, never gated. observe() decides cluster health, and a working
cluster on a slow disk is working — returning an issue there would fail
customers' clusters on a latency threshold. The warning is latched on the
transition instead, because the control plane republishes on its own timer and a
line every interval is how a real signal gets tuned out.
An absent sample is not a fast one. Too few fsyncs to average, an etcd serving
no metrics and an older AMI all arrive as nil, and the field is omitted rather
than defaulted so none of them can be read as healthy. The minimum sample count
is the same lesson as the slow-shard floor: a mean over three fsyncs on a
just-started etcd is a number with no meaning.
* fix: log an etcd fsync baseline and name the two no-sample cases apart
A healthy cluster logs no health lines, so the fsync figure reached the
daemon on every report and was never surfaced — a passing cell left nothing
for a later slow reading to be compared against. Log the first known sample.
Split the "unknown" sentinel into nometrics (etcd served none) and warmup
(too few fsyncs to average): different faults, different fixes. The payload
guard now tests for a number rather than the sentinel word, so a future
sentinel cannot leak into the JSON as a bare value.
* feat: measure fdatasync latency in the diskperf gate
The two pinned profiles are buffered-bypassing throughput jobs at queue
depth 32. Neither measures the cost of making a write durable, which is what
a consensus datastore pays per raft entry and stalls on long before it runs
short of bandwidth. A 5ms barrier on bare metal went unseen as a result.
syncwrite-4k is the etcd-shaped profile: sequential 4k, queue depth 1,
fdatasync every write, buffered psync so the sync has a page cache to flush.
fio reports the sync distribution apart from the write stream, so the parser
reads it separately — gating on write clat would score the time to reach the
cache and call it durability.
No figures captured on this hardware yet, so the baseline entry is zero and
the job records without gating.
* fix: use the real otelcol checksums asset name in the EKS server image
The checksums file is named for the release repo and the distribution and
carries no version, unlike the tarball beside it. The guessed name 404'd and
took the image build with it.
* fix: extract otelcol as root in the EKS server image
The upstream tarball records its build uid (runner/docker, 1001:118), which
does not exist in the image. tar running as root tries to restore it and
fails outright rather than degrading, taking the build with it.
---------
Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>
Ben Duncan