The existing docker push e2e test pushes a single 4KB scratch layer, too
small to exercise predastore's storage path for a real multi-hundred-MB
blob. Add a 3-layer (100+80+60MiB) push/pull case that verifies re-pulled
layer digests against the pushed image and directly checks Content-Length
on a raw HTTP GET of the largest blob, targeting a reported defect where
large blob GETs returned 200 with content-length: 0. Re-pull uses
`docker builder prune -af`, not just `rmi`, since buildx's build cache can
otherwise satisfy the pull without touching the registry.
* ci: benchmark Predastore across a real 3-node Spinifex env
The nightly throughput job runs three s3d processes on loopback aliases on a
single box against local NVMe, so nothing measures Predastore in the shape it
ships in: three machines, with reads crossing a real NIC between hosts.
This provisions a 3-node env the way the multi-node e2e job does, then hands
its three gates to the existing harness through PERF_EXTERNAL_HOSTS. The
harness skips its own start and stop and measures them directly, so the
workloads and analysis are unchanged and the two numbers stay comparable.
It runs on a systemd LAN runner rather than the ephemeral pool because a pool
VM sits behind the geneve overlay at MTU 1408, which would put the runner's
own network path inside the measurement.
* ci: trigger the 3-node benchmark on its own branch for a first run
A workflow_dispatch only resolves against the default branch, so this cannot
be dispatched until it is merged. The push trigger is scoped to this branch
and comes back out once the workflow is on dev.
* fix: measure as the identity the provisioned cluster actually mints
spx admin init generates a random access key pair at bootstrap and writes it
to node1's ~/.aws/credentials, so the benchmark's built-in key is one this
cluster has never heard of and every run died on the first CreateBucket.
The pair is read and used inside a single step, the way e2e-ddil.yml does it:
a step-level env: block is echoed verbatim into the run log, a shell variable
is not.
* ci: default the 3-node benchmark to the compare preset
smoke was the right default while the pipeline was unproven. It is green now,
and compare is what the nightly loopback run uses, so it is the preset whose
numbers can be set beside it.
* ci: make the cluster's ref a knob, pinned to v1.18.0 for one run
The harness stays on this branch while the cluster is built from whatever ref
is asked for, so an old release is measured with the same ruler as the current
one. run-info.txt records the deployed ref rather than the harness checkout.
The v1.18.0 default is temporary: a push event carries no dispatch inputs, and
this cannot be dispatched until it is on dev. It comes back out after the run.
* test: probe ranged GETs, which no workload here exercises
Warp reads whole objects, so the throughput figures stay healthy while ranged
GETs read the whole object per request. The probe reads the same boundary
crossing range from a 64 MiB and a 256 MiB object: under read amplification
the cost tracks object size, and otherwise the two match.
Cluster ref is pinned to the fix branch for this run and comes back out after.
* test: run the ranged GET probe against dev to prove it fails there
A probe that has only ever been green proves nothing about the fix. Pointing
the cluster at dev, which still has the defect, is the control that shows the
probe detects it. The pin comes back out after.
* ci: drop the branch push trigger and the cluster ref pin
Both were scaffolding to run this before it existed on dev. The workflow is
dispatch only again, and cluster_ref defaults to the branch under test.
All three need unexported access and cannot move to an external test
package: the daemon suite drives shared package fixtures and reads vmMgr,
resourceMgr and localStatePath, while the other two assert on buildArgs
and volumeVBConfig directly.
They were added on dev without the marker, so check-test-package.sh
failed for anyone running preflight on dev, where the base ref is
origin/main rather than origin/dev.
The controlplane egress selector also carries the authentication
token-webhook dial, which targets 127.0.0.1:8443 on the control-plane VM.
Tunnelling it through konnectivity hands that dial to a randomly chosen
agent, where nothing is listening, so the authenticator returns an error
rather than a decision and every bearer-token request fails closed with a
bare 401.
It only surfaced once a second agent existed. The control-plane node runs
an agent itself, so while it is the only backend every tunnelled dial
lands back home and authentication works by luck. The apiserver's webhook
client also holds its connection for the 90s IdleConnTimeout, so there is
one backend selection per connection rather than per request.
TestEKS/IRSAPod hit both conditions every run: it builds a kubeconfig and
then waits ~105s for its nodegroup to go ACTIVE before issuing any
request, which both expires the pooled connection and brings up the
cluster's second agent.
Only the cluster egress needs the tunnel, for kubelet exec/logs/
portforward. This matches the upstream reference configuration.
The three ci-single runners map to env3/bottlebrush, env10/banksia and
env11/wattle, and the cell landed on whichever was free. Only bottlebrush
has NVMe; banksia and wattle are Dells with none, which is where every
etcd flush measurement to date was taken.
The eks suite boots its control plane from the eks-node AMI, which only
the nightly built. Bare metal had no EKS or mkosi step at all, so naming
the suite there produced a run that could never pass.
dev did not build: the WALBaseDir call sites landed while go.mod still
pinned viperblock v1.18.0, which predates the field.
constructMountedVB passed WALBaseDir to the nbdkit plugin but not to its
own engine, so the daemon would look for WALs under BaseDir while the
plugin wrote them under WALBaseDir.
A WAL fsync shares the device queue with predastore, the chunk cache and
every other volume on the node. Measured on the e2e stack, competing
filesystem traffic alone moves fsync p99 from 4.7ms to 22ms with no
change to viperblock, which is the tail an etcd guest reports as its
commit latency.
wal_base_dir under [nodes.<n>.viperblock] routes the WAL elsewhere. It
reaches both the short-lived VBs the service opens and, through the
nbdkit argv, the plugin backing each mounted volume. Unset keeps the WAL
under base_dir.
* fix: report CPU and memory pressure in the EKS control-plane health diagnostic
An unhealthy control plane reported readyz, etcd reachability and etcd-disk
free space. When etcd goes unreachable with disk:ok and no OOM kill, that set
cannot distinguish a starved guest from a genuine etcd fault, so add the
loadavg and MemAvailable that separate them.
An empty /readyz body also rendered as readyz:[none], which reads as "no
subcheck failed" when it means the probe never answered. Report that as
unreachable instead.
* fix: sample the node's load, not just the runner host's
Every runner agent runs on one host while their nodes sit on several, so the
existing sample measured the runner's host and was labelled as the hypervisor.
It happens to coincide when the node is on that same host and misleads
otherwise. Relabel it and add a node-side sample, which is where the guests are
actually scheduled and so where a starved guest shows up.
* feat: build the mkosi EKS image in CI and give its control plane telemetry
The Ubuntu/systemd EKS image is what customers run, and nothing built it: the
composition existed in mkosi-build.sh's image_profiles table but had no caller,
so a change that broke it was found by hand or not at all. Cell-25 builds the
Alpine image, which leaves the published one with no gate. Add a Makefile target
beside the Alpine one and a job in the existing path-filtered image workflow.
The guest also collected nothing. k3s has served embedded-etcd metrics on
127.0.0.1:2381 since the daemon started setting etcd-expose-metrics, and no
collector ever read them, so an etcd stall was only ever visible as
"etcd:unreachable" in the state report — which cannot separate a slow disk from
a dead one. Ship otelcol-contrib pinned to the version the cluster hosts already
run, scraping etcd and guest host metrics.
The unit is gated on ConditionPathExists for its endpoint file rather than
started unconditionally: whether the control-plane guest can reach the sink from
the isolated CP VPC is unverified, and a collector with nowhere to export is a
restart loop, not telemetry.
* fix: report etcd fsync latency in the control-plane health diagnostic
disk:ok measures free space, which says nothing about how long a write takes.
etcd stalls on fsync latency long before it runs out of room, and this guest's
disk is an EBS volume served by viperblock, so a fsync is a network round trip.
An unhealthy control plane reporting healthy memory and disk was uninterpretable
without it.
Derived from the histogram's sum and count rather than a bucket percentile: two
greps against an interpolation this script has no business doing, and the
difference between a healthy single-digit reading and a stalled one is an order
of magnitude, not a tail. Rides the state report's existing gateway path, so it
needs no route the CP VPC does not already grant, and lands on both images
because the script is shared.
The scrape's failure is guarded explicitly. Under set -e a bare assignment from
a curl against a port nothing answers aborts the whole diagnosis, dropping the
reason field exactly when it is wanted.
* feat: publish etcd fsync on every control-plane report, not only failing ones
A figure that appears only once the apiserver is already unhealthy has no
baseline to be read against, which is the same gap a passing cell that uploads
no journal leaves: the number is unanchored when it finally arrives. Move the
scrape into its own helper, emit it on healthy reports too, and carry it as a
typed field rather than only inside the reason string.
Slowness is logged, never gated. observe() decides cluster health, and a working
cluster on a slow disk is working — returning an issue there would fail
customers' clusters on a latency threshold. The warning is latched on the
transition instead, because the control plane republishes on its own timer and a
line every interval is how a real signal gets tuned out.
An absent sample is not a fast one. Too few fsyncs to average, an etcd serving
no metrics and an older AMI all arrive as nil, and the field is omitted rather
than defaulted so none of them can be read as healthy. The minimum sample count
is the same lesson as the slow-shard floor: a mean over three fsyncs on a
just-started etcd is a number with no meaning.
* fix: log an etcd fsync baseline and name the two no-sample cases apart
A healthy cluster logs no health lines, so the fsync figure reached the
daemon on every report and was never surfaced — a passing cell left nothing
for a later slow reading to be compared against. Log the first known sample.
Split the "unknown" sentinel into nometrics (etcd served none) and warmup
(too few fsyncs to average): different faults, different fixes. The payload
guard now tests for a number rather than the sentinel word, so a future
sentinel cannot leak into the JSON as a bare value.
* feat: measure fdatasync latency in the diskperf gate
The two pinned profiles are buffered-bypassing throughput jobs at queue
depth 32. Neither measures the cost of making a write durable, which is what
a consensus datastore pays per raft entry and stalls on long before it runs
short of bandwidth. A 5ms barrier on bare metal went unseen as a result.
syncwrite-4k is the etcd-shaped profile: sequential 4k, queue depth 1,
fdatasync every write, buffered psync so the sync has a page cache to flush.
fio reports the sync distribution apart from the write stream, so the parser
reads it separately — gating on write clat would score the time to reach the
cache and call it durability.
No figures captured on this hardware yet, so the baseline entry is zero and
the job records without gating.
* fix: use the real otelcol checksums asset name in the EKS server image
The checksums file is named for the release repo and the distribution and
carries no version, unlike the tarball beside it. The guessed name 404'd and
took the image build with it.
* fix: extract otelcol as root in the EKS server image
The upstream tarball records its build uid (runner/docker, 1001:118), which
does not exist in the image. tar running as root tries to restore it and
fails outright rather than degrading, taking the build with it.
---------
Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>
* fix: stop the e2e analyser inventing failures and let a detach outlast its seal
go-junit-report emits <error message="No test result found"> for a === RUN with
no matching result line, and attaches the test's own captured log as the body.
The analyser counted those as failures, so the baremetal cell reported 25
failures against a junit file recording 1, with log output reading as
assertions. Separate the marker from a real error in both the parse and the
roll-up, count the unresolved cases and say so in the summary.
The gateway gave a detach 30s while the daemon budgets 180s for the ebs.unmount
seal and 180s more for the QMP unplug chain, so any detach whose seal replayed a
WAL was reported as ServerInternal while it went on to succeed.
Also switch four nightly job gates from always() to !cancelled(): always()
overrides cancellation, so a cancelled run kept starting jobs.
* fix: serialise the e2e workflows on one group and retry a dropped libvirt
e2e.yml had no concurrency block at all, so every push to dev could provision
alongside every other push and alongside the nightly. All four workflows that
build on these hypervisors now share e2e-hypervisors, queued rather than
cancelled. A per-environment group is not expressible: GitHub picks the group
before it picks a runner, and the environment is a property of the runner.
tofu apply also failed outright on a transient libvirt connect, which fails a
45-minute job in 90 seconds and reports it as a test failure. retry_run retries
only when that attempt's own output names a connection failure, so a real
failure still fails once.
Also collect the unfiltered journal from the matrix cells and the node journal
from cell-21: the unit filter drops the kernel and OOM lines that explain why
one node behaved unlike its siblings, and a passing run's journal is what makes
a failing one readable.
* fix: build the baremetal junit from the unfiltered suite log
run-baremetal-e2e.sh streams the full suite output to suite-output.log and a
grep-filtered summary to its stdout, which the workflow tees into
test-baremetal.log. That filter anchors --- PASS at column 0 and Go indents
every subtest result by four spaces, so the junit was generated from a log with
no subtest results in it: go-junit-report saw each === RUN with no outcome and
emitted <error message="No test result found">.
17 of cell-28's 39 testcases and 93 of cell-20's 120 were that. Regenerating
cell-28 from suite-output.log gives 39 tests, 0 failures, 0 errors.
* fix: converge released EIPs instead of trusting a fire-and-forget teardown
A disassociate leaves the ENI running, so the orphan sweep's owning-port test
never fires for it. The teardown then rests entirely on vpc.delete-nat, which is
core NATS and is lost whenever vpcd is restarting — and the intent KV watch is
UpdatesOnly, so the same window hides the write that would have driven a repair
pass. The dnat_and_snat row and its host /32 route to a released public address
survived until something else happened to touch them, which for a live ENI was
nothing at all.
Give the sweep the second test it was missing: AddEIP is the only writer of
dnat_and_snat, so the floating IPs intent asks for are the complete set of
addresses that may hold one, and a row for any other address is stale however
alive its port. Ports and external IPs travel together in LiveEIPs, both unioned
from the start-of-pass snapshot and the fresh re-read, so a mid-pass launch is
still spared. The host unbind now runs even when the row had already gone, which
is the state a crash between the OVN delete and the unbind leaves.
Run that sweep on the startup pass too. Startup is the pass that follows the
blind window, so skipping it deferred repair of exactly the changes that were
missed. The port-group and ENI-port sweeps stay gated there for the in-flight
create they were written for; the EIP sweep re-reads intent immediately before
deciding and abandons the whole sweep if that read fails, which is the same
protection by a stronger route.
* fix: keep a release verify off the hypervisors an e2e run is using
release.yml is the fifth workflow that provisions on the ci-single pool and had
no concurrency block at all, so a tag push could land a tofu apply on the same
libvirt host a nightly cell was mid-provision on. The group goes on the verify
job rather than the workflow: the build jobs touch no hypervisor and should not
queue behind a nightly.
* fix: stop cell-27 handing a 50-minute suite a 50-minute job
The rds suite is budgeted 50m in suite-sets.sh and cell-27 took the matrix
default of 50 minutes for the whole job, so provisioning, install, the suite and
teardown had to fit inside the suite's allowance alone. It never could: the last
measured run spent 6 minutes reaching the suite and was killed 44 minutes into
it, mid-TestAutomatedBackups. A job timeout reports "The operation was canceled"
and nothing else, which is why this cell has no verdict in any nightly rather
than a failure anyone could read.
cell-27 gets 75 minutes, and the select job now refuses to start a run whose job
budget does not clear its suite budget plus a setup allowance. The inner timeout
firing first is the property that matters: go test says where it stopped and
what failed, and the job timeout says nothing at all. cell-29 already had the
same fix applied by hand; nothing stopped the next suite from repeating it.
* fix: converge host EIP plumbing against intent, not against the NAT row
The released-EIP sweep landed and did not fire. Its premise was that a stranded
dnat_and_snat row is what a stranded host route can be found by, and on the path
that actually strands one there is no row left. AddEIP scrubs the predecessor
row sharing the private IP before installing the new one, so when reconcile
re-added the instance's auto-assigned address it silently deleted the released
EIP's row — and applyEIPs runs ahead of the sweep, which then walked a table
with nothing stale in it while the /32 route and its proxy-ARP stayed up.
Give the sweep the host's own inventory. ListEIPIngress reads the EIPs back from
the FORWARD accepts EnsureEIPIngress stamps with its own comment, so a binding is
found by what installed it rather than by a row some other path is free to
delete. Anything plumbed that intent no longer asks for is torn down, which also
covers a crash between the OVN delete and the unbind. A failed inventory fails
the pass: an empty list and an unreadable one are indistinguishable, and reading
the second as the first is a sweep that reports success having done nothing.
Close the window inside the pass too. The predecessor scrub now reads the row it
is about to drop and releases that address's host state, so the strand does not
exist between applyEIPs and the sweep rather than merely being repaired a step
later.
* fix: retry the release verify's libvirt calls like every other job does
The verify job drives the same libvirt provider on the same hypervisors as the
e2e cells, where a dropped connection under concurrent load is the steady state
and is retried. Here it was a hard failure that reads as a broken release. It
was skipped earlier only because the job checks out mulga and not spinifex, so
the shared helpers were not on disk — a sparse checkout of .github/scripts costs
nothing and makes apply and destroy behave like their e2e counterparts.
* fix: do not report a rebooting DB instance available on the pre-reboot agent's health
RegisterDBInstance stamped a fresh LastSeen while leaving Agent.EngineHealth at
the value the previous agent process reported. After a reboot the new agent
registers before it has probed anything, so the record read as a healthy engine
with a beat newer than the transition, and the reconciler moved the instance to
available while PostgreSQL was still starting up. A client that connects the
moment the status turns green then gets "the database system is starting up".
A register now discards the previous process's verdict and records when this
agent process started. The restart reconciler requires that registration to be
newer than the transition, which also closes the window where a beat lands
between the engine being stopped and the VM going down.
* test: record the credential kubectl sent when an EKS call comes back Unauthorized
The IRSA pod subtest's first kubectl is the gateway-CA configmap, which lands
before the pod dumps are registered, so a 401 there left no evidence at all. A
verbose read-only probe reports whether an Authorization header was sent, which
separates a token the apiserver rejected from one the exec plugin never made.
* fix: report a starting engine promptly, and budget CI jobs from the suite
The agent waited a full heartbeat interval before its first beat, so a restart
could not be observed as finished for up to 30 seconds after the engine was
already serving. Beat at startup instead, and report every 5 seconds until the
engine has answered once — the control plane is actively waiting on that beat,
and the steady-state cadence it hands back is for a database already serving.
The rds suite is the one most sensitive to what else the hypervisor is doing:
its DB VMs serialise behind a 4 GiB semaphore, so its wall clock is set by how
long it waits for the budget. The same pass ran 45m beside four nightly cells
and overran 50m beside nine, panicking with two tests still running. Raise its
budget to 60m.
That budget was then larger than the jobs meant to contain it. A job cancelled
on timeout reports only "The operation was canceled" and takes the artifacts
with it, so the inner `go test` timeout must always fire first — and it could
not for rds on e2e.yml, nor for storagefault or diskperf, which have asked for
90m and 180m inside a 60m job since they were added. Both workflows now resolve
their job budget from suite-sets.sh rather than carrying a literal: e2e.yml
from the largest requested suite, e2e-suites.yml per matrix leg, so a leg is
bounded by its own suite and not by the longest one beside it. The default push
set resolves to exactly the timeouts both workflows had before.
* fix: check a permutation cell's job budget against the suites it runs
The guard only saw cells that name a suite, so the nine that run a topology's
nightly subset were never checked against it — the same class of mistake the
guard exists to catch, in the majority of the cells. Bound each by the longest
suite in its subset: max() and not sum(), because whichever suite hangs must
panic on its own timeout first, and a sum would demand a budget the set has
never come close to using. Every cell clears it today.
* fix: report why a PostgreSQL engine would not start, and stop failing a stop that landed
Three defects, all found by the first rds pass that ever ran deep enough to
reach them.
A wait that saw an instance reach `failed` called t.Fatalf from inside the
polling condition, and testify runs that condition on its own goroutine. The
Goexit killed the tick and left the collector empty, which reads as the
condition being satisfied — so the wait returned successfully on an instance it
had just declared broken, logged "reached status available" about one that was
failed, and let the test carry on against it. Six conditions across four suites
call Fatal or require the same way. EventuallyErr now runs its condition on the
test's goroutine, which makes all six behave as written.
A DB VM stop has two drivers: the API call and the reconciler resuming one whose
caller it assumes has died. When they overlap the loser is told the VM is not in
a state to be stopped — which is the state the stop was heading for — and the
customer's StopDBInstance returned InvalidDBInstanceState for a stop that had
just completed. Both paths now treat a refused command the way they already
treat an unanswered one: it says nothing about the VM, so the fleet view
settles it.
PostgreSQL had no equivalent of the engine log tail MariaDB has carried since it
was added, so a postmaster that refuses a setting produced "engine did not
respond on /run/postgresql:5432" and nothing else — on a guest with no SSH,
whose console records only that pg_ctl could not start the server. The probe now
quotes the tail of the log the image already directs the postmaster to, and the
reboot and modify paths carry the agent's reason into the failure the way the
create path always has. The reason reaches the customer through StatusInfos.
The rds suite's budget goes to 90m and cell-27's job to 100m. Its wall clock is
what the DB-VM queue costs rather than what the tests do — single tests logged
11m, 22m and 26m waiting for budget — and the pass above still had three of
seventeen tests running at 60m.
* fix: keep a DB VM recoverable when its agent cannot start, and flush the volume before a snapshot
An rds-agent that failed to construct exited, so a transient IMDS credential
resolution took the guest down for good: rds-datadir and rds-init cascade on the
missing handoff, the engine never starts, and the control plane sees only silence
until it declares the instance failed ten minutes later. Construction is retried
like every other boot step instead.
OpenRC skips start_post when the start itself fails, which is the one case with
no diagnostic at all — the console carried nothing but "died". The log mirror
moves out of the handoff wait so a failed start uses it too.
A snapshot quiesced the engine and nothing else. The engine repairs its own files
from the WAL, but postgresql.conf sits beside them and it does not: a restore
came up on a torn tail and the postmaster stopped on a syntax error at the
unterminated include hook. The quiesce now flushes the whole data mount, and the
hook is renamed into place rather than appended, so a boot that finds a half
written one replaces it instead of appending after its open quote.
* fix: flush the config cloud-init wrote before a reset can take it
A DB instance rebooted seconds after it went available came back with no
RDS_GATEWAY_URL, so rds-agent could not reach the control plane and the engine
never started. cloud-init writes agent.env once, per-instance, and never again;
a reboot is a QMP system_reset with no guest shutdown, so the file was still
dirty page cache when the machine reset under it and the next boot read nothing.
The handoff wait now syncs, which is the point at which the agent has proved it
read that file.
* fix: stop dropping an EIP teardown published while vpcd is restarting
DisassociateAddress publishes vpc.delete-nat fire-and-forget. On run
34046316024 vpcd disconnected from NATS at 03:51:26.73 and its replacement
subscribed at 03:51:27.09; the publish landed at 03:51:26.92, in the gap, and
went nowhere. ReleaseAddress then returned the address to the pool two minutes
later with its /32 host route still delivering to the guest that had held it.
vpc.delete-nat now uses request-reply like vpc.add-nat, and retries only on
no-responders — which returns immediately, so the cost is paid solely when
nothing is listening. A teardown that still fails is logged as an error naming
the address rather than lost silently. Neither is fatal to the caller: the
reconciler's orphan sweep remains the backstop.
vpc.add-nat on the EIP path stays fire-and-forget. The shared helper blocks it
on the flows barrier for up to 45s, and associating an address should not
inherit that.
Two reconcile passes in that run had the address absent from intent and did not
sweep its host route, and the journal cannot say why: the sweep logged only when
it acted, so a route it kept was indistinguishable from one it never saw. It now
logs the bindings it found, kept and wanted.
* ci: render one run-level table for the e2e nightly
Reading a nightly meant opening nineteen jobs, and the Actions UI does not
separate a cell that failed from one cancelled out from under it — a
distinction every investigation of this suite has needed, and one the run
34046316024 analysis had to reach for the jobs API to get.
A final job, needs: every cell and if: always(), renders cell, topology,
result, duration, runner and per-cell test counts into the step summary, with
run-level totals, the three slowest cells and the failing test names.
Result, duration and runner come from the jobs API so a cell that died before
writing junit still gets a row. Counts come from a new junit-only artifact per
cell rather than the full cell artifact, so the roll-up does not pull nineteen
journals it has no use for.
Verified against run 34046316024: 14 success, 2 cancelled, 1 failure, with the
two cancellations shown as cancelled rather than folded into failure.
* fix: assert EC2's actual EIP semantics, and probe EKS auth on an authenticated path
natuplink asserted that an instance's auto-assigned public IP is still plumbed
after an EIP associated to the same instance is torn down. EC2 does the
opposite: associating an EIP releases the auto-assigned address back into the
pool, and it cannot be reused. Our datapath already does that — run 34059879666
shows 192.168.0.72's ingress removed at 07:09:44.000, 36ms before .73's was
installed — so the test was asserting behaviour neither AWS nor we have.
It now asserts the release at the point it happens, and that no public delivery
remains after the teardown. This surfaced only once the teardown itself started
working: the run's other assertions passed, including .73's removal, which was
the failure before.
Our records still disagree with our datapath — the ENI keeps the released
address, which is why termination tears the old one down twice and never the
current one. That is the remaining half of mulga-7zc9w and is not fixed here;
it needs an IPAM release on the associate path.
Separately, AuthDiagnostics read /version, which is anonymously readable, so its
200 was consistent with both branches it claimed to distinguish. It now runs
auth whoami, reads kube-system, and invokes the credential plugin directly,
reporting exit code, stderr and whether a token was produced — never the token,
since CI logs outlive it.
* test: capture per-guest state when a spread guest never answers SSH
TestMultinodeSpread's only failure dump is dumpPlacementNATDiag, which collects
OVN, OVS, ARP and route state aimed at the NAT and EIP half of the test. The
failure it actually hits is a guest that never accepted SSH — run 34040324968,
the guest on node3 at 10.100.2.5 — and for that it recorded no console, no port
binding and no tap state. There was no way to tell from the artifact whether the
guest booted at all.
Now dumps, per guest: the console, the NB logical switch port, the SB port
binding, the OVS interface backing its tap, and br-int's port list. Probed on
every node rather than the expected one, because a port bound on the wrong
chassis is exactly what a hosting-node-only dump cannot show.
Covers all three guests, not just the failing one, so a working guest is
available as the control.
* ci: sample hypervisor load either side of every cell
Eight runner names map onto three hypervisors, so cells that never share a
runner still share a host: when cell 10's SSH wait expired on run 34040324968,
cells 22, 17, 18 and 28 were concurrently live, two of them multi-node. Nothing
in the artifacts measured host load, which left co-tenancy as an explanation
that could be neither confirmed nor ruled out.
Records loadavg, MemAvailable, SwapFree, the cpu line carrying steal, and the
QEMU process count before and after each cell, into the cell artifact.
* ci: note that main's missing concurrency block is why the nightly runs ungrouped
Three refs carry three different groupings: main has no concurrency block at
all, dev uses e2e-nightly, this branch uses e2e-hypervisors. Scheduled runs load
the workflow from the default branch, so every cron nightly runs ungrouped and
cannot queue against anything — which is how a dispatch of 34050630627 ran
alongside the scheduled 34046316024 and cost it cells 24 and 27.
e2e-hypervisors is the right group and deliberately broader than dev's, since
e2e.yml, e2e-suites.yml and e2e-ddil.yml draw on the same runner pools. Nothing
on a feature branch can fix this; the block has to reach main.
* fix: stop the detach seal, EIP teardown and key-pair races failing the nightly
Four independent faults the full-suite nightly surfaced, each of which had
been masked while the heavy cells were skipped.
unmountVolume SIGKILLed nbdkit the moment the request arrived. Removing the
guest device has already started the plugin's own Close, which drains to the
backend and leaves a seal receipt, so the kill landed 61ms into that flush and
threw it away — the receipt path never fired once across a whole run. What was
left was 74 orphaned WAL files for the fallback seal to replay, which ran past
the caller's 180s deadline and returned DetachVolume ServerInternal with the
volume still attached. It now terminates gracefully first, bounded by the
plugin's own drain budget, and only escalates if that does not take.
DeleteEIP returned early when the dnat_and_snat row was already absent, before
the neighbour flush and the host unbind. An absent row is not a reassignment —
nobody owns the address — so the /32 route and proxy-ARP entry were orphaned
and a disassociate left the address still answering from the host. The
reassignment cases keep their early return.
EnsureKeyPair was not atomic: a failure after CreateKeyPair left the pair on
the cluster with no cleanup registered, so every later caller failed with
InvalidKeyPair.Duplicate in a test that never ran. The describe that follows a
create also had no tolerance for the propagation window on a multi-node
cluster; it now retries, and still reports a duplicate the moment it sees one.
An admission refusal could not name its cause. The static vCPU/memory budget
and the live MemAvailable clamp both surface as InsufficientInstanceCapacity,
and they have different fixes; the subscription transition now records which
gate closed.
* fix: identify the default-deny by priority, not by being a drop
DenyIsNotEthertypeScoped bucketed every drop ACL by direction, so the last one
seen won each direction. The platform WAN egress block added at priority 1049
is also a drop and is legitimately ip4-scoped, so it displaced the default-deny
in the map and the assertion fired on its qualifier instead:
from-lport deny on sg_c94870c0d68288dc6 is IPv4-scoped:
"inport == @sg_c94870c0d68288dc6 && ip4 && ip4.dst != {10.0.0.0/8, ...}"
Select the two default-denies by their own priorities. What the test asserts is
unchanged — the default-deny still must carry no ethertype qualifier, for the
reason denyIngressACL states: OVN allows on no-match, so an ip4-scoped deny
lets every other ethertype through.
* fix: drain every EIP on an ENI at teardown, not just the first
findByENI returned the first matching record and stopped, so an ENI holding
more than one EIP had all but one left marked associated to an interface being
deleted — with the host route still installed and no record left naming the ENI
to find them. Teardown now clears them all.
* ci: serialise e2e-nightly runs so two do not share the hypervisors
Only iso-single and iso-multi were guarded, by the iso-qemu-rack group. Every
other job could overlap a second run of the workflow, with both sets of cells
building clusters on the same five hosts. Queue rather than cancel: a dispatch
is usually a verification run whose result someone is waiting on.
* fix: point cell-24 stress work root at the runner nvme
The fault-injection harness ignores TMPDIR for large-object and
multipart-upload, defaulting them to $HOME/.cache/predastore-e2e because
TMPDIR is commonly tmpfs. On this runner $HOME is a 110G root disk with
14G free while TMPDIR is a 938G nvme, so the preflight guard passed on a
filesystem the scenarios that need disk never touch.
Set STRESS_WORK_ROOT alongside TMPDIR and check free space on both at a
threshold that covers the 1GiBx8 multipart case, so a run that would skip
a size for want of disk fails preflight instead of reporting a pass with
less coverage.
* fix: give cell-24 room for the full thirteen-scenario stress suite
The scenarios run serially and are ~35 minutes of work on their own,
measured from the per-scenario predastore matrix. The job allowed 30, so
it could not have passed whatever the disk did.
The matrix job gated all 32 of its steps on a step output. A job whose every
step is skipped still completes green, so a dispatch that ran one cell reported
as a nineteen-job pass and was indistinguishable in the UI from a full one. The
only tell was duration: a deselected cell finished in 11-12 seconds.
Every green run in the recent history is such a dispatch, and every run that
actually executed the suite has failed. There was no known-good baseline; there
was a display bug.
A job-level if cannot fix it — matrix is not available in that context, since
it is evaluated before the matrix expands. So the selection moves ahead of the
matrix: a select job filters the cell table and the matrix is built from what
comes back. A cell that was not selected produces no job at all, which is
plainer than skipped and cannot be read as a pass.
The table moves to .github/e2e-cells.json so both the workflow and the filter
read one copy.
* feat: block outbound SMTP egress via platform OVN ACL
Add a platform egress ACL that drops outbound TCP to 25/465/587 for public
destinations on every guest, matching AWS's default outbound-SMTP block. The
control is global by default with an operator-held per-VPC exemption; tenants
cannot open the ports via a security group because the block sits above the
tenant-allow priority band. Private destinations are always exempt.
Config exposes blocked_ports_wan (nil defaults to [25,465,587], [] disables)
and egress_block_exempt_vpcs. Adds unit tests for the ACL builder, the SG
manager wiring and the config resolution, plus an e2e that digs gmail.com MX
and confirms a 5s SMTP connect is blocked while other WAN egress works.
* docs: note the multi-node sync + restart requirement for the egress block
* feat: map virtio-blk virtqueues across an IOThread pool
Boot volumes ran every virtqueue through a single IOThread. Size a pool
from the vCPU count and hand QEMU an iothread-vq-mapping so the queues
spread across it. Falls back to the scalar iothread= form for MMIO
machines, single-queue devices and a pool of one.
SPINIFEX_BLK_IOTHREADS is experiment scaffolding for sweeping the pool
size without a rebuild, and comes out before this merges.
* feat: name QEMU guests with debug-threads=on
Config.Name held the instance ID but was never emitted, so QEMU left
every thread called qemu-system-x86 and a host-side pidstat -t could
not tell an IOThread from a vCPU thread. Emit -name guest=<id> with
debug-threads=on, which is what libvirt does for the same reason.
* refactor: drop the IOThread pool-size experiment override
SPINIFEX_BLK_IOTHREADS existed to sweep the pool size on dev-prod
without a rebuild and redeploy per value. The sweep is done, so the
derived size is the only path and buildDrives keeps its own doc comment.
The public, formation and DNS accept rules carried no address family, so
on a node that autoconfigures a global IPv6 from an upstream Router
Advertisement they served the console, predastore S3 gate, AWS gateway
and DNS on that public address with none of the source scoping applied on
IPv4. Qualify them with meta nfproto ipv4 so v6 falls to the default
drop; the scoped rules are already v4-only via ip saddr.
* refactor: extract DescribeInstanceStatus selection and projection
The request semantics — the instance-ID filter, filter parsing, the visibility
check, state membership defaulting to running-only, and the status filters that
can only run against a built frame — live inside the node handler's vmMgr.View
closure, so nothing else can apply them. A second caller answering the same
request from cached records would have to restate them, and the copy that
drifts is the one no customer request exercises until it does.
StatusSelection parses the request once; StatusFor applies it to one VM and
projects, returning nil when the request does not cover it. Per-VM rather than
per-slice so the node keeps reading VM fields inside its lock exactly as before.
buildInstanceStatus takes az rather than reading s.config.AZ, because a caller
projecting another node's record must use the AZ on that record. The node
passes its own config value, so behaviour is unchanged. systemImpaired stays a
parameter: it is node-local memory pressure only an answering node can report.
No behaviour change; the twenty existing status tests pass untouched.
* feat: read node liveness from the heartbeat store
A gateway answering from cached records has no way to tell a node that is
quietly healthy from one that has died, because an idle node writes no records
and record age measures activity rather than liveness. Liveness reads the
heartbeats the daemon already publishes and judges a node stale after three and
a half missed beats.
Unknown is a distinct answer from Stale and the two must not collapse. A node
nobody has ever heard from may simply never have published, and a store the
gateway cannot reach is not evidence that anything died — both report Unknown,
and a failed read clears the previous answers rather than keeping them.
The threshold is deliberately tighter than the IPsec barrier's six beats. That
one gates a cluster encryption decision where calling a live node dead churns a
security barrier; this one decides whether to admit the control plane cannot
confirm a host, where the costly error is the opposite one. The IPsec window is
left alone. A test in the daemon asserts the publish interval and the gateway's
copy of it have not drifted, since the gateway cannot import that package.
* feat: report impaired status for instances on a node that stopped answering
DescribeInstanceStatus fans out over NATS and returns whatever the nodes send
back. A node that is down sends nothing, so its instances disappear from the
answer entirely and a dead host reads as no host.
Backfill from the instance record cache: for any cached instance no node
answered for, synthesise a status frame and mark the system status from the
heartbeat store. A stale node reports impaired; an unreadable heartbeat store
reports insufficient-data, because not knowing a host is healthy is not knowing
it is not.
A frame an answering node returned is never touched. The covered set is what
guarantees that, and it is asserted directly rather than only through the
handler, where dedupStatuses being first-writer-wins makes either guard alone
sufficient. A live node's omission is its own answer and is not backfilled, and
a cache that has not finished its first whole-set sync contributes nothing.
The zero-value StatusSynthesis disables the whole path, so a gateway wired
without a cache answers exactly as it did before.
* fix: use fan-out responder identity, not heartbeat age, to gate status synthesis
DescribeInstanceStatus previously skipped synthesising an instance whenever
its owning node's heartbeat was still live, even when that node did not
answer this particular fan-out. A node that goes down loses its instances
from the response entirely until its heartbeat crosses the stale threshold.
Switch the Gather call to ExpectedResponders so it tracks per-node identity,
and use SuccessResponders to decide whether an omission was deliberate.
Heartbeat age now only selects the label (impaired vs insufficient-data),
never whether to synthesise. Falls back to the old heartbeat-only rule when
any frame cannot be attributed to a node.
* test: tag existing status-synthesis tests with node identity
answerWith never set X-Node-ID, so every existing test in
status_synthesis_test.go silently ran down the Unidentified fallback
branch instead of the responder-set path real daemons take. Give
answerWith a node-ID parameter (reusing subscribeAsNode) and retag each
call to match the record it is about, so these tests exercise the
success-responder guard they are named for. Keep exactly one test,
via a new answerAnonymously helper, on the fallback branch deliberately.
* test: tag DescribeInstanceStatus_test.go replies with node identity
Same rot as status_synthesis_test.go: raw nc.Subscribe replies carry no
X-Node-ID, so every describe here also sat on the 3s Gather deadline.
These tests mostly pass StatusSynthesis{}, so synthesize bails at
Records == nil before the fallback branch is ever reached — the payoff
is dropping the per-test timeout, not fixing a correctness gap.
Julian Sommer