mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
Bryn Mailer
18aa67b0e9
* feat: add the e2e guest disk-performance gate Nothing measured guest disk performance in CI, which is why a quadratic index scan shipped in viperblock 35dd9f2 and survived three subsequent fixes to adjacent code. It is also why the two availability clauses at the end of the ebs-performance plan's Stage 0 have never been evaluated: a sustained random-write workload took the guest, and then the host, down, and nothing since has demonstrated that it no longer does. tests/e2e/diskperf/ launches an instance, attaches a dedicated blank volume per repetition, and drives a pinned fio profile over guest SSH. Two profiles: randwrite-4k at 4 jobs x 4 GiB, the shape the exit criterion names, and randrw-70-30-4k, the mixed workload the headline IOPS came from. The target is a raw device with direct=1 rather than a file on the boot disk, so guest page cache, filesystem allocation and journal traffic are out of the path. Two classes of assertion, kept apart because folding them together lets a run that left the host unreachable pass on a respectable IOPS figure. Availability is hard pass/fail and absolute, so it gates from the first run: the job completes inside its budget, a guest responsiveness probe answers throughout on a second SSH channel, and the host's own sshd answers throughout. The guest probe reads the root volume while the workload runs on a different one, so a pass is evidence of isolation between volumes rather than of the job merely being slow. The host probe is a TCP connect plus an SSH banner read, which needs no credential and still proves the kernel scheduled sshd. Throughput and latency are banded against a committed baseline: fail above 25% regression, warn above 10%, per-metric median across repetitions. baseline.json is committed and embedded, with every figure zeroed, and a zero reference records and warns rather than failing. Only the availability class gates until numbers are captured on the bare-metal cell; a throughput band nobody measured would be a fabricated gate. A fresh volume per repetition is the reset. Extent count grows monotonically, so reusing a volume makes the gate order-dependent and drifts its own numbers upward over time, slowly normalising the regression it exists to catch. Each job is its own guest invocation and fio's JSON is fetched by a separate command, so a wedged job loses only itself and reports as a budget breach rather than a parse failure. e2e-baremetal.yml gains suites, suite-env and cell inputs so one reusable workflow serves both the correctness cell and nightly cell 28, which queues behind cell 20 on the shared concurrency group. Cells 25-27 are the dedicated eks, ecs and rds cells. The per-suite Go test timeout is resolved from e2e_suite_timeout in suite-sets.sh, where diskperf declares its 180-minute budget beside rds's rather than in a second place. The gate's own arithmetic runs in preflight via test-harness: band edges, the unmeasured cases, per-stream metric selection, fio JSON parsing, and the probe accounting including a slow-but-successful answer counting as a breach. A gate whose comparison logic is untested is the failure mode the plan is otherwise full of. The extent-count ceiling named as a fourth availability assertion is not implemented: it needs viperblock.index.entries from S1.5, which does not exist. * feat: let a manual bare-metal run choose its e2e suites A dispatch run only sees inputs declared under workflow_dispatch, which had none, so suites and suite-env resolved empty and fell back to the correctness set. Capturing a diskperf baseline by hand was therefore unreachable without a workflow_call caller. * fix: build the bare-metal suites the run step will execute The build step packs pre-built binaries into the tarball and did not see the suite selection, so a selected suite was absent from the node unless it happened to be one of the three the build script hardcoded. * feat: commit the measured disk-performance baseline Medians of three repetitions on the bare-metal cell. The throughput and latency bands now gate instead of warning, so a 2x regression in either job fails the suite. randwrite-4k's write tail is two orders above the mixed job's. It is recorded as measured rather than zeroed: an ungated metric would miss the regression class the gate exists to catch.
87 lines
4.3 KiB
Bash
87 lines
4.3 KiB
Bash
# shellcheck shell=bash
|
|
# shellcheck disable=SC2034 # every variable here is consumed by the sourcing workflow
|
|
# Source from a workflow `run:` block:
|
|
# . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
|
|
#
|
|
# Which e2e suites each topology can run. Sourced by every workflow that drives
|
|
# suites (e2e.yml, e2e-suites.yml, e2e-nightly.yml) so the lists cannot drift
|
|
# apart — they previously disagreed three ways.
|
|
#
|
|
# Eligibility is a property of the suite, not of the workflow: a suite belongs
|
|
# to a topology when all of its assertions are reachable there. lb is
|
|
# multi-only for exactly that reason — on a single node every internet-facing
|
|
# subtest skips for want of a peer, so a single-node lb run reports green
|
|
# having exercised a fraction of the suite.
|
|
|
|
# Suites runnable against a one-node environment.
|
|
#
|
|
# partialblock gates merges because it earned it, not because it is new: on one
|
|
# node, same binary and workload, it fails against the pre-fix engine (1758 of
|
|
# 8192 halves reverted a generation) and passes against the fixed one (0 of
|
|
# 8192). A suite that has never been shown to go red for the reason it claims
|
|
# does not belong here.
|
|
# rds is single-only for the opposite reason to lb: v1 has no multi-AZ
|
|
# behaviour, so a second node adds no assertion. Its DB VMs are capped at four
|
|
# concurrent by the suite's own semaphore, which is what keeps it inside the
|
|
# job's 50-minute budget alongside the suites above.
|
|
# quota belongs to both topologies because the gate it asserts is gateway code
|
|
# that runs identically on either, and because a quota left switched on would
|
|
# cap every suite sharing the environment — so it must be proved to restore
|
|
# itself wherever it runs, not only on one node.
|
|
E2E_SUITES_SINGLE="single iam cert eks ecs storagegrowth partialblock rds quota"
|
|
|
|
# Suites runnable against a multi-node environment.
|
|
E2E_SUITES_MULTI="multinode lb cert quota"
|
|
|
|
# grep -xE alternation form, for narrowing a requested set to the eligible one.
|
|
E2E_SUITES_SINGLE_RE="$(printf '%s' "${E2E_SUITES_SINGLE}" | tr ' ' '|')"
|
|
E2E_SUITES_MULTI_RE="$(printf '%s' "${E2E_SUITES_MULTI}" | tr ' ' '|')"
|
|
|
|
# What every e2e-nightly permutation cell must prove, deliberately narrower
|
|
# than the two sets above: those cells have a 35-minute budget and exist to show
|
|
# that every install / network / host-OS combination boots and serves. The
|
|
# service suites are not deferred to the PR gate — eks, ecs and rds each run
|
|
# nightly in their own dedicated cell, because each needs a guest AMI built
|
|
# first and a service failure should not mask whether a host OS boots. These
|
|
# are subsets, and e2e_suites_assert_subset below enforces that.
|
|
E2E_SUITES_NIGHTLY_SINGLE="single cert iam"
|
|
E2E_SUITES_NIGHTLY_MULTI="multinode cert lb"
|
|
|
|
# How long a suite gets, for the same reason the lists above live here: the
|
|
# timeout was written out separately in each workflow and drifted, so rds ran
|
|
# 50m on the PR gate and 30m nightly and panicked there. A suite needing more
|
|
# than the default declares it once, here.
|
|
# e2e_suite_timeout <suite>
|
|
e2e_suite_timeout() {
|
|
case "$1" in
|
|
# rds serialises its DB VMs behind a 4 GiB semaphore, so its wall clock is
|
|
# set by how long it waits for the budget, not by how long a test runs.
|
|
rds) echo "50m" ;;
|
|
# diskperf runs two 16 GiB fio profiles per repetition, each on a volume it
|
|
# creates first. One pass is ~20 minutes on the bare-metal cell and a
|
|
# baseline capture at SPINIFEX_DISKPERF_REPS=3 is three times that.
|
|
diskperf) echo "180m" ;;
|
|
*) echo "30m" ;;
|
|
esac
|
|
}
|
|
|
|
# Fail loudly when a narrowed list names a suite its topology cannot run. This
|
|
# is the guard that stops the lists drifting again: a suite added to a nightly
|
|
# subset without also being declared eligible above stops the job here rather
|
|
# than silently skipping half its assertions in CI.
|
|
# e2e_suites_assert_subset <label> <subset> <eligible-set>
|
|
e2e_suites_assert_subset() {
|
|
local label="$1" subset="$2" eligible="$3" suite found eligible_suite
|
|
for suite in ${subset}; do
|
|
found=0
|
|
for eligible_suite in ${eligible}; do
|
|
if [ "${suite}" = "${eligible_suite}" ]; then found=1; break; fi
|
|
done
|
|
if [ "${found}" -ne 1 ]; then
|
|
echo "::error::${label} names suite '${suite}', which is not eligible for that topology (eligible: ${eligible}). Fix the list in .github/scripts/suite-sets.sh." >&2
|
|
return 1
|
|
fi
|
|
done
|
|
return 0
|
|
}
|