mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-20 11:21:49 +00:00
Bryn Mailer
18aa67b0e9
* feat: add the e2e guest disk-performance gate Nothing measured guest disk performance in CI, which is why a quadratic index scan shipped in viperblock 35dd9f2 and survived three subsequent fixes to adjacent code. It is also why the two availability clauses at the end of the ebs-performance plan's Stage 0 have never been evaluated: a sustained random-write workload took the guest, and then the host, down, and nothing since has demonstrated that it no longer does. tests/e2e/diskperf/ launches an instance, attaches a dedicated blank volume per repetition, and drives a pinned fio profile over guest SSH. Two profiles: randwrite-4k at 4 jobs x 4 GiB, the shape the exit criterion names, and randrw-70-30-4k, the mixed workload the headline IOPS came from. The target is a raw device with direct=1 rather than a file on the boot disk, so guest page cache, filesystem allocation and journal traffic are out of the path. Two classes of assertion, kept apart because folding them together lets a run that left the host unreachable pass on a respectable IOPS figure. Availability is hard pass/fail and absolute, so it gates from the first run: the job completes inside its budget, a guest responsiveness probe answers throughout on a second SSH channel, and the host's own sshd answers throughout. The guest probe reads the root volume while the workload runs on a different one, so a pass is evidence of isolation between volumes rather than of the job merely being slow. The host probe is a TCP connect plus an SSH banner read, which needs no credential and still proves the kernel scheduled sshd. Throughput and latency are banded against a committed baseline: fail above 25% regression, warn above 10%, per-metric median across repetitions. baseline.json is committed and embedded, with every figure zeroed, and a zero reference records and warns rather than failing. Only the availability class gates until numbers are captured on the bare-metal cell; a throughput band nobody measured would be a fabricated gate. A fresh volume per repetition is the reset. Extent count grows monotonically, so reusing a volume makes the gate order-dependent and drifts its own numbers upward over time, slowly normalising the regression it exists to catch. Each job is its own guest invocation and fio's JSON is fetched by a separate command, so a wedged job loses only itself and reports as a budget breach rather than a parse failure. e2e-baremetal.yml gains suites, suite-env and cell inputs so one reusable workflow serves both the correctness cell and nightly cell 28, which queues behind cell 20 on the shared concurrency group. Cells 25-27 are the dedicated eks, ecs and rds cells. The per-suite Go test timeout is resolved from e2e_suite_timeout in suite-sets.sh, where diskperf declares its 180-minute budget beside rds's rather than in a second place. The gate's own arithmetic runs in preflight via test-harness: band edges, the unmeasured cases, per-stream metric selection, fio JSON parsing, and the probe accounting including a slow-but-successful answer counting as a breach. A gate whose comparison logic is untested is the failure mode the plan is otherwise full of. The extent-count ceiling named as a fourth availability assertion is not implemented: it needs viperblock.index.entries from S1.5, which does not exist. * feat: let a manual bare-metal run choose its e2e suites A dispatch run only sees inputs declared under workflow_dispatch, which had none, so suites and suite-env resolved empty and fell back to the correctness set. Capturing a diskperf baseline by hand was therefore unreachable without a workflow_call caller. * fix: build the bare-metal suites the run step will execute The build step packs pre-built binaries into the tarball and did not see the suite selection, so a selected suite was absent from the node unless it happened to be one of the three the build script hardcoded. * feat: commit the measured disk-performance baseline Medians of three repetitions on the bare-metal cell. The throughput and latency bands now gate instead of warning, so a 2x regression in either job fails the suite. randwrite-4k's write tail is two orders above the mixed job's. It is recorded as measured rather than zeroed: an ungated metric would miss the regression class the gate exists to catch.
79 lines
3.5 KiB
Makefile
79 lines
3.5 KiB
Makefile
PKG ?= ./cert/... ./lb/... ./eks/... ./ecs/... ./iam/... ./ecr/... ./bedrock/... ./multinode/... ./baremetal/... ./reboot/... ./tofu/...
|
|
ARTIFACT_DIR ?= /tmp/spinifex-e2e-artifacts
|
|
TIMEOUT ?= 30m
|
|
MODE ?= single
|
|
# RUN restricts the run to tests matching a regex, for packages holding several
|
|
# independent long-running suites. Empty means every test in PKG, so this is a
|
|
# no-op unless asked for. Needed where one package's tests are not
|
|
# interchangeable: storagegrowth holds workloads that are each driven with their
|
|
# own environment and whose assertions are deliberately hostile, so running a
|
|
# sibling by accident reports a failure that says nothing about the run.
|
|
RUN ?=
|
|
# GOTESTSUM=1 emits JUnit XML + JSON streams for CI. Suite tag derives from the
|
|
# first path component of PKG so per-suite targets get distinct artifact names.
|
|
GOTESTSUM ?=
|
|
SUITE_TAG ?= $(notdir $(patsubst %/...,%,$(firstword $(PKG))))
|
|
|
|
# Pre-built test binaries. `e2e-build` cross-compiles each named suite into
|
|
# BIN_DIR/<suite>.test so CI can ship binaries (not source + go toolchain) to
|
|
# the remote VM.
|
|
BIN_DIR ?= $(CURDIR)/_bin
|
|
SUITES_BUILD ?= cert lb eks ecs single iam ecr bedrock gpu natuplink multinode storagegrowth partialblock rds quota diskperf
|
|
|
|
.PHONY: e2e e2e-quota e2e-cert e2e-lb e2e-eks e2e-ecs e2e-single e2e-iam e2e-ecr e2e-bedrock e2e-multinode e2e-bm e2e-reboot e2e-tofu e2e-gpu e2e-natuplink e2e-storagegrowth e2e-partialblock e2e-diskperf e2e-rds e2e-all clean e2e-build
|
|
|
|
RUN_FLAG = $(if $(RUN),-run '$(RUN)',)
|
|
|
|
ifeq ($(GOTESTSUM),1)
|
|
E2E_CMD = gotestsum \
|
|
--junitfile $(ARTIFACT_DIR)/junit-$(SUITE_TAG).xml \
|
|
--jsonfile $(ARTIFACT_DIR)/gotest-$(SUITE_TAG).json \
|
|
--format=testname \
|
|
-- -tags=e2e -timeout=$(TIMEOUT) $(RUN_FLAG) $(PKG)
|
|
else
|
|
E2E_CMD = go test -tags=e2e -v -timeout=$(TIMEOUT) $(RUN_FLAG) $(PKG)
|
|
endif
|
|
|
|
e2e:
|
|
@mkdir -p $(ARTIFACT_DIR)
|
|
SPINIFEX_E2E=1 \
|
|
SPINIFEX_MODE=$(MODE) \
|
|
ARTIFACT_DIR=$(ARTIFACT_DIR) \
|
|
$(E2E_CMD)
|
|
|
|
e2e-cert: ; $(MAKE) e2e PKG=./cert/...
|
|
e2e-lb: ; $(MAKE) e2e PKG=./lb/...
|
|
e2e-eks: ; $(MAKE) e2e PKG=./eks/...
|
|
e2e-ecs: ; $(MAKE) e2e PKG=./ecs/...
|
|
e2e-single: ; $(MAKE) e2e PKG=./single/...
|
|
e2e-iam: ; $(MAKE) e2e PKG=./iam/...
|
|
e2e-ecr: ; $(MAKE) e2e PKG=./ecr/...
|
|
e2e-bedrock: ; $(MAKE) e2e PKG=./bedrock/...
|
|
e2e-multinode: ; $(MAKE) e2e PKG=./multinode/... MODE=multinode
|
|
e2e-bm: ; $(MAKE) e2e PKG=./baremetal/... MODE=baremetal
|
|
e2e-reboot: ; $(MAKE) e2e PKG=./reboot/...
|
|
e2e-tofu: ; $(MAKE) e2e PKG=./tofu/...
|
|
e2e-gpu: ; $(MAKE) e2e PKG=./gpu/...
|
|
e2e-natuplink: ; $(MAKE) e2e PKG=./natuplink/...
|
|
e2e-storagegrowth: ; $(MAKE) e2e PKG=./storagegrowth/...
|
|
e2e-partialblock: ; $(MAKE) e2e PKG=./partialblock/...
|
|
# Two 16 GiB fio profiles per repetition, each on a freshly created volume, so
|
|
# a pass is ~20 minutes on the bare-metal cell. CI resolves the same budget
|
|
# from e2e_suite_timeout in .github/scripts/suite-sets.sh.
|
|
e2e-diskperf: ; $(MAKE) e2e PKG=./diskperf/... TIMEOUT=180m
|
|
# One instance's `available` wait is 23 minutes, so a single wedged create eats
|
|
# most of a 30m package timeout before the tests still running get to report.
|
|
e2e-rds: ; $(MAKE) e2e PKG=./rds/... TIMEOUT=45m
|
|
e2e-quota: ; $(MAKE) e2e PKG=./quota/...
|
|
e2e-all: ; $(MAKE) e2e
|
|
|
|
e2e-build:
|
|
@mkdir -p $(BIN_DIR)
|
|
@for s in $(SUITES_BUILD); do \
|
|
echo "==> building $$s.test"; \
|
|
go test -c -tags=e2e -o $(BIN_DIR)/$$s.test ./$$s/ || exit 1; \
|
|
done
|
|
|
|
clean:
|
|
rm -rf $(ARTIFACT_DIR) $(BIN_DIR)
|