mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
Ben Duncan
8ab96b29a1
* fix: replicate every JetStream KV bucket across the cluster A KV bucket created without a replica count gets one replica, and one replica lives on one JetStream-chosen server recorded in no config. Losing, restarting or partitioning that server makes the bucket return "no responders available" from every node at once, so a single-node event becomes a cluster-wide outage. The reconciler leader-lease buckets were the worst case: acquiring a lease is a write, so a lease bucket without a quorum cannot be acquired by anyone, and every reconciler sharing it stops cluster-wide at exactly the moment a node has failed. The cluster's node count is now declared once, in config.LoadConfig, and read by the one helper every bucket creation goes through. Every process that creates a bucket loads the cluster config first, so no service can forget to pass it and no signature has to carry it. An undeclared size is an error at creation rather than a silent fall back to one replica. Existing buckets are raised rather than left behind: a service opening one raises it, the daemon sweeps them all after formation, and "spx admin kv replicas --repair" does it on demand. Raising reads the live stream config and changes only the replica count, and never lowers one. Removes the ~30 replicas/clusterSize parameters that only forwarded a number, and JetStreamManager.UpdateReplicas, which set len(nodes) with no clamp and would have failed outright on a cluster of six or more. Proved by a repo-wide invariant test pinning bucket creation to the one helper, a whole-cluster sweep in every multi-node nightly cell, and a new kvquorum suite (cell 33) that takes away the node leading the most buckets and requires the survivors to keep answering for all of them. * chore: name cell 33 in the nightly dispatch input * fix: a KV bucket that cannot be raised must still open The first version of the replica fix treated a raise it could not place as a hard error, and on a real three-node cluster that turned the fix into the outage it prevents. A node whose config named a fourth server wanted four replicas, NATS answered "no suitable peers for placement" for three, and the daemon retried for its full five-minute budget without finishing bootstrap. The bucket was present, quorate and serving on three nodes throughout. A config naming a node before that node serves is the ordinary middle of growing a cluster, so that is a state to tolerate rather than refuse. The line is now drawn between the two cases instead of around both. Creating a bucket below the cluster's replica count still fails, because that is the defect and a fallback there reintroduces it where it is hardest to notice. Failing to raise an existing bucket warns and hands the bucket back, and the raise is retried on every open besides the daemon sweep and --repair. This matters most in kvlease, which runs on every reconcile tick: refusing there would stop every reconciler sharing a lease bucket that was working. getOrCreateBucket now attaches before it creates. A create asking for more replicas than the cluster can place is refused before the server looks at the name, so with create-first "this bucket already exists" was an inference from which error came back, and the wrong one comes back on a single-node server. RaiseAllBucketReplicas attempts every bucket and joins the failures, and --repair keeps going too. Stopping at the first would leave the rest of a cluster short because one bucket could not be placed. Also adds nats.ServerName/NodeFromServerName beside the template that renders server_name, with a test tying them together. The kvquorum suite had mapped stream peers to nodes by comparing them to node names, which they are not: they are the node name behind a "spinifex-nats-" prefix, so nothing matched and the node-loss test could not choose a victim. * test: wait for the right NATS peer count after a node rejoins /routez reports a node's routes to its peers, not its own membership, so a fully formed three-node cluster is two peers per node. The kvquorum suite waited for three and spent its whole five-minute rejoin budget on a cluster that had already reformed. Every other caller passes len(Nodes)-1; the helper's doc comment now says so. The assertion the test exists for had already passed: node1 was leading 18 of 47 buckets when it was taken away, and node2 and node3 each still answered for all 47. * fix: replicate KV buckets across the largest odd count a cluster can place A bucket was replicated across min(nodes, 5) servers, which stores a fourth full copy on a four-node cluster and still survives exactly one loss — the same tolerance R3 gives for three quarters of the write amplification. A Raft majority needs an odd count, so the rule becomes the largest odd count the cluster can place, capped at five: 1, 3, 3, 5, 5. It is one expression with two entry points, so no caller can hold a stale copy of the policy and keep passing after the policy changes. Three retry loops could not tell a configuration fault from a cluster that had yet to form. An undeclared cluster size never becomes declared by waiting, so each of them spent its whole budget on one and then reported quorum as the cause. They now stop on a permanent error and say what is actually wrong. This is what the integration suite was hitting: it builds its gateway by hand and never loads a cluster config, so the size was never declared and the IAM retry sat in its sleep until the test timed out. Both ends are fixed. The audit reported configuration alone. A bucket at three replicas with one peer offline and one lagging is one failure from unavailable and read as ok. BucketReport now carries the leader and a count of the peers JetStream itself calls current and reachable, quorum is measured against that count, and spx admin kv replicas exits 0, 1 or 2 so a deployment gate can tell a problem --repair fixes from one needing a human. Declaration is write-once: daemon.Start refuses a conflicting redeclaration because it is about to create buckets, and it declares before JetStream init rather than after it. config.LoadConfig warns instead, because it also reads config files that are not this process's own cluster. spx admin init --nodes 2 is refused — no replica count on two nodes both survives a loss and accepts writes — and single-node replication is asserted in the single e2e suite, so it runs in every nightly host permutation. * fix: withhold the vLLM serving image until its artifact is published The catalog offers ubuntu-26.04-vllm-serving-x86_64, and the qcow2 it names 404s on iso.mulgadc.com, so spx admin images import would fail for every operator who tried it. Its GPU siblings are published; this one never was. The entry moves to WithheldImages rather than being deleted, because the entry is the definition — the URL, the checksum and the two tags resolveServingAMI filters on. Publishing the artifact is then a move back, not a retype. Bedrock already handles the image not being registered: resolveServingAMI returns ErrServingAMINotFound, which is the honest answer while there is nothing to launch. The bedrock tag contract is still pinned, now against the map the definition lives in, and it additionally asserts the entry is not offered for import. The e2e reachability walk covers AvailableImages alone, so a withheld entry is out of it by construction. * fix: drop the cluster size from the call sites dev added Two constructors on dev still pass a replica count that no longer exists as a parameter: the cluster size is a declared process fact, so EKSServiceDeps has no ClusterSize field and NewIAMServiceImpl takes three arguments. The merge is textually clean and breaks the build, which is the shape of collision a signature removal always has. * docs: explain what an operator sees when a cluster server fails The replication section said how many copies a bucket has and stopped there, which leaves the question an operator actually has during an incident: one server is gone, what still works. It now covers the majority side carrying on, an isolated server refusing changes on purpose and why that is the behaviour you want, guests continuing to run while anything that has to be recorded does not, the risk of reading state from a server that is cut off, and a returning server catching up by itself. The surrounding prose reads from the operator's side rather than the implementation's: what you see and what to do, not which consensus protocol is underneath.
91 lines
4.4 KiB
Makefile
91 lines
4.4 KiB
Makefile
PKG ?= ./cert/... ./lb/... ./eks/... ./ecs/... ./iam/... ./ecr/... ./bedrock/... ./multinode/... ./baremetal/... ./reboot/... ./tofu/...
|
|
ARTIFACT_DIR ?= /tmp/spinifex-e2e-artifacts
|
|
TIMEOUT ?= 30m
|
|
MODE ?= single
|
|
# RUN restricts the run to tests matching a regex, for packages holding several
|
|
# independent long-running suites. Empty means every test in PKG, so this is a
|
|
# no-op unless asked for. Needed where one package's tests are not
|
|
# interchangeable: storagegrowth holds workloads that are each driven with their
|
|
# own environment and whose assertions are deliberately hostile, so running a
|
|
# sibling by accident reports a failure that says nothing about the run.
|
|
RUN ?=
|
|
# GOTESTSUM=1 emits JUnit XML + JSON streams for CI. Suite tag derives from the
|
|
# first path component of PKG so per-suite targets get distinct artifact names.
|
|
GOTESTSUM ?=
|
|
SUITE_TAG ?= $(notdir $(patsubst %/...,%,$(firstword $(PKG))))
|
|
|
|
# Pre-built test binaries. `e2e-build` cross-compiles each named suite into
|
|
# BIN_DIR/<suite>.test so CI can ship binaries (not source + go toolchain) to
|
|
# the remote VM.
|
|
BIN_DIR ?= $(CURDIR)/_bin
|
|
SUITES_BUILD ?= cert lb eks ecs single iam ecr bedrock gpu natuplink multinode storagegrowth partialblock rds quota diskperf storagefault instancerecovery lbrecovery kvquorum
|
|
|
|
.PHONY: e2e e2e-quota e2e-cert e2e-lb e2e-eks e2e-ecs e2e-single e2e-iam e2e-ecr e2e-bedrock e2e-multinode e2e-bm e2e-reboot e2e-tofu e2e-gpu e2e-natuplink e2e-storagegrowth e2e-partialblock e2e-diskperf e2e-rds e2e-storagefault e2e-instancerecovery e2e-lbrecovery e2e-kvquorum e2e-all clean e2e-build
|
|
|
|
RUN_FLAG = $(if $(RUN),-run '$(RUN)',)
|
|
|
|
ifeq ($(GOTESTSUM),1)
|
|
E2E_CMD = gotestsum \
|
|
--junitfile $(ARTIFACT_DIR)/junit-$(SUITE_TAG).xml \
|
|
--jsonfile $(ARTIFACT_DIR)/gotest-$(SUITE_TAG).json \
|
|
--format=testname \
|
|
-- -tags=e2e -timeout=$(TIMEOUT) $(RUN_FLAG) $(PKG)
|
|
else
|
|
E2E_CMD = go test -tags=e2e -v -timeout=$(TIMEOUT) $(RUN_FLAG) $(PKG)
|
|
endif
|
|
|
|
e2e:
|
|
@mkdir -p $(ARTIFACT_DIR)
|
|
SPINIFEX_E2E=1 \
|
|
SPINIFEX_MODE=$(MODE) \
|
|
ARTIFACT_DIR=$(ARTIFACT_DIR) \
|
|
$(E2E_CMD)
|
|
|
|
e2e-cert: ; $(MAKE) e2e PKG=./cert/...
|
|
e2e-lb: ; $(MAKE) e2e PKG=./lb/...
|
|
e2e-eks: ; $(MAKE) e2e PKG=./eks/...
|
|
e2e-ecs: ; $(MAKE) e2e PKG=./ecs/...
|
|
e2e-single: ; $(MAKE) e2e PKG=./single/...
|
|
e2e-iam: ; $(MAKE) e2e PKG=./iam/...
|
|
e2e-ecr: ; $(MAKE) e2e PKG=./ecr/...
|
|
e2e-bedrock: ; $(MAKE) e2e PKG=./bedrock/...
|
|
e2e-multinode: ; $(MAKE) e2e PKG=./multinode/... MODE=multinode
|
|
e2e-bm: ; $(MAKE) e2e PKG=./baremetal/... MODE=baremetal
|
|
e2e-reboot: ; $(MAKE) e2e PKG=./reboot/...
|
|
e2e-tofu: ; $(MAKE) e2e PKG=./tofu/...
|
|
e2e-gpu: ; $(MAKE) e2e PKG=./gpu/...
|
|
e2e-natuplink: ; $(MAKE) e2e PKG=./natuplink/...
|
|
e2e-storagegrowth: ; $(MAKE) e2e PKG=./storagegrowth/...
|
|
e2e-partialblock: ; $(MAKE) e2e PKG=./partialblock/...
|
|
# Two 16 GiB fio profiles per repetition, each on a freshly created volume, so
|
|
# a pass is ~20 minutes on the bare-metal cell. CI resolves the same budget
|
|
# from e2e_suite_timeout in .github/scripts/suite-sets.sh.
|
|
e2e-diskperf: ; $(MAKE) e2e PKG=./diskperf/... TIMEOUT=180m
|
|
# One instance's `available` wait is 23 minutes, so a single wedged create eats
|
|
# most of a 30m package timeout before the tests still running get to report.
|
|
e2e-rds: ; $(MAKE) e2e PKG=./rds/... TIMEOUT=45m
|
|
e2e-quota: ; $(MAKE) e2e PKG=./quota/...
|
|
# Freezes predastore, so its budget covers a deliberate outage plus the
|
|
# recovery and re-verification that follow it, twice over for the two tests.
|
|
e2e-storagefault: ; $(MAKE) e2e PKG=./storagefault/... TIMEOUT=45m
|
|
# Takes a node away twice and freezes predastore once. Almost all of its budget
|
|
# is waiting for staleness, a boot and a pause that is measured in minutes.
|
|
e2e-instancerecovery: ; $(MAKE) e2e PKG=./instancerecovery/... TIMEOUT=60m
|
|
# Stands a load balancer and its backends up before it breaks anything, then
|
|
# takes the load balancer's node away and brings it back.
|
|
e2e-lbrecovery: ; $(MAKE) e2e PKG=./lbrecovery/... TIMEOUT=75m
|
|
# Audits every KV bucket, then takes a node away and brings it back. Most of its
|
|
# budget is the node stopping and starting, not the assertions.
|
|
e2e-kvquorum: ; $(MAKE) e2e PKG=./kvquorum/... MODE=multinode TIMEOUT=40m
|
|
e2e-all: ; $(MAKE) e2e
|
|
|
|
e2e-build:
|
|
@mkdir -p $(BIN_DIR)
|
|
@for s in $(SUITES_BUILD); do \
|
|
echo "==> building $$s.test"; \
|
|
go test -c -tags=e2e -o $(BIN_DIR)/$$s.test ./$$s/ || exit 1; \
|
|
done
|
|
|
|
clean:
|
|
rm -rf $(ARTIFACT_DIR) $(BIN_DIR)
|