mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 09:01:31 +00:00
Josh Siviour
8e11908dfb
* fix: report CPU and memory pressure in the EKS control-plane health diagnostic An unhealthy control plane reported readyz, etcd reachability and etcd-disk free space. When etcd goes unreachable with disk:ok and no OOM kill, that set cannot distinguish a starved guest from a genuine etcd fault, so add the loadavg and MemAvailable that separate them. An empty /readyz body also rendered as readyz:[none], which reads as "no subcheck failed" when it means the probe never answered. Report that as unreachable instead. * fix: sample the node's load, not just the runner host's Every runner agent runs on one host while their nodes sit on several, so the existing sample measured the runner's host and was labelled as the hypervisor. It happens to coincide when the node is on that same host and misleads otherwise. Relabel it and add a node-side sample, which is where the guests are actually scheduled and so where a starved guest shows up. * feat: build the mkosi EKS image in CI and give its control plane telemetry The Ubuntu/systemd EKS image is what customers run, and nothing built it: the composition existed in mkosi-build.sh's image_profiles table but had no caller, so a change that broke it was found by hand or not at all. Cell-25 builds the Alpine image, which leaves the published one with no gate. Add a Makefile target beside the Alpine one and a job in the existing path-filtered image workflow. The guest also collected nothing. k3s has served embedded-etcd metrics on 127.0.0.1:2381 since the daemon started setting etcd-expose-metrics, and no collector ever read them, so an etcd stall was only ever visible as "etcd:unreachable" in the state report — which cannot separate a slow disk from a dead one. Ship otelcol-contrib pinned to the version the cluster hosts already run, scraping etcd and guest host metrics. The unit is gated on ConditionPathExists for its endpoint file rather than started unconditionally: whether the control-plane guest can reach the sink from the isolated CP VPC is unverified, and a collector with nowhere to export is a restart loop, not telemetry. * fix: report etcd fsync latency in the control-plane health diagnostic disk:ok measures free space, which says nothing about how long a write takes. etcd stalls on fsync latency long before it runs out of room, and this guest's disk is an EBS volume served by viperblock, so a fsync is a network round trip. An unhealthy control plane reporting healthy memory and disk was uninterpretable without it. Derived from the histogram's sum and count rather than a bucket percentile: two greps against an interpolation this script has no business doing, and the difference between a healthy single-digit reading and a stalled one is an order of magnitude, not a tail. Rides the state report's existing gateway path, so it needs no route the CP VPC does not already grant, and lands on both images because the script is shared. The scrape's failure is guarded explicitly. Under set -e a bare assignment from a curl against a port nothing answers aborts the whole diagnosis, dropping the reason field exactly when it is wanted. * feat: publish etcd fsync on every control-plane report, not only failing ones A figure that appears only once the apiserver is already unhealthy has no baseline to be read against, which is the same gap a passing cell that uploads no journal leaves: the number is unanchored when it finally arrives. Move the scrape into its own helper, emit it on healthy reports too, and carry it as a typed field rather than only inside the reason string. Slowness is logged, never gated. observe() decides cluster health, and a working cluster on a slow disk is working — returning an issue there would fail customers' clusters on a latency threshold. The warning is latched on the transition instead, because the control plane republishes on its own timer and a line every interval is how a real signal gets tuned out. An absent sample is not a fast one. Too few fsyncs to average, an etcd serving no metrics and an older AMI all arrive as nil, and the field is omitted rather than defaulted so none of them can be read as healthy. The minimum sample count is the same lesson as the slow-shard floor: a mean over three fsyncs on a just-started etcd is a number with no meaning. * fix: log an etcd fsync baseline and name the two no-sample cases apart A healthy cluster logs no health lines, so the fsync figure reached the daemon on every report and was never surfaced — a passing cell left nothing for a later slow reading to be compared against. Log the first known sample. Split the "unknown" sentinel into nometrics (etcd served none) and warmup (too few fsyncs to average): different faults, different fixes. The payload guard now tests for a number rather than the sentinel word, so a future sentinel cannot leak into the JSON as a bare value. * feat: measure fdatasync latency in the diskperf gate The two pinned profiles are buffered-bypassing throughput jobs at queue depth 32. Neither measures the cost of making a write durable, which is what a consensus datastore pays per raft entry and stalls on long before it runs short of bandwidth. A 5ms barrier on bare metal went unseen as a result. syncwrite-4k is the etcd-shaped profile: sequential 4k, queue depth 1, fdatasync every write, buffered psync so the sync has a page cache to flush. fio reports the sync distribution apart from the write stream, so the parser reads it separately — gating on write clat would score the time to reach the cache and call it durability. No figures captured on this hardware yet, so the baseline entry is zero and the job records without gating. * fix: use the real otelcol checksums asset name in the EKS server image The checksums file is named for the release repo and the distribution and carries no version, unlike the tarball beside it. The guessed name 404'd and took the image build with it. * fix: extract otelcol as root in the EKS server image The upstream tarball records its build uid (runner/docker, 1001:118), which does not exist in the image. tar running as root tries to restore it and fails outright rather than degrading, taking the build with it. --------- Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>
Spinifex Scripts
Node installation, dev-environment lifecycle, guest image builds, and verification tooling for Spinifex.
Install and node setup
| Script | What it does |
|---|---|
setup.sh |
The production binary installer, served from install.mulgadc.com. Installs deps, users, sudoers, files, systemd units, logrotate and udev rules. SETUP_STAGES runs a subset; ISO_BUILD=1 runs it inside the ISO builder's chroot. |
setup-ovn.sh |
Bootstraps a compute node for OVN VPC networking: installs OVN/OVS, creates br-int, configures the WAN bridge (bridged, DHCP, or routed-NAT uplink), sets the chassis identity, and applies overlay sysctls. --management also starts the OVN central services. |
install-node.sh |
Forms a multi-node cluster across hosts that are already running Spinifex: rebuilds the OVN database in clustered form, runs spx admin init on the first host and spx admin join on the rest, then verifies. The scripted form of install-multi-node. Destroys the joining nodes' logical network state and crypto — see its --help. |
dev-install.sh |
Full local dev setup through the production installer: builds from source, packs a tarball, runs setup.sh, initialises the cluster, and starts services under systemd. |
clone-deps.sh |
Clones or updates the viperblock and predastore checkouts alongside spinifex for cross-repo development. |
Dev environment
| Script | What it does |
|---|---|
reset-dev-env.sh |
Single-node reset: stops services, wipes /etc/, /var/lib/, /var/log/ and /run/spinifex, rebuilds, reinstalls via dev-install.sh, and relaunches a smoke-test instance. Preserves the existing node/network topology from spinifex.toml. |
dev-env/*.conf |
Topology profiles for reset-dev-env.sh --profile=NAME, declaring which bridge carries the wan, lan and vpc planes: single-nic (all collapsed onto wan), two-nic (vpc folds onto lan), three-plane (a NIC each). |
Verification and benchmarks
| Script | What it does |
|---|---|
smoke-test.sh |
End-to-end sanity check on a running node: imports an SSH keypair and the Ubuntu AMI, then launches an instance. --create-vpc builds the VPC, gateway, subnet and routes first and tears them down after; --nodes N launches through a spread placement group so each instance lands on a different physical server. TEST_GPU=1 adds GPU passthrough checks. |
verify-network.sh |
Asserts a node's datapath matches its declared planes — EIP reachability, which plane carries Geneve, and which interfaces services bind. Catches what smoke-test.sh cannot, since none of its traffic leaves the box. |
test-ddil-phase1.sh |
systemd-level smoke test for daemon-local-autonomy tier 1. Destructive on the local node: stops and starts spinifex-nats and restarts spinifex-daemon, restoring both via a cleanup trap. |
check-coverage.sh |
Fails the build if total Go coverage falls below the minimum threshold, excluding packages exempt from the requirement. |
diff-coverage.sh |
Coverage check scoped to changed lines only, against an auto-detected base ref (HEAD~1 on main, origin/main on dev, otherwise origin/dev). |
run-bench.sh |
Launches a benchmark against running Ubuntu instances, auto-detecting the AMI for the host architecture. Tracks nbdkit and perf alongside the run. |
disk-performance.sh |
Runs a fio random 70/30 read-write benchmark swept across 4k/16k/128k/1M. Writes JSON results to /tmp/spinifex-disk-bench; override with BENCH_DIR, OUT_DIR, SIZE, JOBS, BLOCK_SIZES. |
network-performance.sh |
iperf throughput from several clients to one server, driven over SSH, writing a summary.txt of Gbit/s per client. --server-ip sets the address clients dial, so pointing it at instance private addresses measures the VPC overlay rather than the external pool. |
workload-performance.sh |
In-guest memory, disk, CPU and network workload test. Installs Go and runs the Badger test suite with jemalloc disabled and reduced parallelism. |
Guest images
| Script | What it does |
|---|---|
build-system-image.sh |
Builds a minimal system AMI (Alpine or Ubuntu) from a manifest. All customisation happens inside the libguestfs appliance, so the build touches no host block device, no host mount, and needs no sudo. --import also registers it as an AMI. |
publish-system-image.sh |
Converts a built raw image to compressed qcow2, writes the .sha256 sidecar the catalog verifier expects, and uploads both to the R2 bucket behind iso.mulgadc.com/system-ami/. |
build-microvm-image.sh |
Builds the minimal Alpine kernel + initramfs used by the QEMU microvm machine type and the lb-agent, carrying haproxy (ALB) and nginx-stream (NLB) data planes. |
mkosi-build.sh |
Runs an mkosi system-image build inside the pinned builder container, so a runner needs only Docker and the toolchain cannot drift. --image names an output and expands to the right profile list. |
mkosi-publish.sh |
Publishes an mkosi-built image to R2 from inside the same pinned container — qcow2 conversion, .sha256 sidecar, and upload. |
mkosi-common.sh |
Shared plumbing sourced by both mkosi scripts. Builds the toolchain image once, with the builder account matched to the invoking user so bind-mounted output is not written back as root. |
mkosi-builder.Dockerfile |
The pinned image-build toolchain. Debian-hosted regardless of target distro, since mkosi bootstraps the target from its own archive. |
images/ — per-image build inputs
Each subdirectory holds a manifest.conf, init scripts, and a setup.sh that build-system-image.sh runs inside the libguestfs appliance after packages and files are placed. *_test.sh files are shell unit tests for the helper of the same name.
| Path | What it builds |
|---|---|
ecs-agent/ |
The Alpine spinifex-ecs-node container-instance AMI (containerd + ecs-agent). setup.sh sets OpenRC exec bits, wires the CNI bin path, and applies the serial-console tweaks. |
ecs-agent-gpu/ |
The Ubuntu ECS GPU container-instance AMI. Same CNI and containerd wiring adjusted for Ubuntu's package layout, plus the headless NVIDIA driver and nvidia-container-toolkit. |
eks-node/ |
The unified Alpine EKS node AMI (K3s server or agent, selected at first boot). setup.sh installs the pinned, SHA-verified K3s binary and the role init scripts. |
eks-node-gpu/ |
The Ubuntu EKS GPU worker AMI. Agent role only — GPU nodes never run the control plane — plus the NVIDIA GPU stack. |
rds-postgres/ |
The spinifex-rds-postgres AMI. setup.sh sets exec bits on the OpenRC services and creates the root-only directory cloud-init drops the agent env file and gateway CA into. |
rds-mariadb/ |
The spinifex-rds-mariadb AMI. Same layout as rds-postgres/, plus the /etc/my.cnf.d include drop-in and the conf.d/mariadb dependency that keeps the packaged service off an unbootstrapped datadir. |
ubuntu-gpu-{amd,nvidia}-setup.sh |
Chroot setup for GPU-capable Ubuntu guest images: ROCm CLI tools and AMD firmware, or the DKMS-built headless NVIDIA driver. Guest-side only — host VFIO setup is spx admin gpu setup. |
images/eks-node/ — in-guest helpers
Installed into the EKS node AMI and run at boot or on a timer. All host communication goes through the AWS gateway over SigV4; the VM never speaks NATS directly.
| Script | What it does |
|---|---|
eks-node-role.sh |
First-boot role selector. Reads SPINIFEX_K3S_ROLE from the cloud-init env file and enables the matching services — nothing is enabled at bake time. |
k3s-prestart.sh |
Server prestart: refuses to start on a restore-block marker, derives config.yaml from its skeleton, stages the konnectivity-agent and CoreDNS manifests, and waits for the token-webhook kubeconfig. |
k3s-agent-prestart.sh |
Agent prestart shared by the OpenRC and systemd units. Requires K3S_URL and K3S_TOKEN, then resolves the kubelet provider-id from IMDSv2 for EKS parity. |
k3s-first-boot.sh |
Runs once after the server is healthy: reads the node-token and admin kubeconfig, rewrites the kubeconfig server address to the cluster NLB endpoint, and publishes both to the host. |
konnectivity-server-prestart.sh |
Waits for the K3s CA and admin kubeconfig, mints the konnectivity serving cert, and writes the apiserver replica count for the caller to source. |
mulga-eks-addon-sync.sh |
Pulls staged managed-addon descriptors through the gateway, renders each bundle into the K3s auto-deploy dir, and reports per-addon delivery status back. |
mulga-eks-etcd-snapshot.sh |
Snapshots K3s embedded etcd to predastore. Installed into two crond periodic dirs and picks its tier — frequent or daily — from the dir it was invoked from. |
mulga-eks-k3s-recovery.sh |
Applies a control-plane etcd-quorum recovery directive before k3s starts. Pull-based and a no-op unless the reconciler escalated a wedged control plane. |
mulga-eks-state-report.sh |
Publishes server health (healthz, node count) to the cluster reconciler. The apiserver binds the VPC node-ip, which the host daemon cannot probe directly. |
mulga-eks-provider-id.sh |
Writes the K3s config drop-in supplying the metadata the EBS CSI driver needs from IMDS: provider-id plus the region, zone and instance-type labels. |
mulga-ebs-byid.sh |
Mints the EBS-style /dev/disk/by-id symlink the upstream EBS CSI node plugin resolves, from the virtio-blk serial. Alpine's busybox mdev has no eudev to do it. |
mulga-mgmt-net.sh |
Boot oneshot bringing a multi-NIC system VM's interfaces up by MAC from the QEMU fw_cfg netcfg blob, before cloud-init's network stage. cloud-init cannot reliably pick the right NIC out of two. |
mulga-vpc-mtu.sh |
Boot oneshot pinning the VPC data NIC MTU so flannel sizes its VXLAN overlay to fit the OVN geneve underlay. Otherwise large frames are silently dropped while handshakes pass. |
Launch helpers
| Script | What it does |
|---|---|
launch-gpu.sh |
Launches a GPU instance from the AMD or NVIDIA GPU AMI, selecting the AMI from the instance type's GPU brand. --amd / --nvidia override the detection. |
ecs-node-bringup.sh |
Launches an ECS container-instance VM. Renders the cloud-init user-data seeding the agent's control-plane config, IAM credentials and gateway CA — there is no ECS-specific launch API, by design. |
demo/ — GPU demo
| Script | What it does |
|---|---|
deploy.sh |
Deploys and starts the four-instance GPU demo (dashboard, YOLO, Ollama, qwen3-vl) from the *_IP and VIDEO_PATH env vars. |
dashboard_server.py |
Serves the demo web UI and proxies the streams from the GPU instances. |
yolo_stream.py |
Runs YOLO detection over a video file, serving an MJPEG stream of annotated frames and an SSE stream of qwen3-vl scene descriptions. |