mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
Josh Siviour
8e11908dfb
* fix: report CPU and memory pressure in the EKS control-plane health diagnostic An unhealthy control plane reported readyz, etcd reachability and etcd-disk free space. When etcd goes unreachable with disk:ok and no OOM kill, that set cannot distinguish a starved guest from a genuine etcd fault, so add the loadavg and MemAvailable that separate them. An empty /readyz body also rendered as readyz:[none], which reads as "no subcheck failed" when it means the probe never answered. Report that as unreachable instead. * fix: sample the node's load, not just the runner host's Every runner agent runs on one host while their nodes sit on several, so the existing sample measured the runner's host and was labelled as the hypervisor. It happens to coincide when the node is on that same host and misleads otherwise. Relabel it and add a node-side sample, which is where the guests are actually scheduled and so where a starved guest shows up. * feat: build the mkosi EKS image in CI and give its control plane telemetry The Ubuntu/systemd EKS image is what customers run, and nothing built it: the composition existed in mkosi-build.sh's image_profiles table but had no caller, so a change that broke it was found by hand or not at all. Cell-25 builds the Alpine image, which leaves the published one with no gate. Add a Makefile target beside the Alpine one and a job in the existing path-filtered image workflow. The guest also collected nothing. k3s has served embedded-etcd metrics on 127.0.0.1:2381 since the daemon started setting etcd-expose-metrics, and no collector ever read them, so an etcd stall was only ever visible as "etcd:unreachable" in the state report — which cannot separate a slow disk from a dead one. Ship otelcol-contrib pinned to the version the cluster hosts already run, scraping etcd and guest host metrics. The unit is gated on ConditionPathExists for its endpoint file rather than started unconditionally: whether the control-plane guest can reach the sink from the isolated CP VPC is unverified, and a collector with nowhere to export is a restart loop, not telemetry. * fix: report etcd fsync latency in the control-plane health diagnostic disk:ok measures free space, which says nothing about how long a write takes. etcd stalls on fsync latency long before it runs out of room, and this guest's disk is an EBS volume served by viperblock, so a fsync is a network round trip. An unhealthy control plane reporting healthy memory and disk was uninterpretable without it. Derived from the histogram's sum and count rather than a bucket percentile: two greps against an interpolation this script has no business doing, and the difference between a healthy single-digit reading and a stalled one is an order of magnitude, not a tail. Rides the state report's existing gateway path, so it needs no route the CP VPC does not already grant, and lands on both images because the script is shared. The scrape's failure is guarded explicitly. Under set -e a bare assignment from a curl against a port nothing answers aborts the whole diagnosis, dropping the reason field exactly when it is wanted. * feat: publish etcd fsync on every control-plane report, not only failing ones A figure that appears only once the apiserver is already unhealthy has no baseline to be read against, which is the same gap a passing cell that uploads no journal leaves: the number is unanchored when it finally arrives. Move the scrape into its own helper, emit it on healthy reports too, and carry it as a typed field rather than only inside the reason string. Slowness is logged, never gated. observe() decides cluster health, and a working cluster on a slow disk is working — returning an issue there would fail customers' clusters on a latency threshold. The warning is latched on the transition instead, because the control plane republishes on its own timer and a line every interval is how a real signal gets tuned out. An absent sample is not a fast one. Too few fsyncs to average, an etcd serving no metrics and an older AMI all arrive as nil, and the field is omitted rather than defaulted so none of them can be read as healthy. The minimum sample count is the same lesson as the slow-shard floor: a mean over three fsyncs on a just-started etcd is a number with no meaning. * fix: log an etcd fsync baseline and name the two no-sample cases apart A healthy cluster logs no health lines, so the fsync figure reached the daemon on every report and was never surfaced — a passing cell left nothing for a later slow reading to be compared against. Log the first known sample. Split the "unknown" sentinel into nometrics (etcd served none) and warmup (too few fsyncs to average): different faults, different fixes. The payload guard now tests for a number rather than the sentinel word, so a future sentinel cannot leak into the JSON as a bare value. * feat: measure fdatasync latency in the diskperf gate The two pinned profiles are buffered-bypassing throughput jobs at queue depth 32. Neither measures the cost of making a write durable, which is what a consensus datastore pays per raft entry and stalls on long before it runs short of bandwidth. A 5ms barrier on bare metal went unseen as a result. syncwrite-4k is the etcd-shaped profile: sequential 4k, queue depth 1, fdatasync every write, buffered psync so the sync has a page cache to flush. fio reports the sync distribution apart from the write stream, so the parser reads it separately — gating on write clat would score the time to reach the cache and call it durability. No figures captured on this hardware yet, so the baseline entry is zero and the job records without gating. * fix: use the real otelcol checksums asset name in the EKS server image The checksums file is named for the release repo and the distribution and carries no version, unlike the tarball beside it. The guessed name 404'd and took the image build with it. * fix: extract otelcol as root in the EKS server image The upstream tarball records its build uid (runner/docker, 1001:118), which does not exist in the image. tar running as root tries to restore it and fails outright rather than degrading, taking the build with it. --------- Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>
76 lines
2.4 KiB
YAML
76 lines
2.4 KiB
YAML
name: Image Scripts
|
|
|
|
# Separate from spinifex.yml on purpose: scripts/images/ and images/ are shell
|
|
# and mkosi assets baked into system images, not Go, so they get their own
|
|
# path-filtered gate instead of running on every Go contributor's push (see
|
|
# ui.yml for the same pattern applied to the frontend).
|
|
on:
|
|
push:
|
|
branches:
|
|
- main
|
|
- dev
|
|
paths:
|
|
- scripts/images/**
|
|
- images/**
|
|
- .github/workflows/image-scripts.yml
|
|
pull_request:
|
|
branches:
|
|
- main
|
|
- dev
|
|
paths:
|
|
- scripts/images/**
|
|
- images/**
|
|
- .github/workflows/image-scripts.yml
|
|
workflow_dispatch:
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
image_scripts:
|
|
name: Image Scripts
|
|
runs-on: ubuntu-26.04
|
|
|
|
steps:
|
|
- name: Checkout Code
|
|
uses: actions/checkout@v7
|
|
|
|
- name: Run image script tests + shellcheck
|
|
run: make test-images
|
|
|
|
# The mkosi EKS image is what customers run, and until now nothing built it:
|
|
# the composition existed in mkosi-build.sh but had no caller, so a change
|
|
# that broke it was only found by hand. Cell-25 still boots the Alpine image,
|
|
# so this is the only gate the Ubuntu one has.
|
|
#
|
|
# Self-hosted because the build needs Docker and several GB of disk, matching
|
|
# publish-system-ami.yml. Its own concurrency group, not e2e-hypervisors: the
|
|
# build provisions no libvirt guest, and sharing that group would queue image
|
|
# builds behind whole nightlies.
|
|
eks_mkosi_image:
|
|
name: Build mkosi EKS image
|
|
runs-on: [self-hosted, ci-single]
|
|
concurrency:
|
|
group: eks-mkosi-image-${{ github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
steps:
|
|
- name: Checkout Code
|
|
uses: actions/checkout@v7
|
|
|
|
# The profile binaries (eks-token-webhook, eks-gateway-publish, ...) are
|
|
# built on the host, not in the builder container, so Go is needed here
|
|
# even though the toolchain image carries mkosi itself.
|
|
- name: Set up Go
|
|
uses: actions/setup-go@v7
|
|
with:
|
|
go-version-file: "go.mod"
|
|
# Self-hosted: the module + build cache already persists on local
|
|
# disk, and cache-save would re-upload hundreds of MB for nothing.
|
|
cache: false
|
|
|
|
# stage_profile_binaries() sets GOWORK=off, so the sub-repo resolves its
|
|
# own go.mod and the sibling monorepo checkouts are not needed here.
|
|
- name: Build the image
|
|
run: make build-eks-node-mkosi-image
|