mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 09:01:31 +00:00
fix: make the EKS control plane's disk latency observable, and give it room to run (#1009)
* fix: report CPU and memory pressure in the EKS control-plane health diagnostic An unhealthy control plane reported readyz, etcd reachability and etcd-disk free space. When etcd goes unreachable with disk:ok and no OOM kill, that set cannot distinguish a starved guest from a genuine etcd fault, so add the loadavg and MemAvailable that separate them. An empty /readyz body also rendered as readyz:[none], which reads as "no subcheck failed" when it means the probe never answered. Report that as unreachable instead. * fix: sample the node's load, not just the runner host's Every runner agent runs on one host while their nodes sit on several, so the existing sample measured the runner's host and was labelled as the hypervisor. It happens to coincide when the node is on that same host and misleads otherwise. Relabel it and add a node-side sample, which is where the guests are actually scheduled and so where a starved guest shows up. * feat: build the mkosi EKS image in CI and give its control plane telemetry The Ubuntu/systemd EKS image is what customers run, and nothing built it: the composition existed in mkosi-build.sh's image_profiles table but had no caller, so a change that broke it was found by hand or not at all. Cell-25 builds the Alpine image, which leaves the published one with no gate. Add a Makefile target beside the Alpine one and a job in the existing path-filtered image workflow. The guest also collected nothing. k3s has served embedded-etcd metrics on 127.0.0.1:2381 since the daemon started setting etcd-expose-metrics, and no collector ever read them, so an etcd stall was only ever visible as "etcd:unreachable" in the state report — which cannot separate a slow disk from a dead one. Ship otelcol-contrib pinned to the version the cluster hosts already run, scraping etcd and guest host metrics. The unit is gated on ConditionPathExists for its endpoint file rather than started unconditionally: whether the control-plane guest can reach the sink from the isolated CP VPC is unverified, and a collector with nowhere to export is a restart loop, not telemetry. * fix: report etcd fsync latency in the control-plane health diagnostic disk:ok measures free space, which says nothing about how long a write takes. etcd stalls on fsync latency long before it runs out of room, and this guest's disk is an EBS volume served by viperblock, so a fsync is a network round trip. An unhealthy control plane reporting healthy memory and disk was uninterpretable without it. Derived from the histogram's sum and count rather than a bucket percentile: two greps against an interpolation this script has no business doing, and the difference between a healthy single-digit reading and a stalled one is an order of magnitude, not a tail. Rides the state report's existing gateway path, so it needs no route the CP VPC does not already grant, and lands on both images because the script is shared. The scrape's failure is guarded explicitly. Under set -e a bare assignment from a curl against a port nothing answers aborts the whole diagnosis, dropping the reason field exactly when it is wanted. * feat: publish etcd fsync on every control-plane report, not only failing ones A figure that appears only once the apiserver is already unhealthy has no baseline to be read against, which is the same gap a passing cell that uploads no journal leaves: the number is unanchored when it finally arrives. Move the scrape into its own helper, emit it on healthy reports too, and carry it as a typed field rather than only inside the reason string. Slowness is logged, never gated. observe() decides cluster health, and a working cluster on a slow disk is working — returning an issue there would fail customers' clusters on a latency threshold. The warning is latched on the transition instead, because the control plane republishes on its own timer and a line every interval is how a real signal gets tuned out. An absent sample is not a fast one. Too few fsyncs to average, an etcd serving no metrics and an older AMI all arrive as nil, and the field is omitted rather than defaulted so none of them can be read as healthy. The minimum sample count is the same lesson as the slow-shard floor: a mean over three fsyncs on a just-started etcd is a number with no meaning. * fix: log an etcd fsync baseline and name the two no-sample cases apart A healthy cluster logs no health lines, so the fsync figure reached the daemon on every report and was never surfaced — a passing cell left nothing for a later slow reading to be compared against. Log the first known sample. Split the "unknown" sentinel into nometrics (etcd served none) and warmup (too few fsyncs to average): different faults, different fixes. The payload guard now tests for a number rather than the sentinel word, so a future sentinel cannot leak into the JSON as a bare value. * feat: measure fdatasync latency in the diskperf gate The two pinned profiles are buffered-bypassing throughput jobs at queue depth 32. Neither measures the cost of making a write durable, which is what a consensus datastore pays per raft entry and stalls on long before it runs short of bandwidth. A 5ms barrier on bare metal went unseen as a result. syncwrite-4k is the etcd-shaped profile: sequential 4k, queue depth 1, fdatasync every write, buffered psync so the sync has a page cache to flush. fio reports the sync distribution apart from the write stream, so the parser reads it separately — gating on write clat would score the time to reach the cache and call it durability. No figures captured on this hardware yet, so the baseline entry is zero and the job records without gating. * fix: use the real otelcol checksums asset name in the EKS server image The checksums file is named for the release repo and the distribution and carries no version, unlike the tarball beside it. The guessed name 404'd and took the image build with it. * fix: extract otelcol as root in the EKS server image The upstream tarball records its build uid (runner/docker, 1001:118), which does not exist in the image. tar running as root tries to restore it and fails outright rather than degrading, taking the build with it. --------- Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>
This commit is contained in:
Josh Siviour
@@ -42,13 +42,19 @@ DISK_MIN_KB=${DISK_MIN_KB:-262144}
|
||||
|
||||
# diagnose emits a compact, JSON-safe reason for an unhealthy apiserver by
|
||||
# reading the ground truth only reachable from inside the guest: the failing
|
||||
# /readyz subchecks, isolated etcd reachability, and the etcd-disk free space.
|
||||
# /readyz subchecks, etcd reachability, etcd-disk free space, and host pressure.
|
||||
diagnose() {
|
||||
# apiserver prints "[+]name ok" for passing subchecks and "[-]name failed"
|
||||
# for failing ones; collect the failing names (etcd, poststarthook/*, ...).
|
||||
failed=$(kubectl get --raw='/readyz?verbose' 2>/dev/null \
|
||||
| awk '/^\[-\]/{sub(/^\[-\]/,""); sub(/[[:space:]].*$/,""); printf "%s%s", sep, $0; sep=" "}')
|
||||
[ -n "${failed}" ] || failed=none
|
||||
# for failing ones. An empty body means the probe never answered, which is a
|
||||
# different fault from "every subcheck passed" and must not read the same.
|
||||
readyz=$(kubectl get --raw='/readyz?verbose' 2>/dev/null)
|
||||
if [ -z "${readyz}" ]; then
|
||||
failed=unreachable
|
||||
else
|
||||
failed=$(printf '%s\n' "${readyz}" \
|
||||
| awk '/^\[-\]/{sub(/^\[-\]/,""); sub(/[[:space:]].*$/,""); printf "%s%s", sep, $0; sep=" "}')
|
||||
[ -n "${failed}" ] || failed=none
|
||||
fi
|
||||
|
||||
if kubectl get --raw='/healthz/etcd' 2>/dev/null | grep -q '^ok$'; then
|
||||
etcd=ok
|
||||
@@ -63,7 +69,51 @@ diagnose() {
|
||||
disk=ok
|
||||
fi
|
||||
|
||||
printf 'readyz:[%s]; etcd:%s; disk:%s' "${failed}" "${etcd}" "${disk}"
|
||||
# etcd stalls under CPU or memory pressure long before it faults on its own,
|
||||
# so an unhealthy CP reporting disk:ok is uninterpretable without these two.
|
||||
load=$(awk '{print $1}' "${LOADAVG_FILE:-/proc/loadavg}" 2>/dev/null)
|
||||
memavail_kb=$(awk '/^MemAvailable:/{print $2}' "${MEMINFO_FILE:-/proc/meminfo}" 2>/dev/null)
|
||||
[ -n "${load}" ] || load=unknown
|
||||
[ -n "${memavail_kb}" ] || memavail_kb=unknown
|
||||
|
||||
printf 'readyz:[%s]; etcd:%s; disk:%s; fsync:%sms; load:%s; memavail:%sk' \
|
||||
"${failed}" "${etcd}" "${disk}" "$(etcd_fsync_ms)" "${load}" "${memavail_kb}"
|
||||
}
|
||||
|
||||
# etcd_fsync_ms prints the mean WAL fsync in milliseconds, or "nometrics" when
|
||||
# etcd served none, or "warmup" when it has not done enough to average.
|
||||
#
|
||||
# disk:ok in diagnose() is free space, which says nothing about how long a write
|
||||
# takes. etcd stalls on fsync latency long before it runs out of room, and this
|
||||
# guest's disk is an EBS volume served by viperblock, so a fsync here is a
|
||||
# network round trip. Healthy is single-digit milliseconds.
|
||||
#
|
||||
# Mean rather than a percentile: the histogram's _sum and _count are two greps,
|
||||
# where a bucket percentile is an interpolation this script has no business
|
||||
# doing, and a stall shows as an order of magnitude rather than a tail.
|
||||
etcd_fsync_ms() {
|
||||
# `|| metrics=` is load-bearing under set -e: a scrape against a port
|
||||
# nothing answers exits non-zero, and a bare assignment would abort the
|
||||
# caller — dropping the diagnosis exactly when it is wanted.
|
||||
metrics=$(curl -fsS --max-time 2 "${ETCD_METRICS_URL:-http://127.0.0.1:2381/metrics}" 2>/dev/null) || metrics=
|
||||
# "etcd served nothing" and "etcd has not done enough work yet" have
|
||||
# different fixes — the first is a missing endpoint, the second is a guest
|
||||
# that just booted — so they must not read the same. Collapsing both to one
|
||||
# word is the mistake readyz:[none] made.
|
||||
if [ -z "${metrics}" ]; then
|
||||
echo nometrics
|
||||
return
|
||||
fi
|
||||
# A minimum sample count, not just a non-zero one. A mean over the handful
|
||||
# of fsyncs a just-started etcd has done is a number with no meaning, and
|
||||
# reporting it invites the same mistake as judging a throughput floor on a
|
||||
# 297-byte object.
|
||||
fsync=$(printf '%s\n' "${metrics}" | awk -v min="${FSYNC_MIN_COUNT:-100}" '
|
||||
/^etcd_disk_wal_fsync_duration_seconds_sum/ {sum=$2}
|
||||
/^etcd_disk_wal_fsync_duration_seconds_count/ {count=$2}
|
||||
END {if (count >= min) printf "%.1f", (sum / count) * 1000}')
|
||||
[ -n "${fsync}" ] || fsync=warmup
|
||||
echo "${fsync}"
|
||||
}
|
||||
|
||||
publish_report() {
|
||||
@@ -100,12 +150,27 @@ publish_report() {
|
||||
echo "=== end mulga-eks CP diag ==="
|
||||
} > "${CONSOLE:-/dev/console}" 2>&1 || true
|
||||
fi
|
||||
# fsync goes out on every report, healthy or not. A figure that only appears
|
||||
# once the control plane is already failing has no baseline to be read
|
||||
# against, which is the same gap a passing cell that uploads no journal
|
||||
# leaves. Emitted as a bare JSON number, or omitted entirely when unknown so
|
||||
# a missing sample is never mistaken for a fast one.
|
||||
fsync_field=
|
||||
fsync_ms=$(etcd_fsync_ms)
|
||||
# Tested for being a number rather than against the sentinel words: this
|
||||
# emits a bare JSON value, so anything non-numeric reaching it produces a
|
||||
# payload the daemon cannot parse at all. A new sentinel must not be able
|
||||
# to break the report by being added.
|
||||
case "${fsync_ms}" in
|
||||
'' | *[!0-9.]*) : ;;
|
||||
*) fsync_field=$(printf '"fsync_ms":%s,' "${fsync_ms}") ;;
|
||||
esac
|
||||
if [ -n "${reason}" ]; then
|
||||
payload=$(printf '{"healthz":"%s","node_count":%s,"nodegroup_ready":%s,"reason":"%s","ts":%s}' \
|
||||
"${health}" "${node_count}" "${nodegroup_ready}" "${reason}" "$(date +%s)")
|
||||
payload=$(printf '{"healthz":"%s","node_count":%s,"nodegroup_ready":%s,%s"reason":"%s","ts":%s}' \
|
||||
"${health}" "${node_count}" "${nodegroup_ready}" "${fsync_field}" "${reason}" "$(date +%s)")
|
||||
else
|
||||
payload=$(printf '{"healthz":"%s","node_count":%s,"nodegroup_ready":%s,"ts":%s}' \
|
||||
"${health}" "${node_count}" "${nodegroup_ready}" "$(date +%s)")
|
||||
payload=$(printf '{"healthz":"%s","node_count":%s,"nodegroup_ready":%s,%s"ts":%s}' \
|
||||
"${health}" "${node_count}" "${nodegroup_ready}" "${fsync_field}" "$(date +%s)")
|
||||
fi
|
||||
printf '%s' "${payload}" | eks-gateway-publish -channel state 2>&1 \
|
||||
| logger -t mulga-eks-state-report
|
||||
|
||||
@@ -49,6 +49,15 @@ cat > "${STUBBIN}/logger" <<'EOF'
|
||||
:
|
||||
EOF
|
||||
|
||||
# curl stub: serves the etcd /metrics scrape. Empty ETCD_METRICS_BODY exits
|
||||
# non-zero like a real -fsS against a port nothing is listening on, which is the
|
||||
# case the fsync field has to survive.
|
||||
cat > "${STUBBIN}/curl" <<'EOF'
|
||||
#!/bin/sh
|
||||
[ -n "${ETCD_METRICS_BODY:-}" ] || exit 7
|
||||
printf '%s\n' "${ETCD_METRICS_BODY}"
|
||||
EOF
|
||||
|
||||
chmod +x "${STUBBIN}"/*
|
||||
PATH="${STUBBIN}:${PATH}"
|
||||
export PATH
|
||||
@@ -61,6 +70,14 @@ EKS_ACCOUNT_ID=000000000001
|
||||
EKS_CLUSTER_NAME=demo3
|
||||
EOF
|
||||
|
||||
# Pressure fixtures: the agent reads these instead of /proc so the load and
|
||||
# memavail fields are deterministic here.
|
||||
LOADAVG_FILE="${WORK}/loadavg"
|
||||
MEMINFO_FILE="${WORK}/meminfo"
|
||||
echo "7.25 6.10 5.00 3/512 9001" > "${LOADAVG_FILE}"
|
||||
printf 'MemTotal: 4193768 kB\nMemAvailable: 262144 kB\n' > "${MEMINFO_FILE}"
|
||||
export LOADAVG_FILE MEMINFO_FILE
|
||||
|
||||
FAILS=0
|
||||
fail() { echo "FAIL: $*"; FAILS=$((FAILS + 1)); }
|
||||
pass() { echo "ok: $*"; }
|
||||
@@ -105,7 +122,7 @@ run_agent
|
||||
P=$(cat "${WORK}/payload.json")
|
||||
case "${P}" in *'"healthz":"fail"'*) pass "unhealthy: healthz fail" ;; *) fail "unhealthy: healthz wrong: ${P}" ;; esac
|
||||
case "${P}" in *'"nodegroup_ready":{}'*) pass "unhealthy: nodegroup_ready empty (no kubectl node query on a failing apiserver)" ;; *) fail "unhealthy: nodegroup_ready wrong: ${P}" ;; esac
|
||||
case "${P}" in *'"reason":"readyz:[etcd poststarthook/start-service-ip-repair-controllers]; etcd:unreachable; disk:ok"'*) pass "unhealthy: reason names failing subchecks + etcd + disk" ;; *) fail "unhealthy: reason wrong: ${P}" ;; esac
|
||||
case "${P}" in *'"reason":"readyz:[etcd poststarthook/start-service-ip-repair-controllers]; etcd:unreachable; disk:ok; fsync:nometricsms; load:7.25; memavail:262144k"'*) pass "unhealthy: reason names failing subchecks + etcd + disk + fsync + pressure" ;; *) fail "unhealthy: reason wrong: ${P}" ;; esac
|
||||
case "${P}" in *'"reason":"'*'"'*'"'*) : ;; esac
|
||||
C=$(cat "${WORK}/console.out")
|
||||
case "${C}" in *'mulga-eks CP unhealthy'*) pass "unhealthy: console banner emitted" ;; *) fail "unhealthy: console banner missing: ${C}" ;; esac
|
||||
@@ -121,6 +138,96 @@ run_agent
|
||||
P=$(cat "${WORK}/payload.json")
|
||||
case "${P}" in *'etcd:ok; disk:low:1024k'*) pass "disk-low: reason flags low etcd disk" ;; *) fail "disk-low: reason wrong: ${P}" ;; esac
|
||||
|
||||
# --- Case 4: readyz itself never answers -> unreachable, not "none" ---
|
||||
# A wedged apiserver returns an empty body; reporting that as "none" reads as
|
||||
# "no subcheck failed", which is the opposite of what happened.
|
||||
HEALTHZ_BODY=fail
|
||||
READYZ_BODY=
|
||||
ETCD_BODY=error
|
||||
DF_AVAIL_KB=9000000
|
||||
export HEALTHZ_BODY READYZ_BODY ETCD_BODY DF_AVAIL_KB
|
||||
run_agent
|
||||
P=$(cat "${WORK}/payload.json")
|
||||
case "${P}" in *'readyz:[unreachable]'*) pass "wedged: empty readyz reads unreachable, not none" ;; *) fail "wedged: reason wrong: ${P}" ;; esac
|
||||
case "${P}" in *'load:7.25; memavail:262144k'*) pass "wedged: pressure captured alongside" ;; *) fail "wedged: pressure missing: ${P}" ;; esac
|
||||
|
||||
# --- Case 5: etcd serves its metrics -> fsync mean reported in ms ---
|
||||
# The case the field exists for: memory and free space both fine, so a slow
|
||||
# fsync is the only remaining reading. 4.2s over 700 fsyncs is 6.0ms.
|
||||
HEALTHZ_BODY=fail
|
||||
READYZ_BODY=
|
||||
ETCD_BODY=error
|
||||
DF_AVAIL_KB=9000000
|
||||
ETCD_METRICS_BODY=$(printf 'etcd_disk_wal_fsync_duration_seconds_sum 4.2\netcd_disk_wal_fsync_duration_seconds_count 700\n')
|
||||
export HEALTHZ_BODY READYZ_BODY ETCD_BODY DF_AVAIL_KB ETCD_METRICS_BODY
|
||||
run_agent
|
||||
P=$(cat "${WORK}/payload.json")
|
||||
case "${P}" in *'fsync:6.0ms'*) pass "fsync: mean derived from histogram sum/count" ;; *) fail "fsync: wrong: ${P}" ;; esac
|
||||
|
||||
# --- Case 6: a stalled disk is legible against the healthy case above ---
|
||||
# 63s over 700 fsyncs is 90ms — fifteen times case 5 on the same shape of input.
|
||||
ETCD_METRICS_BODY=$(printf 'etcd_disk_wal_fsync_duration_seconds_sum 63\netcd_disk_wal_fsync_duration_seconds_count 700\n')
|
||||
export ETCD_METRICS_BODY
|
||||
run_agent
|
||||
P=$(cat "${WORK}/payload.json")
|
||||
case "${P}" in *'fsync:90.0ms'*) pass "fsync: a stalled disk reads far above a healthy one" ;; *) fail "fsync: stall wrong: ${P}" ;; esac
|
||||
|
||||
# --- Case 7: no fsyncs recorded yet -> unknown, never a divide by zero ---
|
||||
ETCD_METRICS_BODY=$(printf 'etcd_disk_wal_fsync_duration_seconds_sum 0\netcd_disk_wal_fsync_duration_seconds_count 0\n')
|
||||
export ETCD_METRICS_BODY
|
||||
run_agent
|
||||
P=$(cat "${WORK}/payload.json")
|
||||
case "${P}" in *'fsync:warmupms'*) pass "fsync: zero count reads warmup, not a division error" ;; *) fail "fsync: zero-count wrong: ${P}" ;; esac
|
||||
|
||||
# --- Case 8: too few fsyncs to mean anything -> unknown, not a fast reading ---
|
||||
# 3 fsyncs on a just-started etcd averages to a number with no meaning. Same
|
||||
# mistake as judging a throughput floor on a 297-byte object.
|
||||
ETCD_METRICS_BODY=$(printf 'etcd_disk_wal_fsync_duration_seconds_sum 0.012\netcd_disk_wal_fsync_duration_seconds_count 3\n')
|
||||
export ETCD_METRICS_BODY
|
||||
run_agent
|
||||
P=$(cat "${WORK}/payload.json")
|
||||
case "${P}" in *'fsync:warmupms'*) pass "fsync: a sample below the minimum count reads warmup" ;; *) fail "fsync: min-count wrong: ${P}" ;; esac
|
||||
|
||||
# --- Case 9: fsync is published on a HEALTHY report, not only a failing one ---
|
||||
# A figure that appears only once the CP is already failing has no baseline to
|
||||
# be read against.
|
||||
HEALTHZ_BODY=ok
|
||||
NODES_BODY=$(printf 'ip-1 Ready <none> 1d v1.32.5\n')
|
||||
NODES_LABELED_BODY=$(printf 'ip-1 Ready <none> 1d v1.32.5 ng-a\n')
|
||||
ETCD_METRICS_BODY=$(printf 'etcd_disk_wal_fsync_duration_seconds_sum 4.2\netcd_disk_wal_fsync_duration_seconds_count 700\n')
|
||||
export HEALTHZ_BODY NODES_BODY NODES_LABELED_BODY ETCD_METRICS_BODY
|
||||
run_agent
|
||||
P=$(cat "${WORK}/payload.json")
|
||||
case "${P}" in *'"healthz":"ok"'*) pass "healthy-fsync: report is healthy" ;; *) fail "healthy-fsync: not healthy: ${P}" ;; esac
|
||||
case "${P}" in *'"fsync_ms":6.0'*) pass "healthy-fsync: fsync published on a healthy report" ;; *) fail "healthy-fsync: fsync missing: ${P}" ;; esac
|
||||
case "${P}" in *'"reason"'*) fail "healthy-fsync: healthy report must carry no reason: ${P}" ;; *) pass "healthy-fsync: no reason on a healthy report" ;; esac
|
||||
|
||||
# --- Case 10: an unknown fsync is omitted, never emitted as a fast number ---
|
||||
ETCD_METRICS_BODY=
|
||||
export ETCD_METRICS_BODY
|
||||
run_agent
|
||||
P=$(cat "${WORK}/payload.json")
|
||||
case "${P}" in *'"fsync_ms"'*) fail "omitted: unknown fsync must not appear in the payload: ${P}" ;; *) pass "omitted: unknown fsync is absent, not reported as fast" ;; esac
|
||||
# The payload must still be valid JSON with the field gone.
|
||||
case "${P}" in *'"nodegroup_ready":{"ng-a":1},"ts":'*) pass "omitted: payload stays well-formed without the field" ;; *) fail "omitted: payload malformed: ${P}" ;; esac
|
||||
|
||||
# --- Case 11: no metrics endpoint reads differently from a warming etcd ---
|
||||
# Both leave the payload field out, but they have different fixes — a missing
|
||||
# endpoint against a guest that only just booted — so the reason must separate
|
||||
# them. This is the readyz:[none] mistake, not repeated.
|
||||
HEALTHZ_BODY=fail
|
||||
READYZ_BODY=
|
||||
ETCD_BODY=error
|
||||
ETCD_METRICS_BODY=
|
||||
export HEALTHZ_BODY READYZ_BODY ETCD_BODY ETCD_METRICS_BODY
|
||||
run_agent
|
||||
P=$(cat "${WORK}/payload.json")
|
||||
case "${P}" in *'fsync:nometricsms'*) pass "nometrics: an unreachable scrape is distinct from a warming etcd" ;; *) fail "nometrics: wrong: ${P}" ;; esac
|
||||
|
||||
# A sentinel must never reach the payload as a bare JSON value. The guard is
|
||||
# numeric, so a new sentinel cannot silently produce unparseable output.
|
||||
case "${P}" in *'"fsync_ms":nometrics'*) fail "nometrics: sentinel leaked into the payload as a number: ${P}" ;; *) pass "nometrics: sentinel kept out of the JSON value" ;; esac
|
||||
|
||||
if [ "${FAILS}" -eq 0 ]; then
|
||||
echo "PASS: all mulga-eks-state-report cases"
|
||||
exit 0
|
||||
|
||||
Reference in New Issue
Block a user