Balance internal/api race shards by measured run time

The four Backend tests (api-N) shards took equal-count quarters of go
test's list, which left them at 4.0, 10.6, 4.0 and 16.8 minutes on the
first PR run, and at 1.9, 8.0, 1.5 and 10.3 minutes of go test time on
the following main push (run 36560070496). Almost all of it sits in a
few tests. One contract mock chart test runs about six minutes on the
runner, and the 28 integration server tests at the end of the list take
about twenty seconds each, so the fourth quarter held nearly half the
package.

Shards are still contiguous runs of go test's own order, since some
tests depend on package state from their neighbours, but the cut points
now come from estimated seconds.
.github/scripts/select-internal-api-shard.sh weighs each test from
.github/scripts/internal-api-test-seconds.txt (the 199 tests at 0.5s or
more, seeded from a local -race -json run scaled to that main run's
shard totals, with a 0.05s default for anything unlisted), binary
searches the smallest per-shard budget that fits the list into four
slices, and fills up to it. The weights only move cut points. Every test
is assigned exactly one shard in a single pass, so a stale file can
unbalance shards but never drop or repeat a test, and no shard is ever
empty.

With today's list the slices hold 1187, 23, 2556 and 21 tests, estimated
at 1.9, 6.9, 6.5 and 6.7 minutes of tests, or about 9.5 minutes per job
with the roughly 2.6 minutes of setup and race compile. The 2556-test
slice would need a 124 KB -run pattern, so a shard passes -skip of the
other shards' tests whenever that is shorter, which runs the same tests
in the same order.
This commit is contained in:
courtmanr@gmail.com
2026-09-29 13:14:14 +01:00
parent fcbb7e58ad
commit 06d78b66d3
6 changed files with 466 additions and 15 deletions
+28 -10
View File
@@ -323,9 +323,9 @@ jobs:
# internal/api alone took ~27 of the ~31 minutes of the old single
# `Backend tests (api)` job under -race, which made it the critical path of
# every pull request. Its top-level tests are split across these shards
# as contiguous runs of the package's own test order, and the
# `Backend tests (api)` job below keeps the required check name by passing
# only when every shard passed.
# as contiguous runs of the package's own test order, cut by estimated run
# time, and the `Backend tests (api)` job below keeps the required check
# name by passing only when every shard passed.
backend-api:
name: Backend tests (api-${{ matrix.index }})
needs: changes
@@ -379,9 +379,15 @@ jobs:
# tests still lean on package state left by the tests just before
# them (for example admin bypass or session store globals), and an
# interleaved split by name broke dozens of them, while contiguous
# slices keep those neighbours together. Listing with -race builds
# the same test binary the run then reuses from cache. Benchmarks
# are left out because plain go test never runs them.
# slices keep those neighbours together. The slices are cut by
# estimated run time from .github/scripts/internal-api-test-seconds.txt
# rather than by count, because a handful of tests take most of the
# time (one contract test runs about six minutes and the integration
# server tail about twenty seconds per test). Unlisted tests get a
# small default weight, and the weights only move the cut points, so
# coverage never depends on them. Listing with -race builds the same
# test binary the run then reuses from cache. Benchmarks are left out
# because plain go test never runs them.
run: |
set -euo pipefail
listing=$(go test -race -list . ./internal/api)
@@ -391,21 +397,33 @@ jobs:
exit 1
fi
total=$(printf '%s\n' "$tests" | wc -l | tr -d ' ')
selected=$(printf '%s\n' "$tests" | awk -v n="$API_SHARD_COUNT" -v i="$API_SHARD_INDEX" -v total="$total" \
'NR > int(total * i / n) && NR <= int(total * (i + 1) / n)')
selected=$(printf '%s\n' "$tests" | bash .github/scripts/select-internal-api-shard.sh \
.github/scripts/internal-api-test-seconds.txt "$API_SHARD_COUNT" "$API_SHARD_INDEX")
if [ -z "$selected" ]; then
echo "::error::internal/api shard ${API_SHARD_INDEX} of ${API_SHARD_COUNT} resolved to an empty test list"
exit 1
fi
others=$(printf '%s\n' "$tests" | grep -vxF -f <(printf '%s\n' "$selected") || true)
# Name whichever side is shorter. A shard holding most of the tests
# is passed as -skip of every other shard's tests, which runs the
# same set in the same order.
pattern="^($(printf '%s\n' "$selected" | paste -sd '|' -))\$"
filter=(-run "$pattern")
if [ -n "$others" ]; then
skip="^($(printf '%s\n' "$others" | paste -sd '|' -))\$"
if [ "${#skip}" -lt "${#pattern}" ]; then
pattern=$skip
filter=(-run . -skip "$pattern")
fi
fi
# Linux rejects a single argument over 128 KiB. Fail with a clear
# instruction instead of an opaque exec error.
if [ "${#pattern}" -gt 120000 ]; then
echo "::error::internal/api shard pattern is ${#pattern} bytes; add a shard index and raise API_SHARD_COUNT"
exit 1
fi
echo "internal/api shard ${API_SHARD_INDEX} of ${API_SHARD_COUNT} runs $(printf '%s\n' "$selected" | wc -l | tr -d ' ') of ${total} tests"
go test -race -timeout 50m -run "$pattern" ./internal/api
echo "internal/api shard ${API_SHARD_INDEX} of ${API_SHARD_COUNT} runs $(printf '%s\n' "$selected" | wc -l | tr -d ' ') of ${total} tests (${selected%%$'\n'*} .. ${selected##*$'\n'})"
go test -race -timeout 50m "${filter[@]}" ./internal/api
# Required check name. Branch protection matches `Backend tests (api)`
# exactly, so this job must keep that name and must fail unless every