The first beta's release workflow failed twice in backend_tests:
TestHostAgentCredentialFileHandoff, added on 2026-10-11 and so new to
every release job, compiles pulse-agent itself and was killed at exactly
180s with "build agent from this source: signal: killed".
The release runner starts each API shard with a small pinned GOMAXPROCS,
and the test's child compiler inherits it. A cold agent build is about
250 CPU-seconds, so under the pin it serializes past the test's fixed
three-minute limit. The release job also has no Go build cache, and the
race-enabled test binary the runner compiles does not warm this plain
build. Pull request jobs keep a warm cache, which is why the test passed
there.
Reproduced on pulse-dev with the shard's conditions, a prebuilt test
binary, GOMAXPROCS=1 and an empty cache: the test fails at 180.02s with
the identical message. With the agent built once beforehand, unpinned, in
99s, the same pinned run passes in 16s.
Build the agent once in the runner, after compiling the test binary and
before any shard starts, with the full CPU count. The runner test pins
that order so the warm-up cannot drift after the shards, and the contract
records why. Against the previous runner the pin fails with the expected
message. The release preflight and policy suites pass.
Contract-Neutral: Release test runner ordering only; no published artifact, tag, signing, install or customer contract changes.
The three-shard backend gate landed with a 16 GiB admission requirement
that exceeded the 8-vCPU PVE worker's own idle availability (measured
16.1-16.7 GiB, and 14.1-14.9 GiB beside the sibling release compilers),
so the next release would have hard-failed at admission before running a
single test. Direct probes on the worker measured the complete gate at a
~6-7.5 GiB footprint (8.9 GiB MemAvailable floor from a 16.4 GiB start,
zero swap), so admission now requires 10 GiB for three shards and 8 GiB
for two, and degrades the shard count instead of failing the release
when the bounded wait expires without headroom.
Shard CPU is now weighted by planned test volume from the plan manifest.
Top-level tests execute serially per test-binary process, so width mainly
buys runtime, GC, and race-detector headroom for the 3595-test prefix
shard; the ~15-test wait-bound tails cannot use it. Probes measured the
prefix shard at 569s with 2 procs versus 484s with 4, with total
allocation still equal to the worker's vCPU count.
Validated by three direct runs of the canonical script on the worker:
the full gate passed in 8m10s wall (auto-admitted three shards, procs
4/2/2, shard walls 484s/331s/224s) against the 12m09s two-shard release
job measured in run 32514803052. One rig run also proved the gate fails
closed: a stale contract pin failed the non-API graph and the harness
terminated every descendant shard process.
rcourtman