Calibrate internal/api shard weights to CI and record per-test seconds

The weights from #2324 came from one local run and left the api shards at
4.5, 8.8, 17.1 and 12.5 minutes on CI. Most of the 17.1 was not test time.
That shard ran `-run . -skip <1231 names>`, and the test binary caches only
its last compiled pattern, so alternating the two recompiled the 62 KB skip
regex for every test and subtest (about 50 ms each under -race locally, 30s
against 1.3s for 300 trivial tests). Two runs put that shard at 887s and
875s against roughly 450s of tests. The skip path now passes -skip alone.

Every api shard now runs go test -json through
.github/scripts/record-internal-api-test-seconds.py. It prints what plain go
test would (package lines and the output of failing or unfinished tests),
exits non-zero on any failure behind pipefail, and writes each top-level
test's seconds to an internal-api-test-seconds-N artifact that is uploaded
even when the shard fails. .github/scripts/refresh-internal-api-test-seconds.py
rebuilds the weights file from those artifacts (gh run download, median
across runs, DEFAULT_WEIGHT set to the mean of the unlisted tests, which the
selector now reads from the file).

Until those artifacts exist, the weights are a local -race -json run with
each region of go test's order scaled to what CI reported for it.
Equal-count quarters averaged 107, 437, 100 and 787s over four runs, and the
weighted cut gave 131s for the first 1187 tests, 350s for the next 23 and
608s for the last 21 over two. The last quarter and the last 21 tests
disagree on how the tail splits, so the tail takes the midpoint of an exact
fit and a pooled factor.

With these weights four shards still predict a slowest shard near 10
minutes, and a 20% slower runner on the tail would take it past the 11
minutes of Frontend and rest-1. Five shards predict about 131, 325, 280-340,
330-355 and 330-365s of tests, or 4.5 to 8.5 minutes a job with about 2.3
minutes of setup, so API_SHARD_COUNT goes to 5. Every test still runs
exactly once in contiguous go test order, and all five slices passed locally
from their new starting points. The required Backend tests (api) verdict
keeps its name.

.github/workflows/build-and-test.yml

@@ -337,9 +337,9 @@ jobs:
       fail-fast: false
       matrix:
         # Keep API_SHARD_COUNT equal to the number of indexes listed here.
-        index: [0, 1, 2, 3]
+        index: [0, 1, 2, 3, 4]
     env:
-      API_SHARD_COUNT: 4
+      API_SHARD_COUNT: 5
 
     steps:
       - name: Report documentation-only change
@@ -387,7 +387,12 @@ jobs:
         # small default weight, and the weights only move the cut points, so
         # coverage never depends on them. Listing with -race builds the same
         # test binary the run then reuses from cache. Benchmarks are left out
-        # because plain go test never runs them.
+        # because plain go test never runs them. The run goes through
+        # -json so every shard records each test's seconds on the runner
+        # (uploaded below and fed back into the weights by
+        # refresh-internal-api-test-seconds.py). The recorder prints only
+        # package lines and the output of failing or unfinished tests, like
+        # plain go test, and pipefail keeps a go test failure fatal.
         run: |
           set -euo pipefail
           listing=$(go test -race -list . ./internal/api)
@@ -406,14 +411,17 @@ jobs:
           others=$(printf '%s\n' "$tests" | grep -vxF -f <(printf '%s\n' "$selected") || true)
           # Name whichever side is shorter. A shard holding most of the tests
           # is passed as -skip of every other shard's tests, which runs the
-          # same set in the same order.
+          # same set in the same order. Never pair -skip with -run. The test
+          # binary caches only the last compiled pattern, so alternating the
+          # two recompiles the long skip pattern for every test and subtest,
+          # which added about seven minutes to such a shard.
           pattern="^($(printf '%s\n' "$selected" | paste -sd '|' -))\$"
           filter=(-run "$pattern")
           if [ -n "$others" ]; then
             skip="^($(printf '%s\n' "$others" | paste -sd '|' -))\$"
             if [ "${#skip}" -lt "${#pattern}" ]; then
               pattern=$skip
-              filter=(-run . -skip "$pattern")
+              filter=(-skip "$pattern")
             fi
           fi
           # Linux rejects a single argument over 128 KiB. Fail with a clear
@@ -423,7 +431,19 @@ jobs:
             exit 1
           fi
           echo "internal/api shard ${API_SHARD_INDEX} of ${API_SHARD_COUNT} runs $(printf '%s\n' "$selected" | wc -l | tr -d ' ') of ${total} tests (${selected%%$'\n'*} .. ${selected##*$'\n'})"
-          go test -race -timeout 50m "${filter[@]}" ./internal/api
+          timings="$RUNNER_TEMP/internal-api-test-seconds"
+          mkdir -p "$timings"
+          go test -race -timeout 50m -json "${filter[@]}" ./internal/api \
+            | python3 .github/scripts/record-internal-api-test-seconds.py "$timings/api-${API_SHARD_INDEX}.txt"
+
+      - name: Upload internal/api test timings
+        if: always() && needs.changes.outputs.code == 'true'
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: internal-api-test-seconds-${{ matrix.index }}
+          path: ${{ runner.temp }}/internal-api-test-seconds/
+          if-no-files-found: warn
+          retention-days: 30
 
   # Required check name. Branch protection matches `Backend tests (api)`
   # exactly, so this job must keep that name and must fail unless every