docs(mcp): attribute the cold start — the app is 0.5s of it, the platform is the rest

The boot marks answered the question on the first production cold start, and the
answer overturns the reasoning behind an infrastructure change made earlier today.

  +244ms  node started (startup + module loading)
  +431ms  corpora read from disk          (+187ms)
  +491ms  index read, validated, hydrated  (+60ms for 3.12 MB)
  +504ms  listening
  +677ms  query embedder warm             (background)

Against a 31.5s cold start, the application accounts for half a second. The other
~98% is Container Apps activation: KEDA reacting, pod scheduling, mounting the
236 MB image, starting the container. The container also boots faster than a
development laptop (504ms vs 1273ms), so nothing here is CPU-starved.

Consequences recorded rather than quietly dropped:

- Raising the container to 0.5 vCPU was justified by the embedding model load on
  the cold-start path. Prebaked, that load is 173ms and runs in the background
  where only the first search could wait on it. The justification does not survive
  measurement. The setting stays — it plausibly helps concurrent requests, and at
  scale-to-zero the cost difference is negligible — but the Bicep comment now says
  what is true instead of what was assumed, so nobody re-derives the wrong reason.
- Index size is irrelevant to startup (60ms for 3.12 MB), so the corpus can grow
  freely, and the embedding rounding earns its keep on file size alone.
- The only lever that would actually remove the cold start is minReplicas: 1, at
  roughly 7x the monthly free consumption grant.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Oliver Kieselbach
2026-07-21 22:34:24 +02:00
parent 8305f5c850
commit eca528b9b3
2 changed files with 37 additions and 24 deletions
+22 -17
View File
@@ -215,29 +215,34 @@ sequence collapses to a single instant however it is queried. The server therefo
times itself — `[boot +Nms]` marks from `process.uptime()`, which includes Node's own
startup and module loading.
On a development machine the whole app boot is ~1.3 s:
Measured in production on a real cold start (31.5 s to first byte that time):
```
+860ms node started (startup + module loading)
+1199ms corpora read from disk (+339ms)
+1242ms index read, validated, hydrated (+43ms <- for 3.12 MB)
+1273ms listening (+31ms)
+1497ms query embedder warm (+224ms, background)
+244ms node started (startup + module loading)
+431ms corpora read from disk (+187ms)
+491ms index read, validated, hydrated (+60ms <- for 3.12 MB)
+504ms listening
+677ms query embedder warm (background)
```
Two things follow, and both contradict plausible assumptions:
**The application accounts for half a second of it.** The other ~31 s — 98% — is
Container Apps activation: KEDA reacting to the request, scheduling a pod, mounting
the 236 MB image, starting the container. The container also boots *faster* than a
development laptop (504 ms against 1273 ms), so nothing here is CPU-starved.
- **Index size is not the cold-start cost.** Parsing and hydrating 3.12 MB takes 43 ms.
Growing the corpus further is cheap.
- **The prebaked embedder is not either** — 224 ms, and it loads in the background, so
only the first *search* could ever wait on it.
Three conclusions, and two of them overturn assumptions this file previously stated:
The dominant local cost is Node startup and module loading. Whether that, or platform
activation (pod scheduling, container start), dominates the production 24.5 s is what
the `[boot]` marks answer on the next cold start. If they show the app listening after
a few seconds, the remainder is platform time that no CPU or index change can touch —
and `minReplicas: 1` would be the only real lever, at roughly 7× the monthly free
consumption grant.
- **Index size is irrelevant to startup.** 3.12 MB hydrates in 60 ms. Growing the
corpus is effectively free, and the embedding rounding earns its keep on file size
alone, not on boot time.
- **Raising the container to 0.5 vCPU was not justified by cold start.** The stated
reason was the embedding model load; prebaked, that is 173 ms and runs in the
background, where only the first *search* could ever wait on it. The extra vCPU may
still help concurrent requests, but the cold-start argument does not survive
measurement.
- **The only real lever is `minReplicas: 1`**, which removes activation entirely at
roughly 7× the monthly free consumption grant. Shrinking the image would shave the
mount, but that is a fraction of platform time, not the bulk of it.
# Cross-repo coupling
+15 -7
View File
@@ -192,13 +192,21 @@ resource containerApp 'Microsoft.App/containerApps@2024-03-01' = {
{
name: 'mcp-server'
image: '${acr.properties.loginServer}/${containerAppName}:${imageTag}'
// 0.5 vCPU is deliberate, not a default. The corpus embeddings are
// precomputed at image build time, so steady-state search is cheap —
// but a scale-to-zero cold start still loads the ~23 MB ONNX embedding
// model to embed the incoming QUERY, and that is seconds of pure CPU on
// the critical path of the first search. Doubling the vCPU roughly
// halves it. Container Apps only accepts fixed CPU/memory pairs, so the
// memory moves with it. Do not "optimize" this back to 0.25.
// 0.5 vCPU, but NOT for the reason originally given. It was raised from
// 0.25 to shorten the scale-to-zero cold start, on the assumption that
// loading the ONNX embedding model dominated it. Measurement disproved
// that: the app boots in 504 ms in production (see the [boot] marks in
// the container log and docs/mcp/docs-corpus.md), of which the prebaked
// embedder is 173 ms in the background. Of a 31 s cold start, ~98% is
// Container Apps activation — pod scheduling, image mount, container
// start — which no CPU setting touches.
//
// It stays at 0.5 because it plausibly helps CONCURRENT requests
// (maxReplicas is 1, so one replica serves up to 10 in-flight calls) and
// because at scale-to-zero the container bills only while active, making
// the difference negligible. If cold start ever needs to actually go
// away, the only lever is minReplicas: 1. Container Apps accepts fixed
// CPU/memory pairs, so memory moves with the CPU.
resources: {
cpu: json('0.5')
memory: '1.0Gi'