From eca528b9b374c71463c57866d2170394e00a0218 Mon Sep 17 00:00:00 2001 From: Oliver Kieselbach Date: Tue, 21 Jul 2026 22:34:24 +0200 Subject: [PATCH] =?UTF-8?q?docs(mcp):=20attribute=20the=20cold=20start=20?= =?UTF-8?q?=E2=80=94=20the=20app=20is=200.5s=20of=20it,=20the=20platform?= =?UTF-8?q?=20is=20the=20rest?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The boot marks answered the question on the first production cold start, and the answer overturns the reasoning behind an infrastructure change made earlier today. +244ms node started (startup + module loading) +431ms corpora read from disk (+187ms) +491ms index read, validated, hydrated (+60ms for 3.12 MB) +504ms listening +677ms query embedder warm (background) Against a 31.5s cold start, the application accounts for half a second. The other ~98% is Container Apps activation: KEDA reacting, pod scheduling, mounting the 236 MB image, starting the container. The container also boots faster than a development laptop (504ms vs 1273ms), so nothing here is CPU-starved. Consequences recorded rather than quietly dropped: - Raising the container to 0.5 vCPU was justified by the embedding model load on the cold-start path. Prebaked, that load is 173ms and runs in the background where only the first search could wait on it. The justification does not survive measurement. The setting stays — it plausibly helps concurrent requests, and at scale-to-zero the cost difference is negligible — but the Bicep comment now says what is true instead of what was assumed, so nobody re-derives the wrong reason. - Index size is irrelevant to startup (60ms for 3.12 MB), so the corpus can grow freely, and the embedding rounding earns its keep on file size alone. - The only lever that would actually remove the cold start is minReplicas: 1, at roughly 7x the monthly free consumption grant. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/mcp/docs-corpus.md | 39 ++++++++++++++++++++++----------------- infra/mcp-server.bicep | 22 +++++++++++++++------- 2 files changed, 37 insertions(+), 24 deletions(-) diff --git a/docs/mcp/docs-corpus.md b/docs/mcp/docs-corpus.md index 8bdc8508..83bdbe7b 100644 --- a/docs/mcp/docs-corpus.md +++ b/docs/mcp/docs-corpus.md @@ -215,29 +215,34 @@ sequence collapses to a single instant however it is queried. The server therefo times itself — `[boot +Nms]` marks from `process.uptime()`, which includes Node's own startup and module loading. -On a development machine the whole app boot is ~1.3 s: +Measured in production on a real cold start (31.5 s to first byte that time): ``` -+860ms node started (startup + module loading) -+1199ms corpora read from disk (+339ms) -+1242ms index read, validated, hydrated (+43ms <- for 3.12 MB) -+1273ms listening (+31ms) -+1497ms query embedder warm (+224ms, background) ++244ms node started (startup + module loading) ++431ms corpora read from disk (+187ms) ++491ms index read, validated, hydrated (+60ms <- for 3.12 MB) ++504ms listening ++677ms query embedder warm (background) ``` -Two things follow, and both contradict plausible assumptions: +**The application accounts for half a second of it.** The other ~31 s — 98% — is +Container Apps activation: KEDA reacting to the request, scheduling a pod, mounting +the 236 MB image, starting the container. The container also boots *faster* than a +development laptop (504 ms against 1273 ms), so nothing here is CPU-starved. -- **Index size is not the cold-start cost.** Parsing and hydrating 3.12 MB takes 43 ms. - Growing the corpus further is cheap. -- **The prebaked embedder is not either** — 224 ms, and it loads in the background, so - only the first *search* could ever wait on it. +Three conclusions, and two of them overturn assumptions this file previously stated: -The dominant local cost is Node startup and module loading. Whether that, or platform -activation (pod scheduling, container start), dominates the production 24.5 s is what -the `[boot]` marks answer on the next cold start. If they show the app listening after -a few seconds, the remainder is platform time that no CPU or index change can touch — -and `minReplicas: 1` would be the only real lever, at roughly 7× the monthly free -consumption grant. +- **Index size is irrelevant to startup.** 3.12 MB hydrates in 60 ms. Growing the + corpus is effectively free, and the embedding rounding earns its keep on file size + alone, not on boot time. +- **Raising the container to 0.5 vCPU was not justified by cold start.** The stated + reason was the embedding model load; prebaked, that is 173 ms and runs in the + background, where only the first *search* could ever wait on it. The extra vCPU may + still help concurrent requests, but the cold-start argument does not survive + measurement. +- **The only real lever is `minReplicas: 1`**, which removes activation entirely at + roughly 7× the monthly free consumption grant. Shrinking the image would shave the + mount, but that is a fraction of platform time, not the bulk of it. # Cross-repo coupling diff --git a/infra/mcp-server.bicep b/infra/mcp-server.bicep index c6b0220b..9208d6d1 100644 --- a/infra/mcp-server.bicep +++ b/infra/mcp-server.bicep @@ -192,13 +192,21 @@ resource containerApp 'Microsoft.App/containerApps@2024-03-01' = { { name: 'mcp-server' image: '${acr.properties.loginServer}/${containerAppName}:${imageTag}' - // 0.5 vCPU is deliberate, not a default. The corpus embeddings are - // precomputed at image build time, so steady-state search is cheap — - // but a scale-to-zero cold start still loads the ~23 MB ONNX embedding - // model to embed the incoming QUERY, and that is seconds of pure CPU on - // the critical path of the first search. Doubling the vCPU roughly - // halves it. Container Apps only accepts fixed CPU/memory pairs, so the - // memory moves with it. Do not "optimize" this back to 0.25. + // 0.5 vCPU, but NOT for the reason originally given. It was raised from + // 0.25 to shorten the scale-to-zero cold start, on the assumption that + // loading the ONNX embedding model dominated it. Measurement disproved + // that: the app boots in 504 ms in production (see the [boot] marks in + // the container log and docs/mcp/docs-corpus.md), of which the prebaked + // embedder is 173 ms in the background. Of a 31 s cold start, ~98% is + // Container Apps activation — pod scheduling, image mount, container + // start — which no CPU setting touches. + // + // It stays at 0.5 because it plausibly helps CONCURRENT requests + // (maxReplicas is 1, so one replica serves up to 10 in-flight calls) and + // because at scale-to-zero the container bills only while active, making + // the difference negligible. If cold start ever needs to actually go + // away, the only lever is minReplicas: 1. Container Apps accepts fixed + // CPU/memory pairs, so memory moves with the CPU. resources: { cpu: json('0.5') memory: '1.0Gi'