Telemetry follow-ups from the 7d tool-quality review:
- tool_call log lines now carry a capped errorMessage (thrown message or
leading toolError text) so error drilldowns name the failure class
instead of guessing from args; absent on success.
- get_rule_stats omits per-rule daily trend rows by default (they drove
50% of calls over the response cap at platform scope); new
includeTrends arg opts back in, trendsNote marks the omission.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
withToolTelemetry now receives the tool args and inspects the RESULT: handlers
never throw (toolError returns isError:true), so the previous isError flag was
effectively always false. Each MCP_TOOL_LOGGING line now carries soft-error
status, resultChars + overCap (vs. the anthropic/maxResultSizeChars hint),
a size-bounded args summary and the caller scope — the only telemetry that
covers backend-free tools (search_docs, validate_rule, get_resource).
New signals: search_zero_hit on the three search tools (unmet-demand signal)
and tool_call_rejected for JSON-RPC-level Zod/unknown-tool rejections, sniffed
at the /mcp route since the SDK answers those before any handler runs.
Bicep default toolLoggingEnabled=true (desired state; live flag via
az containerapp update). Docs: docs/mcp/tool-telemetry.md with KQL examples.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A German question returns zero results from search_docs, not poor ones: both the
documentation and the rules are English, and all-MiniLM-L6-v2 is an English-only
model, so the query vector lands nowhere near any chunk. For a German-speaking
admin driving an MCP client in their own language, the tool is effectively blind
and gives no hint why.
The caller is itself a language model, so the cheapest correct fix is to say so:
both tool descriptions now instruct it to translate the question to English before
searching and to answer the user in their own language. No infrastructure, no
model change, no ranking impact.
A multilingual embedding model would handle it natively and was considered, but it
is substantially larger — which matters on a scale-to-zero container — and it would
change English retrieval as well, so it would have to be measured against
scripts/eval-docs-search.ts rather than swapped in on the assumption that more
languages cannot hurt.
Verified: 447 tests pass, tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The boot marks answered the question on the first production cold start, and the
answer overturns the reasoning behind an infrastructure change made earlier today.
+244ms node started (startup + module loading)
+431ms corpora read from disk (+187ms)
+491ms index read, validated, hydrated (+60ms for 3.12 MB)
+504ms listening
+677ms query embedder warm (background)
Against a 31.5s cold start, the application accounts for half a second. The other
~98% is Container Apps activation: KEDA reacting, pod scheduling, mounting the
236 MB image, starting the container. The container also boots faster than a
development laptop (504ms vs 1273ms), so nothing here is CPU-starved.
Consequences recorded rather than quietly dropped:
- Raising the container to 0.5 vCPU was justified by the embedding model load on
the cold-start path. Prebaked, that load is 173ms and runs in the background
where only the first search could wait on it. The justification does not survive
measurement. The setting stays — it plausibly helps concurrent requests, and at
scale-to-zero the cost difference is negligible — but the Bicep comment now says
what is true instead of what was assumed, so nobody re-derives the wrong reason.
- Index size is irrelevant to startup (60ms for 3.12 MB), so the corpus can grow
freely, and the embedding rounding earns its keep on file size alone.
- The only lever that would actually remove the cold start is minReplicas: 1, at
roughly 7x the monthly free consumption grant.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A cold start after scale-to-zero measures 24.5s to first byte against 60ms warm,
and nothing outside the process could say why. Log Analytics stamps every console
line of one ingestion batch with the same time, so the boot sequence collapses to
a single instant however it is queried, and piping stdout locally buffers until
exit. So the server times itself: `[boot +Nms]` marks derived from
process.uptime(), which unlike an in-process timer also covers Node's own startup
and module loading.
Measuring the precomputed path locally already contradicts two assumptions worth
recording:
+860ms node started (startup + module loading)
+1199ms corpora read from disk (+339ms)
+1242ms index read, validated, hydrated (+43ms for 3.12 MB)
+1273ms listening (+31ms)
+1497ms query embedder warm (+224ms, background)
The index is 43ms, not seconds — its size is not the cold-start cost, and growing
the corpus is cheap. The prebaked embedder is 224ms and loads in the background,
so only the first search could ever wait on it. That was the stated reason for
raising the container to 0.5 vCPU, which now looks like a weaker justification
than it did; the production marks will settle it. If they show the app listening
after a few seconds, the rest of the 24.5s is platform activation that neither CPU
nor index changes can touch.
Verified: 447 tests pass, marks appear on both boot paths locally. Incidentally
the run also confirmed the corpus hash guard and the staleness signal both work —
the local docs repo has moved to 6333d56 while the deployed image reports d8cb7f4,
so the precomputed index was correctly rejected and recomputed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The live test found a third recall gap. The fallback only topped up when semantic
search returned FEWER than topK results — but the more common failure is a full
list of unconvincing ones. Querying "WOW6432Node" returned three chunks scoring
~0.29, none containing the word, while the single chunk that does was never
considered at all.
The fix is a ranking change, so it was measured rather than reasoned about. Adds
scripts/eval-docs-search.ts, which scores strategies over 2465 queries in three
deliberately different classes: every chunk heading (easy by construction, so a
regression detector), rare identifiers (the class this fallback serves), and
hand-written questions (the reality anchor — not derived from the corpus, so it
shares little vocabulary with it). Embeddings are computed once and each strategy
is a pure function over the same candidates, so sweeping a threshold costs no
extra inference.
known-item literal handwritten MRR fixed broken
append-only (before) 71.9% 66.2% 1.000 - -
weak < 0.30 71.9% 72.6% 1.000 142 0
weak < 0.35 73.7% 81.1% 1.000 335 0
weak < 0.40 76.3% 88.2% 0.944 502 3
weak < 0.45 77.2% 93.0% 0.889 613 5
0.35 is strictly dominant — better in every class, worse in none — so that is
what ships. Higher thresholds keep repairing identifier queries but start
demoting hand-written questions, the class that most resembles real use, so the
gain stops being free. When the head is weak, exact matches move to the front
while at least one semantic result is kept, so a weak-but-correct hit is never
pushed out entirely.
The measurement also resized the problem: identifier queries were succeeding only
66.2% of the time, so the reported case was one of roughly 750, not an outlier.
That is the part a single anecdote could not have shown, and the reason the
harness exists before the fix rather than after it.
Verified: 447 tests pass, including new cases for the weak path and for leaving a
confident ranking untouched. On the real corpus "WOW6432Node" now returns the
chunk that contains it at rank 1, while "how do I deploy the agent with Intune"
(0.819) and "what firewall URLs do I need to allow" (0.374) are unchanged and the
negative controls stay empty.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two measured recall gaps in search_docs, both silent.
1. Truncation. MiniLM's context is 256 tokens; 87 of 324 chunks exceeded it and
13% of all tokens never reached the ranking — the tokenizer drops the tail
without complaint. Concretely, in built-in-rules.md :: Apps (734 tokens) the
strings ANALYZE-APP-013, ANALYZE-OFFICE-001, ANALYZE-CORR-003 and Click-to-Run
all sit past the cutoff, so "proxy configuration blocking app downloads" did not
retrieve that section at all.
Shrinking chunks cannot fix this: the corpus ranges 2.43 to 5.81 characters per
token, so no character budget bounds tokens, and a budget safe for the worst case
(~620 chars) would shred prose and tables alike. So the chunk stays whole and the
embedding is split instead — index() cuts long text into 250-token windows with
40 overlap, a document carries embeddings[][], and search() scores it by its BEST
window. Max rather than mean: averaging would blur a long section's several
topics into one mediocre direction, the same dilution that makes large context
windows worse than chunking.
The tokenizer is a build-time dependency only (vocabulary, not ONNX weights,
~93 ms); production boots from precomputed vectors and never loads it.
2. No general fallback. Proper nouns embed poorly — "RealmJoin" occurs in 3
chunks yet scores under the threshold. Fuzzy matching was built, measured and
rejected: at a threshold low enough to surface RealmJoin (0.28), Fuse also
returns 5 hits for "SCEPman", a word this corpus contains zero times. Literal
containment has neither problem, and every provider already exposes it. So a
short result set is now topped up via lexicalMatch on distinctive query terms,
requiring two matching terms as corroboration — without that rule "how do I bake
sourdough bread" matched a page about LLM providers, because "bake" is a
substring of "baked into the image". Keyword hits are appended and flagged, never
sorted together with cosine scores.
Also sharpens the tool description with the measured reason to prefer this over
fetching the docs site (~1k tokens for topK=3 against ~9k raw / ~600k rendered,
version-matched, and matching on meaning), and replaces the growing tail of
positional docs parameters with a DocsSearchBundle.
Verified: token coverage 87% -> 100%; "Office Click-to-Run install never
finished" 0.318 -> 0.442; "proxy configuration blocking app downloads" absent ->
0.559 at rank 1; the ten reference queries keep byte-identical scores and rank 1
(short chunks are one window, so the same vector as before — no long-chunk bias).
The fallback is not a dead path: across 904 rare corpus tokens queried alone it
contributes hits for 431, while negative controls stay empty. Precompute output
is accepted by the boot validator and a mutated corpus still rejected; index
2.63 -> 3.12 MB for 324 chunks / 424 vectors. The rules corpus gains the same
treatment unplanned: 128 rules now carry 172 vectors. 445 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The MCP server could answer telemetry questions ("why did this session fail")
but not product ones ("how do I deploy the agent", "where is my data stored").
Those answers already exist in the autopilotmonitor-docs bundle; search_docs
makes them semantically searchable.
Deliberately a separate index from search_knowledge: rules and prose answer
different questions, and sharing one topK would dilute both. Registered for
every role — published documentation carries no tenant data — and skipped
entirely when no corpus was baked in, so a doc-less build advertises no
broken tool.
Corpus handling:
- Chunked at h2/h3 and <details>/<summary>, each chunk prefixed with a
"Page > Section > Subsection" breadcrumb so a mid-page hit is self-describing.
- Newlines are normalized before any parsing. This is a correctness
precondition, not cosmetics: the bundle ships CRLF and JS `.` does not match
\r, so /^(#{2,3})\s+(.*)$/ matched zero headings — silently, degrading every
page to unlabelled paragraph splits.
- Two size exceptions: tables are never split (a split table leaves `| --- |`
rows with no header), and unbroken changelog lists split at line level (left
whole, their tails fall outside the model window and become unsearchable).
- Short sections are kept — a one-line answer is often the best hit, and the
breadcrumb supplies the context a short body lacks.
Build and delivery:
- Docs are embedded at image build time into an optional third
PrecomputedSection. Optional so a checkout without the docs repo does not
invalidate the rules and event-type vectors; the reverse (server has docs,
file does not) is staleness and is rejected.
- The deploy workflow checks the public docs repo into the build context and
passes its SHA as DOCS_COMMIT, which /health reports alongside the chunk
count — editing the docs does not update a running server, so staleness is
made checkable rather than memorable.
- Embeddings are rounded to 6 decimals on serialize: 55% smaller index (2.63
vs 5.88 MB) for a ~1e-5 norm error against a 1e-3 tolerance. indexPrecomputed
now asserts unit norm, since the hydrate path never checked it.
Infra: the Bicep template was two changes behind the live deployment (EU
cutover and ACR hardening, both applied directly in Azure). Reconciled against
the running resources — managed-identity image pull replaces the admin
credentials and the acr-password secret — and infra/ is no longer gitignored so
it stops drifting. Container raised to 0.5 vCPU / 1.0 GiB for the cold-start
query-embedder load. The template does not model the mcp.autopilotmonitor.com
custom domain, so a full deployment would remove it; the header warns about it.
Verified: 428 tests pass; 45 pages produce 324 chunks; sample queries hit the
right page top-1/2; all 8 derived URLs return 200 live; the precompute output
is accepted by the boot validator and a mutated corpus is rejected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Oliver Kieselbach