feat(mcp): search_docs — customer documentation as a third search corpus

The MCP server could answer telemetry questions ("why did this session fail")
but not product ones ("how do I deploy the agent", "where is my data stored").
Those answers already exist in the autopilotmonitor-docs bundle; search_docs
makes them semantically searchable.

Deliberately a separate index from search_knowledge: rules and prose answer
different questions, and sharing one topK would dilute both. Registered for
every role — published documentation carries no tenant data — and skipped
entirely when no corpus was baked in, so a doc-less build advertises no
broken tool.

Corpus handling:
- Chunked at h2/h3 and <details>/<summary>, each chunk prefixed with a
  "Page > Section > Subsection" breadcrumb so a mid-page hit is self-describing.
- Newlines are normalized before any parsing. This is a correctness
  precondition, not cosmetics: the bundle ships CRLF and JS `.` does not match
  \r, so /^(#{2,3})\s+(.*)$/ matched zero headings — silently, degrading every
  page to unlabelled paragraph splits.
- Two size exceptions: tables are never split (a split table leaves `| --- |`
  rows with no header), and unbroken changelog lists split at line level (left
  whole, their tails fall outside the model window and become unsearchable).
- Short sections are kept — a one-line answer is often the best hit, and the
  breadcrumb supplies the context a short body lacks.

Build and delivery:
- Docs are embedded at image build time into an optional third
  PrecomputedSection. Optional so a checkout without the docs repo does not
  invalidate the rules and event-type vectors; the reverse (server has docs,
  file does not) is staleness and is rejected.
- The deploy workflow checks the public docs repo into the build context and
  passes its SHA as DOCS_COMMIT, which /health reports alongside the chunk
  count — editing the docs does not update a running server, so staleness is
  made checkable rather than memorable.
- Embeddings are rounded to 6 decimals on serialize: 55% smaller index (2.63
  vs 5.88 MB) for a ~1e-5 norm error against a 1e-3 tolerance. indexPrecomputed
  now asserts unit norm, since the hydrate path never checked it.

Infra: the Bicep template was two changes behind the live deployment (EU
cutover and ACR hardening, both applied directly in Azure). Reconciled against
the running resources — managed-identity image pull replaces the admin
credentials and the acr-password secret — and infra/ is no longer gitignored so
it stops drifting. Container raised to 0.5 vCPU / 1.0 GiB for the cold-start
query-embedder load. The template does not model the mcp.autopilotmonitor.com
custom domain, so a full deployment would remove it; the header warns about it.

Verified: 428 tests pass; 45 pages produce 324 chunks; sample queries hit the
right page top-1/2; all 8 derived URLs return 200 live; the precompute output
is accepted by the boot validator and a mutated corpus is rejected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs/index.md

@@ -28,6 +28,10 @@ https://docs.autopilotmonitor.com (separate repository).
 
 * [Business Timestamps](backend/business-timestamps.md) - Why the Azure Tables system Timestamp is never authoritative (migrations reset it), and the OccurredUtc + RowKey-decode compensation for AuditLogs/OpsEvents/Events after the 2026-07-18 migration.
 
+# MCP Server
+
+* [MCP Docs Corpus](mcp/docs-corpus.md) - How the published customer documentation becomes the `search_docs` corpus: CRLF-safe heading chunking, build-time embedding as a third precomputed section, and why a docs edit needs an MCP redeploy.
+
 # Security & Identity
 
 * [MCP OAuth Flow](mcp-oauth-flow.md) - Who authenticates where when connecting an AI client to the MCP server; two identities, three parties plus Entra ID.

docs/log.md

@@ -1,5 +1,9 @@
 # Log
 
+## 2026-07-21
+
+* **Creation**: Added `mcp/docs-corpus.md` (new `mcp/` section) — the MCP server gains `search_docs`, a third semantic corpus built from the published customer documentation repository (`autopilotmonitor-docs`, checked out into the Docker build context by the deploy workflow). Documents the h2/h3 + `<details>` chunking with breadcrumb prefixes, the two deliberate size exceptions (tables never split, unbroken changelog lists split at line level), the CRLF precondition (`.` does not match `\r` in JS — headings silently stop matching, which produced 286 wrong-but-plausible chunks before it was found), the optional `docs` section in `search-index.json` and its validation asymmetry, embedding rounding to 6 decimals, and the cross-repo coupling: a docs edit reaches clients only after an MCP redeploy, made checkable via `DOCS_COMMIT` on `/health`. Container resources raised to 0.5 vCPU / 1.0 GiB for the cold-start query-embedder load.
+
 ## 2026-07-20
 
 * **Update**: Extended `rules/gather-rule-phase-scoping.md` with the `phase_exit` trigger — the closing bookend to `phase_change`, for one-shot collection when a phase is LEFT (e.g. "state at the end of Account Setup"). Documents the evaluation-before-phase-advance rule (triggerPhase match and scope gate see the phase being left), firing into `Failed`, the disjoint `exit:` dedup key space, and why no ConfigVersion bump is needed (no new field — old agents simply never fire the rule).

docs/mcp/docs-corpus.md

@@ -0,0 +1,159 @@
+---
+type: Concept
+title: MCP Docs Corpus — search_docs and the Cross-Repo Build Coupling
+description: How the published customer documentation becomes a third semantic search corpus in the MCP server — CRLF-safe heading chunking, build-time embedding, the corpus hash guard, and why editing the docs does not update a running server.
+resource: /src/McpServer/autopilot-monitor-mcp/src/docs-corpus.ts
+tags:
+  - mcp
+  - search
+  - embeddings
+  - documentation
+  - build
+timestamp: 2026-07-21T19:30:00+02:00
+---
+
+# Purpose
+
+The MCP server answers two different kinds of question, and they need two
+different corpora.
+
+* *"Why did this enrollment fail?"* → analysis rules, gather rules, IME log
+  patterns. That is `search_knowledge`, backed by `knowledge-base.ts`.
+* *"How do I deploy the agent?" / "Where is my data stored?"* → the published
+  customer documentation. That is `search_docs`, backed by `docs-corpus.ts`.
+
+They are deliberately **separate indexes**. Merged into one, they would compete
+for the same `topK` and dilute each other: a rules query would surface prose
+pages, and a product question would surface rule definitions.
+
+# Schema
+
+## Corpus source
+
+The corpus is the `autopilotmonitor-docs` repository — a public OKF bundle
+published at `docs.autopilotmonitor.com`. It is **not** vendored into this repo.
+The deploy workflow checks it out into `mcp-docs/` of the Docker build context;
+the Dockerfile copies it to `/app/docs` in both the build and the final stage
+(byte-identical, or the boot-time corpus hash rejects the precomputed index).
+
+`SUMMARY.md` and `log.md` are skipped as navigation scaffolding. `index.md` is
+kept — it is the documentation map and answers "what documentation exists" — but
+gets no `url`, because the bundle deliberately keeps it out of `SUMMARY.md` and
+GitBook therefore never renders it.
+
+## Chunking
+
+A whole page cannot be one document: `trust/security-faq.md` is 36 KB, while
+MiniLM-L6-v2 truncates at ~256 tokens. An un-chunked page would silently embed
+only its first paragraph.
+
+Chunks are cut at **h2/h3 headings**, which in this bundle are also the semantic
+boundaries — `security-faq.md` carries 37 h3s, one per question.
+`<details><summary>Q</summary>` blocks (used by `troubleshooting/faq.md`) are
+treated the same way, with the summary as the section title. Each chunk is
+prefixed with a `Page Title › Section › Subsection` breadcrumb so a hit lifted
+out of the middle of a page is self-describing to both the embedder and the
+reader.
+
+Two deliberate exceptions to the ~1500-character budget:
+
+* **Markdown tables are never split.** A table broken across chunks leaves
+  `| --- |` rows with no header — unreadable in a result and meaningless to
+  embed. Tables carry real answers here (the role matrix, the settings
+  reference), so an over-long table is allowed to exceed the budget.
+* **Unbroken lists split at line level.** The changelog pages are one bullet
+  list per release with no blank lines, so paragraph splitting cannot touch
+  them; left whole, everything past the model's window would be truncated away
+  and become unsearchable.
+
+Short sections are **kept**, not filtered. A one-line answer ("Is it free?" →
+"Yes, the Community plan is free") is often the most valuable hit, and the
+breadcrumb supplies the context the short body lacks. Only structural residue
+(under 25 alphanumeric characters) is dropped.
+
+## CRLF is a correctness precondition
+
+The docs repo is authored on Windows and ships CRLF. In JavaScript `.` does not
+match `\r`, so a pattern like `/^(#{2,3})\s+(.*)$/` matches **nothing** on a CRLF
+file: `(.*)` stops before the `\r` and `$` never reaches end-of-string.
+
+This was observed during development, and the failure mode is what makes it
+dangerous: no crash, no error — headings and frontmatter simply stop being
+recognized and every page silently degrades to unlabelled paragraph splits. The
+first working build produced 286 plausible-looking chunks that were almost all
+wrong; after normalizing, `security-faq.md` went from 29 blind splits to 46 real
+question sections.
+
+`normalizeNewlines()` runs before any parsing. Do not remove it, and do not
+"fix" a future parsing bug here by adding `\r?` to one pattern.
+
+## Build-time embedding
+
+Like the rules corpus, docs are embedded once at Docker build time by
+`precompute-embeddings.js <rulesDir> <outFile> [docsDir]`, into a third
+`PrecomputedSection` (`docs`) of `search-index.json`. The container never embeds
+a corpus at boot — only the incoming query.
+
+The `docs` section is **optional** in the file format, which is what makes a
+local checkout without the docs repo work: `validatePrecomputedIndex` skips it
+when the booting server loaded zero doc chunks. The asymmetry is intentional —
+if the server *has* a docs corpus and the file does *not*, that is staleness and
+is rejected like any other mismatch.
+
+Embeddings are rounded to 6 decimals on serialize. A float64 component
+serializes to ~20 characters of JSON, so 384 dims cost ~7.8 KB per document;
+rounding roughly halves the index file, which is `JSON.parse`d on every
+scale-to-zero cold start. The induced L2 norm error (~1e-5 over 384 dims) sits
+two orders of magnitude inside `UNIT_NORM_EPSILON`.
+
+## Registration
+
+`search_docs` is registered for **every role**, unlike the platform tools: this
+is public documentation carrying no tenant data, and a tenant user asking "how do
+I deploy the agent" has as much business here as a Global Admin. It is skipped
+entirely when the corpus is empty, so a doc-less build advertises no broken tool.
+
+# Cross-repo coupling
+
+**Editing the documentation does not update a running MCP server.** The corpus is
+baked into the image, so only re-running the *Deploy MCP Server* workflow picks
+up doc changes.
+
+That coupling is made checkable rather than memorable: the build passes the docs
+checkout SHA as `--build-arg DOCS_COMMIT`, and `/health` reports
+`docs: { commit, chunks, sections }`. Comparing that commit against
+`git rev-parse HEAD` in `autopilotmonitor-docs` answers "is the deployed
+documentation stale?" directly.
+
+Automatic redeploy on a docs push was considered and rejected: it would need a
+PAT secret in the docs repo and would turn every typo fix into a container
+deployment.
+
+# Examples
+
+Retrieval measured against the real bundle (323 chunks, all-MiniLM-L6-v2):
+
+```
+"where is my data stored"            → 0.566  troubleshooting/faq.md :: General › Where is my data stored?
+"how do I deploy the agent with Intune" → 0.819  troubleshooting/faq.md :: Setup & Agent › How do I deploy the agent?
+"which roles exist and what can they do" → 0.500  trust/security-faq.md :: Identity and Access › What roles exist?
+"how do I set up Teams notifications"    → 0.592  reference/settings.md  :: Tenant › Notifications
+"what firewall URLs do I need to allow"  → 0.374  reference/network-endpoints.md :: (intro)
+```
+
+The 0.25 `minScore` default is tuned to that observed band — short keyword
+queries on all-MiniLM score low, and a relevant-but-marginal hit lands around
+0.34.
+
+# Citations
+
+* `/src/McpServer/autopilot-monitor-mcp/src/docs-corpus.ts` — loader, frontmatter
+  parser, chunker
+* `/src/McpServer/autopilot-monitor-mcp/src/tools/search.ts` — `search_docs`
+  registration
+* `/src/McpServer/autopilot-monitor-mcp/src/precomputed-index.ts` — `docs`
+  section and its validation asymmetry
+* `/src/McpServer/autopilot-monitor-mcp/Dockerfile` — `mcp-docs/` copy,
+  `DOCS_COMMIT`
+* `/.github/workflows/deploy-mcp.yml` — docs checkout and SHA capture
+* [MCP OAuth Flow](../mcp-oauth-flow.md) — the other MCP concept document