mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-24 20:53:06 +00:00
Oliver Kieselbach
62599d972c
Two measured recall gaps in search_docs, both silent. 1. Truncation. MiniLM's context is 256 tokens; 87 of 324 chunks exceeded it and 13% of all tokens never reached the ranking — the tokenizer drops the tail without complaint. Concretely, in built-in-rules.md :: Apps (734 tokens) the strings ANALYZE-APP-013, ANALYZE-OFFICE-001, ANALYZE-CORR-003 and Click-to-Run all sit past the cutoff, so "proxy configuration blocking app downloads" did not retrieve that section at all. Shrinking chunks cannot fix this: the corpus ranges 2.43 to 5.81 characters per token, so no character budget bounds tokens, and a budget safe for the worst case (~620 chars) would shred prose and tables alike. So the chunk stays whole and the embedding is split instead — index() cuts long text into 250-token windows with 40 overlap, a document carries embeddings[][], and search() scores it by its BEST window. Max rather than mean: averaging would blur a long section's several topics into one mediocre direction, the same dilution that makes large context windows worse than chunking. The tokenizer is a build-time dependency only (vocabulary, not ONNX weights, ~93 ms); production boots from precomputed vectors and never loads it. 2. No general fallback. Proper nouns embed poorly — "RealmJoin" occurs in 3 chunks yet scores under the threshold. Fuzzy matching was built, measured and rejected: at a threshold low enough to surface RealmJoin (0.28), Fuse also returns 5 hits for "SCEPman", a word this corpus contains zero times. Literal containment has neither problem, and every provider already exposes it. So a short result set is now topped up via lexicalMatch on distinctive query terms, requiring two matching terms as corroboration — without that rule "how do I bake sourdough bread" matched a page about LLM providers, because "bake" is a substring of "baked into the image". Keyword hits are appended and flagged, never sorted together with cosine scores. Also sharpens the tool description with the measured reason to prefer this over fetching the docs site (~1k tokens for topK=3 against ~9k raw / ~600k rendered, version-matched, and matching on meaning), and replaces the growing tail of positional docs parameters with a DocsSearchBundle. Verified: token coverage 87% -> 100%; "Office Click-to-Run install never finished" 0.318 -> 0.442; "proxy configuration blocking app downloads" absent -> 0.559 at rank 1; the ten reference queries keep byte-identical scores and rank 1 (short chunks are one window, so the same vector as before — no long-chunk bias). The fallback is not a dead path: across 904 rare corpus tokens queried alone it contributes hits for 431, while negative controls stay empty. Precompute output is accepted by the boot validator and a mutated corpus still rejected; index 2.63 -> 3.12 MB for 324 chunks / 424 vectors. The rules corpus gains the same treatment unplanned: 128 rules now carry 172 vectors. 445 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>