mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-24 12:42:38 +00:00
Oliver Kieselbach
62599d972c
Two measured recall gaps in search_docs, both silent. 1. Truncation. MiniLM's context is 256 tokens; 87 of 324 chunks exceeded it and 13% of all tokens never reached the ranking — the tokenizer drops the tail without complaint. Concretely, in built-in-rules.md :: Apps (734 tokens) the strings ANALYZE-APP-013, ANALYZE-OFFICE-001, ANALYZE-CORR-003 and Click-to-Run all sit past the cutoff, so "proxy configuration blocking app downloads" did not retrieve that section at all. Shrinking chunks cannot fix this: the corpus ranges 2.43 to 5.81 characters per token, so no character budget bounds tokens, and a budget safe for the worst case (~620 chars) would shred prose and tables alike. So the chunk stays whole and the embedding is split instead — index() cuts long text into 250-token windows with 40 overlap, a document carries embeddings[][], and search() scores it by its BEST window. Max rather than mean: averaging would blur a long section's several topics into one mediocre direction, the same dilution that makes large context windows worse than chunking. The tokenizer is a build-time dependency only (vocabulary, not ONNX weights, ~93 ms); production boots from precomputed vectors and never loads it. 2. No general fallback. Proper nouns embed poorly — "RealmJoin" occurs in 3 chunks yet scores under the threshold. Fuzzy matching was built, measured and rejected: at a threshold low enough to surface RealmJoin (0.28), Fuse also returns 5 hits for "SCEPman", a word this corpus contains zero times. Literal containment has neither problem, and every provider already exposes it. So a short result set is now topped up via lexicalMatch on distinctive query terms, requiring two matching terms as corroboration — without that rule "how do I bake sourdough bread" matched a page about LLM providers, because "bake" is a substring of "baked into the image". Keyword hits are appended and flagged, never sorted together with cosine scores. Also sharpens the tool description with the measured reason to prefer this over fetching the docs site (~1k tokens for topK=3 against ~9k raw / ~600k rendered, version-matched, and matching on meaning), and replaces the growing tail of positional docs parameters with a DocsSearchBundle. Verified: token coverage 87% -> 100%; "Office Click-to-Run install never finished" 0.318 -> 0.442; "proxy configuration blocking app downloads" absent -> 0.559 at rank 1; the ten reference queries keep byte-identical scores and rank 1 (short chunks are one window, so the same vector as before — no long-chunk bias). The fallback is not a dead path: across 904 rare corpus tokens queried alone it contributes hits for 431, while negative controls stay empty. Precompute output is accepted by the boot validator and a mutated corpus still rejected; index 2.63 -> 3.12 MB for 324 chunks / 424 vectors. The rules corpus gains the same treatment unplanned: 128 rules now carry 172 vectors. 445 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
AutopilotMonitor
Advanced monitoring and troubleshooting solution for Windows Autopilot deployments. Gain full visibility into every enrollment session with a detailed event timeline, fleet health dashboards, and session reporting. Define custom analysis rules to automatically detect issues and gather rules to collect targeted evidence. Retrieve diagnostics packages on demand, configure agent settings like auto-reboot behavior and automatic timezone adjustment — all managed centrally from the web dashboard.
Private Preview
Autopilot Monitor is currently running as a Private Preview. Visit autopilotmonitor.com to request access and learn more.
Overview
Autopilot Monitor provides real-time tracking, intelligent diagnostics, and automated troubleshooting for Windows Autopilot enrollment processes. It consists of:
- Bootstrap Script — PowerShell script deployed via Intune that starts monitoring early in the enrollment process
- Monitoring Agent — Lightweight .NET application that collects telemetry and evidence during enrollment
- Backend API — Azure Functions-based ingestion and processing pipeline
- Web Dashboard — Next.js application for real-time monitoring and fleet analytics
Architecture
For detailed information about the system architecture, components, and data flow, see Architecture Documentation.
Documentation
Full admin documentation is available at docs.autopilotmonitor.com
License
This project uses a split licensing model:
- MIT License — Agent (
src/Agent/) and Shared library (src/Shared/) — unrestricted use on end-user devices - AGPL-3.0 — Backend (
src/Backend/), Web Dashboard (src/Web/), and MCP Server (src/McpServer/) — server-side components remain open source
See LICENSE for full details.