AutopilotMonitor/docs/log.md

Log

2026-07-23

• Creation: Added agent/esp-policy-provider-stall.md — a customer diagnosed that pre-provisioned devices with the ConfigMgr client as a blocking app occasionally park the user ESP at "Apps (Identifying)" for days: only a ConfigMgr provider is registered under EnrollmentStatusTracking\Device\Setup\Apps\PolicyProviders (no Sidecar = IME) and TrackingPoliciesCreated never becomes 1 — a wait the CSP defines without any timeout; renaming the key to Sidecar unblocks immediately. The agent now ships the always-on esp_policy_provider_stalled tripwire (60-s tick, 15-min continuous-incompleteness dwell, one-shot per provider, StartupEventGate fingerprint across restarts, purely observational via the InformationalEvent rail — no toggle, no ConfigVersion bump), generalized to any provider in both Setup/Apps and DevicePreparation kinds rather than hardcoding the ConfigMgr case. Companion change: SOFTWARE\Microsoft\Windows\Autopilot joins the gather-rule registry allowlist so tenants can target the subtree with custom rules (e.g. not_exists on the Sidecar key).

• Creation: Added agent/secureboot-ca2023-detection.md — ANALYZE-SEC-001 warned on devices whose firmware already carried the 2023 certificate, because the agent only read the SecureBoot\Servicing\UEFICA2023Status registry value, which mirrors the Windows Update rollout rather than the firmware (customer report). The agent now reads the UEFI db/KEK variables natively (GetFirmwareEnvironmentVariableExW with SeSystemEnvironmentPrivilege enabled on the process token — first privilege-adjustment code in the repo) and emits per-certificate booleans on secureboot_status. Suppression rides on a one-sided marker, uefiCa2023FirmwareConfirmed, emitted only on positive confirmation: the engine's not_exists precondition fails on any present value, so a two-sided boolean could not distinguish "confirmed" from "checked and missing", while the marker's absence keeps old agents and firmware-missing devices on the v2 path. Rule v3.0.0 is therefore safe to seed before the agent release.

• Update: url-registry.md — the first release after the alias migration surfaced Front Door caching as a correctness hazard: the alias served the previous version.json (X-Cache: TCP_REMOTE_HIT) after the origin had rotated, and since manifest and ZIP are independent cache entries, a stale manifest with a fresh ZIP fails the bootstrap SHA-256 check. Route caching on /agent/* is now disabled and every blob upload stamps Cache-Control: no-cache (CI Send-Blob, local build.ps1, versions/*.json manifests) — all agent-container blobs rotate in place, so nothing there is safely cacheable.

• Update: agent/decision-engine.md — the RealmJoin completion gate has a third terminal, FirstDeploymentIncomplete. Session 224b2087 showed why: an interactive logon 17 s after DeploymentPhase 101 makes RealmJoin reclassify the run as a secondary-user deployment, whose branch writes 200/210 and never 110 — the gate stayed closed until the 60-min hard timeout although Hello and desktop were long resolved. realmjoin_phase_changed is now a typed RealmJoinPhaseChanged signal, so RealmJoinFacts.LastDeploymentPhase is persisted across agent restarts (this also fixes every realmjoin_timeout event claiming "last phase: 0"); a transition out of 100/101 into 200/210 releases the gate with the realmjoin_first_deployment_incomplete Warning, and the RealmJoinDetected replay path evaluates the same rule for restarts that swallow the transition. A first observation of 210 deliberately does not release — session 6f1959c0 started at 210 and only deployed afterwards.

• Creation: Added url-registry.md — all well-known hosts (own subdomains, both blob accounts, API origin, Graph, Entra login) are now defined once per component (Constants.cs, web utils/config.ts, MCP src/config.ts) and referenced everywhere else; three guard tests fail the build on any literal outside the registries. Written because the EU cutover missed two hardcoded copies of the legacy blob host — LatestVersionsService (feeds the portal's version badge) and ValidateBootstrapCodeFunction — which kept working only through the fail-soft legacy mirror in build-agent.yml; those two plus the agent SelfUpdater now use the download.autopilotmonitor.com alias (verified serving all four artifacts first). The portal's bootstrap allow-list accepts alias AND legacy host during the transition, which imposes a deploy order: web before (or with) backend. The sweep also surfaced a fourth copy of the www/portal host pair in middleware.ts (now derived from the registry via hostRouting.ts) and a backend short-link built on the bare apex domain, which only worked through a registrar-level 301.

2026-07-22

• Creation: Added versioning.md — agent, backend and MCP now share one version shape (<major>.<minor>.<build> + 7-char commit + build timestamp) and one implementation of the build-number reservation (.github/scripts/Request-BuildNumber.ps1, previously copy-pasted between build-agent.yml and build.ps1). The backend had been pinned at a hand-written 1.0.0 since inception and now starts at 1.5.1000; the MCP server's package.json version was bumped by hand and now starts at 1.5.300, and it reports its own commit on /health instead of only the documentation bundle's. Both deploy workflows poll their health endpoint until it reports the build just deployed and publish versions/<component>.json only then, so a manifest that disagrees with the live endpoint means the instance drifted rather than the pipeline failing halfway. agent/build-counter-blob.md keeps the agent-specific parts and now points at the shared protocol; its counter URL was still the pre-cutover storage account.

• Creation: Added rules/gather-rule-guardrails.md — the gather-rule allowlists are enforced only on the agent (the backend validates no collectorType, target, or parameters), so a collector that forgets to call its guard is the vulnerability. Written because two had: LogParserCollector read any path in rule.Target, bypassing even the C:\Users hard block that every other file collector honours, and EventLogCollector read any channel including Security — while the customer-facing documentation already promised both were allowlisted. Both now guard, with logparser checking the directory for wildcard targets (Path.GetFullPath throws on *) and re-checking each resolved path. Added an event-log channel allowlist (eventLogChannels) plus hard-blocked channels held in C# rather than JSON, so a parse error cannot lift them. Also fixed the command timeout, which was inert: ReadToEnd() blocks until the pipe closes, so WaitForExit(30000) was only reached after the process had already exited and a hung command pinned its worker forever — now read asynchronously and killed at the timeout.

• Update: Added .github/workflows/ci.yml — the full suite (backend, agent V2 Core, DecisionCore, SummaryDialog, web, MCP) now runs on every pull request and every push to main, plus a job that fails if the generated guardrail files drift from rules/guardrails.json. Previously no workflow ran tests on a PR: the deploy workflows build without testing, and build-agent.yml tested only DecisionCore.Tests on manual dispatch — while trust/security-faq.md already told customers that automated tests run in CI. Added CodeQL (C# and TypeScript) and Dependabot (nuget, npm ×2, actions). Marked OfficeBinaryWatcherTests [Collection("SerialThreading")]: its bounded re-probe races the shared ThreadPool and failed once under full-suite load.

2026-07-21

• Update: search_docs and search_knowledge now state in their tool descriptions that queries must be in English. Both corpora and all-MiniLM-L6-v2 are English-only, and the failure is total rather than gradual — a German question returns zero results, not weak ones. The caller is a language model, so translating the question before the call and answering in the user's language costs nothing; a multilingual embedding model would be the alternative but is much larger and would have to be measured against the eval harness first, since it changes English retrieval too.

• Update: Added scripts/eval-docs-search.ts, an offline retrieval evaluation harness (2465 queries in three classes: chunk headings as a regression detector, rare identifiers, and hand-written questions as the reality anchor), and used it to close a third search_docs gap found during the live test. The keyword fallback only fired when semantic search returned fewer than topK; a pasted identifier like WOW6432Node instead returned three chunks at ~0.29 that did not contain the word while the one chunk that did was never considered. It now also fires on a weak semantic head (<0.35), letting exact matches take the front while keeping one semantic result. The threshold is measured, not guessed: 0.35 fixes 335 queries and breaks 0 (identifier recall 66.2% → 81.1%), while 0.40 fixes more but starts demoting hand-written questions. The measurement also resized the problem — identifier queries were only succeeding 66.2% of the time, so the reported case was one of roughly 750.

• Update: Closed the two measured recall gaps in search_docs (mcp/docs-corpus.md). (1) Token windows — 87 of 324 chunks exceeded the model's 256-token context and 13% of all tokens never reached the ranking; a character budget cannot fix this (the corpus ranges 2.43–5.81 chars/token), so long text is now embedded as 250-token windows with 40 overlap and scored by its BEST window (embeddings: number[][], max not mean). Token coverage 87% → 100%; "proxy configuration blocking app downloads" went from absent to 0.559 at rank 1; the ten reference queries kept byte-identical scores. Costs +31% vectors, index 2.63 → 3.12 MB. The rules corpus gained the same treatment (128 rules → 172 vectors). (2) Keyword fallback — when the semantic pass returns fewer than topK, results are topped up via literal lexicalMatch on distinctive query terms, requiring two matching terms for corroboration (one incidental substring match returned a page about LLM providers for a sourdough query). Fuzzy matching was measured and rejected: at a threshold low enough to surface "RealmJoin" it invents 5 hits for "SCEPman", which the corpus never mentions. Also sharpened the tool description with the measured context argument for preferring it over fetching the docs site.

• Creation: Added mcp/docs-corpus.md (new mcp/ section) — the MCP server gains search_docs, a third semantic corpus built from the published customer documentation repository (autopilotmonitor-docs, checked out into the Docker build context by the deploy workflow). Documents the h2/h3 + <details> chunking with breadcrumb prefixes, the two deliberate size exceptions (tables never split, unbroken changelog lists split at line level), the CRLF precondition (. does not match \r in JS — headings silently stop matching, which produced 286 wrong-but-plausible chunks before it was found), the optional docs section in search-index.json and its validation asymmetry, embedding rounding to 6 decimals, and the cross-repo coupling: a docs edit reaches clients only after an MCP redeploy, made checkable via DOCS_COMMIT on /health. Container resources raised to 0.5 vCPU / 1.0 GiB for the cold-start query-embedder load.

2026-07-20

• Update: rules/gather-rule-phase-scoping.md — the gather-rule form now shows the phase-scope and emit-mode controls only where they can change behavior (supportsPhaseScope / supportsEmitMode), nulls hidden-control state out of the payload, and rejects a scope mode selected but left unfilled. Offering all three "when" controls unconditionally allowed combinations that could only suppress a one-shot rule, plus a silent no-op state.

• Update: Extended rules/gather-rule-phase-scoping.md with the phase_exit trigger — the closing bookend to phase_change, for one-shot collection when a phase is LEFT (e.g. "state at the end of Account Setup"). Documents the evaluation-before-phase-advance rule (triggerPhase match and scope gate see the phase being left), firing into Failed, the disjoint exit: dedup key space, and why no ConfigVersion bump is needed (no new field — old agents simply never fire the rule).

• Update: Reworked agent/mdm-reboot-coalescing.md after the first real enrollment (session b2e890c1): 2800 records are now aggregated per burst into ONE neutral Info event (rebootUris[]/uriCount/firstRebootUri; watermark persisted at flush so a pre-emit reboot kill loses nothing); the raw event no longer claims a reboot — ANALYZE-ESP-005 is gated on an actually-observed system_reboot_detected. Removed the Shell-Core esp_reboot_coalescing event entirely: the "RebootCoalescing" token appears in the routine SubcategoryProcessing_Started bootstrap marker on every enrollment (false positive by construction). Verified 2800 description shape recorded.

• Creation: Added agent/mdm-reboot-coalescing.md — the agent now attributes the "unexpected mid-ESP reboot + second sign-in" pattern (PMPC research) to the device-assigned policy URIs that forced it: new MdmRebootPolicyTracker (DM-Enterprise EventID 2800, per-URI mdm_policy_reboot_required events, cross-restart RecordId watermark), a RebootCoalescing branch in ShellCoreTracker (esp_reboot_coalescing, backfill-emitting by design), and the ANALYZE-ESP-005 advisory (reassign device → user groups). ConfigVersion 35; event types deliberately distinct from system_reboot_detected so RebootCount stays uninflated.

• Creation: Added backend/business-timestamps.md (new backend/ section) — the 2026-07-18 migration reset every row's system Timestamp, exposing that AuditLogs/OpsEvents/Events had (unknowingly) displayed and filtered on it all along; documents the OccurredUtc column, the RowKey decoders and RowKey-range date/retention clauses in BusinessTimestamp, the GUID-guard trap, the audit/ops backfill endpoint, and the "system Timestamp is never authoritative" migration rule.

• Creation: Added rules/gather-rule-phase-scoping.md (new rules/ section) — gather rules gain activePhases (run only during listed phases), activeFromPhase (sticky latch from a phase onwards, never via Failed) and emitMode: "on_change" (canonical-hash result dedup with suppressedPolls observability), ConfigVersion 34. Includes the prerequisite backend fix: the custom-rule toggle PUT no longer wipes the stored rule definition (partial-merge in GatherRuleService.UpdateRuleAsync).

2026-07-18

• Creation: Added agent/endpoint-migration.md — config-channel agent re-home (MigrateToApiBaseUrl on GET agent/config, ConfigVersion 33): backend or per-tenant API-URL moves without DNS indirection (impossible due to mTLS/Flex-TLS). Live-fetch-only with cache strip, AgentEndpointMigrationRules allowlist validated on both sides, one hop, kill-switch wins. Built as the lesson from the 2026-07-18 WEU→GWC cutover, which hard-cut all in-flight agents.

2026-07-17

• Creation: Added agent/build-counter-blob.md — agent build numbers now come from a shared public counter blob (buildcounter-v{N}.txt, reserve-before-build with ETag-CAS) so local build.ps1 and CI build-agent.yml builds can never mint the same number. Documents the CI parity fixes (SummaryDialog.exe.config in the ZIP, EXE-vs-manifest version check, hash-oracle update gated on stable cutover) and the Sigstore provenance attestation on CI-built ZIPs.

2026-07-16

• Creation: Added agent/autopilot-ztd-diagnostics.md — reference for Windows' own ZTD diagnostic surfaces (ModernDeployment Autopilot event IDs incl. 807/809/815/908, the Diagnostics\Autopilot registry key, deployment-service endpoints, known-issue error-code map with KB fixes). Backs the agent's ZtdEvidence collector (ztdVerdict on autopilot_profile_missing) and the backend known-issue rules; sources carry a re-check RSS feed.

2026-07-13

• Removal: Moved customer-graph-add-on-permissions.md out of this contributor bundle into the customer documentation repository (reference/optional-graph-permissions.md). It was customer-facing product content and did not belong in the tech-docs bundle; removed the entry from index.md.

• Creation: Added agent/hello-wizard-unskip.md — session 772fe502 fix: the HelloWizardStarted decision signal (prevention veto + un-skip cure for the policy-disabled Hello shortcut) and the HelloTracker confirmation second read against flip-flopping WHfB CSP values.

2026-07-11

• Creation: Added the agent/ section — three concept documents on the V2 agent: agent/overview.md (runtime, collectors, telemetry pipeline), agent/decision-engine.md (DecisionCore reducer, signals, A/B/C completion arms, invariants), agent/logs-and-persistence.md (persisted artifacts, signal log vs spool, recovery/replay flow) plus agent/index.md.

• Creation: Converted docs/ into an OKF v0.1 knowledge bundle — added YAML frontmatter to architecture.md, mcp-oauth-flow.md, and customer-graph-add-on-permissions.md; created index.md and this log.