AutopilotMonitor/docs/log.md

Log

2026-07-27

• Creation: Added backend/device-journeys.md — F2 device history / First-Time-Right: the device key (trim + case-fold serial, junk/placeholder exclusion incl. the agent's Unknown WMI sentinel), journey grouping over the terminal set Succeeded/Failed/Incomplete (first success ends a journey, a >30-day gap from the previous attempt's CompletedAt starts one, WhiteGlove's single session row is ONE attempt, open sessions are never attempts), the DeviceHistories chain (cap 20, written inline at the terminal seam, healed by the sweep, deleted-session refs pruned tombstone-driven, deliberately not in the per-session deletion manifest) and the additive DeviceJourneyAggregates daily rows (window rate = ratio of sums — no rolling row needed, 180-day retention). Surfaces: the session-detail "Attempt N" banner with server-computed attempt numbers (virtual-attempt rule for live sessions), the Fleet Health First-time-right section (≥20-journey gate, attempt histogram, repeat-devices violator list), and the MCP get_device_history tool (serial mode / fleet FTR mode).

2026-07-25

• Update: agent/secureboot-ca2023-detection.md — review follow-up on 90012d9 hardened the privilege scope, the repo's only privilege-adjustment code and therefore the precedent for any future one. The process token moved from a raw IntPtr to a SafeHandle assigned by the P/Invoke marshaler, so the CLR owns it from the moment OpenProcessToken returns and the critical finalizer releases it even when Dispose never runs — previously an async exception between the successful open and the caller's using leaked the handle with the privilege enabled and no owner left to restore it. Dispose became idempotent and now releases the handle in a finally, so a throwing restore cannot strand it; the restore obligation is recorded immediately after the successful AdjustTokenPrivileges with nothing throwable in between. The whole enable → read → restore sequence runs under a static gate because the privilege is adjusted on the process token: a concurrent reader would see "already enabled", correctly skip its own restore, and then lose the privilege mid-read when the first reader disposed, reporting privilege_denied on a machine that holds it.

2026-07-24

• Creation: Added backend/server-actions-on-demand-diagnostics.md — the ServerAction channel (queue on Sessions.PendingActionsJson, deliver piggy-backed on the next ingest response, at-least-once with Type|RuleId|QueuedAt dedup and sequential dispatch) and the on-demand log collection built on it: the portal's always-visible Collect Logs button, the new EndpointPolicy.TenantAdminOrOperator tier with the mandatory in-function per-type re-gate (Operators queue only request_diagnostics), the member-readable diagnosticsUploadConfigured feature flag, the admin quick-config path that must queue rotate_config before request_diagnostics (works against the deployed fleet — a one-time override would not, the mode gate is agent-side), sequence-based progress matching immune to agent clock skew, and the offboarding cascade's hosted-prefix sweep closing the latest-wins orphan-blob gap.

• Update: agent/esp-policy-provider-stall.md — the customer's follow-up on issue #106 corrected the field-case mechanics: TrackingPoliciesCreated=1 WAS set under the foreign ConfigMgr provider, so the ESP ignores a foreign provider's completion values and waits on the Intune-registered Sidecar provider by name (the rename to Sidecar creates no tracking entries yet unblocks instantly). The completeness-keyed detector would have missed exactly this — the incident it was built for. The tripwire is now two-armed under one event type discriminated by reason: provider_incomplete (unchanged dwell on continuously-incomplete providers) and sidecar_provider_missing (≥ 1 Setup/Apps provider registered without Sidecar, completeness deliberately ignored, clock reset when Sidecar registers later). Sidecar-missing wins the reason when both arms cross in one round; the cross-restart fingerprint gains a sidecarMissing sentinel; payload adds reason and sidecarMissingForMinutes. Evidence for the name-keyed wait: Niehaus' documented Intune WAP payload always registers Sidecar (ConfigMgr only via a co-management settings policy) and Microsoft's ESP troubleshooting models app tracking exclusively through the Sidecar subkey.

• Creation: Added agent/esp-policy-provider-stall.md — a customer diagnosed that pre-provisioned devices with the ConfigMgr client as a blocking app occasionally park the user ESP at "Apps (Identifying)" for days: only a ConfigMgr provider is registered under EnrollmentStatusTracking\Device\Setup\Apps\PolicyProviders (no Sidecar = IME) and TrackingPoliciesCreated never becomes 1 — a wait the CSP defines without any timeout; renaming the key to Sidecar unblocks immediately. The agent now ships the always-on esp_policy_provider_stalled tripwire (60-s tick, 15-min continuous-incompleteness dwell, one-shot per provider, StartupEventGate fingerprint across restarts, purely observational via the InformationalEvent rail — no toggle, no ConfigVersion bump), generalized to any provider in both Setup/Apps and DevicePreparation kinds rather than hardcoding the ConfigMgr case. Companion change: SOFTWARE\Microsoft\Windows\Autopilot joins the gather-rule registry allowlist so tenants can target the subtree with custom rules (e.g. not_exists on the Sidecar key).

• Creation: Added agent/secureboot-ca2023-detection.md — ANALYZE-SEC-001 warned on devices whose firmware already carried the 2023 certificate, because the agent only read the SecureBoot\Servicing\UEFICA2023Status registry value, which mirrors the Windows Update rollout rather than the firmware (customer report). The agent now reads the UEFI db/KEK variables natively (GetFirmwareEnvironmentVariableExW with SeSystemEnvironmentPrivilege enabled on the process token — first privilege-adjustment code in the repo) and emits per-certificate booleans on secureboot_status. Suppression rides on a one-sided marker, uefiCa2023FirmwareConfirmed, emitted only on positive confirmation: the engine's not_exists precondition fails on any present value, so a two-sided boolean could not distinguish "confirmed" from "checked and missing", while the marker's absence keeps old agents and firmware-missing devices on the v2 path. Rule v3.0.0 is therefore safe to seed before the agent release.

• Update: url-registry.md — the first release after the alias migration surfaced Front Door caching as a correctness hazard: the alias served the previous version.json (X-Cache: TCP_REMOTE_HIT) after the origin had rotated, and since manifest and ZIP are independent cache entries, a stale manifest with a fresh ZIP fails the bootstrap SHA-256 check. Route caching on /agent/* is now disabled and every blob upload stamps Cache-Control: no-cache (CI Send-Blob, local build.ps1, versions/*.json manifests) — all agent-container blobs rotate in place, so nothing there is safely cacheable.

• Update: agent/decision-engine.md — the RealmJoin completion gate has a third terminal, FirstDeploymentIncomplete. Session 224b2087 showed why: an interactive logon 17 s after DeploymentPhase 101 makes RealmJoin reclassify the run as a secondary-user deployment, whose branch writes 200/210 and never 110 — the gate stayed closed until the 60-min hard timeout although Hello and desktop were long resolved. realmjoin_phase_changed is now a typed RealmJoinPhaseChanged signal, so RealmJoinFacts.LastDeploymentPhase is persisted across agent restarts (this also fixes every realmjoin_timeout event claiming "last phase: 0"); a transition out of 100/101 into 200/210 releases the gate with the realmjoin_first_deployment_incomplete Warning, and the RealmJoinDetected replay path evaluates the same rule for restarts that swallow the transition. A first observation of 210 deliberately does not release — session 6f1959c0 started at 210 and only deployed afterwards.

• Creation: Added url-registry.md — all well-known hosts (own subdomains, both blob accounts, API origin, Graph, Entra login) are now defined once per component (Constants.cs, web utils/config.ts, MCP src/config.ts) and referenced everywhere else; three guard tests fail the build on any literal outside the registries. Written because the EU cutover missed two hardcoded copies of the legacy blob host — LatestVersionsService (feeds the portal's version badge) and ValidateBootstrapCodeFunction — which kept working only through the fail-soft legacy mirror in build-agent.yml; those two plus the agent SelfUpdater now use the download.autopilotmonitor.com alias (verified serving all four artifacts first). The portal's bootstrap allow-list accepts alias AND legacy host during the transition, which imposes a deploy order: web before (or with) backend. The sweep also surfaced a fourth copy of the www/portal host pair in middleware.ts (now derived from the registry via hostRouting.ts) and a backend short-link built on the bare apex domain, which only worked through a registrar-level 301.

2026-07-22

• Creation: Added versioning.md — agent, backend and MCP now share one version shape (<major>.<minor>.<build> + 7-char commit + build timestamp) and one implementation of the build-number reservation (.github/scripts/Request-BuildNumber.ps1, previously copy-pasted between build-agent.yml and build.ps1). The backend had been pinned at a hand-written 1.0.0 since inception and now starts at 1.5.1000; the MCP server's package.json version was bumped by hand and now starts at 1.5.300, and it reports its own commit on /health instead of only the documentation bundle's. Both deploy workflows poll their health endpoint until it reports the build just deployed and publish versions/<component>.json only then, so a manifest that disagrees with the live endpoint means the instance drifted rather than the pipeline failing halfway. agent/build-counter-blob.md keeps the agent-specific parts and now points at the shared protocol; its counter URL was still the pre-cutover storage account.

• Creation: Added rules/gather-rule-guardrails.md — the gather-rule allowlists are enforced only on the agent (the backend validates no collectorType, target, or parameters), so a collector that forgets to call its guard is the vulnerability. Written because two had: LogParserCollector read any path in rule.Target, bypassing even the C:\Users hard block that every other file collector honours, and EventLogCollector read any channel including Security — while the customer-facing documentation already promised both were allowlisted. Both now guard, with logparser checking the directory for wildcard targets (Path.GetFullPath throws on *) and re-checking each resolved path. Added an event-log channel allowlist (eventLogChannels) plus hard-blocked channels held in C# rather than JSON, so a parse error cannot lift them. Also fixed the command timeout, which was inert: ReadToEnd() blocks until the pipe closes, so WaitForExit(30000) was only reached after the process had already exited and a hung command pinned its worker forever — now read asynchronously and killed at the timeout.

• Update: Added .github/workflows/ci.yml — the full suite (backend, agent V2 Core, DecisionCore, SummaryDialog, web, MCP) now runs on every pull request and every push to main, plus a job that fails if the generated guardrail files drift from rules/guardrails.json. Previously no workflow ran tests on a PR: the deploy workflows build without testing, and build-agent.yml tested only DecisionCore.Tests on manual dispatch — while trust/security-faq.md already told customers that automated tests run in CI. Added CodeQL (C# and TypeScript) and Dependabot (nuget, npm ×2, actions). Marked OfficeBinaryWatcherTests [Collection("SerialThreading")]: its bounded re-probe races the shared ThreadPool and failed once under full-suite load.

2026-07-21

• Update: search_docs and search_knowledge now state in their tool descriptions that queries must be in English. Both corpora and all-MiniLM-L6-v2 are English-only, and the failure is total rather than gradual — a German question returns zero results, not weak ones. The caller is a language model, so translating the question before the call and answering in the user's language costs nothing; a multilingual embedding model would be the alternative but is much larger and would have to be measured against the eval harness first, since it changes English retrieval too.

• Update: Added scripts/eval-docs-search.ts, an offline retrieval evaluation harness (2465 queries in three classes: chunk headings as a regression detector, rare identifiers, and hand-written questions as the reality anchor), and used it to close a third search_docs gap found during the live test. The keyword fallback only fired when semantic search returned fewer than topK; a pasted identifier like WOW6432Node instead returned three chunks at ~0.29 that did not contain the word while the one chunk that did was never considered. It now also fires on a weak semantic head (<0.35), letting exact matches take the front while keeping one semantic result. The threshold is measured, not guessed: 0.35 fixes 335 queries and breaks 0 (identifier recall 66.2% → 81.1%), while 0.40 fixes more but starts demoting hand-written questions. The measurement also resized the problem — identifier queries were only succeeding 66.2% of the time, so the reported case was one of roughly 750.

• Update: Closed the two measured recall gaps in search_docs (mcp/docs-corpus.md). (1) Token windows — 87 of 324 chunks exceeded the model's 256-token context and 13% of all tokens never reached the ranking; a character budget cannot fix this (the corpus ranges 2.43–5.81 chars/token), so long text is now embedded as 250-token windows with 40 overlap and scored by its BEST window (embeddings: number[][], max not mean). Token coverage 87% → 100%; "proxy configuration blocking app downloads" went from absent to 0.559 at rank 1; the ten reference queries kept byte-identical scores. Costs +31% vectors, index 2.63 → 3.12 MB. The rules corpus gained the same treatment (128 rules → 172 vectors). (2) Keyword fallback — when the semantic pass returns fewer than topK, results are topped up via literal lexicalMatch on distinctive query terms, requiring two matching terms for corroboration (one incidental substring match returned a page about LLM providers for a sourdough query). Fuzzy matching was measured and rejected: at a threshold low enough to surface "RealmJoin" it invents 5 hits for "SCEPman", which the corpus never mentions. Also sharpened the tool description with the measured context argument for preferring it over fetching the docs site.

• Creation: Added mcp/docs-corpus.md (new mcp/ section) — the MCP server gains search_docs, a third semantic corpus built from the published customer documentation repository (autopilotmonitor-docs, checked out into the Docker build context by the deploy workflow). Documents the h2/h3 + <details> chunking with breadcrumb prefixes, the two deliberate size exceptions (tables never split, unbroken changelog lists split at line level), the CRLF precondition (. does not match \r in JS — headings silently stop matching, which produced 286 wrong-but-plausible chunks before it was found), the optional docs section in search-index.json and its validation asymmetry, embedding rounding to 6 decimals, and the cross-repo coupling: a docs edit reaches clients only after an MCP redeploy, made checkable via DOCS_COMMIT on /health. Container resources raised to 0.5 vCPU / 1.0 GiB for the cold-start query-embedder load.

2026-07-20

• Update: rules/gather-rule-phase-scoping.md — the gather-rule form now shows the phase-scope and emit-mode controls only where they can change behavior (supportsPhaseScope / supportsEmitMode), nulls hidden-control state out of the payload, and rejects a scope mode selected but left unfilled. Offering all three "when" controls unconditionally allowed combinations that could only suppress a one-shot rule, plus a silent no-op state.

• Update: Extended rules/gather-rule-phase-scoping.md with the phase_exit trigger — the closing bookend to phase_change, for one-shot collection when a phase is LEFT (e.g. "state at the end of Account Setup"). Documents the evaluation-before-phase-advance rule (triggerPhase match and scope gate see the phase being left), firing into Failed, the disjoint exit: dedup key space, and why no ConfigVersion bump is needed (no new field — old agents simply never fire the rule).

• Update: Reworked agent/mdm-reboot-coalescing.md after the first real enrollment (session b2e890c1): 2800 records are now aggregated per burst into ONE neutral Info event (rebootUris[]/uriCount/firstRebootUri; watermark persisted at flush so a pre-emit reboot kill loses nothing); the raw event no longer claims a reboot — ANALYZE-ESP-005 is gated on an actually-observed system_reboot_detected. Removed the Shell-Core esp_reboot_coalescing event entirely: the "RebootCoalescing" token appears in the routine SubcategoryProcessing_Started bootstrap marker on every enrollment (false positive by construction). Verified 2800 description shape recorded.

• Creation: Added agent/mdm-reboot-coalescing.md — the agent now attributes the "unexpected mid-ESP reboot + second sign-in" pattern (PMPC research) to the device-assigned policy URIs that forced it: new MdmRebootPolicyTracker (DM-Enterprise EventID 2800, per-URI mdm_policy_reboot_required events, cross-restart RecordId watermark), a RebootCoalescing branch in ShellCoreTracker (esp_reboot_coalescing, backfill-emitting by design), and the ANALYZE-ESP-005 advisory (reassign device → user groups). ConfigVersion 35; event types deliberately distinct from system_reboot_detected so RebootCount stays uninflated.

• Creation: Added backend/business-timestamps.md (new backend/ section) — the 2026-07-18 migration reset every row's system Timestamp, exposing that AuditLogs/OpsEvents/Events had (unknowingly) displayed and filtered on it all along; documents the OccurredUtc column, the RowKey decoders and RowKey-range date/retention clauses in BusinessTimestamp, the GUID-guard trap, the audit/ops backfill endpoint, and the "system Timestamp is never authoritative" migration rule.

• Creation: Added rules/gather-rule-phase-scoping.md (new rules/ section) — gather rules gain activePhases (run only during listed phases), activeFromPhase (sticky latch from a phase onwards, never via Failed) and emitMode: "on_change" (canonical-hash result dedup with suppressedPolls observability), ConfigVersion 34. Includes the prerequisite backend fix: the custom-rule toggle PUT no longer wipes the stored rule definition (partial-merge in GatherRuleService.UpdateRuleAsync).

2026-07-18

• Creation: Added agent/endpoint-migration.md — config-channel agent re-home (MigrateToApiBaseUrl on GET agent/config, ConfigVersion 33): backend or per-tenant API-URL moves without DNS indirection (impossible due to mTLS/Flex-TLS). Live-fetch-only with cache strip, AgentEndpointMigrationRules allowlist validated on both sides, one hop, kill-switch wins. Built as the lesson from the 2026-07-18 WEU→GWC cutover, which hard-cut all in-flight agents.

2026-07-26

• Creation: Added backend/time-attribution.md — the F1 enrollment time attribution: exact-partition invariant against the session's authoritative DurationSeconds, two-window WhiteGlove model, segment mapping from phase-declaration events, positive-evidence ESP-blocking join with "up to" what-if bounds, reboot spans from event gaps around lastBootUtc, and the SessionTimeBreakdowns / TimeAttributionAggregates tables with their terminal-seam + maintenance-sweep compute paths and read surfaces (web lane, fleet panel, MCP get_time_attribution).

2026-07-17

• Creation: Added agent/build-counter-blob.md — agent build numbers now come from a shared public counter blob (buildcounter-v{N}.txt, reserve-before-build with ETag-CAS) so local build.ps1 and CI build-agent.yml builds can never mint the same number. Documents the CI parity fixes (SummaryDialog.exe.config in the ZIP, EXE-vs-manifest version check, hash-oracle update gated on stable cutover) and the Sigstore provenance attestation on CI-built ZIPs.

2026-07-16

• Creation: Added agent/autopilot-ztd-diagnostics.md — reference for Windows' own ZTD diagnostic surfaces (ModernDeployment Autopilot event IDs incl. 807/809/815/908, the Diagnostics\Autopilot registry key, deployment-service endpoints, known-issue error-code map with KB fixes). Backs the agent's ZtdEvidence collector (ztdVerdict on autopilot_profile_missing) and the backend known-issue rules; sources carry a re-check RSS feed.

2026-07-13

• Removal: Moved customer-graph-add-on-permissions.md out of this contributor bundle into the customer documentation repository (reference/optional-graph-permissions.md). It was customer-facing product content and did not belong in the tech-docs bundle; removed the entry from index.md.

• Creation: Added agent/hello-wizard-unskip.md — session 772fe502 fix: the HelloWizardStarted decision signal (prevention veto + un-skip cure for the policy-disabled Hello shortcut) and the HelloTracker confirmation second read against flip-flopping WHfB CSP values.

2026-07-11

• Creation: Added the agent/ section — three concept documents on the V2 agent: agent/overview.md (runtime, collectors, telemetry pipeline), agent/decision-engine.md (DecisionCore reducer, signals, A/B/C completion arms, invariants), agent/logs-and-persistence.md (persisted artifacts, signal log vs spool, recovery/replay flow) plus agent/index.md.

• Creation: Converted docs/ into an OKF v0.1 knowledge bundle — added YAML frontmatter to architecture.md, mcp-oauth-flow.md, and customer-graph-add-on-permissions.md; created index.md and this log.