diff --git a/docs/agent/index.md b/docs/agent/index.md index 5d9421b9..984f0320 100644 --- a/docs/agent/index.md +++ b/docs/agent/index.md @@ -7,5 +7,6 @@ How the on-device agent works: * [Logs, Persistence & Crash Recovery](logs-and-persistence.md) - every persisted file and why it exists; signal log vs spool; snapshot/journal; the replay-based recovery flow and quarantine. * [Hello Wizard Un-Skip](hello-wizard-unskip.md) - why a single "WHfB disabled" policy read never decides completion: the HelloWizardStarted signal (prevention + cure) and the HelloTracker confirmation second read. * [Autopilot ZTD Diagnostics](autopilot-ztd-diagnostics.md) - Windows' own diagnostic surfaces for the profile-download flow: ModernDeployment event IDs (807/815/164/...), the Diagnostics\Autopilot registry key, deployment-service endpoints, and the known-issue error-code map. +* [MDM Reboot Coalescing](mdm-reboot-coalescing.md) - attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: 2800 watcher with cross-restart watermark, Shell-Core RebootCoalescing corroboration, ANALYZE-ESP-005 advisory. Reading order for newcomers: overview → decision engine → persistence. diff --git a/docs/agent/mdm-reboot-coalescing.md b/docs/agent/mdm-reboot-coalescing.md new file mode 100644 index 00000000..3faca9e5 --- /dev/null +++ b/docs/agent/mdm-reboot-coalescing.md @@ -0,0 +1,109 @@ +--- +type: Concept +title: MDM Reboot Coalescing — attributing the mid-ESP reboot and second sign-in to policy URIs +description: How the agent detects device-assigned MDM policies that force a coalesced reboot during ESP DeviceSetup (DM-Enterprise EventID 2800 + Shell-Core 62407 RebootCoalescing), and the ANALYZE-ESP-005 advisory that makes the invisible "second sign-in" fixable. +resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals +tags: + - agent + - esp + - reboot + - policy-attribution + - second-sign-in +timestamp: 2026-07-20T00:00:00+02:00 +--- + +# MDM Reboot Coalescing + +Device-assigned MDM policies whose CSP URIs match the OS reboot-required catalog +(`HKLM\SOFTWARE\Microsoft\Provisioning\SyncML\RebootRequiredURIs`) force a **coalesced +reboot at the end of ESP DeviceSetup** — the user sees an unexpected restart and a +**second sign-in screen** before AccountSetup. This is expected OS behavior, invisible in +the Intune console; most admins simply live with it, not knowing that reassigning the +profiles to user groups removes the reboot entirely. Source research: Patch My PC, +["Avoid the second sign-in screen after Autopilot"](https://patchmypc.com/blog/autopilot-unexpected-reboot-autopilot-second-login-screen/). + +The session already counts reboots (`system_reboot_detected` → `session.rebootCount`); +this feature adds the **cause attribution**. + +# Signal chain + +1. **EventID 2800**, `Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin` + — logged by the SyncML engine once per policy URI that matched the reboot-required + catalog, during device-policy application in DeviceSetup. +2. **EventID 62407**, `Microsoft-Windows-Shell-Core/Operational`, description containing + `RebootCoalescing` — ESP/CloudExperienceHost initiating the coalesced reboot at the end + of DeviceSetup. +3. The reboot kills the agent; the post-reboot restart re-observes both channels via + backfill (the .evtx persists across the reboot). + +# Detectors + +## `MdmRebootPolicyTracker` (new collector, `MdmRebootPolicyWatcherHost`) + +Structural clone of `WindowsUpdateTracker`: live `EventLogWatcher` on the DM-Enterprise +Admin channel (EventID 2800) + startup backfill (default 60 min lookback) + **cross-restart +RecordId watermark** (`mdm-reboot-watermark.json`). The watermark is mandatory here: the +coalesced reboot restarts the agent, and without it every reboot-forcing URI would be +re-emitted after the very reboot it caused. + +Per 2800 record it emits one `mdm_policy_reboot_required` event (Warning, +`ImmediateUpload=true` — the announced reboot will kill the agent, flush now) with +`Data.rebootUri` (the CSP URI), `recordId`, `backfilled`, `description` (truncated 1000), +and a fail-soft `omadmRebootRequiredFlag` corroboration probe of +`HKLM\SOFTWARE\Microsoft\Provisioning\OMADM\SyncML\RebootRequired` (null = unknown, never +blocks the emit). Per-URI emission (not aggregated) is deliberate: an aggregate would need +a flush point that races the reboot kill. + +**Tolerant matching**: the exact 2800 message text is unverified — emission is gated on +the EventID only. URI extraction prefers structured EventData values starting with `./` +and falls back to a permissive regex over the description (trailing punctuation trimmed). +Events with an unparseable URI are still emitted (timeline-visible, rule stays silent, and +the captured description tells us how to fix the extraction). Exact texts are validated +post-deploy on the E2E VM and recorded here. + +Config (internal steering only, no admin UI — this is an agent-native collector, NOT an +`IAgentAnalyzer`): `Collectors.MdmRebootPolicyWatcherEnabled` (default true), +`Collectors.MdmRebootPolicyBackfillLookbackMinutes` (default 60). ConfigVersion 35. + +## `ShellCoreTracker` — `RebootCoalescing` branch + +The already-watched 62407 stream gets a fourth branch (checked AFTER +WhiteGlove/failure/exiting so their semantics stay untouched): description containing +`RebootCoalescing` → one `esp_reboot_coalescing` event (Warning, ImmediateUpload), pure +informational — no `FinalizingSetup` transition, no C# event raise, no DecisionCore +involvement. + +**Deliberate deviation**: unlike the other 62407 branches, the backfill path DOES emit +this event. The live emit races the very reboot that kills the process; the post-restart +5-min backfill re-emission is the only reliable delivery. A live+backfill cross-process +duplicate is possible and accepted (fire-once bool resets per process; both copies carry +the identical historical timestamp; the backfill copy is marked `backfilled:true`; the +analyze rule uses exists-semantics). The shared 5-min lookback is NOT enlarged — the +rule's required condition is the watermarked 2800 event; coalescing is optional +corroboration only. + +# Advisory — ANALYZE-ESP-005 + +"MDM Policy Forced a Mid-ESP Reboot (Second Sign-In)", severity `warning`, +`markSessionAsFailedDefault: false`. Required: `mdm_policy_reboot_required.rebootUri` +exists (first URI interpolates as `{{rebootUri}}`). Optional corroboration: +`esp_reboot_coalescing` observed (+20 confidence) and ≥2 reboot URIs (+10) on base 70 / +threshold 70. Remediation: map each URI to its Intune profile and reassign device → user +groups (or exclude the Autopilot device group), or accept + communicate the restart. +Because `{{token}}` interpolation is scalar-only, the explanation points at the timeline +for the complete URI list — each URI is its own event. + +# Non-goals / invariants + +* The new event types are **deliberately distinct from `system_reboot_detected`** — the + backend RebootCount matches that string exactly in two places + (`EventIngestProcessor.Classification` + the terminal recount); attribution events must + never inflate the reboot count. +* No session-level field / SessionInfoCard change in this iteration (follow-up candidate). +* The `RebootRequiredURIs` catalog itself is not read (format unverified); only the 2800 + events (which carry the actually-triggered URIs) are consumed. + +# Citations + +* [Patch My PC — Avoid the second sign-in screen after Autopilot](https://patchmypc.com/blog/autopilot-unexpected-reboot-autopilot-second-login-screen/) +* [Microsoft Learn — Enrollment Status Page tracking](https://learn.microsoft.com/en-us/autopilot/enrollment-status) diff --git a/docs/index.md b/docs/index.md index 5ea1b038..cadebde6 100644 --- a/docs/index.md +++ b/docs/index.md @@ -18,6 +18,7 @@ https://docs.autopilotmonitor.com (separate repository). * [V2 Agent](agent/index.md) - How the on-device agent works: [runtime overview](agent/overview.md), [decision engine](agent/decision-engine.md) (reducer, signals, completion arms), [logs & persistence](agent/logs-and-persistence.md) (signal log, snapshot, crash recovery), [Hello wizard un-skip](agent/hello-wizard-unskip.md) (why a single "WHfB disabled" read never decides completion), [Autopilot ZTD diagnostics](agent/autopilot-ztd-diagnostics.md) (event IDs, diagnostic registry, endpoints, error-code map). * [Build Counter Blob](agent/build-counter-blob.md) - How agent build numbers are minted: shared counter blob with ETag-CAS so local and CI builds never collide; CI additionally attests build provenance (Sigstore). * [Agent Endpoint Migration](agent/endpoint-migration.md) - How the backend re-homes agents to a new API base URL via the config channel (backend move or per-tenant region move): live-fetch-only, allowlist-validated on both sides, one hop, kill wins. +* [MDM Reboot Coalescing](agent/mdm-reboot-coalescing.md) - Attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: DM-Enterprise 2800 watcher with cross-restart watermark, Shell-Core RebootCoalescing corroboration, ANALYZE-ESP-005 advisory. # Rules diff --git a/docs/log.md b/docs/log.md index 3729632f..574d4095 100644 --- a/docs/log.md +++ b/docs/log.md @@ -2,6 +2,8 @@ ## 2026-07-20 +* **Creation**: Added `agent/mdm-reboot-coalescing.md` — the agent now attributes the "unexpected mid-ESP reboot + second sign-in" pattern (PMPC research) to the device-assigned policy URIs that forced it: new `MdmRebootPolicyTracker` (DM-Enterprise EventID 2800, per-URI `mdm_policy_reboot_required` events, cross-restart RecordId watermark), a `RebootCoalescing` branch in `ShellCoreTracker` (`esp_reboot_coalescing`, backfill-emitting by design), and the `ANALYZE-ESP-005` advisory (reassign device → user groups). ConfigVersion 35; event types deliberately distinct from `system_reboot_detected` so RebootCount stays uninflated. + * **Creation**: Added `backend/business-timestamps.md` (new `backend/` section) — the 2026-07-18 migration reset every row's system Timestamp, exposing that AuditLogs/OpsEvents/Events had (unknowingly) displayed and filtered on it all along; documents the `OccurredUtc` column, the RowKey decoders and RowKey-range date/retention clauses in `BusinessTimestamp`, the GUID-guard trap, the audit/ops backfill endpoint, and the "system Timestamp is never authoritative" migration rule. * **Creation**: Added `rules/gather-rule-phase-scoping.md` (new `rules/` section) — gather rules gain `activePhases` (run only during listed phases), `activeFromPhase` (sticky latch from a phase onwards, never via Failed) and `emitMode: "on_change"` (canonical-hash result dedup with `suppressedPolls` observability), ConfigVersion 34. Includes the prerequisite backend fix: the custom-rule toggle PUT no longer wipes the stored rule definition (partial-merge in `GatherRuleService.UpdateRuleAsync`). diff --git a/rules/analyze/ANALYZE-ESP-005.json b/rules/analyze/ANALYZE-ESP-005.json new file mode 100644 index 00000000..4cb16873 --- /dev/null +++ b/rules/analyze/ANALYZE-ESP-005.json @@ -0,0 +1,74 @@ +{ + "$schema": "../schema/analyze-rule.schema.json", + "ruleId": "ANALYZE-ESP-005", + "title": "MDM Policy Forced a Mid-ESP Reboot (Second Sign-In)", + "description": "Detects the 'unexpected reboot + second sign-in screen' pattern: one or more device-assigned MDM policies matched the OS reboot-required catalog during ESP DeviceSetup, and Windows coalesced them into a restart at the end of DeviceSetup — forcing the user through a second sign-in before AccountSetup. Attributes the reboot to the specific CSP policy URIs so the profiles can be reassigned.", + "severity": "warning", + "category": "esp", + "version": "1.0.0", + "author": "Autopilot Monitor", + "enabled": true, + "trigger": "single", + "baseConfidence": 70, + "conditions": [ + { + "signal": "reboot_forcing_policy_uri", + "source": "event_data", + "eventType": "mdm_policy_reboot_required", + "dataField": "rebootUri", + "operator": "exists", + "required": true + }, + { + "signal": "reboot_uri_count", + "source": "event_count", + "eventType": "mdm_policy_reboot_required", + "operator": "count_gte", + "value": "1", + "required": false + }, + { + "signal": "esp_coalesced_reboot_observed", + "source": "event_type", + "eventType": "esp_reboot_coalescing", + "operator": "exists", + "required": false + } + ], + "confidenceFactors": [ + { "signal": "esp_coalesced_reboot_observed", "condition": "exists", "weight": 20 }, + { "signal": "mdm_policy_reboot_required", "condition": "count >= 2", "weight": 10 } + ], + "confidenceThreshold": 70, + "explanation": "During ESP DeviceSetup, a device-assigned MDM policy matched the OS reboot-required catalog and Windows scheduled a **coalesced reboot** at the end of DeviceSetup — the user saw an unexpected restart and a **second sign-in screen** before AccountSetup continued.\n\nFirst reboot-forcing policy URI: `{{rebootUri}}`\n\nWhen several policies force a reboot they are coalesced into one restart; **each** reboot-forcing policy appears as its own `mdm_policy_reboot_required` event on the session timeline — check the timeline for the complete URI list.\n\nThis is expected OS behavior, not a failure — but it degrades the enrollment experience, confuses users, and is invisible in the Intune console.", + "remediation": [ + { + "title": "Reassign the reboot-forcing policies to user groups", + "steps": [ + "Open each `mdm_policy_reboot_required` timeline event and note the CSP URI (e.g. `./Device/Vendor/MSFT/...`)", + "Map each URI to its Intune profile (Settings Catalog / configuration profile) — the URI path names the CSP area", + "Reassign those profiles from device groups to user groups so they apply during/after AccountSetup instead of DeviceSetup", + "Alternatively, exclude the Autopilot device group from the profile and deliver the setting post-enrollment" + ] + }, + { + "title": "Or accept the reboot and set expectations", + "steps": [ + "If the policy must be device-assigned (e.g. a security baseline), keep it and document the expected mid-ESP restart", + "Communicate to users that a restart and second sign-in during provisioning is expected for these deployments" + ] + } + ], + "relatedDocs": [ + { + "title": "Patch My PC: Avoid the second sign-in screen after Autopilot", + "url": "https://patchmypc.com/blog/autopilot-unexpected-reboot-autopilot-second-login-screen/" + }, + { + "title": "Microsoft Learn: Enrollment Status Page tracking", + "url": "https://learn.microsoft.com/en-us/autopilot/enrollment-status" + } + ], + "markSessionAsFailedDefault": false, + "tags": ["esp", "reboot", "coalescing", "second-sign-in", "policy-attribution", "user-experience"] +} diff --git a/rules/dist/analyze-rules.json b/rules/dist/analyze-rules.json index 5c6fe740..f4143bf0 100644 --- a/rules/dist/analyze-rules.json +++ b/rules/dist/analyze-rules.json @@ -1785,6 +1785,94 @@ "enrollment" ] }, + { + "ruleId": "ANALYZE-ESP-005", + "title": "MDM Policy Forced a Mid-ESP Reboot (Second Sign-In)", + "description": "Detects the 'unexpected reboot + second sign-in screen' pattern: one or more device-assigned MDM policies matched the OS reboot-required catalog during ESP DeviceSetup, and Windows coalesced them into a restart at the end of DeviceSetup — forcing the user through a second sign-in before AccountSetup. Attributes the reboot to the specific CSP policy URIs so the profiles can be reassigned.", + "severity": "warning", + "category": "esp", + "version": "1.0.0", + "author": "Autopilot Monitor", + "enabled": true, + "trigger": "single", + "baseConfidence": 70, + "conditions": [ + { + "signal": "reboot_forcing_policy_uri", + "source": "event_data", + "eventType": "mdm_policy_reboot_required", + "dataField": "rebootUri", + "operator": "exists", + "required": true + }, + { + "signal": "reboot_uri_count", + "source": "event_count", + "eventType": "mdm_policy_reboot_required", + "operator": "count_gte", + "value": "1", + "required": false + }, + { + "signal": "esp_coalesced_reboot_observed", + "source": "event_type", + "eventType": "esp_reboot_coalescing", + "operator": "exists", + "required": false + } + ], + "confidenceFactors": [ + { + "signal": "esp_coalesced_reboot_observed", + "condition": "exists", + "weight": 20 + }, + { + "signal": "mdm_policy_reboot_required", + "condition": "count >= 2", + "weight": 10 + } + ], + "confidenceThreshold": 70, + "explanation": "During ESP DeviceSetup, a device-assigned MDM policy matched the OS reboot-required catalog and Windows scheduled a **coalesced reboot** at the end of DeviceSetup — the user saw an unexpected restart and a **second sign-in screen** before AccountSetup continued.\n\nFirst reboot-forcing policy URI: `{{rebootUri}}`\n\nWhen several policies force a reboot they are coalesced into one restart; **each** reboot-forcing policy appears as its own `mdm_policy_reboot_required` event on the session timeline — check the timeline for the complete URI list.\n\nThis is expected OS behavior, not a failure — but it degrades the enrollment experience, confuses users, and is invisible in the Intune console.", + "remediation": [ + { + "title": "Reassign the reboot-forcing policies to user groups", + "steps": [ + "Open each `mdm_policy_reboot_required` timeline event and note the CSP URI (e.g. `./Device/Vendor/MSFT/...`)", + "Map each URI to its Intune profile (Settings Catalog / configuration profile) — the URI path names the CSP area", + "Reassign those profiles from device groups to user groups so they apply during/after AccountSetup instead of DeviceSetup", + "Alternatively, exclude the Autopilot device group from the profile and deliver the setting post-enrollment" + ] + }, + { + "title": "Or accept the reboot and set expectations", + "steps": [ + "If the policy must be device-assigned (e.g. a security baseline), keep it and document the expected mid-ESP restart", + "Communicate to users that a restart and second sign-in during provisioning is expected for these deployments" + ] + } + ], + "relatedDocs": [ + { + "title": "Patch My PC: Avoid the second sign-in screen after Autopilot", + "url": "https://patchmypc.com/blog/autopilot-unexpected-reboot-autopilot-second-login-screen/" + }, + { + "title": "Microsoft Learn: Enrollment Status Page tracking", + "url": "https://learn.microsoft.com/en-us/autopilot/enrollment-status" + } + ], + "markSessionAsFailedDefault": false, + "tags": [ + "esp", + "reboot", + "coalescing", + "second-sign-in", + "policy-attribution", + "user-experience" + ] + }, { "ruleId": "ANALYZE-ID-001", "title": "Expected Machine Certificate Not Found", diff --git a/src/Agent/AutopilotMonitor.Agent.V2.Core.Tests/Monitoring/ShellCoreTrackerRebootCoalescingTests.cs b/src/Agent/AutopilotMonitor.Agent.V2.Core.Tests/Monitoring/ShellCoreTrackerRebootCoalescingTests.cs new file mode 100644 index 00000000..033c046e --- /dev/null +++ b/src/Agent/AutopilotMonitor.Agent.V2.Core.Tests/Monitoring/ShellCoreTrackerRebootCoalescingTests.cs @@ -0,0 +1,195 @@ +#nullable enable +using System; +using System.Collections.Generic; +using System.Linq; +using AutopilotMonitor.Agent.V2.Core.Logging; +using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals; +using AutopilotMonitor.Agent.V2.Core.Orchestration; +using AutopilotMonitor.Agent.V2.Core.Tests.Harness; +using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration; +using AutopilotMonitor.DecisionCore.Engine; +using AutopilotMonitor.DecisionCore.Signals; +using AutopilotMonitor.Shared; +using Xunit; + +namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring +{ + /// + /// Shell-Core 62407 "RebootCoalescing" branch — ESP-initiated coalesced reboot (MDM policies + /// forced a mid-ESP restart → second sign-in). Pure informational corroboration for the + /// MdmRebootPolicyTracker's per-URI 2800 attribution: no FinalizingSetup transition, no C# + /// event raise, and — as a DELIBERATE deviation from the other 62407 backfill branches — the + /// backfill path DOES emit (the live emit races the very reboot that kills the process). + /// + public sealed class ShellCoreTrackerRebootCoalescingTests + { + private static readonly DateTime ClockNow = new DateTime(2026, 7, 20, 10, 5, 0, DateTimeKind.Utc); + private const string CoalescingDescription = + "CloudExperienceHost Web App Event 2. Name: 'CommercialOOBE_DeviceSetup_RebootCoalescing', Value: '...'."; + + private static (ShellCoreTracker tracker, FakeSignalIngressSink sink) MakeTracker(TempDirectory tmp) + { + var sink = new FakeSignalIngressSink(); + var logger = new AgentLogger(tmp.Path, AgentLogLevel.Info); + var tracker = new ShellCoreTracker( + sessionId: "S1", + tenantId: "T1", + post: new InformationalEventPost(sink, new VirtualClock(ClockNow)), + logger: logger, + helloTracker: null); + return (tracker, sink); + } + + private static IReadOnlyList ByType(FakeSignalIngressSink sink, string eventType) => + sink.Posted.Where(p => + p.Kind == DecisionSignalKind.InformationalEvent + && p.Payload != null + && p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et) + && et == eventType).ToList(); + + private static IReadOnlyDictionary Data(FakeSignalIngressSink.PostedSignal s) => + (IReadOnlyDictionary)s.TypedPayload!; + + [Fact] + public void ProcessEvent_62407_RebootCoalescing_EmitsWarning_ImmediateUpload() + { + using var tmp = new TempDirectory(); + var (tracker, sink) = MakeTracker(tmp); + using (tracker) + { + var sourceTime = ClockNow.AddSeconds(-5); + tracker.ProcessEvent( + eventId: ShellCoreTracker.EventId_ShellCore_WebAppEvent, + description: CoalescingDescription, + timestamp: sourceTime, + providerName: "Microsoft-Windows-Shell-Core", + isBackfill: false); + + var s = Assert.Single(ByType(sink, Constants.EventTypes.EspRebootCoalescing)); + Assert.Equal("Warning", s.Payload![SignalPayloadKeys.Severity]); + Assert.Equal("true", s.Payload![SignalPayloadKeys.ImmediateUpload]); + Assert.Equal(sourceTime, s.OccurredAtUtc); + Assert.Equal(CoalescingDescription, Data(s)["description"]); + } + } + + [Fact] + public void ProcessEvent_62407_RebootCoalescing_FiresOncePerRun() + { + using var tmp = new TempDirectory(); + var (tracker, sink) = MakeTracker(tmp); + using (tracker) + { + tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent, + CoalescingDescription, ClockNow, "Microsoft-Windows-Shell-Core", isBackfill: false); + tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent, + CoalescingDescription, ClockNow.AddSeconds(1), "Microsoft-Windows-Shell-Core", isBackfill: false); + + Assert.Single(ByType(sink, Constants.EventTypes.EspRebootCoalescing)); + } + } + + [Fact] + public void ProcessEvent_62407_RebootCoalescing_DoesNotTriggerEspSemantics() + { + // The coalescing record is corroboration only — it must not trip the ESP-exit / + // FinalizingSetup / failure machinery. + using var tmp = new TempDirectory(); + var (tracker, sink) = MakeTracker(tmp); + using (tracker) + { + var raised = new List(); + tracker.FinalizingSetupPhaseTriggered += (_, reason) => raised.Add($"finalizing:{reason}"); + tracker.EspExited += (_, _) => raised.Add("espExited"); + tracker.EspFailureDetected += (_, type) => raised.Add($"failure:{type}"); + tracker.WhiteGloveCompleted += (_, _) => raised.Add("whiteGlove"); + + tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent, + CoalescingDescription, ClockNow, "Microsoft-Windows-Shell-Core", isBackfill: false); + + Assert.Empty(raised); + Assert.False(tracker.IsEspExitedForTest); + Assert.Empty(ByType(sink, Constants.EventTypes.EspExiting)); + Assert.Empty(ByType(sink, Constants.EventTypes.EspFailure)); + } + } + + [Fact] + public void ProcessEvent_62407_ExistingBranches_Unaffected() + { + // Ordering regression guard: WhiteGlove/failure/exiting still classify as before — + // the RebootCoalescing check comes AFTER them. + using var tmp = new TempDirectory(); + var (tracker, sink) = MakeTracker(tmp); + using (tracker) + { + tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent, + "Name: 'CommercialOOBE_ESPProgress_Page_Exiting'", ClockNow, "p", isBackfill: false); + tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent, + "Name: 'CommercialOOBE_ESPProgress_WhiteGlove_Success'", ClockNow, "p", isBackfill: false); + + Assert.Single(ByType(sink, Constants.EventTypes.EspExiting)); + Assert.Single(ByType(sink, Constants.EventTypes.WhiteGloveComplete)); + Assert.Empty(ByType(sink, Constants.EventTypes.EspRebootCoalescing)); + } + } + + [Fact] + public void Backfill_RebootCoalescing_Emits_WithBackfilledTrue_AndHistoricalTimestamp() + { + // Deliberate deviation from the other 62407 backfill branches: after the coalesced + // reboot restarts the agent, the backfill re-emission is the only reliable delivery. + using var tmp = new TempDirectory(); + var (tracker, sink) = MakeTracker(tmp); + using (tracker) + { + var historical = ClockNow.AddMinutes(-3); + tracker.HandleBackfillRecord( + ShellCoreTracker.EventId_ShellCore_WebAppEvent, CoalescingDescription, historical); + + var s = Assert.Single(ByType(sink, Constants.EventTypes.EspRebootCoalescing)); + Assert.Equal(historical, s.OccurredAtUtc); + Assert.Equal(true, Data(s)["backfilled"]); + Assert.Equal("true", s.Payload![SignalPayloadKeys.ImmediateUpload]); + } + } + + [Fact] + public void Backfill_RebootCoalescing_AfterLiveFire_DoesNotDoubleEmit_SameProcess() + { + // Same-process dedup via the fire-once flag. (Cross-process live+backfill duplication + // is possible and accepted — exists-semantics rule, identical historical timestamp.) + using var tmp = new TempDirectory(); + var (tracker, sink) = MakeTracker(tmp); + using (tracker) + { + tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent, + CoalescingDescription, ClockNow, "p", isBackfill: false); + tracker.HandleBackfillRecord( + ShellCoreTracker.EventId_ShellCore_WebAppEvent, CoalescingDescription, ClockNow.AddMinutes(-1)); + + Assert.Single(ByType(sink, Constants.EventTypes.EspRebootCoalescing)); + } + } + + [Fact] + public void Backfill_RebootCoalescing_DoesNotFallThroughToExitOrFailureHandling() + { + using var tmp = new TempDirectory(); + var (tracker, sink) = MakeTracker(tmp); + using (tracker) + { + var raised = new List(); + tracker.FinalizingSetupPhaseTriggered += (_, reason) => raised.Add($"finalizing:{reason}"); + tracker.EspExited += (_, _) => raised.Add("espExited"); + tracker.EspFailureDetected += (_, type) => raised.Add($"failure:{type}"); + + tracker.HandleBackfillRecord( + ShellCoreTracker.EventId_ShellCore_WebAppEvent, CoalescingDescription, ClockNow.AddMinutes(-2)); + + Assert.Empty(raised); + Assert.False(tracker.IsEspExitedForTest); + } + } + } +} diff --git a/src/Agent/AutopilotMonitor.Agent.V2.Core.Tests/Monitoring/SystemSignals/MdmRebootPolicyTrackerTests.cs b/src/Agent/AutopilotMonitor.Agent.V2.Core.Tests/Monitoring/SystemSignals/MdmRebootPolicyTrackerTests.cs new file mode 100644 index 00000000..a67394d6 --- /dev/null +++ b/src/Agent/AutopilotMonitor.Agent.V2.Core.Tests/Monitoring/SystemSignals/MdmRebootPolicyTrackerTests.cs @@ -0,0 +1,235 @@ +#nullable enable +using System; +using System.Collections.Generic; +using System.Linq; +using AutopilotMonitor.Agent.V2.Core.Logging; +using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals; +using AutopilotMonitor.Agent.V2.Core.Orchestration; +using AutopilotMonitor.Agent.V2.Core.Tests.Harness; +using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration; +using AutopilotMonitor.DecisionCore.Engine; +using AutopilotMonitor.DecisionCore.Signals; +using AutopilotMonitor.Shared; +using Xunit; + +namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals +{ + /// + /// MdmRebootPolicyTracker — attributes the "unexpected reboot + second sign-in" pattern to the + /// device-assigned policy URIs that forced the coalesced reboot (DM-Enterprise EventID 2800). + /// Drives the primitive test seam (no real + /// EventRecord, which is abstract + Windows-only), mirroring the WindowsUpdateTracker pattern. + /// The registry probe is always overridden — tests never touch the live registry. + /// + public sealed class MdmRebootPolicyTrackerTests : IDisposable + { + private static readonly DateTime At = new DateTime(2026, 7, 20, 9, 30, 0, DateTimeKind.Utc); + private const string SampleUri = "./Device/Vendor/MSFT/Policy/Config/DeviceGuard/EnableVirtualizationBasedSecurity"; + + private readonly TempDirectory _tmp = new TempDirectory(); + private readonly FakeSignalIngressSink _sink; + private readonly MdmRebootPolicyTracker _tracker; + private long _nextRecordId = 1; + + public MdmRebootPolicyTrackerTests() + { + _sink = new FakeSignalIngressSink(); + var post = new InformationalEventPost(_sink, new VirtualClock(At)); + var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info); + _tracker = new MdmRebootPolicyTracker( + sessionId: "sess-mdm", + tenantId: "tenant-mdm", + post: post, + logger: logger, + backfillEnabled: false, + stateDirectory: null); // in-memory dedup for most tests + _tracker.RebootRequiredFlagProbe = () => null; // registry untouched by default + } + + public void Dispose() => _tmp.Dispose(); + + private void Emit(string? rebootUri = SampleUri, string? description = null, long? recordId = null, bool isBackfill = false, DateTime? timeCreatedUtc = null) + { + _tracker.ProcessEvent( + eventId: MdmRebootPolicyTracker.EventId_PolicyRebootRequired, + recordId: recordId ?? _nextRecordId++, + timeCreatedUtc: timeCreatedUtc ?? At, + rebootUri: rebootUri, + formattedDescription: description, + isBackfill: isBackfill); + } + + private IReadOnlyList Emitted() => + _sink.Posted.Where(p => + p.Kind == DecisionSignalKind.InformationalEvent + && p.Payload != null + && p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et) + && et == Constants.EventTypes.MdmPolicyRebootRequired).ToList(); + + private static IReadOnlyDictionary Data(FakeSignalIngressSink.PostedSignal s) => + (IReadOnlyDictionary)s.TypedPayload!; + + [Fact] + public void ProcessEvent_EmitsWarning_WithUri_AndImmediateUpload() + { + Emit(); + + var s = Assert.Single(Emitted()); + Assert.Equal("Warning", s.Payload![SignalPayloadKeys.Severity]); + Assert.Equal("true", s.Payload![SignalPayloadKeys.ImmediateUpload]); + + var data = Data(s); + Assert.Equal(SampleUri, data["rebootUri"]); + Assert.Equal(MdmRebootPolicyTracker.EventId_PolicyRebootRequired, data["windowsEventId"]); + Assert.Equal(MdmRebootPolicyTracker.Channel, data["eventLogChannel"]); + } + + [Fact] + public void ProcessEvent_NoUri_StillEmits_WithoutRebootUriField() + { + // Tolerant contract: an unparsed URI must stay timeline-visible (with the captured + // description telling us how to fix the extraction) — the analyze rule keys on + // "rebootUri exists" and simply stays silent. + Emit(rebootUri: null, description: "Some unexpected 2800 wording"); + + var s = Assert.Single(Emitted()); + var data = Data(s); + Assert.False(data.ContainsKey("rebootUri")); + Assert.Equal("Some unexpected 2800 wording", data["description"]); + } + + [Fact] + public void BackfilledEvent_UsesEventTimeForTimelineTimestamp_AndMarksBackfilled() + { + var eventTime = At.AddMinutes(-25); // policy applied well before agent start + Emit(isBackfill: true, timeCreatedUtc: eventTime); + + var s = Assert.Single(Emitted()); + Assert.Equal(eventTime, s.OccurredAtUtc); + Assert.Equal(true, Data(s)["backfilled"]); + } + + [Fact] + public void SameRecordId_ProcessedTwice_EmitsOnce() + { + Emit(recordId: 42); + Emit(recordId: 42); + + Assert.Single(Emitted()); + } + + [Fact] + public void BackfillEventWithLowerRecordId_AfterLiveEvent_IsStillEmitted() + { + // Same contract as WindowsUpdateTracker: the live watcher is armed BEFORE the backfill + // scan, so a live record with a higher RecordId can arrive first — the older, + // never-emitted backfill record must not be suppressed by a high-water mark. + Emit(recordId: 100); + Emit(recordId: 50, isBackfill: true); + + Assert.Equal(2, Emitted().Count); + } + + [Fact] + public void NegativeRecordId_IsNeverDeduped() + { + Emit(recordId: -1); + Emit(recordId: -1); + + Assert.Equal(2, Emitted().Count); + } + + [Fact] + public void Watermark_PersistsAcrossTrackerInstances() + { + // The coalesced reboot restarts the agent and the post-reboot backfill re-reads the + // same 2800 records — the persisted watermark must make that re-read a no-op. + var post = new InformationalEventPost(_sink, new VirtualClock(At)); + var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info); + + var first = new MdmRebootPolicyTracker("s", "t", post, logger, backfillEnabled: false, stateDirectory: _tmp.Path); + first.RebootRequiredFlagProbe = () => null; + first.ProcessEvent(MdmRebootPolicyTracker.EventId_PolicyRebootRequired, recordId: 500, + timeCreatedUtc: At, rebootUri: SampleUri, formattedDescription: null, isBackfill: false); + + var second = new MdmRebootPolicyTracker("s", "t", post, logger, backfillEnabled: false, stateDirectory: _tmp.Path); + second.RebootRequiredFlagProbe = () => null; + second.LoadWatermark(); + second.ProcessEvent(MdmRebootPolicyTracker.EventId_PolicyRebootRequired, recordId: 500, + timeCreatedUtc: At, rebootUri: SampleUri, formattedDescription: null, isBackfill: true); + + Assert.Single(Emitted()); + } + + [Fact] + public void RebootRequiredFlagProbe_True_AddsField_Null_OmitsIt() + { + _tracker.RebootRequiredFlagProbe = () => true; + Emit(recordId: 1); + _tracker.RebootRequiredFlagProbe = () => null; + Emit(recordId: 2); + + var emitted = Emitted(); + Assert.Equal(2, emitted.Count); + Assert.Equal(true, Data(emitted[0])["omadmRebootRequiredFlag"]); + Assert.False(Data(emitted[1]).ContainsKey("omadmRebootRequiredFlag")); + } + + [Fact] + public void RebootRequiredFlagProbe_Throwing_IsFailSoft() + { + _tracker.RebootRequiredFlagProbe = () => throw new InvalidOperationException("boom"); + Emit(); + + var s = Assert.Single(Emitted()); + Assert.False(Data(s).ContainsKey("omadmRebootRequiredFlag")); + } + + [Fact] + public void BuildXPath_Targets2800() + { + Assert.Equal("*[System[(EventID=2800)]]", MdmRebootPolicyTracker.BuildXPath()); + } + + // ------------------------------------------------------------------ URI extraction + + [Fact] + public void ExtractRebootUri_PrefersEventDataValue_OverDescription() + { + var eventData = new Dictionary { ["Data0"] = " " + SampleUri + " " }; + var uri = MdmRebootPolicyTracker.ExtractRebootUri(eventData, "unrelated ./Other/Uri text"); + Assert.Equal(SampleUri, uri); + } + + [Fact] + public void ExtractRebootUri_RegexFallback_FromDescription() + { + var uri = MdmRebootPolicyTracker.ExtractRebootUri( + new Dictionary { ["Data0"] = "not a uri" }, + $"The following URI has triggered a reboot: {SampleUri}."); + Assert.Equal(SampleUri, uri); + } + + [Fact] + public void ExtractRebootUri_ReturnsNull_WhenNothingUriLike() + { + Assert.Null(MdmRebootPolicyTracker.ExtractRebootUri( + new Dictionary { ["Data0"] = "plain" }, "no uri here")); + Assert.Null(MdmRebootPolicyTracker.ExtractRebootUri(null, null)); + } + + [Fact] + public void ParseEventData_CollectsNamedAndPositionalData() + { + const string xml = + "" + + "2800" + + "./Device/Vendor/MSFT/Xpositional" + + ""; + + var parsed = MdmRebootPolicyTracker.ParseEventData(xml); + Assert.Equal("./Device/Vendor/MSFT/X", parsed["Uri"]); + Assert.Equal("positional", parsed["Data0"]); + } + } +} diff --git a/src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/MdmRebootPolicyTracker.cs b/src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/MdmRebootPolicyTracker.cs new file mode 100644 index 00000000..f6508bfb --- /dev/null +++ b/src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/MdmRebootPolicyTracker.cs @@ -0,0 +1,519 @@ +using System; +using System.Collections.Generic; +using System.Diagnostics.Eventing.Reader; +using System.IO; +using System.Linq; +using System.Text.RegularExpressions; +using System.Xml.Linq; +using AutopilotMonitor.Agent.V2.Core.Logging; +using AutopilotMonitor.Agent.V2.Core.Orchestration; +using AutopilotMonitor.Shared; +using AutopilotMonitor.Shared.Models; +using Newtonsoft.Json; + +namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals +{ + /// + /// Watches Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin for + /// EventID 2800 — logged once per device-assigned MDM policy whose CSP URI matches the OS + /// reboot-required catalog — and forwards each as a mdm_policy_reboot_required event + /// carrying the reboot-forcing URI. + /// + /// This attributes the "unexpected reboot + second sign-in screen" pattern (PMPC research): + /// device-assigned policies applied during ESP DeviceSetup can demand a reboot; Windows + /// coalesces them into one restart at the end of DeviceSetup, forcing the user through a second + /// sign-in before AccountSetup. Admins are blind to this in the Intune console — the attributed + /// URIs make it actionable (reassign the profiles to user groups and the reboot disappears). + /// + /// + /// Backfill. Device policies can apply before the agent starts, so a startup backfill + /// scans the lookback window. The .evtx persists across the coalesced reboot, so the + /// post-reboot agent restart re-reads the same records — which is why the + /// cross-restart RecordId watermark (WindowsUpdateTracker pattern) is mandatory here: + /// without it every reboot-forcing URI would be re-emitted after the very reboot it caused. + /// + /// + /// Tolerant matching. The exact 2800 message text is unverified (validated post-deploy on + /// the E2E VM): emission is gated on the EventID only, never on message wording. URI extraction + /// prefers structured EventData values that look like an OMA-DM URI (./…) and falls back + /// to a permissive regex over the formatted description. Events with an unparseable URI are + /// still emitted (with the captured description) so the timeline shows them and the captured + /// text tells us how to fix the extraction — the analyze rule keys on rebootUri exists + /// and simply stays silent for those. + /// + /// + internal sealed class MdmRebootPolicyTracker : IDisposable + { + internal const string Channel = "Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin"; + internal const int EventId_PolicyRebootRequired = 2800; + + internal const string WatermarkStateFileName = "mdm-reboot-watermark.json"; + + /// Registry key whose RebootRequired value/subkey flags a pending OMA-DM coalesced reboot. + internal const string OmadmSyncMlKeyPath = @"SOFTWARE\Microsoft\Provisioning\OMADM\SyncML"; + + // Permissive OMA-DM URI shape: "./Device/Vendor/MSFT/...", "./User/...", "./Vendor/...". + // Kept deliberately broad — the exact 2800 text is unverified; post-deploy E2E validates + // and tightens this against real records if needed. + private static readonly Regex UriPattern = new Regex( + @"\./[A-Za-z0-9_\-./\[\]{}%~+]+", RegexOptions.Compiled); + + private readonly AgentLogger _logger; + private readonly string _sessionId; + private readonly string _tenantId; + private readonly InformationalEventPost _post; + private readonly bool _backfillEnabled; + private readonly int _backfillLookbackMinutes; + private readonly string _stateDirectory; + + private EventLogWatcher _watcher; + + // Dedup — same contract as WindowsUpdateTracker (see its field docs): immutable + // cross-restart boundary from disk + intra-run HashSet (NOT a high-water mark, because the + // live watcher is armed before the backfill scan and can deliver higher RecordIds first). + private long _restartWatermark = -1; + private readonly HashSet _seenThisRun = new HashSet(); + private long _maxEmittedRecordId = -1; + private readonly object _watermarkLock = new object(); + + /// + /// Fail-soft corroboration probe for the OMA-DM RebootRequired flag — null = unknown + /// (key absent or unreadable). Settable test seam; the default reads the live registry. + /// + internal Func RebootRequiredFlagProbe { get; set; } = ReadOmadmRebootRequiredFlag; + + public MdmRebootPolicyTracker( + string sessionId, + string tenantId, + InformationalEventPost post, + AgentLogger logger, + bool backfillEnabled = true, + int backfillLookbackMinutes = 60, + string stateDirectory = null) + { + _sessionId = sessionId ?? throw new ArgumentNullException(nameof(sessionId)); + _tenantId = tenantId ?? throw new ArgumentNullException(nameof(tenantId)); + _post = post ?? throw new ArgumentNullException(nameof(post)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + _backfillEnabled = backfillEnabled; + _backfillLookbackMinutes = backfillLookbackMinutes; + _stateDirectory = stateDirectory != null ? Environment.ExpandEnvironmentVariables(stateDirectory) : null; + } + + public void Start() + { + LoadWatermark(); + StartWatcher(); + + if (_backfillEnabled && _backfillLookbackMinutes > 0) + BackfillRecentEvents(); + else + _logger.Info("MdmRebootPolicy backfill disabled by config"); + } + + public void Stop() + { + if (_watcher == null) return; + try + { + _watcher.Enabled = false; + _watcher.Dispose(); + _logger.Info("MdmRebootPolicy watcher stopped"); + } + catch (Exception ex) + { + _logger.Error("Error stopping MdmRebootPolicy watcher", ex); + } + finally + { + _watcher = null; + } + } + + public void Dispose() => Stop(); + + // ----------------------------------------------------------------------- + // Watcher lifecycle + // ----------------------------------------------------------------------- + + private void StartWatcher() + { + try + { + var query = new EventLogQuery(Channel, PathType.LogName, BuildXPath()); + _watcher = new EventLogWatcher(query); + _watcher.EventRecordWritten += OnEventRecordWritten; + _watcher.Enabled = true; + _logger.Info($"MdmRebootPolicy watcher started: {Channel} (EventID {EventId_PolicyRebootRequired})"); + } + catch (EventLogNotFoundException) + { + _logger.Warning($"MdmRebootPolicy event log not found: {Channel} (normal on non-MDM-enrolled test environments)"); + } + catch (UnauthorizedAccessException ex) + { + _logger.Warning($"MdmRebootPolicy watcher access denied for {Channel}: {ex.Message}"); + } + catch (Exception ex) + { + _logger.Error($"Failed to start MdmRebootPolicy watcher for {Channel}", ex); + // MON-D1: surface a dead watcher as one-shot telemetry, not just a local log line. + CollectorDegradationReporter.Report(_post, _sessionId, _tenantId, + collectorName: "MdmRebootPolicyTracker", reason: $"watcher_arm_failed:{Channel}", ex: ex); + } + } + + /// Targeted-EventID XPath filter. Exposed as internal for tests. + internal static string BuildXPath() => + $"*[System[(EventID={EventId_PolicyRebootRequired})]]"; + + // ----------------------------------------------------------------------- + // Event processing + // ----------------------------------------------------------------------- + + private void OnEventRecordWritten(object sender, EventRecordWrittenEventArgs e) + { + if (e.EventRecord == null) return; + try + { + ProcessRecord(e.EventRecord, isBackfill: false); + } + catch (Exception ex) + { + _logger.Error("Error processing MdmRebootPolicy event", ex); + } + } + + private void ProcessRecord(EventRecord record, bool isBackfill) + { + var recordId = record.RecordId ?? -1; + + // Cheap dedup gate BEFORE the expensive XML/FormatDescription work. + if (IsAlreadyProcessed(recordId)) + return; + + string xml = null; + try { xml = record.ToXml(); } + catch { /* fall back to description-only parse below */ } + + string description = null; + try { description = record.FormatDescription(); } + catch { /* some events lack formatting resources */ } + + var rebootUri = ExtractRebootUri(ParseEventData(xml), description); + + ProcessEvent( + eventId: record.Id, + recordId: recordId, + timeCreatedUtc: record.TimeCreated?.ToUniversalTime(), + rebootUri: rebootUri, + formattedDescription: description, + isBackfill: isBackfill); + } + + /// + /// Core processing extracted to primitive inputs so tests can drive it without synthesizing + /// an abstract, Windows-only . Mirrors the WindowsUpdateTracker + /// test-seam pattern. + /// + internal void ProcessEvent( + int eventId, + long recordId, + DateTime? timeCreatedUtc, + string rebootUri, + string formattedDescription, + bool isBackfill) + { + if (!MarkProcessed(recordId)) + return; // already emitted (cross-restart or duplicate delivery) + + var data = new Dictionary + { + { "windowsEventId", eventId }, + { "backfilled", isBackfill }, + { "eventLogChannel", Channel }, + }; + if (recordId >= 0) data["recordId"] = recordId; + if (timeCreatedUtc.HasValue) data["eventTime"] = timeCreatedUtc.Value.ToString("o"); + if (!string.IsNullOrEmpty(rebootUri)) data["rebootUri"] = rebootUri; + if (!string.IsNullOrEmpty(formattedDescription)) + { + data["description"] = formattedDescription.Length > 1000 + ? formattedDescription.Substring(0, 1000) + "…" + : formattedDescription; + } + + var flag = SafeProbeRebootRequiredFlag(); + if (flag.HasValue) data["omadmRebootRequiredFlag"] = flag.Value; + + _post.Emit(new EnrollmentEvent + { + SessionId = _sessionId, + TenantId = _tenantId, + // The record's own time, not UtcNow: backfilled pre-agent (and post-reboot re-read) + // records must land on the timeline at when the policy actually demanded the reboot. + Timestamp = timeCreatedUtc ?? DateTime.UtcNow, + EventType = Constants.EventTypes.MdmPolicyRebootRequired, + Severity = EventSeverity.Warning, + Source = "MdmRebootPolicyWatcher", + Phase = EnrollmentPhase.Unknown, + Message = !string.IsNullOrEmpty(rebootUri) + ? $"MDM policy requires a reboot during ESP: {rebootUri}" + : "MDM policy requires a reboot during ESP (URI not parsed from event — see description)", + Data = data, + // The reboot this event announces will kill the agent — flush now. + ImmediateUpload = true, + }); + } + + /// + /// Extracts the reboot-forcing OMA-DM URI. Preference order: (1) a structured EventData + /// value that starts with ./ (the CSP-URI shape), (2) a permissive regex over the + /// formatted description. Returns null when nothing URI-like is found — the caller still + /// emits, just without the rebootUri field. + /// + internal static string ExtractRebootUri(Dictionary eventData, string description) + { + if (eventData != null) + { + foreach (var value in eventData.Values) + { + var trimmed = value?.Trim(); + if (!string.IsNullOrEmpty(trimmed) && trimmed.StartsWith("./", StringComparison.Ordinal)) + return trimmed; + } + } + + if (!string.IsNullOrEmpty(description)) + { + var match = UriPattern.Match(description); + if (match.Success) + { + // The permissive pattern can swallow sentence punctuation trailing the URI + // ("...Security."). OMA-DM URI segments never end in punctuation — trim it. + var candidate = match.Value.TrimEnd('.', ',', ';', ':', '\'', '"', ')'); + if (candidate.Length > 2) return candidate; + } + } + + return null; + } + + /// + /// Extracts <Data Name="...">value</Data> pairs from an event's rendered + /// XML, keyed case-insensitively. Also collects UNNAMED Data elements under positional keys + /// (Data0, Data1, …) — DM-Enterprise events frequently carry positional-only + /// payloads. Namespace-agnostic; returns an empty map on null/malformed XML — never throws. + /// + internal static Dictionary ParseEventData(string xml) + { + var result = new Dictionary(StringComparer.OrdinalIgnoreCase); + if (string.IsNullOrEmpty(xml)) return result; + + try + { + var doc = XDocument.Parse(xml); + var positionalIndex = 0; + foreach (var el in doc.Descendants().Where(e => e.Name.LocalName == "Data")) + { + var nameAttr = el.Attribute("Name"); + var key = nameAttr != null && !string.IsNullOrEmpty(nameAttr.Value) + ? nameAttr.Value + : "Data" + positionalIndex++; + if (!result.ContainsKey(key)) + result[key] = el.Value; + } + } + catch + { + // Malformed / unexpected XML — best effort, return what we have. + } + + return result; + } + + // ----------------------------------------------------------------------- + // OMA-DM RebootRequired flag probe (fail-soft corroboration) + // ----------------------------------------------------------------------- + + private bool? SafeProbeRebootRequiredFlag() + { + try + { + return RebootRequiredFlagProbe?.Invoke(); + } + catch + { + return null; // corroboration only — never let the probe break the emit + } + } + + /// + /// Reads the pending-reboot flag under . The exact shape is + /// unverified (PMPC research names the key, not the value type), so both a value and a + /// subkey named RebootRequired count as "pending". Fail-soft: null = unknown + /// (key absent / unreadable), matching the EspSkipConfigurationProbe convention. + /// + internal static bool? ReadOmadmRebootRequiredFlag() + { + try + { + using (var key = Microsoft.Win32.Registry.LocalMachine.OpenSubKey(OmadmSyncMlKeyPath)) + { + if (key == null) return null; + if (key.GetValue("RebootRequired") != null) return true; + using (var sub = key.OpenSubKey("RebootRequired")) + { + return sub != null ? (bool?)true : false; + } + } + } + catch + { + return null; + } + } + + // ----------------------------------------------------------------------- + // Backfill + // ----------------------------------------------------------------------- + + private void BackfillRecentEvents() + { + try + { + var lookbackMs = (long)_backfillLookbackMinutes * 60 * 1000; + var xpath = $"*[System[(EventID={EventId_PolicyRebootRequired}) and TimeCreated[timediff(@SystemTime) <= {lookbackMs}]]]"; + + _logger.Info($"MdmRebootPolicy backfill: scanning {Channel} " + + $"(lookback={_backfillLookbackMinutes}min, restartWatermark={_restartWatermark})"); + + var query = new EventLogQuery(Channel, PathType.LogName, xpath); + + int processed = 0; + using (var reader = new EventLogReader(query)) + { + EventRecord record; + while ((record = reader.ReadEvent()) != null) + { + using (record) + { + ProcessRecord(record, isBackfill: true); + processed++; + } + } + } + + if (processed == 0) + _logger.Debug($"MdmRebootPolicy backfill: no EventID {EventId_PolicyRebootRequired} records in last {_backfillLookbackMinutes} minutes"); + else + _logger.Info($"MdmRebootPolicy backfill: scanned {processed} record(s)"); + } + catch (EventLogNotFoundException) + { + _logger.Warning($"MdmRebootPolicy event log not found during backfill: {Channel} (normal on non-MDM-enrolled test environments)"); + } + catch (UnauthorizedAccessException ex) + { + _logger.Warning($"MdmRebootPolicy backfill access denied: {ex.Message}"); + } + catch (Exception ex) + { + _logger.Warning($"MdmRebootPolicy backfill failed: {ex.Message}"); + } + } + + // ----------------------------------------------------------------------- + // Watermark dedup (cross-restart) — WindowsUpdateTracker pattern verbatim + // ----------------------------------------------------------------------- + + private bool IsAlreadyProcessed(long recordId) + { + if (recordId < 0) return false; // no RecordId → cannot dedup, let it through + lock (_watermarkLock) + { + return recordId <= _restartWatermark || _seenThisRun.Contains(recordId); + } + } + + /// + /// Atomically claims . Returns true if this is the first time we + /// see it (caller should emit), false if it was already processed — either in a PRIOR run + /// (at/below the restart watermark) or already this run (in the seen-set). Records without + /// a RecordId (-1) are always emitted, never tracked or persisted. + /// + private bool MarkProcessed(long recordId) + { + if (recordId < 0) return true; + + long toPersist = -1; + lock (_watermarkLock) + { + if (recordId <= _restartWatermark || !_seenThisRun.Add(recordId)) + return false; + + if (recordId > _maxEmittedRecordId) + { + _maxEmittedRecordId = recordId; + toPersist = _maxEmittedRecordId; + } + } + if (toPersist >= 0) PersistWatermark(toPersist); + return true; + } + + internal void LoadWatermark() + { + if (string.IsNullOrEmpty(_stateDirectory)) return; + + var filePath = Path.Combine(_stateDirectory, WatermarkStateFileName); + if (!File.Exists(filePath)) return; + + try + { + var json = File.ReadAllText(filePath); + var state = JsonConvert.DeserializeObject(json); + if (state != null) + { + lock (_watermarkLock) + { + _restartWatermark = state.LastRecordId; + _maxEmittedRecordId = state.LastRecordId; + } + _logger.Info($"MdmRebootPolicy watermark loaded: lastRecordId={state.LastRecordId} (persisted {state.PersistedUtc:O})"); + } + } + catch (Exception ex) + { + _logger.Warning($"Failed to load MdmRebootPolicy watermark: {ex.Message}"); + } + } + + private void PersistWatermark(long recordId) + { + if (string.IsNullOrEmpty(_stateDirectory)) return; + + try + { + Directory.CreateDirectory(_stateDirectory); + var filePath = Path.Combine(_stateDirectory, WatermarkStateFileName); + var state = new WatermarkState { LastRecordId = recordId, PersistedUtc = DateTime.UtcNow }; + var json = JsonConvert.SerializeObject(state, Formatting.Indented); + var tempPath = filePath + ".tmp"; + File.WriteAllText(tempPath, json); + File.Copy(tempPath, filePath, overwrite: true); + try { File.Delete(tempPath); } catch { } + } + catch (Exception ex) + { + _logger.Warning($"Failed to persist MdmRebootPolicy watermark: {ex.Message}"); + } + } + + internal sealed class WatermarkState + { + public long LastRecordId { get; set; } + public DateTime PersistedUtc { get; set; } + } + } +} diff --git a/src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/ShellCoreTracker.cs b/src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/ShellCoreTracker.cs index 61bacf9b..c93cf113 100644 --- a/src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/ShellCoreTracker.cs +++ b/src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/ShellCoreTracker.cs @@ -16,6 +16,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals /// CommercialOOBE_ESPProgress_Page_Exiting — normal ESP exit /// CommercialOOBE_ESPProgress_WhiteGlove_Success — WhiteGlove complete /// CommercialOOBE_ESPProgress_Failure/_Timeout/_Abort/WhiteGlove_Failed — ESP failure + /// "RebootCoalescing" (DeviceSetup) — ESP-initiated coalesced reboot + /// (MDM policies forced a mid-ESP restart → second sign-in; corroborates the + /// MdmRebootPolicyTracker's per-URI 2800 attribution events) /// /// Raises , , /// and . Cross-notifies the @@ -47,6 +50,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals private bool _espExitDetected; private bool _whiteGloveDetected; private bool _helloWizardStartDetected; + private bool _rebootCoalescingDetected; private readonly object _stateLock = new object(); /// @@ -247,6 +251,27 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals _logger.Info("ESP phase exit detected - detected via Shell-Core event 62407"); } + else if (description.IndexOf("RebootCoalescing", StringComparison.OrdinalIgnoreCase) >= 0) + { + // ESP-initiated coalesced reboot (end of DeviceSetup): MDM policies forced a + // restart, the user will see a second sign-in screen. Checked AFTER the + // WhiteGlove/failure/exiting patterns so their semantics stay untouched even + // if a description ever carried both tokens. Fire-once per process run — the + // reboot restarts the agent, and the fresh run re-observes it via backfill. + lock (_stateLock) + { + if (_rebootCoalescingDetected) return; + _rebootCoalescingDetected = true; + } + + eventType = Constants.EventTypes.EspRebootCoalescing; + severity = EventSeverity.Warning; + message = "ESP initiated a coalesced reboot (RebootCoalescing) — device will restart and show a second sign-in"; + // No FinalizingSetup transition, no C# event raise — pure informational + // corroboration for the MdmRebootPolicyTracker's per-URI attribution. + + _logger.Warning("ESP coalesced reboot detected via Shell-Core event 62407 (RebootCoalescing)"); + } else { return; @@ -421,6 +446,29 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals return; } + if (description.IndexOf("RebootCoalescing", StringComparison.OrdinalIgnoreCase) >= 0) + { + // DELIBERATE deviation from the other 62407 backfill branches (which only raise C# + // events): this is a pure informational type with no reducer/subscriber — the + // emission IS the signal, and the live emit races the very reboot that kills the + // process. Re-emitting after the restart is the only reliable delivery path. A + // live+backfill cross-process duplicate is possible (no watermark here); accepted: + // the analyze rule uses exists-semantics, both copies carry the identical historical + // timestamp, and the backfill copy is marked backfilled:true. + bool shouldEmit; + lock (_stateLock) + { + shouldEmit = !_rebootCoalescingDetected; + _rebootCoalescingDetected = true; + } + if (shouldEmit) + { + _logger.Info($"Backfill: ESP coalesced reboot (RebootCoalescing) found in recent Shell-Core logs (originalAt={occurredAtUtc:o})"); + EmitRebootCoalescing(description, occurredAtUtc, isBackfill: true); + } + return; // a RebootCoalescing record is not an ESP exit/failure — stop here + } + if (EspExitingPattern.IsMatch(description)) { bool shouldNotify = false; @@ -459,6 +507,39 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals } } + /// + /// Emits the esp_reboot_coalescing event with the standard Shell-Core eventData + /// shape. Used by the backfill path (the live path flows through the shared emit tail in + /// ); adds the + /// backfilled marker that distinguishes an accepted post-restart re-emission. + /// + private void EmitRebootCoalescing(string description, DateTime occurredAtUtc, bool isBackfill) + { + var eventData = new Dictionary + { + { "windowsEventId", EventId_ShellCore_WebAppEvent }, + { "providerName", "" }, + { "description", description }, + { "eventLogChannel", ShellCoreEventLogChannel }, + { "eventTime", occurredAtUtc.ToString("o") }, + { "backfilled", isBackfill } + }; + + _post.Emit(new EnrollmentEvent + { + SessionId = _sessionId, + TenantId = _tenantId, + Timestamp = occurredAtUtc, + EventType = Constants.EventTypes.EspRebootCoalescing, + Severity = EventSeverity.Warning, + Source = "EspAndHelloTracker", + Phase = EnrollmentPhase.Unknown, + Message = "ESP initiated a coalesced reboot (RebootCoalescing) — device will restart and show a second sign-in", + Data = eventData, + ImmediateUpload = true + }); + } + // ===================================================================== // Pattern helpers // ===================================================================== diff --git a/src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/DefaultComponentFactory.cs b/src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/DefaultComponentFactory.cs index e23069f4..6ebd12ce 100644 --- a/src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/DefaultComponentFactory.cs +++ b/src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/DefaultComponentFactory.cs @@ -335,6 +335,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration osBuildChangedProvider: () => osBuildChangeHost.BuildChanged)); } + // MDM reboot-policy watcher — subscribes to DeviceManagement-Enterprise-Diagnostics- + // Provider/Admin EventID 2800 and attributes the "unexpected reboot + second sign-in" + // pattern to the device-assigned policy URIs that forced the coalesced reboot. + // Agent-native collector (NOT an IAgentAnalyzer); the config flag is an internal + // steering lever only. + if (collectors.MdmRebootPolicyWatcherEnabled) + { + hosts.Add(new MdmRebootPolicyWatcherHost( + sessionId: sessionId, + tenantId: tenantId, + logger: logger, + ingress: ingress, + clock: clock, + backfillLookbackMinutes: collectors.MdmRebootPolicyBackfillLookbackMinutes, + stateDirectory: _stateDirectory)); + } + // ----- Peripheral hosts (event-only; driven by remote-config toggles) -------------- // V1 parity (PeriodicCollectorManager) — combine Performance + AgentSelfMetrics under diff --git a/src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/Hosts/MdmRebootPolicyWatcherHost.cs b/src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/Hosts/MdmRebootPolicyWatcherHost.cs new file mode 100644 index 00000000..d913c856 --- /dev/null +++ b/src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/Hosts/MdmRebootPolicyWatcherHost.cs @@ -0,0 +1,68 @@ +#nullable enable +using System; +using System.Threading; +using AutopilotMonitor.Agent.V2.Core.Logging; +using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals; +using AutopilotMonitor.DecisionCore.Engine; +using AutopilotMonitor.DecisionCore.Signals; + +namespace AutopilotMonitor.Agent.V2.Core.Orchestration +{ + /// + /// Standalone host for the — watches + /// Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin (EventID + /// 2800) for MDM policies that force a coalesced reboot during ESP DeviceSetup and forwards + /// them via the shared single-rail ingress. Mirrors : + /// builds its own and delegates Start/Stop to the tracker. + /// + internal sealed class MdmRebootPolicyWatcherHost : ICollectorHost + { + public string Name => "MdmRebootPolicyTracker"; + + private readonly MdmRebootPolicyTracker _tracker; + private readonly AgentLogger _logger; + private int _disposed; + + public MdmRebootPolicyWatcherHost( + string sessionId, + string tenantId, + AgentLogger logger, + ISignalIngressSink ingress, + IClock clock, + int backfillLookbackMinutes, + string stateDirectory) + { + if (ingress == null) throw new ArgumentNullException(nameof(ingress)); + if (clock == null) throw new ArgumentNullException(nameof(clock)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + + var post = new InformationalEventPost(ingress, clock, logger); + _tracker = new MdmRebootPolicyTracker( + sessionId: sessionId, + tenantId: tenantId, + post: post, + logger: logger, + backfillEnabled: backfillLookbackMinutes > 0, + backfillLookbackMinutes: backfillLookbackMinutes, + stateDirectory: stateDirectory); + } + + public void Start() + { + _tracker.Start(); + _logger.Info("MdmRebootPolicyWatcherHost: started."); + } + + public void Stop() + { + try { _tracker.Stop(); } + catch (Exception ex) { _logger.Warning($"MdmRebootPolicyWatcherHost: stop failed: {ex.Message}"); } + } + + public void Dispose() + { + if (Interlocked.Exchange(ref _disposed, 1) == 1) return; + Stop(); + } + } +} diff --git a/src/Backend/AutopilotMonitor.Functions/Functions/Config/GetAgentConfigFunction.cs b/src/Backend/AutopilotMonitor.Functions/Functions/Config/GetAgentConfigFunction.cs index 362bccc4..71075ce7 100644 --- a/src/Backend/AutopilotMonitor.Functions/Functions/Config/GetAgentConfigFunction.cs +++ b/src/Backend/AutopilotMonitor.Functions/Functions/Config/GetAgentConfigFunction.cs @@ -270,7 +270,7 @@ namespace AutopilotMonitor.Functions.Functions.Config var response = req.CreateResponse(HttpStatusCode.OK); await response.WriteAsJsonAsync(new AgentConfigResponse { - ConfigVersion = 34, // Gather rule phase scoping (ActivePhases/ActiveFromPhase) + EmitMode on_change anti-spam + ConfigVersion = 35, // MDM reboot-policy watcher (mdm_policy_reboot_required / esp_reboot_coalescing) UploadIntervalSeconds = 10, SelfDestructOnComplete = tenantConfig.SelfDestructOnComplete ?? true, KeepLogFile = tenantConfig.KeepLogFile ?? false, diff --git a/src/McpServer/autopilot-monitor-mcp/src/resource-catalog.ts b/src/McpServer/autopilot-monitor-mcp/src/resource-catalog.ts index f106c87f..98bdf6a1 100644 --- a/src/McpServer/autopilot-monitor-mcp/src/resource-catalog.ts +++ b/src/McpServer/autopilot-monitor-mcp/src/resource-catalog.ts @@ -24,6 +24,8 @@ export const EVENT_TYPES_CATALOG = { 'esp_failure_settle_started', 'esp_failure_settle_recovered', 'esp_exiting', + 'esp_reboot_coalescing', + 'mdm_policy_reboot_required', 'esp_config_detected', 'completion_check', 'completion_waiting', diff --git a/src/Shared/AutopilotMonitor.Shared/Constants.cs b/src/Shared/AutopilotMonitor.Shared/Constants.cs index a55c5d8f..c609bf06 100644 --- a/src/Shared/AutopilotMonitor.Shared/Constants.cs +++ b/src/Shared/AutopilotMonitor.Shared/Constants.cs @@ -405,6 +405,18 @@ namespace AutopilotMonitor.Shared public const string OsBuildChanged = "os_build_changed"; // OS build (CurrentBuild.UBR) differs across an agent restart — an update was installed during enrollment public const string WindowsUpdateChannelCensus = "windows_update_channel_census"; // One-shot EventID histogram of WU Client + UpdateOrchestrator channels when os_build_changed but zero targeted WU events captured + // MDM-policy-forced coalesced reboot during ESP (MdmRebootPolicyTracker + ShellCoreTracker; + // PMPC "unexpected reboot / second sign-in" research). Device-assigned policies whose CSP + // URIs match the OS reboot-required catalog force a coalesced reboot at the end of + // DeviceSetup — the user sees an unexpected restart and a second sign-in screen before + // AccountSetup. 2800 (DeviceManagement-Enterprise-Diagnostics-Provider/Admin) fires once + // per reboot-forcing URI; Shell-Core 62407 "RebootCoalescing" records ESP initiating the + // reboot. Deliberately DISTINCT from system_reboot_detected: the backend RebootCount + // matches that string exactly (EventIngestProcessor.Classification + terminal recount) — + // these attribution events must never inflate the reboot count. + public const string MdmPolicyRebootRequired = "mdm_policy_reboot_required"; // DM-Enterprise EventID 2800 — Data.rebootUri = CSP URI that demands the reboot (one event per URI) + public const string EspRebootCoalescing = "esp_reboot_coalescing"; // Shell-Core 62407 "RebootCoalescing" — ESP-initiated coalesced reboot imminent (second sign-in follows) + // TEMPORARY: shadow SM rollout verification — remove when CompletionStateMachine is promoted to primary public const string ShadowDiscrepancy = "shadow_discrepancy"; diff --git a/src/Shared/AutopilotMonitor.Shared/Models/Config/AgentConfigResponse.cs b/src/Shared/AutopilotMonitor.Shared/Models/Config/AgentConfigResponse.cs index cb5d1b94..2b03537e 100644 --- a/src/Shared/AutopilotMonitor.Shared/Models/Config/AgentConfigResponse.cs +++ b/src/Shared/AutopilotMonitor.Shared/Models/Config/AgentConfigResponse.cs @@ -482,6 +482,28 @@ namespace AutopilotMonitor.Shared.Models /// public bool WindowsUpdateChannelCensusEnabled { get; set; } = true; + // ----------------------------------------------------------------------- + // MDM reboot-policy watcher (DeviceManagement-Enterprise-Diagnostics-Provider/Admin) + // ----------------------------------------------------------------------- + + /// + /// Master switch for the MDM reboot-policy watcher. When enabled the agent subscribes to + /// Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin EventID + /// 2800 and emits one mdm_policy_reboot_required event per device-assigned policy + /// URI that forces a coalesced reboot during ESP DeviceSetup (the "unexpected reboot + + /// second sign-in" pattern). Internal steering lever only — not surfaced in any admin UI; + /// disable by pushing config server-side. Default: true. + /// + public bool MdmRebootPolicyWatcherEnabled { get; set; } = true; + + /// + /// Lookback window in minutes for the MDM reboot-policy backfill scan on startup. Device + /// policies can apply before the agent starts, and after the coalesced reboot the restarted + /// agent must re-scan past the reboot gap (the cross-restart watermark keeps re-reads from + /// re-emitting). 0 = backfill disabled. Default: 60 minutes. + /// + public int MdmRebootPolicyBackfillLookbackMinutes { get; set; } = 60; + /// /// Creates default collector configuration ///