feat(agent): attribute mid-ESP coalesced reboots to MDM policy URIs (second sign-in detection)

Device-assigned MDM policies whose CSP URIs match the OS reboot-required
catalog force a coalesced reboot at the end of ESP DeviceSetup - the user
sees an unexpected restart and a second sign-in before AccountSetup.
Invisible in the Intune console; admins simply lived with it. Now attributed
and actionable (PMPC research).

- MdmRebootPolicyTracker (+host): watches DeviceManagement-Enterprise-
  Diagnostics-Provider/Admin EventID 2800, emits one mdm_policy_reboot_required
  per reboot-forcing URI (Warning, ImmediateUpload; tolerant EventData-first
  URI extraction with regex fallback; fail-soft OMADM RebootRequired probe).
  Cross-restart RecordId watermark (mdm-reboot-watermark.json) - mandatory,
  the coalesced reboot restarts the agent and the backfill re-reads the log.
- ShellCoreTracker: RebootCoalescing branch on the already-watched 62407
  stream -> esp_reboot_coalescing corroboration event; deliberate deviation:
  emits on backfill too (live emit races the reboot kill).
- ANALYZE-ESP-005 advisory (warning): names the first URI via {{rebootUri}},
  remediation = reassign profiles device -> user groups; +20 confidence on
  coalescing corroboration, +10 on >=2 URIs.
- Config: Collectors.MdmRebootPolicyWatcherEnabled (default ON, internal
  steering only - agent-native collector, NOT an IAgentAnalyzer, no admin UI),
  MdmRebootPolicyBackfillLookbackMinutes=60; ConfigVersion 35.
- Event types deliberately distinct from system_reboot_detected so the
  backend RebootCount (exact-string match, 2 sites) stays uninflated.
- Docs: docs/agent/mdm-reboot-coalescing.md (+index/log). Exact 2800/62407
  message texts are unverified until E2E VM validation (tolerant matching).

Tests: Agent.V2.Core 1868/1868 (21 new), Functions 3099/3099, MCP 400 passed
(event-types drift), solution build 0 errors.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Oliver Kieselbach
2026-07-20 18:05:11 +02:00
parent 6d1ff8d7f0
commit a654c7309e
16 changed files with 1427 additions and 1 deletions
+1
View File
@@ -7,5 +7,6 @@ How the on-device agent works:
* [Logs, Persistence & Crash Recovery](logs-and-persistence.md) - every persisted file and why it exists; signal log vs spool; snapshot/journal; the replay-based recovery flow and quarantine.
* [Hello Wizard Un-Skip](hello-wizard-unskip.md) - why a single "WHfB disabled" policy read never decides completion: the HelloWizardStarted signal (prevention + cure) and the HelloTracker confirmation second read.
* [Autopilot ZTD Diagnostics](autopilot-ztd-diagnostics.md) - Windows' own diagnostic surfaces for the profile-download flow: ModernDeployment event IDs (807/815/164/...), the Diagnostics\Autopilot registry key, deployment-service endpoints, and the known-issue error-code map.
* [MDM Reboot Coalescing](mdm-reboot-coalescing.md) - attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: 2800 watcher with cross-restart watermark, Shell-Core RebootCoalescing corroboration, ANALYZE-ESP-005 advisory.
Reading order for newcomers: overview → decision engine → persistence.
+109
View File
@@ -0,0 +1,109 @@
---
type: Concept
title: MDM Reboot Coalescing — attributing the mid-ESP reboot and second sign-in to policy URIs
description: How the agent detects device-assigned MDM policies that force a coalesced reboot during ESP DeviceSetup (DM-Enterprise EventID 2800 + Shell-Core 62407 RebootCoalescing), and the ANALYZE-ESP-005 advisory that makes the invisible "second sign-in" fixable.
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals
tags:
- agent
- esp
- reboot
- policy-attribution
- second-sign-in
timestamp: 2026-07-20T00:00:00+02:00
---
# MDM Reboot Coalescing
Device-assigned MDM policies whose CSP URIs match the OS reboot-required catalog
(`HKLM\SOFTWARE\Microsoft\Provisioning\SyncML\RebootRequiredURIs`) force a **coalesced
reboot at the end of ESP DeviceSetup** — the user sees an unexpected restart and a
**second sign-in screen** before AccountSetup. This is expected OS behavior, invisible in
the Intune console; most admins simply live with it, not knowing that reassigning the
profiles to user groups removes the reboot entirely. Source research: Patch My PC,
["Avoid the second sign-in screen after Autopilot"](https://patchmypc.com/blog/autopilot-unexpected-reboot-autopilot-second-login-screen/).
The session already counts reboots (`system_reboot_detected` → `session.rebootCount`);
this feature adds the **cause attribution**.
# Signal chain
1. **EventID 2800**, `Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin`
— logged by the SyncML engine once per policy URI that matched the reboot-required
catalog, during device-policy application in DeviceSetup.
2. **EventID 62407**, `Microsoft-Windows-Shell-Core/Operational`, description containing
`RebootCoalescing` — ESP/CloudExperienceHost initiating the coalesced reboot at the end
of DeviceSetup.
3. The reboot kills the agent; the post-reboot restart re-observes both channels via
backfill (the .evtx persists across the reboot).
# Detectors
## `MdmRebootPolicyTracker` (new collector, `MdmRebootPolicyWatcherHost`)
Structural clone of `WindowsUpdateTracker`: live `EventLogWatcher` on the DM-Enterprise
Admin channel (EventID 2800) + startup backfill (default 60 min lookback) + **cross-restart
RecordId watermark** (`mdm-reboot-watermark.json`). The watermark is mandatory here: the
coalesced reboot restarts the agent, and without it every reboot-forcing URI would be
re-emitted after the very reboot it caused.
Per 2800 record it emits one `mdm_policy_reboot_required` event (Warning,
`ImmediateUpload=true` — the announced reboot will kill the agent, flush now) with
`Data.rebootUri` (the CSP URI), `recordId`, `backfilled`, `description` (truncated 1000),
and a fail-soft `omadmRebootRequiredFlag` corroboration probe of
`HKLM\SOFTWARE\Microsoft\Provisioning\OMADM\SyncML\RebootRequired` (null = unknown, never
blocks the emit). Per-URI emission (not aggregated) is deliberate: an aggregate would need
a flush point that races the reboot kill.
**Tolerant matching**: the exact 2800 message text is unverified — emission is gated on
the EventID only. URI extraction prefers structured EventData values starting with `./`
and falls back to a permissive regex over the description (trailing punctuation trimmed).
Events with an unparseable URI are still emitted (timeline-visible, rule stays silent, and
the captured description tells us how to fix the extraction). Exact texts are validated
post-deploy on the E2E VM and recorded here.
Config (internal steering only, no admin UI — this is an agent-native collector, NOT an
`IAgentAnalyzer`): `Collectors.MdmRebootPolicyWatcherEnabled` (default true),
`Collectors.MdmRebootPolicyBackfillLookbackMinutes` (default 60). ConfigVersion 35.
## `ShellCoreTracker` — `RebootCoalescing` branch
The already-watched 62407 stream gets a fourth branch (checked AFTER
WhiteGlove/failure/exiting so their semantics stay untouched): description containing
`RebootCoalescing` → one `esp_reboot_coalescing` event (Warning, ImmediateUpload), pure
informational — no `FinalizingSetup` transition, no C# event raise, no DecisionCore
involvement.
**Deliberate deviation**: unlike the other 62407 branches, the backfill path DOES emit
this event. The live emit races the very reboot that kills the process; the post-restart
5-min backfill re-emission is the only reliable delivery. A live+backfill cross-process
duplicate is possible and accepted (fire-once bool resets per process; both copies carry
the identical historical timestamp; the backfill copy is marked `backfilled:true`; the
analyze rule uses exists-semantics). The shared 5-min lookback is NOT enlarged — the
rule's required condition is the watermarked 2800 event; coalescing is optional
corroboration only.
# Advisory — ANALYZE-ESP-005
"MDM Policy Forced a Mid-ESP Reboot (Second Sign-In)", severity `warning`,
`markSessionAsFailedDefault: false`. Required: `mdm_policy_reboot_required.rebootUri`
exists (first URI interpolates as `{{rebootUri}}`). Optional corroboration:
`esp_reboot_coalescing` observed (+20 confidence) and ≥2 reboot URIs (+10) on base 70 /
threshold 70. Remediation: map each URI to its Intune profile and reassign device → user
groups (or exclude the Autopilot device group), or accept + communicate the restart.
Because `{{token}}` interpolation is scalar-only, the explanation points at the timeline
for the complete URI list — each URI is its own event.
# Non-goals / invariants
* The new event types are **deliberately distinct from `system_reboot_detected`** — the
backend RebootCount matches that string exactly in two places
(`EventIngestProcessor.Classification` + the terminal recount); attribution events must
never inflate the reboot count.
* No session-level field / SessionInfoCard change in this iteration (follow-up candidate).
* The `RebootRequiredURIs` catalog itself is not read (format unverified); only the 2800
events (which carry the actually-triggered URIs) are consumed.
# Citations
* [Patch My PC — Avoid the second sign-in screen after Autopilot](https://patchmypc.com/blog/autopilot-unexpected-reboot-autopilot-second-login-screen/)
* [Microsoft Learn — Enrollment Status Page tracking](https://learn.microsoft.com/en-us/autopilot/enrollment-status)
+1
View File
@@ -18,6 +18,7 @@ https://docs.autopilotmonitor.com (separate repository).
* [V2 Agent](agent/index.md) - How the on-device agent works: [runtime overview](agent/overview.md), [decision engine](agent/decision-engine.md) (reducer, signals, completion arms), [logs & persistence](agent/logs-and-persistence.md) (signal log, snapshot, crash recovery), [Hello wizard un-skip](agent/hello-wizard-unskip.md) (why a single "WHfB disabled" read never decides completion), [Autopilot ZTD diagnostics](agent/autopilot-ztd-diagnostics.md) (event IDs, diagnostic registry, endpoints, error-code map).
* [Build Counter Blob](agent/build-counter-blob.md) - How agent build numbers are minted: shared counter blob with ETag-CAS so local and CI builds never collide; CI additionally attests build provenance (Sigstore).
* [Agent Endpoint Migration](agent/endpoint-migration.md) - How the backend re-homes agents to a new API base URL via the config channel (backend move or per-tenant region move): live-fetch-only, allowlist-validated on both sides, one hop, kill wins.
* [MDM Reboot Coalescing](agent/mdm-reboot-coalescing.md) - Attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: DM-Enterprise 2800 watcher with cross-restart watermark, Shell-Core RebootCoalescing corroboration, ANALYZE-ESP-005 advisory.
# Rules
+2
View File
@@ -2,6 +2,8 @@
## 2026-07-20
* **Creation**: Added `agent/mdm-reboot-coalescing.md` — the agent now attributes the "unexpected mid-ESP reboot + second sign-in" pattern (PMPC research) to the device-assigned policy URIs that forced it: new `MdmRebootPolicyTracker` (DM-Enterprise EventID 2800, per-URI `mdm_policy_reboot_required` events, cross-restart RecordId watermark), a `RebootCoalescing` branch in `ShellCoreTracker` (`esp_reboot_coalescing`, backfill-emitting by design), and the `ANALYZE-ESP-005` advisory (reassign device → user groups). ConfigVersion 35; event types deliberately distinct from `system_reboot_detected` so RebootCount stays uninflated.
* **Creation**: Added `backend/business-timestamps.md` (new `backend/` section) — the 2026-07-18 migration reset every row's system Timestamp, exposing that AuditLogs/OpsEvents/Events had (unknowingly) displayed and filtered on it all along; documents the `OccurredUtc` column, the RowKey decoders and RowKey-range date/retention clauses in `BusinessTimestamp`, the GUID-guard trap, the audit/ops backfill endpoint, and the "system Timestamp is never authoritative" migration rule.
* **Creation**: Added `rules/gather-rule-phase-scoping.md` (new `rules/` section) — gather rules gain `activePhases` (run only during listed phases), `activeFromPhase` (sticky latch from a phase onwards, never via Failed) and `emitMode: "on_change"` (canonical-hash result dedup with `suppressedPolls` observability), ConfigVersion 34. Includes the prerequisite backend fix: the custom-rule toggle PUT no longer wipes the stored rule definition (partial-merge in `GatherRuleService.UpdateRuleAsync`).
+74
View File
@@ -0,0 +1,74 @@
{
"$schema": "../schema/analyze-rule.schema.json",
"ruleId": "ANALYZE-ESP-005",
"title": "MDM Policy Forced a Mid-ESP Reboot (Second Sign-In)",
"description": "Detects the 'unexpected reboot + second sign-in screen' pattern: one or more device-assigned MDM policies matched the OS reboot-required catalog during ESP DeviceSetup, and Windows coalesced them into a restart at the end of DeviceSetup — forcing the user through a second sign-in before AccountSetup. Attributes the reboot to the specific CSP policy URIs so the profiles can be reassigned.",
"severity": "warning",
"category": "esp",
"version": "1.0.0",
"author": "Autopilot Monitor",
"enabled": true,
"trigger": "single",
"baseConfidence": 70,
"conditions": [
{
"signal": "reboot_forcing_policy_uri",
"source": "event_data",
"eventType": "mdm_policy_reboot_required",
"dataField": "rebootUri",
"operator": "exists",
"required": true
},
{
"signal": "reboot_uri_count",
"source": "event_count",
"eventType": "mdm_policy_reboot_required",
"operator": "count_gte",
"value": "1",
"required": false
},
{
"signal": "esp_coalesced_reboot_observed",
"source": "event_type",
"eventType": "esp_reboot_coalescing",
"operator": "exists",
"required": false
}
],
"confidenceFactors": [
{ "signal": "esp_coalesced_reboot_observed", "condition": "exists", "weight": 20 },
{ "signal": "mdm_policy_reboot_required", "condition": "count >= 2", "weight": 10 }
],
"confidenceThreshold": 70,
"explanation": "During ESP DeviceSetup, a device-assigned MDM policy matched the OS reboot-required catalog and Windows scheduled a **coalesced reboot** at the end of DeviceSetup — the user saw an unexpected restart and a **second sign-in screen** before AccountSetup continued.\n\nFirst reboot-forcing policy URI: `{{rebootUri}}`\n\nWhen several policies force a reboot they are coalesced into one restart; **each** reboot-forcing policy appears as its own `mdm_policy_reboot_required` event on the session timeline — check the timeline for the complete URI list.\n\nThis is expected OS behavior, not a failure — but it degrades the enrollment experience, confuses users, and is invisible in the Intune console.",
"remediation": [
{
"title": "Reassign the reboot-forcing policies to user groups",
"steps": [
"Open each `mdm_policy_reboot_required` timeline event and note the CSP URI (e.g. `./Device/Vendor/MSFT/...`)",
"Map each URI to its Intune profile (Settings Catalog / configuration profile) — the URI path names the CSP area",
"Reassign those profiles from device groups to user groups so they apply during/after AccountSetup instead of DeviceSetup",
"Alternatively, exclude the Autopilot device group from the profile and deliver the setting post-enrollment"
]
},
{
"title": "Or accept the reboot and set expectations",
"steps": [
"If the policy must be device-assigned (e.g. a security baseline), keep it and document the expected mid-ESP restart",
"Communicate to users that a restart and second sign-in during provisioning is expected for these deployments"
]
}
],
"relatedDocs": [
{
"title": "Patch My PC: Avoid the second sign-in screen after Autopilot",
"url": "https://patchmypc.com/blog/autopilot-unexpected-reboot-autopilot-second-login-screen/"
},
{
"title": "Microsoft Learn: Enrollment Status Page tracking",
"url": "https://learn.microsoft.com/en-us/autopilot/enrollment-status"
}
],
"markSessionAsFailedDefault": false,
"tags": ["esp", "reboot", "coalescing", "second-sign-in", "policy-attribution", "user-experience"]
}
+88
View File
@@ -1785,6 +1785,94 @@
"enrollment"
]
},
{
"ruleId": "ANALYZE-ESP-005",
"title": "MDM Policy Forced a Mid-ESP Reboot (Second Sign-In)",
"description": "Detects the 'unexpected reboot + second sign-in screen' pattern: one or more device-assigned MDM policies matched the OS reboot-required catalog during ESP DeviceSetup, and Windows coalesced them into a restart at the end of DeviceSetup — forcing the user through a second sign-in before AccountSetup. Attributes the reboot to the specific CSP policy URIs so the profiles can be reassigned.",
"severity": "warning",
"category": "esp",
"version": "1.0.0",
"author": "Autopilot Monitor",
"enabled": true,
"trigger": "single",
"baseConfidence": 70,
"conditions": [
{
"signal": "reboot_forcing_policy_uri",
"source": "event_data",
"eventType": "mdm_policy_reboot_required",
"dataField": "rebootUri",
"operator": "exists",
"required": true
},
{
"signal": "reboot_uri_count",
"source": "event_count",
"eventType": "mdm_policy_reboot_required",
"operator": "count_gte",
"value": "1",
"required": false
},
{
"signal": "esp_coalesced_reboot_observed",
"source": "event_type",
"eventType": "esp_reboot_coalescing",
"operator": "exists",
"required": false
}
],
"confidenceFactors": [
{
"signal": "esp_coalesced_reboot_observed",
"condition": "exists",
"weight": 20
},
{
"signal": "mdm_policy_reboot_required",
"condition": "count >= 2",
"weight": 10
}
],
"confidenceThreshold": 70,
"explanation": "During ESP DeviceSetup, a device-assigned MDM policy matched the OS reboot-required catalog and Windows scheduled a **coalesced reboot** at the end of DeviceSetup — the user saw an unexpected restart and a **second sign-in screen** before AccountSetup continued.\n\nFirst reboot-forcing policy URI: `{{rebootUri}}`\n\nWhen several policies force a reboot they are coalesced into one restart; **each** reboot-forcing policy appears as its own `mdm_policy_reboot_required` event on the session timeline — check the timeline for the complete URI list.\n\nThis is expected OS behavior, not a failure — but it degrades the enrollment experience, confuses users, and is invisible in the Intune console.",
"remediation": [
{
"title": "Reassign the reboot-forcing policies to user groups",
"steps": [
"Open each `mdm_policy_reboot_required` timeline event and note the CSP URI (e.g. `./Device/Vendor/MSFT/...`)",
"Map each URI to its Intune profile (Settings Catalog / configuration profile) — the URI path names the CSP area",
"Reassign those profiles from device groups to user groups so they apply during/after AccountSetup instead of DeviceSetup",
"Alternatively, exclude the Autopilot device group from the profile and deliver the setting post-enrollment"
]
},
{
"title": "Or accept the reboot and set expectations",
"steps": [
"If the policy must be device-assigned (e.g. a security baseline), keep it and document the expected mid-ESP restart",
"Communicate to users that a restart and second sign-in during provisioning is expected for these deployments"
]
}
],
"relatedDocs": [
{
"title": "Patch My PC: Avoid the second sign-in screen after Autopilot",
"url": "https://patchmypc.com/blog/autopilot-unexpected-reboot-autopilot-second-login-screen/"
},
{
"title": "Microsoft Learn: Enrollment Status Page tracking",
"url": "https://learn.microsoft.com/en-us/autopilot/enrollment-status"
}
],
"markSessionAsFailedDefault": false,
"tags": [
"esp",
"reboot",
"coalescing",
"second-sign-in",
"policy-attribution",
"user-experience"
]
},
{
"ruleId": "ANALYZE-ID-001",
"title": "Expected Machine Certificate Not Found",
@@ -0,0 +1,195 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.Shared;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
{
/// <summary>
/// Shell-Core 62407 "RebootCoalescing" branch — ESP-initiated coalesced reboot (MDM policies
/// forced a mid-ESP restart → second sign-in). Pure informational corroboration for the
/// MdmRebootPolicyTracker's per-URI 2800 attribution: no FinalizingSetup transition, no C#
/// event raise, and — as a DELIBERATE deviation from the other 62407 backfill branches — the
/// backfill path DOES emit (the live emit races the very reboot that kills the process).
/// </summary>
public sealed class ShellCoreTrackerRebootCoalescingTests
{
private static readonly DateTime ClockNow = new DateTime(2026, 7, 20, 10, 5, 0, DateTimeKind.Utc);
private const string CoalescingDescription =
"CloudExperienceHost Web App Event 2. Name: 'CommercialOOBE_DeviceSetup_RebootCoalescing', Value: '...'.";
private static (ShellCoreTracker tracker, FakeSignalIngressSink sink) MakeTracker(TempDirectory tmp)
{
var sink = new FakeSignalIngressSink();
var logger = new AgentLogger(tmp.Path, AgentLogLevel.Info);
var tracker = new ShellCoreTracker(
sessionId: "S1",
tenantId: "T1",
post: new InformationalEventPost(sink, new VirtualClock(ClockNow)),
logger: logger,
helloTracker: null);
return (tracker, sink);
}
private static IReadOnlyList<FakeSignalIngressSink.PostedSignal> ByType(FakeSignalIngressSink sink, string eventType) =>
sink.Posted.Where(p =>
p.Kind == DecisionSignalKind.InformationalEvent
&& p.Payload != null
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
&& et == eventType).ToList();
private static IReadOnlyDictionary<string, object> Data(FakeSignalIngressSink.PostedSignal s) =>
(IReadOnlyDictionary<string, object>)s.TypedPayload!;
[Fact]
public void ProcessEvent_62407_RebootCoalescing_EmitsWarning_ImmediateUpload()
{
using var tmp = new TempDirectory();
var (tracker, sink) = MakeTracker(tmp);
using (tracker)
{
var sourceTime = ClockNow.AddSeconds(-5);
tracker.ProcessEvent(
eventId: ShellCoreTracker.EventId_ShellCore_WebAppEvent,
description: CoalescingDescription,
timestamp: sourceTime,
providerName: "Microsoft-Windows-Shell-Core",
isBackfill: false);
var s = Assert.Single(ByType(sink, Constants.EventTypes.EspRebootCoalescing));
Assert.Equal("Warning", s.Payload![SignalPayloadKeys.Severity]);
Assert.Equal("true", s.Payload![SignalPayloadKeys.ImmediateUpload]);
Assert.Equal(sourceTime, s.OccurredAtUtc);
Assert.Equal(CoalescingDescription, Data(s)["description"]);
}
}
[Fact]
public void ProcessEvent_62407_RebootCoalescing_FiresOncePerRun()
{
using var tmp = new TempDirectory();
var (tracker, sink) = MakeTracker(tmp);
using (tracker)
{
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
CoalescingDescription, ClockNow, "Microsoft-Windows-Shell-Core", isBackfill: false);
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
CoalescingDescription, ClockNow.AddSeconds(1), "Microsoft-Windows-Shell-Core", isBackfill: false);
Assert.Single(ByType(sink, Constants.EventTypes.EspRebootCoalescing));
}
}
[Fact]
public void ProcessEvent_62407_RebootCoalescing_DoesNotTriggerEspSemantics()
{
// The coalescing record is corroboration only — it must not trip the ESP-exit /
// FinalizingSetup / failure machinery.
using var tmp = new TempDirectory();
var (tracker, sink) = MakeTracker(tmp);
using (tracker)
{
var raised = new List<string>();
tracker.FinalizingSetupPhaseTriggered += (_, reason) => raised.Add($"finalizing:{reason}");
tracker.EspExited += (_, _) => raised.Add("espExited");
tracker.EspFailureDetected += (_, type) => raised.Add($"failure:{type}");
tracker.WhiteGloveCompleted += (_, _) => raised.Add("whiteGlove");
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
CoalescingDescription, ClockNow, "Microsoft-Windows-Shell-Core", isBackfill: false);
Assert.Empty(raised);
Assert.False(tracker.IsEspExitedForTest);
Assert.Empty(ByType(sink, Constants.EventTypes.EspExiting));
Assert.Empty(ByType(sink, Constants.EventTypes.EspFailure));
}
}
[Fact]
public void ProcessEvent_62407_ExistingBranches_Unaffected()
{
// Ordering regression guard: WhiteGlove/failure/exiting still classify as before —
// the RebootCoalescing check comes AFTER them.
using var tmp = new TempDirectory();
var (tracker, sink) = MakeTracker(tmp);
using (tracker)
{
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
"Name: 'CommercialOOBE_ESPProgress_Page_Exiting'", ClockNow, "p", isBackfill: false);
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
"Name: 'CommercialOOBE_ESPProgress_WhiteGlove_Success'", ClockNow, "p", isBackfill: false);
Assert.Single(ByType(sink, Constants.EventTypes.EspExiting));
Assert.Single(ByType(sink, Constants.EventTypes.WhiteGloveComplete));
Assert.Empty(ByType(sink, Constants.EventTypes.EspRebootCoalescing));
}
}
[Fact]
public void Backfill_RebootCoalescing_Emits_WithBackfilledTrue_AndHistoricalTimestamp()
{
// Deliberate deviation from the other 62407 backfill branches: after the coalesced
// reboot restarts the agent, the backfill re-emission is the only reliable delivery.
using var tmp = new TempDirectory();
var (tracker, sink) = MakeTracker(tmp);
using (tracker)
{
var historical = ClockNow.AddMinutes(-3);
tracker.HandleBackfillRecord(
ShellCoreTracker.EventId_ShellCore_WebAppEvent, CoalescingDescription, historical);
var s = Assert.Single(ByType(sink, Constants.EventTypes.EspRebootCoalescing));
Assert.Equal(historical, s.OccurredAtUtc);
Assert.Equal(true, Data(s)["backfilled"]);
Assert.Equal("true", s.Payload![SignalPayloadKeys.ImmediateUpload]);
}
}
[Fact]
public void Backfill_RebootCoalescing_AfterLiveFire_DoesNotDoubleEmit_SameProcess()
{
// Same-process dedup via the fire-once flag. (Cross-process live+backfill duplication
// is possible and accepted — exists-semantics rule, identical historical timestamp.)
using var tmp = new TempDirectory();
var (tracker, sink) = MakeTracker(tmp);
using (tracker)
{
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
CoalescingDescription, ClockNow, "p", isBackfill: false);
tracker.HandleBackfillRecord(
ShellCoreTracker.EventId_ShellCore_WebAppEvent, CoalescingDescription, ClockNow.AddMinutes(-1));
Assert.Single(ByType(sink, Constants.EventTypes.EspRebootCoalescing));
}
}
[Fact]
public void Backfill_RebootCoalescing_DoesNotFallThroughToExitOrFailureHandling()
{
using var tmp = new TempDirectory();
var (tracker, sink) = MakeTracker(tmp);
using (tracker)
{
var raised = new List<string>();
tracker.FinalizingSetupPhaseTriggered += (_, reason) => raised.Add($"finalizing:{reason}");
tracker.EspExited += (_, _) => raised.Add("espExited");
tracker.EspFailureDetected += (_, type) => raised.Add($"failure:{type}");
tracker.HandleBackfillRecord(
ShellCoreTracker.EventId_ShellCore_WebAppEvent, CoalescingDescription, ClockNow.AddMinutes(-2));
Assert.Empty(raised);
Assert.False(tracker.IsEspExitedForTest);
}
}
}
}
@@ -0,0 +1,235 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.Shared;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
{
/// <summary>
/// MdmRebootPolicyTracker — attributes the "unexpected reboot + second sign-in" pattern to the
/// device-assigned policy URIs that forced the coalesced reboot (DM-Enterprise EventID 2800).
/// Drives the primitive <see cref="MdmRebootPolicyTracker.ProcessEvent"/> test seam (no real
/// EventRecord, which is abstract + Windows-only), mirroring the WindowsUpdateTracker pattern.
/// The registry probe is always overridden — tests never touch the live registry.
/// </summary>
public sealed class MdmRebootPolicyTrackerTests : IDisposable
{
private static readonly DateTime At = new DateTime(2026, 7, 20, 9, 30, 0, DateTimeKind.Utc);
private const string SampleUri = "./Device/Vendor/MSFT/Policy/Config/DeviceGuard/EnableVirtualizationBasedSecurity";
private readonly TempDirectory _tmp = new TempDirectory();
private readonly FakeSignalIngressSink _sink;
private readonly MdmRebootPolicyTracker _tracker;
private long _nextRecordId = 1;
public MdmRebootPolicyTrackerTests()
{
_sink = new FakeSignalIngressSink();
var post = new InformationalEventPost(_sink, new VirtualClock(At));
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
_tracker = new MdmRebootPolicyTracker(
sessionId: "sess-mdm",
tenantId: "tenant-mdm",
post: post,
logger: logger,
backfillEnabled: false,
stateDirectory: null); // in-memory dedup for most tests
_tracker.RebootRequiredFlagProbe = () => null; // registry untouched by default
}
public void Dispose() => _tmp.Dispose();
private void Emit(string? rebootUri = SampleUri, string? description = null, long? recordId = null, bool isBackfill = false, DateTime? timeCreatedUtc = null)
{
_tracker.ProcessEvent(
eventId: MdmRebootPolicyTracker.EventId_PolicyRebootRequired,
recordId: recordId ?? _nextRecordId++,
timeCreatedUtc: timeCreatedUtc ?? At,
rebootUri: rebootUri,
formattedDescription: description,
isBackfill: isBackfill);
}
private IReadOnlyList<FakeSignalIngressSink.PostedSignal> Emitted() =>
_sink.Posted.Where(p =>
p.Kind == DecisionSignalKind.InformationalEvent
&& p.Payload != null
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
&& et == Constants.EventTypes.MdmPolicyRebootRequired).ToList();
private static IReadOnlyDictionary<string, object> Data(FakeSignalIngressSink.PostedSignal s) =>
(IReadOnlyDictionary<string, object>)s.TypedPayload!;
[Fact]
public void ProcessEvent_EmitsWarning_WithUri_AndImmediateUpload()
{
Emit();
var s = Assert.Single(Emitted());
Assert.Equal("Warning", s.Payload![SignalPayloadKeys.Severity]);
Assert.Equal("true", s.Payload![SignalPayloadKeys.ImmediateUpload]);
var data = Data(s);
Assert.Equal(SampleUri, data["rebootUri"]);
Assert.Equal(MdmRebootPolicyTracker.EventId_PolicyRebootRequired, data["windowsEventId"]);
Assert.Equal(MdmRebootPolicyTracker.Channel, data["eventLogChannel"]);
}
[Fact]
public void ProcessEvent_NoUri_StillEmits_WithoutRebootUriField()
{
// Tolerant contract: an unparsed URI must stay timeline-visible (with the captured
// description telling us how to fix the extraction) — the analyze rule keys on
// "rebootUri exists" and simply stays silent.
Emit(rebootUri: null, description: "Some unexpected 2800 wording");
var s = Assert.Single(Emitted());
var data = Data(s);
Assert.False(data.ContainsKey("rebootUri"));
Assert.Equal("Some unexpected 2800 wording", data["description"]);
}
[Fact]
public void BackfilledEvent_UsesEventTimeForTimelineTimestamp_AndMarksBackfilled()
{
var eventTime = At.AddMinutes(-25); // policy applied well before agent start
Emit(isBackfill: true, timeCreatedUtc: eventTime);
var s = Assert.Single(Emitted());
Assert.Equal(eventTime, s.OccurredAtUtc);
Assert.Equal(true, Data(s)["backfilled"]);
}
[Fact]
public void SameRecordId_ProcessedTwice_EmitsOnce()
{
Emit(recordId: 42);
Emit(recordId: 42);
Assert.Single(Emitted());
}
[Fact]
public void BackfillEventWithLowerRecordId_AfterLiveEvent_IsStillEmitted()
{
// Same contract as WindowsUpdateTracker: the live watcher is armed BEFORE the backfill
// scan, so a live record with a higher RecordId can arrive first — the older,
// never-emitted backfill record must not be suppressed by a high-water mark.
Emit(recordId: 100);
Emit(recordId: 50, isBackfill: true);
Assert.Equal(2, Emitted().Count);
}
[Fact]
public void NegativeRecordId_IsNeverDeduped()
{
Emit(recordId: -1);
Emit(recordId: -1);
Assert.Equal(2, Emitted().Count);
}
[Fact]
public void Watermark_PersistsAcrossTrackerInstances()
{
// The coalesced reboot restarts the agent and the post-reboot backfill re-reads the
// same 2800 records — the persisted watermark must make that re-read a no-op.
var post = new InformationalEventPost(_sink, new VirtualClock(At));
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
var first = new MdmRebootPolicyTracker("s", "t", post, logger, backfillEnabled: false, stateDirectory: _tmp.Path);
first.RebootRequiredFlagProbe = () => null;
first.ProcessEvent(MdmRebootPolicyTracker.EventId_PolicyRebootRequired, recordId: 500,
timeCreatedUtc: At, rebootUri: SampleUri, formattedDescription: null, isBackfill: false);
var second = new MdmRebootPolicyTracker("s", "t", post, logger, backfillEnabled: false, stateDirectory: _tmp.Path);
second.RebootRequiredFlagProbe = () => null;
second.LoadWatermark();
second.ProcessEvent(MdmRebootPolicyTracker.EventId_PolicyRebootRequired, recordId: 500,
timeCreatedUtc: At, rebootUri: SampleUri, formattedDescription: null, isBackfill: true);
Assert.Single(Emitted());
}
[Fact]
public void RebootRequiredFlagProbe_True_AddsField_Null_OmitsIt()
{
_tracker.RebootRequiredFlagProbe = () => true;
Emit(recordId: 1);
_tracker.RebootRequiredFlagProbe = () => null;
Emit(recordId: 2);
var emitted = Emitted();
Assert.Equal(2, emitted.Count);
Assert.Equal(true, Data(emitted[0])["omadmRebootRequiredFlag"]);
Assert.False(Data(emitted[1]).ContainsKey("omadmRebootRequiredFlag"));
}
[Fact]
public void RebootRequiredFlagProbe_Throwing_IsFailSoft()
{
_tracker.RebootRequiredFlagProbe = () => throw new InvalidOperationException("boom");
Emit();
var s = Assert.Single(Emitted());
Assert.False(Data(s).ContainsKey("omadmRebootRequiredFlag"));
}
[Fact]
public void BuildXPath_Targets2800()
{
Assert.Equal("*[System[(EventID=2800)]]", MdmRebootPolicyTracker.BuildXPath());
}
// ------------------------------------------------------------------ URI extraction
[Fact]
public void ExtractRebootUri_PrefersEventDataValue_OverDescription()
{
var eventData = new Dictionary<string, string> { ["Data0"] = " " + SampleUri + " " };
var uri = MdmRebootPolicyTracker.ExtractRebootUri(eventData, "unrelated ./Other/Uri text");
Assert.Equal(SampleUri, uri);
}
[Fact]
public void ExtractRebootUri_RegexFallback_FromDescription()
{
var uri = MdmRebootPolicyTracker.ExtractRebootUri(
new Dictionary<string, string> { ["Data0"] = "not a uri" },
$"The following URI has triggered a reboot: {SampleUri}.");
Assert.Equal(SampleUri, uri);
}
[Fact]
public void ExtractRebootUri_ReturnsNull_WhenNothingUriLike()
{
Assert.Null(MdmRebootPolicyTracker.ExtractRebootUri(
new Dictionary<string, string> { ["Data0"] = "plain" }, "no uri here"));
Assert.Null(MdmRebootPolicyTracker.ExtractRebootUri(null, null));
}
[Fact]
public void ParseEventData_CollectsNamedAndPositionalData()
{
const string xml =
"<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'>" +
"<System><EventID>2800</EventID></System>" +
"<EventData><Data Name='Uri'>./Device/Vendor/MSFT/X</Data><Data>positional</Data></EventData>" +
"</Event>";
var parsed = MdmRebootPolicyTracker.ParseEventData(xml);
Assert.Equal("./Device/Vendor/MSFT/X", parsed["Uri"]);
Assert.Equal("positional", parsed["Data0"]);
}
}
}
@@ -0,0 +1,519 @@
using System;
using System.Collections.Generic;
using System.Diagnostics.Eventing.Reader;
using System.IO;
using System.Linq;
using System.Text.RegularExpressions;
using System.Xml.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
using Newtonsoft.Json;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
/// <summary>
/// Watches <c>Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin</c> for
/// EventID 2800 — logged once per device-assigned MDM policy whose CSP URI matches the OS
/// reboot-required catalog — and forwards each as a <c>mdm_policy_reboot_required</c> event
/// carrying the reboot-forcing URI.
/// <para>
/// This attributes the "unexpected reboot + second sign-in screen" pattern (PMPC research):
/// device-assigned policies applied during ESP DeviceSetup can demand a reboot; Windows
/// coalesces them into one restart at the end of DeviceSetup, forcing the user through a second
/// sign-in before AccountSetup. Admins are blind to this in the Intune console — the attributed
/// URIs make it actionable (reassign the profiles to user groups and the reboot disappears).
/// </para>
/// <para>
/// <b>Backfill.</b> Device policies can apply before the agent starts, so a startup backfill
/// scans the lookback window. The .evtx persists across the coalesced reboot, so the
/// post-reboot agent restart re-reads the same records — which is why the
/// <b>cross-restart RecordId watermark</b> (WindowsUpdateTracker pattern) is mandatory here:
/// without it every reboot-forcing URI would be re-emitted after the very reboot it caused.
/// </para>
/// <para>
/// <b>Tolerant matching.</b> The exact 2800 message text is unverified (validated post-deploy on
/// the E2E VM): emission is gated on the EventID only, never on message wording. URI extraction
/// prefers structured EventData values that look like an OMA-DM URI (<c>./…</c>) and falls back
/// to a permissive regex over the formatted description. Events with an unparseable URI are
/// still emitted (with the captured description) so the timeline shows them and the captured
/// text tells us how to fix the extraction — the analyze rule keys on <c>rebootUri exists</c>
/// and simply stays silent for those.
/// </para>
/// </summary>
internal sealed class MdmRebootPolicyTracker : IDisposable
{
internal const string Channel = "Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin";
internal const int EventId_PolicyRebootRequired = 2800;
internal const string WatermarkStateFileName = "mdm-reboot-watermark.json";
/// <summary>Registry key whose <c>RebootRequired</c> value/subkey flags a pending OMA-DM coalesced reboot.</summary>
internal const string OmadmSyncMlKeyPath = @"SOFTWARE\Microsoft\Provisioning\OMADM\SyncML";
// Permissive OMA-DM URI shape: "./Device/Vendor/MSFT/...", "./User/...", "./Vendor/...".
// Kept deliberately broad — the exact 2800 text is unverified; post-deploy E2E validates
// and tightens this against real records if needed.
private static readonly Regex UriPattern = new Regex(
@"\./[A-Za-z0-9_\-./\[\]{}%~+]+", RegexOptions.Compiled);
private readonly AgentLogger _logger;
private readonly string _sessionId;
private readonly string _tenantId;
private readonly InformationalEventPost _post;
private readonly bool _backfillEnabled;
private readonly int _backfillLookbackMinutes;
private readonly string _stateDirectory;
private EventLogWatcher _watcher;
// Dedup — same contract as WindowsUpdateTracker (see its field docs): immutable
// cross-restart boundary from disk + intra-run HashSet (NOT a high-water mark, because the
// live watcher is armed before the backfill scan and can deliver higher RecordIds first).
private long _restartWatermark = -1;
private readonly HashSet<long> _seenThisRun = new HashSet<long>();
private long _maxEmittedRecordId = -1;
private readonly object _watermarkLock = new object();
/// <summary>
/// Fail-soft corroboration probe for the OMA-DM RebootRequired flag — null = unknown
/// (key absent or unreadable). Settable test seam; the default reads the live registry.
/// </summary>
internal Func<bool?> RebootRequiredFlagProbe { get; set; } = ReadOmadmRebootRequiredFlag;
public MdmRebootPolicyTracker(
string sessionId,
string tenantId,
InformationalEventPost post,
AgentLogger logger,
bool backfillEnabled = true,
int backfillLookbackMinutes = 60,
string stateDirectory = null)
{
_sessionId = sessionId ?? throw new ArgumentNullException(nameof(sessionId));
_tenantId = tenantId ?? throw new ArgumentNullException(nameof(tenantId));
_post = post ?? throw new ArgumentNullException(nameof(post));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_backfillEnabled = backfillEnabled;
_backfillLookbackMinutes = backfillLookbackMinutes;
_stateDirectory = stateDirectory != null ? Environment.ExpandEnvironmentVariables(stateDirectory) : null;
}
public void Start()
{
LoadWatermark();
StartWatcher();
if (_backfillEnabled && _backfillLookbackMinutes > 0)
BackfillRecentEvents();
else
_logger.Info("MdmRebootPolicy backfill disabled by config");
}
public void Stop()
{
if (_watcher == null) return;
try
{
_watcher.Enabled = false;
_watcher.Dispose();
_logger.Info("MdmRebootPolicy watcher stopped");
}
catch (Exception ex)
{
_logger.Error("Error stopping MdmRebootPolicy watcher", ex);
}
finally
{
_watcher = null;
}
}
public void Dispose() => Stop();
// -----------------------------------------------------------------------
// Watcher lifecycle
// -----------------------------------------------------------------------
private void StartWatcher()
{
try
{
var query = new EventLogQuery(Channel, PathType.LogName, BuildXPath());
_watcher = new EventLogWatcher(query);
_watcher.EventRecordWritten += OnEventRecordWritten;
_watcher.Enabled = true;
_logger.Info($"MdmRebootPolicy watcher started: {Channel} (EventID {EventId_PolicyRebootRequired})");
}
catch (EventLogNotFoundException)
{
_logger.Warning($"MdmRebootPolicy event log not found: {Channel} (normal on non-MDM-enrolled test environments)");
}
catch (UnauthorizedAccessException ex)
{
_logger.Warning($"MdmRebootPolicy watcher access denied for {Channel}: {ex.Message}");
}
catch (Exception ex)
{
_logger.Error($"Failed to start MdmRebootPolicy watcher for {Channel}", ex);
// MON-D1: surface a dead watcher as one-shot telemetry, not just a local log line.
CollectorDegradationReporter.Report(_post, _sessionId, _tenantId,
collectorName: "MdmRebootPolicyTracker", reason: $"watcher_arm_failed:{Channel}", ex: ex);
}
}
/// <summary>Targeted-EventID XPath filter. Exposed as internal for tests.</summary>
internal static string BuildXPath() =>
$"*[System[(EventID={EventId_PolicyRebootRequired})]]";
// -----------------------------------------------------------------------
// Event processing
// -----------------------------------------------------------------------
private void OnEventRecordWritten(object sender, EventRecordWrittenEventArgs e)
{
if (e.EventRecord == null) return;
try
{
ProcessRecord(e.EventRecord, isBackfill: false);
}
catch (Exception ex)
{
_logger.Error("Error processing MdmRebootPolicy event", ex);
}
}
private void ProcessRecord(EventRecord record, bool isBackfill)
{
var recordId = record.RecordId ?? -1;
// Cheap dedup gate BEFORE the expensive XML/FormatDescription work.
if (IsAlreadyProcessed(recordId))
return;
string xml = null;
try { xml = record.ToXml(); }
catch { /* fall back to description-only parse below */ }
string description = null;
try { description = record.FormatDescription(); }
catch { /* some events lack formatting resources */ }
var rebootUri = ExtractRebootUri(ParseEventData(xml), description);
ProcessEvent(
eventId: record.Id,
recordId: recordId,
timeCreatedUtc: record.TimeCreated?.ToUniversalTime(),
rebootUri: rebootUri,
formattedDescription: description,
isBackfill: isBackfill);
}
/// <summary>
/// Core processing extracted to primitive inputs so tests can drive it without synthesizing
/// an abstract, Windows-only <see cref="EventRecord"/>. Mirrors the WindowsUpdateTracker
/// test-seam pattern.
/// </summary>
internal void ProcessEvent(
int eventId,
long recordId,
DateTime? timeCreatedUtc,
string rebootUri,
string formattedDescription,
bool isBackfill)
{
if (!MarkProcessed(recordId))
return; // already emitted (cross-restart or duplicate delivery)
var data = new Dictionary<string, object>
{
{ "windowsEventId", eventId },
{ "backfilled", isBackfill },
{ "eventLogChannel", Channel },
};
if (recordId >= 0) data["recordId"] = recordId;
if (timeCreatedUtc.HasValue) data["eventTime"] = timeCreatedUtc.Value.ToString("o");
if (!string.IsNullOrEmpty(rebootUri)) data["rebootUri"] = rebootUri;
if (!string.IsNullOrEmpty(formattedDescription))
{
data["description"] = formattedDescription.Length > 1000
? formattedDescription.Substring(0, 1000) + "…"
: formattedDescription;
}
var flag = SafeProbeRebootRequiredFlag();
if (flag.HasValue) data["omadmRebootRequiredFlag"] = flag.Value;
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
// The record's own time, not UtcNow: backfilled pre-agent (and post-reboot re-read)
// records must land on the timeline at when the policy actually demanded the reboot.
Timestamp = timeCreatedUtc ?? DateTime.UtcNow,
EventType = Constants.EventTypes.MdmPolicyRebootRequired,
Severity = EventSeverity.Warning,
Source = "MdmRebootPolicyWatcher",
Phase = EnrollmentPhase.Unknown,
Message = !string.IsNullOrEmpty(rebootUri)
? $"MDM policy requires a reboot during ESP: {rebootUri}"
: "MDM policy requires a reboot during ESP (URI not parsed from event — see description)",
Data = data,
// The reboot this event announces will kill the agent — flush now.
ImmediateUpload = true,
});
}
/// <summary>
/// Extracts the reboot-forcing OMA-DM URI. Preference order: (1) a structured EventData
/// value that starts with <c>./</c> (the CSP-URI shape), (2) a permissive regex over the
/// formatted description. Returns null when nothing URI-like is found — the caller still
/// emits, just without the <c>rebootUri</c> field.
/// </summary>
internal static string ExtractRebootUri(Dictionary<string, string> eventData, string description)
{
if (eventData != null)
{
foreach (var value in eventData.Values)
{
var trimmed = value?.Trim();
if (!string.IsNullOrEmpty(trimmed) && trimmed.StartsWith("./", StringComparison.Ordinal))
return trimmed;
}
}
if (!string.IsNullOrEmpty(description))
{
var match = UriPattern.Match(description);
if (match.Success)
{
// The permissive pattern can swallow sentence punctuation trailing the URI
// ("...Security."). OMA-DM URI segments never end in punctuation — trim it.
var candidate = match.Value.TrimEnd('.', ',', ';', ':', '\'', '"', ')');
if (candidate.Length > 2) return candidate;
}
}
return null;
}
/// <summary>
/// Extracts <c>&lt;Data Name="..."&gt;value&lt;/Data&gt;</c> pairs from an event's rendered
/// XML, keyed case-insensitively. Also collects UNNAMED Data elements under positional keys
/// (<c>Data0</c>, <c>Data1</c>, …) — DM-Enterprise events frequently carry positional-only
/// payloads. Namespace-agnostic; returns an empty map on null/malformed XML — never throws.
/// </summary>
internal static Dictionary<string, string> ParseEventData(string xml)
{
var result = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
if (string.IsNullOrEmpty(xml)) return result;
try
{
var doc = XDocument.Parse(xml);
var positionalIndex = 0;
foreach (var el in doc.Descendants().Where(e => e.Name.LocalName == "Data"))
{
var nameAttr = el.Attribute("Name");
var key = nameAttr != null && !string.IsNullOrEmpty(nameAttr.Value)
? nameAttr.Value
: "Data" + positionalIndex++;
if (!result.ContainsKey(key))
result[key] = el.Value;
}
}
catch
{
// Malformed / unexpected XML — best effort, return what we have.
}
return result;
}
// -----------------------------------------------------------------------
// OMA-DM RebootRequired flag probe (fail-soft corroboration)
// -----------------------------------------------------------------------
private bool? SafeProbeRebootRequiredFlag()
{
try
{
return RebootRequiredFlagProbe?.Invoke();
}
catch
{
return null; // corroboration only — never let the probe break the emit
}
}
/// <summary>
/// Reads the pending-reboot flag under <see cref="OmadmSyncMlKeyPath"/>. The exact shape is
/// unverified (PMPC research names the key, not the value type), so both a value and a
/// subkey named <c>RebootRequired</c> count as "pending". Fail-soft: null = unknown
/// (key absent / unreadable), matching the EspSkipConfigurationProbe convention.
/// </summary>
internal static bool? ReadOmadmRebootRequiredFlag()
{
try
{
using (var key = Microsoft.Win32.Registry.LocalMachine.OpenSubKey(OmadmSyncMlKeyPath))
{
if (key == null) return null;
if (key.GetValue("RebootRequired") != null) return true;
using (var sub = key.OpenSubKey("RebootRequired"))
{
return sub != null ? (bool?)true : false;
}
}
}
catch
{
return null;
}
}
// -----------------------------------------------------------------------
// Backfill
// -----------------------------------------------------------------------
private void BackfillRecentEvents()
{
try
{
var lookbackMs = (long)_backfillLookbackMinutes * 60 * 1000;
var xpath = $"*[System[(EventID={EventId_PolicyRebootRequired}) and TimeCreated[timediff(@SystemTime) <= {lookbackMs}]]]";
_logger.Info($"MdmRebootPolicy backfill: scanning {Channel} " +
$"(lookback={_backfillLookbackMinutes}min, restartWatermark={_restartWatermark})");
var query = new EventLogQuery(Channel, PathType.LogName, xpath);
int processed = 0;
using (var reader = new EventLogReader(query))
{
EventRecord record;
while ((record = reader.ReadEvent()) != null)
{
using (record)
{
ProcessRecord(record, isBackfill: true);
processed++;
}
}
}
if (processed == 0)
_logger.Debug($"MdmRebootPolicy backfill: no EventID {EventId_PolicyRebootRequired} records in last {_backfillLookbackMinutes} minutes");
else
_logger.Info($"MdmRebootPolicy backfill: scanned {processed} record(s)");
}
catch (EventLogNotFoundException)
{
_logger.Warning($"MdmRebootPolicy event log not found during backfill: {Channel} (normal on non-MDM-enrolled test environments)");
}
catch (UnauthorizedAccessException ex)
{
_logger.Warning($"MdmRebootPolicy backfill access denied: {ex.Message}");
}
catch (Exception ex)
{
_logger.Warning($"MdmRebootPolicy backfill failed: {ex.Message}");
}
}
// -----------------------------------------------------------------------
// Watermark dedup (cross-restart) — WindowsUpdateTracker pattern verbatim
// -----------------------------------------------------------------------
private bool IsAlreadyProcessed(long recordId)
{
if (recordId < 0) return false; // no RecordId → cannot dedup, let it through
lock (_watermarkLock)
{
return recordId <= _restartWatermark || _seenThisRun.Contains(recordId);
}
}
/// <summary>
/// Atomically claims <paramref name="recordId"/>. Returns true if this is the first time we
/// see it (caller should emit), false if it was already processed — either in a PRIOR run
/// (at/below the restart watermark) or already this run (in the seen-set). Records without
/// a RecordId (-1) are always emitted, never tracked or persisted.
/// </summary>
private bool MarkProcessed(long recordId)
{
if (recordId < 0) return true;
long toPersist = -1;
lock (_watermarkLock)
{
if (recordId <= _restartWatermark || !_seenThisRun.Add(recordId))
return false;
if (recordId > _maxEmittedRecordId)
{
_maxEmittedRecordId = recordId;
toPersist = _maxEmittedRecordId;
}
}
if (toPersist >= 0) PersistWatermark(toPersist);
return true;
}
internal void LoadWatermark()
{
if (string.IsNullOrEmpty(_stateDirectory)) return;
var filePath = Path.Combine(_stateDirectory, WatermarkStateFileName);
if (!File.Exists(filePath)) return;
try
{
var json = File.ReadAllText(filePath);
var state = JsonConvert.DeserializeObject<WatermarkState>(json);
if (state != null)
{
lock (_watermarkLock)
{
_restartWatermark = state.LastRecordId;
_maxEmittedRecordId = state.LastRecordId;
}
_logger.Info($"MdmRebootPolicy watermark loaded: lastRecordId={state.LastRecordId} (persisted {state.PersistedUtc:O})");
}
}
catch (Exception ex)
{
_logger.Warning($"Failed to load MdmRebootPolicy watermark: {ex.Message}");
}
}
private void PersistWatermark(long recordId)
{
if (string.IsNullOrEmpty(_stateDirectory)) return;
try
{
Directory.CreateDirectory(_stateDirectory);
var filePath = Path.Combine(_stateDirectory, WatermarkStateFileName);
var state = new WatermarkState { LastRecordId = recordId, PersistedUtc = DateTime.UtcNow };
var json = JsonConvert.SerializeObject(state, Formatting.Indented);
var tempPath = filePath + ".tmp";
File.WriteAllText(tempPath, json);
File.Copy(tempPath, filePath, overwrite: true);
try { File.Delete(tempPath); } catch { }
}
catch (Exception ex)
{
_logger.Warning($"Failed to persist MdmRebootPolicy watermark: {ex.Message}");
}
}
internal sealed class WatermarkState
{
public long LastRecordId { get; set; }
public DateTime PersistedUtc { get; set; }
}
}
}
@@ -16,6 +16,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
/// CommercialOOBE_ESPProgress_Page_Exiting — normal ESP exit
/// CommercialOOBE_ESPProgress_WhiteGlove_Success — WhiteGlove complete
/// CommercialOOBE_ESPProgress_Failure/_Timeout/_Abort/WhiteGlove_Failed — ESP failure
/// "RebootCoalescing" (DeviceSetup) — ESP-initiated coalesced reboot
/// (MDM policies forced a mid-ESP restart → second sign-in; corroborates the
/// MdmRebootPolicyTracker's per-URI 2800 attribution events)
///
/// Raises <see cref="FinalizingSetupPhaseTriggered"/>, <see cref="WhiteGloveCompleted"/>,
/// and <see cref="EspFailureDetected"/>. Cross-notifies the <see cref="HelloTracker"/>
@@ -47,6 +50,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
private bool _espExitDetected;
private bool _whiteGloveDetected;
private bool _helloWizardStartDetected;
private bool _rebootCoalescingDetected;
private readonly object _stateLock = new object();
/// <summary>
@@ -247,6 +251,27 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
_logger.Info("ESP phase exit detected - detected via Shell-Core event 62407");
}
else if (description.IndexOf("RebootCoalescing", StringComparison.OrdinalIgnoreCase) >= 0)
{
// ESP-initiated coalesced reboot (end of DeviceSetup): MDM policies forced a
// restart, the user will see a second sign-in screen. Checked AFTER the
// WhiteGlove/failure/exiting patterns so their semantics stay untouched even
// if a description ever carried both tokens. Fire-once per process run — the
// reboot restarts the agent, and the fresh run re-observes it via backfill.
lock (_stateLock)
{
if (_rebootCoalescingDetected) return;
_rebootCoalescingDetected = true;
}
eventType = Constants.EventTypes.EspRebootCoalescing;
severity = EventSeverity.Warning;
message = "ESP initiated a coalesced reboot (RebootCoalescing) — device will restart and show a second sign-in";
// No FinalizingSetup transition, no C# event raise — pure informational
// corroboration for the MdmRebootPolicyTracker's per-URI attribution.
_logger.Warning("ESP coalesced reboot detected via Shell-Core event 62407 (RebootCoalescing)");
}
else
{
return;
@@ -421,6 +446,29 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
return;
}
if (description.IndexOf("RebootCoalescing", StringComparison.OrdinalIgnoreCase) >= 0)
{
// DELIBERATE deviation from the other 62407 backfill branches (which only raise C#
// events): this is a pure informational type with no reducer/subscriber — the
// emission IS the signal, and the live emit races the very reboot that kills the
// process. Re-emitting after the restart is the only reliable delivery path. A
// live+backfill cross-process duplicate is possible (no watermark here); accepted:
// the analyze rule uses exists-semantics, both copies carry the identical historical
// timestamp, and the backfill copy is marked backfilled:true.
bool shouldEmit;
lock (_stateLock)
{
shouldEmit = !_rebootCoalescingDetected;
_rebootCoalescingDetected = true;
}
if (shouldEmit)
{
_logger.Info($"Backfill: ESP coalesced reboot (RebootCoalescing) found in recent Shell-Core logs (originalAt={occurredAtUtc:o})");
EmitRebootCoalescing(description, occurredAtUtc, isBackfill: true);
}
return; // a RebootCoalescing record is not an ESP exit/failure — stop here
}
if (EspExitingPattern.IsMatch(description))
{
bool shouldNotify = false;
@@ -459,6 +507,39 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
}
}
/// <summary>
/// Emits the <c>esp_reboot_coalescing</c> event with the standard Shell-Core eventData
/// shape. Used by the backfill path (the live path flows through the shared emit tail in
/// <see cref="ProcessEvent"/>); <paramref name="isBackfill"/> adds the
/// <c>backfilled</c> marker that distinguishes an accepted post-restart re-emission.
/// </summary>
private void EmitRebootCoalescing(string description, DateTime occurredAtUtc, bool isBackfill)
{
var eventData = new Dictionary<string, object>
{
{ "windowsEventId", EventId_ShellCore_WebAppEvent },
{ "providerName", "" },
{ "description", description },
{ "eventLogChannel", ShellCoreEventLogChannel },
{ "eventTime", occurredAtUtc.ToString("o") },
{ "backfilled", isBackfill }
};
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
Timestamp = occurredAtUtc,
EventType = Constants.EventTypes.EspRebootCoalescing,
Severity = EventSeverity.Warning,
Source = "EspAndHelloTracker",
Phase = EnrollmentPhase.Unknown,
Message = "ESP initiated a coalesced reboot (RebootCoalescing) — device will restart and show a second sign-in",
Data = eventData,
ImmediateUpload = true
});
}
// =====================================================================
// Pattern helpers
// =====================================================================
@@ -335,6 +335,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
osBuildChangedProvider: () => osBuildChangeHost.BuildChanged));
}
// MDM reboot-policy watcher — subscribes to DeviceManagement-Enterprise-Diagnostics-
// Provider/Admin EventID 2800 and attributes the "unexpected reboot + second sign-in"
// pattern to the device-assigned policy URIs that forced the coalesced reboot.
// Agent-native collector (NOT an IAgentAnalyzer); the config flag is an internal
// steering lever only.
if (collectors.MdmRebootPolicyWatcherEnabled)
{
hosts.Add(new MdmRebootPolicyWatcherHost(
sessionId: sessionId,
tenantId: tenantId,
logger: logger,
ingress: ingress,
clock: clock,
backfillLookbackMinutes: collectors.MdmRebootPolicyBackfillLookbackMinutes,
stateDirectory: _stateDirectory));
}
// ----- Peripheral hosts (event-only; driven by remote-config toggles) --------------
// V1 parity (PeriodicCollectorManager) — combine Performance + AgentSelfMetrics under
@@ -0,0 +1,68 @@
#nullable enable
using System;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
namespace AutopilotMonitor.Agent.V2.Core.Orchestration
{
/// <summary>
/// Standalone host for the <see cref="MdmRebootPolicyTracker"/> — watches
/// <c>Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin</c> (EventID
/// 2800) for MDM policies that force a coalesced reboot during ESP DeviceSetup and forwards
/// them via the shared single-rail ingress. Mirrors <see cref="WindowsUpdateWatcherHost"/>:
/// builds its own <see cref="InformationalEventPost"/> and delegates Start/Stop to the tracker.
/// </summary>
internal sealed class MdmRebootPolicyWatcherHost : ICollectorHost
{
public string Name => "MdmRebootPolicyTracker";
private readonly MdmRebootPolicyTracker _tracker;
private readonly AgentLogger _logger;
private int _disposed;
public MdmRebootPolicyWatcherHost(
string sessionId,
string tenantId,
AgentLogger logger,
ISignalIngressSink ingress,
IClock clock,
int backfillLookbackMinutes,
string stateDirectory)
{
if (ingress == null) throw new ArgumentNullException(nameof(ingress));
if (clock == null) throw new ArgumentNullException(nameof(clock));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
var post = new InformationalEventPost(ingress, clock, logger);
_tracker = new MdmRebootPolicyTracker(
sessionId: sessionId,
tenantId: tenantId,
post: post,
logger: logger,
backfillEnabled: backfillLookbackMinutes > 0,
backfillLookbackMinutes: backfillLookbackMinutes,
stateDirectory: stateDirectory);
}
public void Start()
{
_tracker.Start();
_logger.Info("MdmRebootPolicyWatcherHost: started.");
}
public void Stop()
{
try { _tracker.Stop(); }
catch (Exception ex) { _logger.Warning($"MdmRebootPolicyWatcherHost: stop failed: {ex.Message}"); }
}
public void Dispose()
{
if (Interlocked.Exchange(ref _disposed, 1) == 1) return;
Stop();
}
}
}
@@ -270,7 +270,7 @@ namespace AutopilotMonitor.Functions.Functions.Config
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new AgentConfigResponse
{
ConfigVersion = 34, // Gather rule phase scoping (ActivePhases/ActiveFromPhase) + EmitMode on_change anti-spam
ConfigVersion = 35, // MDM reboot-policy watcher (mdm_policy_reboot_required / esp_reboot_coalescing)
UploadIntervalSeconds = 10,
SelfDestructOnComplete = tenantConfig.SelfDestructOnComplete ?? true,
KeepLogFile = tenantConfig.KeepLogFile ?? false,
@@ -24,6 +24,8 @@ export const EVENT_TYPES_CATALOG = {
'esp_failure_settle_started',
'esp_failure_settle_recovered',
'esp_exiting',
'esp_reboot_coalescing',
'mdm_policy_reboot_required',
'esp_config_detected',
'completion_check',
'completion_waiting',
@@ -405,6 +405,18 @@ namespace AutopilotMonitor.Shared
public const string OsBuildChanged = "os_build_changed"; // OS build (CurrentBuild.UBR) differs across an agent restart — an update was installed during enrollment
public const string WindowsUpdateChannelCensus = "windows_update_channel_census"; // One-shot EventID histogram of WU Client + UpdateOrchestrator channels when os_build_changed but zero targeted WU events captured
// MDM-policy-forced coalesced reboot during ESP (MdmRebootPolicyTracker + ShellCoreTracker;
// PMPC "unexpected reboot / second sign-in" research). Device-assigned policies whose CSP
// URIs match the OS reboot-required catalog force a coalesced reboot at the end of
// DeviceSetup — the user sees an unexpected restart and a second sign-in screen before
// AccountSetup. 2800 (DeviceManagement-Enterprise-Diagnostics-Provider/Admin) fires once
// per reboot-forcing URI; Shell-Core 62407 "RebootCoalescing" records ESP initiating the
// reboot. Deliberately DISTINCT from system_reboot_detected: the backend RebootCount
// matches that string exactly (EventIngestProcessor.Classification + terminal recount) —
// these attribution events must never inflate the reboot count.
public const string MdmPolicyRebootRequired = "mdm_policy_reboot_required"; // DM-Enterprise EventID 2800 — Data.rebootUri = CSP URI that demands the reboot (one event per URI)
public const string EspRebootCoalescing = "esp_reboot_coalescing"; // Shell-Core 62407 "RebootCoalescing" — ESP-initiated coalesced reboot imminent (second sign-in follows)
// TEMPORARY: shadow SM rollout verification — remove when CompletionStateMachine is promoted to primary
public const string ShadowDiscrepancy = "shadow_discrepancy";
@@ -482,6 +482,28 @@ namespace AutopilotMonitor.Shared.Models
/// </summary>
public bool WindowsUpdateChannelCensusEnabled { get; set; } = true;
// -----------------------------------------------------------------------
// MDM reboot-policy watcher (DeviceManagement-Enterprise-Diagnostics-Provider/Admin)
// -----------------------------------------------------------------------
/// <summary>
/// Master switch for the MDM reboot-policy watcher. When enabled the agent subscribes to
/// <c>Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin</c> EventID
/// 2800 and emits one <c>mdm_policy_reboot_required</c> event per device-assigned policy
/// URI that forces a coalesced reboot during ESP DeviceSetup (the "unexpected reboot +
/// second sign-in" pattern). Internal steering lever only — not surfaced in any admin UI;
/// disable by pushing config server-side. Default: true.
/// </summary>
public bool MdmRebootPolicyWatcherEnabled { get; set; } = true;
/// <summary>
/// Lookback window in minutes for the MDM reboot-policy backfill scan on startup. Device
/// policies can apply before the agent starts, and after the coalesced reboot the restarted
/// agent must re-scan past the reboot gap (the cross-restart watermark keeps re-reads from
/// re-emitting). 0 = backfill disabled. Default: 60 minutes.
/// </summary>
public int MdmRebootPolicyBackfillLookbackMinutes { get; set; } = 60;
/// <summary>
/// Creates default collector configuration
/// </summary>