mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-24 04:32:32 +00:00
feat(agent): attribute mid-ESP coalesced reboots to MDM policy URIs (second sign-in detection)
Device-assigned MDM policies whose CSP URIs match the OS reboot-required
catalog force a coalesced reboot at the end of ESP DeviceSetup - the user
sees an unexpected restart and a second sign-in before AccountSetup.
Invisible in the Intune console; admins simply lived with it. Now attributed
and actionable (PMPC research).
- MdmRebootPolicyTracker (+host): watches DeviceManagement-Enterprise-
Diagnostics-Provider/Admin EventID 2800, emits one mdm_policy_reboot_required
per reboot-forcing URI (Warning, ImmediateUpload; tolerant EventData-first
URI extraction with regex fallback; fail-soft OMADM RebootRequired probe).
Cross-restart RecordId watermark (mdm-reboot-watermark.json) - mandatory,
the coalesced reboot restarts the agent and the backfill re-reads the log.
- ShellCoreTracker: RebootCoalescing branch on the already-watched 62407
stream -> esp_reboot_coalescing corroboration event; deliberate deviation:
emits on backfill too (live emit races the reboot kill).
- ANALYZE-ESP-005 advisory (warning): names the first URI via {{rebootUri}},
remediation = reassign profiles device -> user groups; +20 confidence on
coalescing corroboration, +10 on >=2 URIs.
- Config: Collectors.MdmRebootPolicyWatcherEnabled (default ON, internal
steering only - agent-native collector, NOT an IAgentAnalyzer, no admin UI),
MdmRebootPolicyBackfillLookbackMinutes=60; ConfigVersion 35.
- Event types deliberately distinct from system_reboot_detected so the
backend RebootCount (exact-string match, 2 sites) stays uninflated.
- Docs: docs/agent/mdm-reboot-coalescing.md (+index/log). Exact 2800/62407
message texts are unverified until E2E VM validation (tolerant matching).
Tests: Agent.V2.Core 1868/1868 (21 new), Functions 3099/3099, MCP 400 passed
(event-types drift), solution build 0 errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -7,5 +7,6 @@ How the on-device agent works:
|
||||
* [Logs, Persistence & Crash Recovery](logs-and-persistence.md) - every persisted file and why it exists; signal log vs spool; snapshot/journal; the replay-based recovery flow and quarantine.
|
||||
* [Hello Wizard Un-Skip](hello-wizard-unskip.md) - why a single "WHfB disabled" policy read never decides completion: the HelloWizardStarted signal (prevention + cure) and the HelloTracker confirmation second read.
|
||||
* [Autopilot ZTD Diagnostics](autopilot-ztd-diagnostics.md) - Windows' own diagnostic surfaces for the profile-download flow: ModernDeployment event IDs (807/815/164/...), the Diagnostics\Autopilot registry key, deployment-service endpoints, and the known-issue error-code map.
|
||||
* [MDM Reboot Coalescing](mdm-reboot-coalescing.md) - attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: 2800 watcher with cross-restart watermark, Shell-Core RebootCoalescing corroboration, ANALYZE-ESP-005 advisory.
|
||||
|
||||
Reading order for newcomers: overview → decision engine → persistence.
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
---
|
||||
type: Concept
|
||||
title: MDM Reboot Coalescing — attributing the mid-ESP reboot and second sign-in to policy URIs
|
||||
description: How the agent detects device-assigned MDM policies that force a coalesced reboot during ESP DeviceSetup (DM-Enterprise EventID 2800 + Shell-Core 62407 RebootCoalescing), and the ANALYZE-ESP-005 advisory that makes the invisible "second sign-in" fixable.
|
||||
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals
|
||||
tags:
|
||||
- agent
|
||||
- esp
|
||||
- reboot
|
||||
- policy-attribution
|
||||
- second-sign-in
|
||||
timestamp: 2026-07-20T00:00:00+02:00
|
||||
---
|
||||
|
||||
# MDM Reboot Coalescing
|
||||
|
||||
Device-assigned MDM policies whose CSP URIs match the OS reboot-required catalog
|
||||
(`HKLM\SOFTWARE\Microsoft\Provisioning\SyncML\RebootRequiredURIs`) force a **coalesced
|
||||
reboot at the end of ESP DeviceSetup** — the user sees an unexpected restart and a
|
||||
**second sign-in screen** before AccountSetup. This is expected OS behavior, invisible in
|
||||
the Intune console; most admins simply live with it, not knowing that reassigning the
|
||||
profiles to user groups removes the reboot entirely. Source research: Patch My PC,
|
||||
["Avoid the second sign-in screen after Autopilot"](https://patchmypc.com/blog/autopilot-unexpected-reboot-autopilot-second-login-screen/).
|
||||
|
||||
The session already counts reboots (`system_reboot_detected` → `session.rebootCount`);
|
||||
this feature adds the **cause attribution**.
|
||||
|
||||
# Signal chain
|
||||
|
||||
1. **EventID 2800**, `Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin`
|
||||
— logged by the SyncML engine once per policy URI that matched the reboot-required
|
||||
catalog, during device-policy application in DeviceSetup.
|
||||
2. **EventID 62407**, `Microsoft-Windows-Shell-Core/Operational`, description containing
|
||||
`RebootCoalescing` — ESP/CloudExperienceHost initiating the coalesced reboot at the end
|
||||
of DeviceSetup.
|
||||
3. The reboot kills the agent; the post-reboot restart re-observes both channels via
|
||||
backfill (the .evtx persists across the reboot).
|
||||
|
||||
# Detectors
|
||||
|
||||
## `MdmRebootPolicyTracker` (new collector, `MdmRebootPolicyWatcherHost`)
|
||||
|
||||
Structural clone of `WindowsUpdateTracker`: live `EventLogWatcher` on the DM-Enterprise
|
||||
Admin channel (EventID 2800) + startup backfill (default 60 min lookback) + **cross-restart
|
||||
RecordId watermark** (`mdm-reboot-watermark.json`). The watermark is mandatory here: the
|
||||
coalesced reboot restarts the agent, and without it every reboot-forcing URI would be
|
||||
re-emitted after the very reboot it caused.
|
||||
|
||||
Per 2800 record it emits one `mdm_policy_reboot_required` event (Warning,
|
||||
`ImmediateUpload=true` — the announced reboot will kill the agent, flush now) with
|
||||
`Data.rebootUri` (the CSP URI), `recordId`, `backfilled`, `description` (truncated 1000),
|
||||
and a fail-soft `omadmRebootRequiredFlag` corroboration probe of
|
||||
`HKLM\SOFTWARE\Microsoft\Provisioning\OMADM\SyncML\RebootRequired` (null = unknown, never
|
||||
blocks the emit). Per-URI emission (not aggregated) is deliberate: an aggregate would need
|
||||
a flush point that races the reboot kill.
|
||||
|
||||
**Tolerant matching**: the exact 2800 message text is unverified — emission is gated on
|
||||
the EventID only. URI extraction prefers structured EventData values starting with `./`
|
||||
and falls back to a permissive regex over the description (trailing punctuation trimmed).
|
||||
Events with an unparseable URI are still emitted (timeline-visible, rule stays silent, and
|
||||
the captured description tells us how to fix the extraction). Exact texts are validated
|
||||
post-deploy on the E2E VM and recorded here.
|
||||
|
||||
Config (internal steering only, no admin UI — this is an agent-native collector, NOT an
|
||||
`IAgentAnalyzer`): `Collectors.MdmRebootPolicyWatcherEnabled` (default true),
|
||||
`Collectors.MdmRebootPolicyBackfillLookbackMinutes` (default 60). ConfigVersion 35.
|
||||
|
||||
## `ShellCoreTracker` — `RebootCoalescing` branch
|
||||
|
||||
The already-watched 62407 stream gets a fourth branch (checked AFTER
|
||||
WhiteGlove/failure/exiting so their semantics stay untouched): description containing
|
||||
`RebootCoalescing` → one `esp_reboot_coalescing` event (Warning, ImmediateUpload), pure
|
||||
informational — no `FinalizingSetup` transition, no C# event raise, no DecisionCore
|
||||
involvement.
|
||||
|
||||
**Deliberate deviation**: unlike the other 62407 branches, the backfill path DOES emit
|
||||
this event. The live emit races the very reboot that kills the process; the post-restart
|
||||
5-min backfill re-emission is the only reliable delivery. A live+backfill cross-process
|
||||
duplicate is possible and accepted (fire-once bool resets per process; both copies carry
|
||||
the identical historical timestamp; the backfill copy is marked `backfilled:true`; the
|
||||
analyze rule uses exists-semantics). The shared 5-min lookback is NOT enlarged — the
|
||||
rule's required condition is the watermarked 2800 event; coalescing is optional
|
||||
corroboration only.
|
||||
|
||||
# Advisory — ANALYZE-ESP-005
|
||||
|
||||
"MDM Policy Forced a Mid-ESP Reboot (Second Sign-In)", severity `warning`,
|
||||
`markSessionAsFailedDefault: false`. Required: `mdm_policy_reboot_required.rebootUri`
|
||||
exists (first URI interpolates as `{{rebootUri}}`). Optional corroboration:
|
||||
`esp_reboot_coalescing` observed (+20 confidence) and ≥2 reboot URIs (+10) on base 70 /
|
||||
threshold 70. Remediation: map each URI to its Intune profile and reassign device → user
|
||||
groups (or exclude the Autopilot device group), or accept + communicate the restart.
|
||||
Because `{{token}}` interpolation is scalar-only, the explanation points at the timeline
|
||||
for the complete URI list — each URI is its own event.
|
||||
|
||||
# Non-goals / invariants
|
||||
|
||||
* The new event types are **deliberately distinct from `system_reboot_detected`** — the
|
||||
backend RebootCount matches that string exactly in two places
|
||||
(`EventIngestProcessor.Classification` + the terminal recount); attribution events must
|
||||
never inflate the reboot count.
|
||||
* No session-level field / SessionInfoCard change in this iteration (follow-up candidate).
|
||||
* The `RebootRequiredURIs` catalog itself is not read (format unverified); only the 2800
|
||||
events (which carry the actually-triggered URIs) are consumed.
|
||||
|
||||
# Citations
|
||||
|
||||
* [Patch My PC — Avoid the second sign-in screen after Autopilot](https://patchmypc.com/blog/autopilot-unexpected-reboot-autopilot-second-login-screen/)
|
||||
* [Microsoft Learn — Enrollment Status Page tracking](https://learn.microsoft.com/en-us/autopilot/enrollment-status)
|
||||
@@ -18,6 +18,7 @@ https://docs.autopilotmonitor.com (separate repository).
|
||||
* [V2 Agent](agent/index.md) - How the on-device agent works: [runtime overview](agent/overview.md), [decision engine](agent/decision-engine.md) (reducer, signals, completion arms), [logs & persistence](agent/logs-and-persistence.md) (signal log, snapshot, crash recovery), [Hello wizard un-skip](agent/hello-wizard-unskip.md) (why a single "WHfB disabled" read never decides completion), [Autopilot ZTD diagnostics](agent/autopilot-ztd-diagnostics.md) (event IDs, diagnostic registry, endpoints, error-code map).
|
||||
* [Build Counter Blob](agent/build-counter-blob.md) - How agent build numbers are minted: shared counter blob with ETag-CAS so local and CI builds never collide; CI additionally attests build provenance (Sigstore).
|
||||
* [Agent Endpoint Migration](agent/endpoint-migration.md) - How the backend re-homes agents to a new API base URL via the config channel (backend move or per-tenant region move): live-fetch-only, allowlist-validated on both sides, one hop, kill wins.
|
||||
* [MDM Reboot Coalescing](agent/mdm-reboot-coalescing.md) - Attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: DM-Enterprise 2800 watcher with cross-restart watermark, Shell-Core RebootCoalescing corroboration, ANALYZE-ESP-005 advisory.
|
||||
|
||||
# Rules
|
||||
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
## 2026-07-20
|
||||
|
||||
* **Creation**: Added `agent/mdm-reboot-coalescing.md` — the agent now attributes the "unexpected mid-ESP reboot + second sign-in" pattern (PMPC research) to the device-assigned policy URIs that forced it: new `MdmRebootPolicyTracker` (DM-Enterprise EventID 2800, per-URI `mdm_policy_reboot_required` events, cross-restart RecordId watermark), a `RebootCoalescing` branch in `ShellCoreTracker` (`esp_reboot_coalescing`, backfill-emitting by design), and the `ANALYZE-ESP-005` advisory (reassign device → user groups). ConfigVersion 35; event types deliberately distinct from `system_reboot_detected` so RebootCount stays uninflated.
|
||||
|
||||
* **Creation**: Added `backend/business-timestamps.md` (new `backend/` section) — the 2026-07-18 migration reset every row's system Timestamp, exposing that AuditLogs/OpsEvents/Events had (unknowingly) displayed and filtered on it all along; documents the `OccurredUtc` column, the RowKey decoders and RowKey-range date/retention clauses in `BusinessTimestamp`, the GUID-guard trap, the audit/ops backfill endpoint, and the "system Timestamp is never authoritative" migration rule.
|
||||
* **Creation**: Added `rules/gather-rule-phase-scoping.md` (new `rules/` section) — gather rules gain `activePhases` (run only during listed phases), `activeFromPhase` (sticky latch from a phase onwards, never via Failed) and `emitMode: "on_change"` (canonical-hash result dedup with `suppressedPolls` observability), ConfigVersion 34. Includes the prerequisite backend fix: the custom-rule toggle PUT no longer wipes the stored rule definition (partial-merge in `GatherRuleService.UpdateRuleAsync`).
|
||||
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
{
|
||||
"$schema": "../schema/analyze-rule.schema.json",
|
||||
"ruleId": "ANALYZE-ESP-005",
|
||||
"title": "MDM Policy Forced a Mid-ESP Reboot (Second Sign-In)",
|
||||
"description": "Detects the 'unexpected reboot + second sign-in screen' pattern: one or more device-assigned MDM policies matched the OS reboot-required catalog during ESP DeviceSetup, and Windows coalesced them into a restart at the end of DeviceSetup — forcing the user through a second sign-in before AccountSetup. Attributes the reboot to the specific CSP policy URIs so the profiles can be reassigned.",
|
||||
"severity": "warning",
|
||||
"category": "esp",
|
||||
"version": "1.0.0",
|
||||
"author": "Autopilot Monitor",
|
||||
"enabled": true,
|
||||
"trigger": "single",
|
||||
"baseConfidence": 70,
|
||||
"conditions": [
|
||||
{
|
||||
"signal": "reboot_forcing_policy_uri",
|
||||
"source": "event_data",
|
||||
"eventType": "mdm_policy_reboot_required",
|
||||
"dataField": "rebootUri",
|
||||
"operator": "exists",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"signal": "reboot_uri_count",
|
||||
"source": "event_count",
|
||||
"eventType": "mdm_policy_reboot_required",
|
||||
"operator": "count_gte",
|
||||
"value": "1",
|
||||
"required": false
|
||||
},
|
||||
{
|
||||
"signal": "esp_coalesced_reboot_observed",
|
||||
"source": "event_type",
|
||||
"eventType": "esp_reboot_coalescing",
|
||||
"operator": "exists",
|
||||
"required": false
|
||||
}
|
||||
],
|
||||
"confidenceFactors": [
|
||||
{ "signal": "esp_coalesced_reboot_observed", "condition": "exists", "weight": 20 },
|
||||
{ "signal": "mdm_policy_reboot_required", "condition": "count >= 2", "weight": 10 }
|
||||
],
|
||||
"confidenceThreshold": 70,
|
||||
"explanation": "During ESP DeviceSetup, a device-assigned MDM policy matched the OS reboot-required catalog and Windows scheduled a **coalesced reboot** at the end of DeviceSetup — the user saw an unexpected restart and a **second sign-in screen** before AccountSetup continued.\n\nFirst reboot-forcing policy URI: `{{rebootUri}}`\n\nWhen several policies force a reboot they are coalesced into one restart; **each** reboot-forcing policy appears as its own `mdm_policy_reboot_required` event on the session timeline — check the timeline for the complete URI list.\n\nThis is expected OS behavior, not a failure — but it degrades the enrollment experience, confuses users, and is invisible in the Intune console.",
|
||||
"remediation": [
|
||||
{
|
||||
"title": "Reassign the reboot-forcing policies to user groups",
|
||||
"steps": [
|
||||
"Open each `mdm_policy_reboot_required` timeline event and note the CSP URI (e.g. `./Device/Vendor/MSFT/...`)",
|
||||
"Map each URI to its Intune profile (Settings Catalog / configuration profile) — the URI path names the CSP area",
|
||||
"Reassign those profiles from device groups to user groups so they apply during/after AccountSetup instead of DeviceSetup",
|
||||
"Alternatively, exclude the Autopilot device group from the profile and deliver the setting post-enrollment"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "Or accept the reboot and set expectations",
|
||||
"steps": [
|
||||
"If the policy must be device-assigned (e.g. a security baseline), keep it and document the expected mid-ESP restart",
|
||||
"Communicate to users that a restart and second sign-in during provisioning is expected for these deployments"
|
||||
]
|
||||
}
|
||||
],
|
||||
"relatedDocs": [
|
||||
{
|
||||
"title": "Patch My PC: Avoid the second sign-in screen after Autopilot",
|
||||
"url": "https://patchmypc.com/blog/autopilot-unexpected-reboot-autopilot-second-login-screen/"
|
||||
},
|
||||
{
|
||||
"title": "Microsoft Learn: Enrollment Status Page tracking",
|
||||
"url": "https://learn.microsoft.com/en-us/autopilot/enrollment-status"
|
||||
}
|
||||
],
|
||||
"markSessionAsFailedDefault": false,
|
||||
"tags": ["esp", "reboot", "coalescing", "second-sign-in", "policy-attribution", "user-experience"]
|
||||
}
|
||||
+88
@@ -1785,6 +1785,94 @@
|
||||
"enrollment"
|
||||
]
|
||||
},
|
||||
{
|
||||
"ruleId": "ANALYZE-ESP-005",
|
||||
"title": "MDM Policy Forced a Mid-ESP Reboot (Second Sign-In)",
|
||||
"description": "Detects the 'unexpected reboot + second sign-in screen' pattern: one or more device-assigned MDM policies matched the OS reboot-required catalog during ESP DeviceSetup, and Windows coalesced them into a restart at the end of DeviceSetup — forcing the user through a second sign-in before AccountSetup. Attributes the reboot to the specific CSP policy URIs so the profiles can be reassigned.",
|
||||
"severity": "warning",
|
||||
"category": "esp",
|
||||
"version": "1.0.0",
|
||||
"author": "Autopilot Monitor",
|
||||
"enabled": true,
|
||||
"trigger": "single",
|
||||
"baseConfidence": 70,
|
||||
"conditions": [
|
||||
{
|
||||
"signal": "reboot_forcing_policy_uri",
|
||||
"source": "event_data",
|
||||
"eventType": "mdm_policy_reboot_required",
|
||||
"dataField": "rebootUri",
|
||||
"operator": "exists",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"signal": "reboot_uri_count",
|
||||
"source": "event_count",
|
||||
"eventType": "mdm_policy_reboot_required",
|
||||
"operator": "count_gte",
|
||||
"value": "1",
|
||||
"required": false
|
||||
},
|
||||
{
|
||||
"signal": "esp_coalesced_reboot_observed",
|
||||
"source": "event_type",
|
||||
"eventType": "esp_reboot_coalescing",
|
||||
"operator": "exists",
|
||||
"required": false
|
||||
}
|
||||
],
|
||||
"confidenceFactors": [
|
||||
{
|
||||
"signal": "esp_coalesced_reboot_observed",
|
||||
"condition": "exists",
|
||||
"weight": 20
|
||||
},
|
||||
{
|
||||
"signal": "mdm_policy_reboot_required",
|
||||
"condition": "count >= 2",
|
||||
"weight": 10
|
||||
}
|
||||
],
|
||||
"confidenceThreshold": 70,
|
||||
"explanation": "During ESP DeviceSetup, a device-assigned MDM policy matched the OS reboot-required catalog and Windows scheduled a **coalesced reboot** at the end of DeviceSetup — the user saw an unexpected restart and a **second sign-in screen** before AccountSetup continued.\n\nFirst reboot-forcing policy URI: `{{rebootUri}}`\n\nWhen several policies force a reboot they are coalesced into one restart; **each** reboot-forcing policy appears as its own `mdm_policy_reboot_required` event on the session timeline — check the timeline for the complete URI list.\n\nThis is expected OS behavior, not a failure — but it degrades the enrollment experience, confuses users, and is invisible in the Intune console.",
|
||||
"remediation": [
|
||||
{
|
||||
"title": "Reassign the reboot-forcing policies to user groups",
|
||||
"steps": [
|
||||
"Open each `mdm_policy_reboot_required` timeline event and note the CSP URI (e.g. `./Device/Vendor/MSFT/...`)",
|
||||
"Map each URI to its Intune profile (Settings Catalog / configuration profile) — the URI path names the CSP area",
|
||||
"Reassign those profiles from device groups to user groups so they apply during/after AccountSetup instead of DeviceSetup",
|
||||
"Alternatively, exclude the Autopilot device group from the profile and deliver the setting post-enrollment"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "Or accept the reboot and set expectations",
|
||||
"steps": [
|
||||
"If the policy must be device-assigned (e.g. a security baseline), keep it and document the expected mid-ESP restart",
|
||||
"Communicate to users that a restart and second sign-in during provisioning is expected for these deployments"
|
||||
]
|
||||
}
|
||||
],
|
||||
"relatedDocs": [
|
||||
{
|
||||
"title": "Patch My PC: Avoid the second sign-in screen after Autopilot",
|
||||
"url": "https://patchmypc.com/blog/autopilot-unexpected-reboot-autopilot-second-login-screen/"
|
||||
},
|
||||
{
|
||||
"title": "Microsoft Learn: Enrollment Status Page tracking",
|
||||
"url": "https://learn.microsoft.com/en-us/autopilot/enrollment-status"
|
||||
}
|
||||
],
|
||||
"markSessionAsFailedDefault": false,
|
||||
"tags": [
|
||||
"esp",
|
||||
"reboot",
|
||||
"coalescing",
|
||||
"second-sign-in",
|
||||
"policy-attribution",
|
||||
"user-experience"
|
||||
]
|
||||
},
|
||||
{
|
||||
"ruleId": "ANALYZE-ID-001",
|
||||
"title": "Expected Machine Certificate Not Found",
|
||||
|
||||
+195
@@ -0,0 +1,195 @@
|
||||
#nullable enable
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
|
||||
using AutopilotMonitor.Agent.V2.Core.Orchestration;
|
||||
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
|
||||
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
|
||||
using AutopilotMonitor.DecisionCore.Engine;
|
||||
using AutopilotMonitor.DecisionCore.Signals;
|
||||
using AutopilotMonitor.Shared;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
|
||||
{
|
||||
/// <summary>
|
||||
/// Shell-Core 62407 "RebootCoalescing" branch — ESP-initiated coalesced reboot (MDM policies
|
||||
/// forced a mid-ESP restart → second sign-in). Pure informational corroboration for the
|
||||
/// MdmRebootPolicyTracker's per-URI 2800 attribution: no FinalizingSetup transition, no C#
|
||||
/// event raise, and — as a DELIBERATE deviation from the other 62407 backfill branches — the
|
||||
/// backfill path DOES emit (the live emit races the very reboot that kills the process).
|
||||
/// </summary>
|
||||
public sealed class ShellCoreTrackerRebootCoalescingTests
|
||||
{
|
||||
private static readonly DateTime ClockNow = new DateTime(2026, 7, 20, 10, 5, 0, DateTimeKind.Utc);
|
||||
private const string CoalescingDescription =
|
||||
"CloudExperienceHost Web App Event 2. Name: 'CommercialOOBE_DeviceSetup_RebootCoalescing', Value: '...'.";
|
||||
|
||||
private static (ShellCoreTracker tracker, FakeSignalIngressSink sink) MakeTracker(TempDirectory tmp)
|
||||
{
|
||||
var sink = new FakeSignalIngressSink();
|
||||
var logger = new AgentLogger(tmp.Path, AgentLogLevel.Info);
|
||||
var tracker = new ShellCoreTracker(
|
||||
sessionId: "S1",
|
||||
tenantId: "T1",
|
||||
post: new InformationalEventPost(sink, new VirtualClock(ClockNow)),
|
||||
logger: logger,
|
||||
helloTracker: null);
|
||||
return (tracker, sink);
|
||||
}
|
||||
|
||||
private static IReadOnlyList<FakeSignalIngressSink.PostedSignal> ByType(FakeSignalIngressSink sink, string eventType) =>
|
||||
sink.Posted.Where(p =>
|
||||
p.Kind == DecisionSignalKind.InformationalEvent
|
||||
&& p.Payload != null
|
||||
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
|
||||
&& et == eventType).ToList();
|
||||
|
||||
private static IReadOnlyDictionary<string, object> Data(FakeSignalIngressSink.PostedSignal s) =>
|
||||
(IReadOnlyDictionary<string, object>)s.TypedPayload!;
|
||||
|
||||
[Fact]
|
||||
public void ProcessEvent_62407_RebootCoalescing_EmitsWarning_ImmediateUpload()
|
||||
{
|
||||
using var tmp = new TempDirectory();
|
||||
var (tracker, sink) = MakeTracker(tmp);
|
||||
using (tracker)
|
||||
{
|
||||
var sourceTime = ClockNow.AddSeconds(-5);
|
||||
tracker.ProcessEvent(
|
||||
eventId: ShellCoreTracker.EventId_ShellCore_WebAppEvent,
|
||||
description: CoalescingDescription,
|
||||
timestamp: sourceTime,
|
||||
providerName: "Microsoft-Windows-Shell-Core",
|
||||
isBackfill: false);
|
||||
|
||||
var s = Assert.Single(ByType(sink, Constants.EventTypes.EspRebootCoalescing));
|
||||
Assert.Equal("Warning", s.Payload![SignalPayloadKeys.Severity]);
|
||||
Assert.Equal("true", s.Payload![SignalPayloadKeys.ImmediateUpload]);
|
||||
Assert.Equal(sourceTime, s.OccurredAtUtc);
|
||||
Assert.Equal(CoalescingDescription, Data(s)["description"]);
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ProcessEvent_62407_RebootCoalescing_FiresOncePerRun()
|
||||
{
|
||||
using var tmp = new TempDirectory();
|
||||
var (tracker, sink) = MakeTracker(tmp);
|
||||
using (tracker)
|
||||
{
|
||||
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
|
||||
CoalescingDescription, ClockNow, "Microsoft-Windows-Shell-Core", isBackfill: false);
|
||||
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
|
||||
CoalescingDescription, ClockNow.AddSeconds(1), "Microsoft-Windows-Shell-Core", isBackfill: false);
|
||||
|
||||
Assert.Single(ByType(sink, Constants.EventTypes.EspRebootCoalescing));
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ProcessEvent_62407_RebootCoalescing_DoesNotTriggerEspSemantics()
|
||||
{
|
||||
// The coalescing record is corroboration only — it must not trip the ESP-exit /
|
||||
// FinalizingSetup / failure machinery.
|
||||
using var tmp = new TempDirectory();
|
||||
var (tracker, sink) = MakeTracker(tmp);
|
||||
using (tracker)
|
||||
{
|
||||
var raised = new List<string>();
|
||||
tracker.FinalizingSetupPhaseTriggered += (_, reason) => raised.Add($"finalizing:{reason}");
|
||||
tracker.EspExited += (_, _) => raised.Add("espExited");
|
||||
tracker.EspFailureDetected += (_, type) => raised.Add($"failure:{type}");
|
||||
tracker.WhiteGloveCompleted += (_, _) => raised.Add("whiteGlove");
|
||||
|
||||
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
|
||||
CoalescingDescription, ClockNow, "Microsoft-Windows-Shell-Core", isBackfill: false);
|
||||
|
||||
Assert.Empty(raised);
|
||||
Assert.False(tracker.IsEspExitedForTest);
|
||||
Assert.Empty(ByType(sink, Constants.EventTypes.EspExiting));
|
||||
Assert.Empty(ByType(sink, Constants.EventTypes.EspFailure));
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ProcessEvent_62407_ExistingBranches_Unaffected()
|
||||
{
|
||||
// Ordering regression guard: WhiteGlove/failure/exiting still classify as before —
|
||||
// the RebootCoalescing check comes AFTER them.
|
||||
using var tmp = new TempDirectory();
|
||||
var (tracker, sink) = MakeTracker(tmp);
|
||||
using (tracker)
|
||||
{
|
||||
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
|
||||
"Name: 'CommercialOOBE_ESPProgress_Page_Exiting'", ClockNow, "p", isBackfill: false);
|
||||
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
|
||||
"Name: 'CommercialOOBE_ESPProgress_WhiteGlove_Success'", ClockNow, "p", isBackfill: false);
|
||||
|
||||
Assert.Single(ByType(sink, Constants.EventTypes.EspExiting));
|
||||
Assert.Single(ByType(sink, Constants.EventTypes.WhiteGloveComplete));
|
||||
Assert.Empty(ByType(sink, Constants.EventTypes.EspRebootCoalescing));
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Backfill_RebootCoalescing_Emits_WithBackfilledTrue_AndHistoricalTimestamp()
|
||||
{
|
||||
// Deliberate deviation from the other 62407 backfill branches: after the coalesced
|
||||
// reboot restarts the agent, the backfill re-emission is the only reliable delivery.
|
||||
using var tmp = new TempDirectory();
|
||||
var (tracker, sink) = MakeTracker(tmp);
|
||||
using (tracker)
|
||||
{
|
||||
var historical = ClockNow.AddMinutes(-3);
|
||||
tracker.HandleBackfillRecord(
|
||||
ShellCoreTracker.EventId_ShellCore_WebAppEvent, CoalescingDescription, historical);
|
||||
|
||||
var s = Assert.Single(ByType(sink, Constants.EventTypes.EspRebootCoalescing));
|
||||
Assert.Equal(historical, s.OccurredAtUtc);
|
||||
Assert.Equal(true, Data(s)["backfilled"]);
|
||||
Assert.Equal("true", s.Payload![SignalPayloadKeys.ImmediateUpload]);
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Backfill_RebootCoalescing_AfterLiveFire_DoesNotDoubleEmit_SameProcess()
|
||||
{
|
||||
// Same-process dedup via the fire-once flag. (Cross-process live+backfill duplication
|
||||
// is possible and accepted — exists-semantics rule, identical historical timestamp.)
|
||||
using var tmp = new TempDirectory();
|
||||
var (tracker, sink) = MakeTracker(tmp);
|
||||
using (tracker)
|
||||
{
|
||||
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
|
||||
CoalescingDescription, ClockNow, "p", isBackfill: false);
|
||||
tracker.HandleBackfillRecord(
|
||||
ShellCoreTracker.EventId_ShellCore_WebAppEvent, CoalescingDescription, ClockNow.AddMinutes(-1));
|
||||
|
||||
Assert.Single(ByType(sink, Constants.EventTypes.EspRebootCoalescing));
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Backfill_RebootCoalescing_DoesNotFallThroughToExitOrFailureHandling()
|
||||
{
|
||||
using var tmp = new TempDirectory();
|
||||
var (tracker, sink) = MakeTracker(tmp);
|
||||
using (tracker)
|
||||
{
|
||||
var raised = new List<string>();
|
||||
tracker.FinalizingSetupPhaseTriggered += (_, reason) => raised.Add($"finalizing:{reason}");
|
||||
tracker.EspExited += (_, _) => raised.Add("espExited");
|
||||
tracker.EspFailureDetected += (_, type) => raised.Add($"failure:{type}");
|
||||
|
||||
tracker.HandleBackfillRecord(
|
||||
ShellCoreTracker.EventId_ShellCore_WebAppEvent, CoalescingDescription, ClockNow.AddMinutes(-2));
|
||||
|
||||
Assert.Empty(raised);
|
||||
Assert.False(tracker.IsEspExitedForTest);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
+235
@@ -0,0 +1,235 @@
|
||||
#nullable enable
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
|
||||
using AutopilotMonitor.Agent.V2.Core.Orchestration;
|
||||
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
|
||||
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
|
||||
using AutopilotMonitor.DecisionCore.Engine;
|
||||
using AutopilotMonitor.DecisionCore.Signals;
|
||||
using AutopilotMonitor.Shared;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
|
||||
{
|
||||
/// <summary>
|
||||
/// MdmRebootPolicyTracker — attributes the "unexpected reboot + second sign-in" pattern to the
|
||||
/// device-assigned policy URIs that forced the coalesced reboot (DM-Enterprise EventID 2800).
|
||||
/// Drives the primitive <see cref="MdmRebootPolicyTracker.ProcessEvent"/> test seam (no real
|
||||
/// EventRecord, which is abstract + Windows-only), mirroring the WindowsUpdateTracker pattern.
|
||||
/// The registry probe is always overridden — tests never touch the live registry.
|
||||
/// </summary>
|
||||
public sealed class MdmRebootPolicyTrackerTests : IDisposable
|
||||
{
|
||||
private static readonly DateTime At = new DateTime(2026, 7, 20, 9, 30, 0, DateTimeKind.Utc);
|
||||
private const string SampleUri = "./Device/Vendor/MSFT/Policy/Config/DeviceGuard/EnableVirtualizationBasedSecurity";
|
||||
|
||||
private readonly TempDirectory _tmp = new TempDirectory();
|
||||
private readonly FakeSignalIngressSink _sink;
|
||||
private readonly MdmRebootPolicyTracker _tracker;
|
||||
private long _nextRecordId = 1;
|
||||
|
||||
public MdmRebootPolicyTrackerTests()
|
||||
{
|
||||
_sink = new FakeSignalIngressSink();
|
||||
var post = new InformationalEventPost(_sink, new VirtualClock(At));
|
||||
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
|
||||
_tracker = new MdmRebootPolicyTracker(
|
||||
sessionId: "sess-mdm",
|
||||
tenantId: "tenant-mdm",
|
||||
post: post,
|
||||
logger: logger,
|
||||
backfillEnabled: false,
|
||||
stateDirectory: null); // in-memory dedup for most tests
|
||||
_tracker.RebootRequiredFlagProbe = () => null; // registry untouched by default
|
||||
}
|
||||
|
||||
public void Dispose() => _tmp.Dispose();
|
||||
|
||||
private void Emit(string? rebootUri = SampleUri, string? description = null, long? recordId = null, bool isBackfill = false, DateTime? timeCreatedUtc = null)
|
||||
{
|
||||
_tracker.ProcessEvent(
|
||||
eventId: MdmRebootPolicyTracker.EventId_PolicyRebootRequired,
|
||||
recordId: recordId ?? _nextRecordId++,
|
||||
timeCreatedUtc: timeCreatedUtc ?? At,
|
||||
rebootUri: rebootUri,
|
||||
formattedDescription: description,
|
||||
isBackfill: isBackfill);
|
||||
}
|
||||
|
||||
private IReadOnlyList<FakeSignalIngressSink.PostedSignal> Emitted() =>
|
||||
_sink.Posted.Where(p =>
|
||||
p.Kind == DecisionSignalKind.InformationalEvent
|
||||
&& p.Payload != null
|
||||
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
|
||||
&& et == Constants.EventTypes.MdmPolicyRebootRequired).ToList();
|
||||
|
||||
private static IReadOnlyDictionary<string, object> Data(FakeSignalIngressSink.PostedSignal s) =>
|
||||
(IReadOnlyDictionary<string, object>)s.TypedPayload!;
|
||||
|
||||
[Fact]
|
||||
public void ProcessEvent_EmitsWarning_WithUri_AndImmediateUpload()
|
||||
{
|
||||
Emit();
|
||||
|
||||
var s = Assert.Single(Emitted());
|
||||
Assert.Equal("Warning", s.Payload![SignalPayloadKeys.Severity]);
|
||||
Assert.Equal("true", s.Payload![SignalPayloadKeys.ImmediateUpload]);
|
||||
|
||||
var data = Data(s);
|
||||
Assert.Equal(SampleUri, data["rebootUri"]);
|
||||
Assert.Equal(MdmRebootPolicyTracker.EventId_PolicyRebootRequired, data["windowsEventId"]);
|
||||
Assert.Equal(MdmRebootPolicyTracker.Channel, data["eventLogChannel"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ProcessEvent_NoUri_StillEmits_WithoutRebootUriField()
|
||||
{
|
||||
// Tolerant contract: an unparsed URI must stay timeline-visible (with the captured
|
||||
// description telling us how to fix the extraction) — the analyze rule keys on
|
||||
// "rebootUri exists" and simply stays silent.
|
||||
Emit(rebootUri: null, description: "Some unexpected 2800 wording");
|
||||
|
||||
var s = Assert.Single(Emitted());
|
||||
var data = Data(s);
|
||||
Assert.False(data.ContainsKey("rebootUri"));
|
||||
Assert.Equal("Some unexpected 2800 wording", data["description"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BackfilledEvent_UsesEventTimeForTimelineTimestamp_AndMarksBackfilled()
|
||||
{
|
||||
var eventTime = At.AddMinutes(-25); // policy applied well before agent start
|
||||
Emit(isBackfill: true, timeCreatedUtc: eventTime);
|
||||
|
||||
var s = Assert.Single(Emitted());
|
||||
Assert.Equal(eventTime, s.OccurredAtUtc);
|
||||
Assert.Equal(true, Data(s)["backfilled"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SameRecordId_ProcessedTwice_EmitsOnce()
|
||||
{
|
||||
Emit(recordId: 42);
|
||||
Emit(recordId: 42);
|
||||
|
||||
Assert.Single(Emitted());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BackfillEventWithLowerRecordId_AfterLiveEvent_IsStillEmitted()
|
||||
{
|
||||
// Same contract as WindowsUpdateTracker: the live watcher is armed BEFORE the backfill
|
||||
// scan, so a live record with a higher RecordId can arrive first — the older,
|
||||
// never-emitted backfill record must not be suppressed by a high-water mark.
|
||||
Emit(recordId: 100);
|
||||
Emit(recordId: 50, isBackfill: true);
|
||||
|
||||
Assert.Equal(2, Emitted().Count);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NegativeRecordId_IsNeverDeduped()
|
||||
{
|
||||
Emit(recordId: -1);
|
||||
Emit(recordId: -1);
|
||||
|
||||
Assert.Equal(2, Emitted().Count);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Watermark_PersistsAcrossTrackerInstances()
|
||||
{
|
||||
// The coalesced reboot restarts the agent and the post-reboot backfill re-reads the
|
||||
// same 2800 records — the persisted watermark must make that re-read a no-op.
|
||||
var post = new InformationalEventPost(_sink, new VirtualClock(At));
|
||||
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
|
||||
|
||||
var first = new MdmRebootPolicyTracker("s", "t", post, logger, backfillEnabled: false, stateDirectory: _tmp.Path);
|
||||
first.RebootRequiredFlagProbe = () => null;
|
||||
first.ProcessEvent(MdmRebootPolicyTracker.EventId_PolicyRebootRequired, recordId: 500,
|
||||
timeCreatedUtc: At, rebootUri: SampleUri, formattedDescription: null, isBackfill: false);
|
||||
|
||||
var second = new MdmRebootPolicyTracker("s", "t", post, logger, backfillEnabled: false, stateDirectory: _tmp.Path);
|
||||
second.RebootRequiredFlagProbe = () => null;
|
||||
second.LoadWatermark();
|
||||
second.ProcessEvent(MdmRebootPolicyTracker.EventId_PolicyRebootRequired, recordId: 500,
|
||||
timeCreatedUtc: At, rebootUri: SampleUri, formattedDescription: null, isBackfill: true);
|
||||
|
||||
Assert.Single(Emitted());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RebootRequiredFlagProbe_True_AddsField_Null_OmitsIt()
|
||||
{
|
||||
_tracker.RebootRequiredFlagProbe = () => true;
|
||||
Emit(recordId: 1);
|
||||
_tracker.RebootRequiredFlagProbe = () => null;
|
||||
Emit(recordId: 2);
|
||||
|
||||
var emitted = Emitted();
|
||||
Assert.Equal(2, emitted.Count);
|
||||
Assert.Equal(true, Data(emitted[0])["omadmRebootRequiredFlag"]);
|
||||
Assert.False(Data(emitted[1]).ContainsKey("omadmRebootRequiredFlag"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RebootRequiredFlagProbe_Throwing_IsFailSoft()
|
||||
{
|
||||
_tracker.RebootRequiredFlagProbe = () => throw new InvalidOperationException("boom");
|
||||
Emit();
|
||||
|
||||
var s = Assert.Single(Emitted());
|
||||
Assert.False(Data(s).ContainsKey("omadmRebootRequiredFlag"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BuildXPath_Targets2800()
|
||||
{
|
||||
Assert.Equal("*[System[(EventID=2800)]]", MdmRebootPolicyTracker.BuildXPath());
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------ URI extraction
|
||||
|
||||
[Fact]
|
||||
public void ExtractRebootUri_PrefersEventDataValue_OverDescription()
|
||||
{
|
||||
var eventData = new Dictionary<string, string> { ["Data0"] = " " + SampleUri + " " };
|
||||
var uri = MdmRebootPolicyTracker.ExtractRebootUri(eventData, "unrelated ./Other/Uri text");
|
||||
Assert.Equal(SampleUri, uri);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ExtractRebootUri_RegexFallback_FromDescription()
|
||||
{
|
||||
var uri = MdmRebootPolicyTracker.ExtractRebootUri(
|
||||
new Dictionary<string, string> { ["Data0"] = "not a uri" },
|
||||
$"The following URI has triggered a reboot: {SampleUri}.");
|
||||
Assert.Equal(SampleUri, uri);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ExtractRebootUri_ReturnsNull_WhenNothingUriLike()
|
||||
{
|
||||
Assert.Null(MdmRebootPolicyTracker.ExtractRebootUri(
|
||||
new Dictionary<string, string> { ["Data0"] = "plain" }, "no uri here"));
|
||||
Assert.Null(MdmRebootPolicyTracker.ExtractRebootUri(null, null));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ParseEventData_CollectsNamedAndPositionalData()
|
||||
{
|
||||
const string xml =
|
||||
"<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'>" +
|
||||
"<System><EventID>2800</EventID></System>" +
|
||||
"<EventData><Data Name='Uri'>./Device/Vendor/MSFT/X</Data><Data>positional</Data></EventData>" +
|
||||
"</Event>";
|
||||
|
||||
var parsed = MdmRebootPolicyTracker.ParseEventData(xml);
|
||||
Assert.Equal("./Device/Vendor/MSFT/X", parsed["Uri"]);
|
||||
Assert.Equal("positional", parsed["Data0"]);
|
||||
}
|
||||
}
|
||||
}
|
||||
+519
@@ -0,0 +1,519 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Diagnostics.Eventing.Reader;
|
||||
using System.IO;
|
||||
using System.Linq;
|
||||
using System.Text.RegularExpressions;
|
||||
using System.Xml.Linq;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using AutopilotMonitor.Agent.V2.Core.Orchestration;
|
||||
using AutopilotMonitor.Shared;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using Newtonsoft.Json;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
{
|
||||
/// <summary>
|
||||
/// Watches <c>Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin</c> for
|
||||
/// EventID 2800 — logged once per device-assigned MDM policy whose CSP URI matches the OS
|
||||
/// reboot-required catalog — and forwards each as a <c>mdm_policy_reboot_required</c> event
|
||||
/// carrying the reboot-forcing URI.
|
||||
/// <para>
|
||||
/// This attributes the "unexpected reboot + second sign-in screen" pattern (PMPC research):
|
||||
/// device-assigned policies applied during ESP DeviceSetup can demand a reboot; Windows
|
||||
/// coalesces them into one restart at the end of DeviceSetup, forcing the user through a second
|
||||
/// sign-in before AccountSetup. Admins are blind to this in the Intune console — the attributed
|
||||
/// URIs make it actionable (reassign the profiles to user groups and the reboot disappears).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Backfill.</b> Device policies can apply before the agent starts, so a startup backfill
|
||||
/// scans the lookback window. The .evtx persists across the coalesced reboot, so the
|
||||
/// post-reboot agent restart re-reads the same records — which is why the
|
||||
/// <b>cross-restart RecordId watermark</b> (WindowsUpdateTracker pattern) is mandatory here:
|
||||
/// without it every reboot-forcing URI would be re-emitted after the very reboot it caused.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Tolerant matching.</b> The exact 2800 message text is unverified (validated post-deploy on
|
||||
/// the E2E VM): emission is gated on the EventID only, never on message wording. URI extraction
|
||||
/// prefers structured EventData values that look like an OMA-DM URI (<c>./…</c>) and falls back
|
||||
/// to a permissive regex over the formatted description. Events with an unparseable URI are
|
||||
/// still emitted (with the captured description) so the timeline shows them and the captured
|
||||
/// text tells us how to fix the extraction — the analyze rule keys on <c>rebootUri exists</c>
|
||||
/// and simply stays silent for those.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
internal sealed class MdmRebootPolicyTracker : IDisposable
|
||||
{
|
||||
internal const string Channel = "Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin";
|
||||
internal const int EventId_PolicyRebootRequired = 2800;
|
||||
|
||||
internal const string WatermarkStateFileName = "mdm-reboot-watermark.json";
|
||||
|
||||
/// <summary>Registry key whose <c>RebootRequired</c> value/subkey flags a pending OMA-DM coalesced reboot.</summary>
|
||||
internal const string OmadmSyncMlKeyPath = @"SOFTWARE\Microsoft\Provisioning\OMADM\SyncML";
|
||||
|
||||
// Permissive OMA-DM URI shape: "./Device/Vendor/MSFT/...", "./User/...", "./Vendor/...".
|
||||
// Kept deliberately broad — the exact 2800 text is unverified; post-deploy E2E validates
|
||||
// and tightens this against real records if needed.
|
||||
private static readonly Regex UriPattern = new Regex(
|
||||
@"\./[A-Za-z0-9_\-./\[\]{}%~+]+", RegexOptions.Compiled);
|
||||
|
||||
private readonly AgentLogger _logger;
|
||||
private readonly string _sessionId;
|
||||
private readonly string _tenantId;
|
||||
private readonly InformationalEventPost _post;
|
||||
private readonly bool _backfillEnabled;
|
||||
private readonly int _backfillLookbackMinutes;
|
||||
private readonly string _stateDirectory;
|
||||
|
||||
private EventLogWatcher _watcher;
|
||||
|
||||
// Dedup — same contract as WindowsUpdateTracker (see its field docs): immutable
|
||||
// cross-restart boundary from disk + intra-run HashSet (NOT a high-water mark, because the
|
||||
// live watcher is armed before the backfill scan and can deliver higher RecordIds first).
|
||||
private long _restartWatermark = -1;
|
||||
private readonly HashSet<long> _seenThisRun = new HashSet<long>();
|
||||
private long _maxEmittedRecordId = -1;
|
||||
private readonly object _watermarkLock = new object();
|
||||
|
||||
/// <summary>
|
||||
/// Fail-soft corroboration probe for the OMA-DM RebootRequired flag — null = unknown
|
||||
/// (key absent or unreadable). Settable test seam; the default reads the live registry.
|
||||
/// </summary>
|
||||
internal Func<bool?> RebootRequiredFlagProbe { get; set; } = ReadOmadmRebootRequiredFlag;
|
||||
|
||||
public MdmRebootPolicyTracker(
|
||||
string sessionId,
|
||||
string tenantId,
|
||||
InformationalEventPost post,
|
||||
AgentLogger logger,
|
||||
bool backfillEnabled = true,
|
||||
int backfillLookbackMinutes = 60,
|
||||
string stateDirectory = null)
|
||||
{
|
||||
_sessionId = sessionId ?? throw new ArgumentNullException(nameof(sessionId));
|
||||
_tenantId = tenantId ?? throw new ArgumentNullException(nameof(tenantId));
|
||||
_post = post ?? throw new ArgumentNullException(nameof(post));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_backfillEnabled = backfillEnabled;
|
||||
_backfillLookbackMinutes = backfillLookbackMinutes;
|
||||
_stateDirectory = stateDirectory != null ? Environment.ExpandEnvironmentVariables(stateDirectory) : null;
|
||||
}
|
||||
|
||||
public void Start()
|
||||
{
|
||||
LoadWatermark();
|
||||
StartWatcher();
|
||||
|
||||
if (_backfillEnabled && _backfillLookbackMinutes > 0)
|
||||
BackfillRecentEvents();
|
||||
else
|
||||
_logger.Info("MdmRebootPolicy backfill disabled by config");
|
||||
}
|
||||
|
||||
public void Stop()
|
||||
{
|
||||
if (_watcher == null) return;
|
||||
try
|
||||
{
|
||||
_watcher.Enabled = false;
|
||||
_watcher.Dispose();
|
||||
_logger.Info("MdmRebootPolicy watcher stopped");
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.Error("Error stopping MdmRebootPolicy watcher", ex);
|
||||
}
|
||||
finally
|
||||
{
|
||||
_watcher = null;
|
||||
}
|
||||
}
|
||||
|
||||
public void Dispose() => Stop();
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Watcher lifecycle
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
private void StartWatcher()
|
||||
{
|
||||
try
|
||||
{
|
||||
var query = new EventLogQuery(Channel, PathType.LogName, BuildXPath());
|
||||
_watcher = new EventLogWatcher(query);
|
||||
_watcher.EventRecordWritten += OnEventRecordWritten;
|
||||
_watcher.Enabled = true;
|
||||
_logger.Info($"MdmRebootPolicy watcher started: {Channel} (EventID {EventId_PolicyRebootRequired})");
|
||||
}
|
||||
catch (EventLogNotFoundException)
|
||||
{
|
||||
_logger.Warning($"MdmRebootPolicy event log not found: {Channel} (normal on non-MDM-enrolled test environments)");
|
||||
}
|
||||
catch (UnauthorizedAccessException ex)
|
||||
{
|
||||
_logger.Warning($"MdmRebootPolicy watcher access denied for {Channel}: {ex.Message}");
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.Error($"Failed to start MdmRebootPolicy watcher for {Channel}", ex);
|
||||
// MON-D1: surface a dead watcher as one-shot telemetry, not just a local log line.
|
||||
CollectorDegradationReporter.Report(_post, _sessionId, _tenantId,
|
||||
collectorName: "MdmRebootPolicyTracker", reason: $"watcher_arm_failed:{Channel}", ex: ex);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Targeted-EventID XPath filter. Exposed as internal for tests.</summary>
|
||||
internal static string BuildXPath() =>
|
||||
$"*[System[(EventID={EventId_PolicyRebootRequired})]]";
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Event processing
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
private void OnEventRecordWritten(object sender, EventRecordWrittenEventArgs e)
|
||||
{
|
||||
if (e.EventRecord == null) return;
|
||||
try
|
||||
{
|
||||
ProcessRecord(e.EventRecord, isBackfill: false);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.Error("Error processing MdmRebootPolicy event", ex);
|
||||
}
|
||||
}
|
||||
|
||||
private void ProcessRecord(EventRecord record, bool isBackfill)
|
||||
{
|
||||
var recordId = record.RecordId ?? -1;
|
||||
|
||||
// Cheap dedup gate BEFORE the expensive XML/FormatDescription work.
|
||||
if (IsAlreadyProcessed(recordId))
|
||||
return;
|
||||
|
||||
string xml = null;
|
||||
try { xml = record.ToXml(); }
|
||||
catch { /* fall back to description-only parse below */ }
|
||||
|
||||
string description = null;
|
||||
try { description = record.FormatDescription(); }
|
||||
catch { /* some events lack formatting resources */ }
|
||||
|
||||
var rebootUri = ExtractRebootUri(ParseEventData(xml), description);
|
||||
|
||||
ProcessEvent(
|
||||
eventId: record.Id,
|
||||
recordId: recordId,
|
||||
timeCreatedUtc: record.TimeCreated?.ToUniversalTime(),
|
||||
rebootUri: rebootUri,
|
||||
formattedDescription: description,
|
||||
isBackfill: isBackfill);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Core processing extracted to primitive inputs so tests can drive it without synthesizing
|
||||
/// an abstract, Windows-only <see cref="EventRecord"/>. Mirrors the WindowsUpdateTracker
|
||||
/// test-seam pattern.
|
||||
/// </summary>
|
||||
internal void ProcessEvent(
|
||||
int eventId,
|
||||
long recordId,
|
||||
DateTime? timeCreatedUtc,
|
||||
string rebootUri,
|
||||
string formattedDescription,
|
||||
bool isBackfill)
|
||||
{
|
||||
if (!MarkProcessed(recordId))
|
||||
return; // already emitted (cross-restart or duplicate delivery)
|
||||
|
||||
var data = new Dictionary<string, object>
|
||||
{
|
||||
{ "windowsEventId", eventId },
|
||||
{ "backfilled", isBackfill },
|
||||
{ "eventLogChannel", Channel },
|
||||
};
|
||||
if (recordId >= 0) data["recordId"] = recordId;
|
||||
if (timeCreatedUtc.HasValue) data["eventTime"] = timeCreatedUtc.Value.ToString("o");
|
||||
if (!string.IsNullOrEmpty(rebootUri)) data["rebootUri"] = rebootUri;
|
||||
if (!string.IsNullOrEmpty(formattedDescription))
|
||||
{
|
||||
data["description"] = formattedDescription.Length > 1000
|
||||
? formattedDescription.Substring(0, 1000) + "…"
|
||||
: formattedDescription;
|
||||
}
|
||||
|
||||
var flag = SafeProbeRebootRequiredFlag();
|
||||
if (flag.HasValue) data["omadmRebootRequiredFlag"] = flag.Value;
|
||||
|
||||
_post.Emit(new EnrollmentEvent
|
||||
{
|
||||
SessionId = _sessionId,
|
||||
TenantId = _tenantId,
|
||||
// The record's own time, not UtcNow: backfilled pre-agent (and post-reboot re-read)
|
||||
// records must land on the timeline at when the policy actually demanded the reboot.
|
||||
Timestamp = timeCreatedUtc ?? DateTime.UtcNow,
|
||||
EventType = Constants.EventTypes.MdmPolicyRebootRequired,
|
||||
Severity = EventSeverity.Warning,
|
||||
Source = "MdmRebootPolicyWatcher",
|
||||
Phase = EnrollmentPhase.Unknown,
|
||||
Message = !string.IsNullOrEmpty(rebootUri)
|
||||
? $"MDM policy requires a reboot during ESP: {rebootUri}"
|
||||
: "MDM policy requires a reboot during ESP (URI not parsed from event — see description)",
|
||||
Data = data,
|
||||
// The reboot this event announces will kill the agent — flush now.
|
||||
ImmediateUpload = true,
|
||||
});
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Extracts the reboot-forcing OMA-DM URI. Preference order: (1) a structured EventData
|
||||
/// value that starts with <c>./</c> (the CSP-URI shape), (2) a permissive regex over the
|
||||
/// formatted description. Returns null when nothing URI-like is found — the caller still
|
||||
/// emits, just without the <c>rebootUri</c> field.
|
||||
/// </summary>
|
||||
internal static string ExtractRebootUri(Dictionary<string, string> eventData, string description)
|
||||
{
|
||||
if (eventData != null)
|
||||
{
|
||||
foreach (var value in eventData.Values)
|
||||
{
|
||||
var trimmed = value?.Trim();
|
||||
if (!string.IsNullOrEmpty(trimmed) && trimmed.StartsWith("./", StringComparison.Ordinal))
|
||||
return trimmed;
|
||||
}
|
||||
}
|
||||
|
||||
if (!string.IsNullOrEmpty(description))
|
||||
{
|
||||
var match = UriPattern.Match(description);
|
||||
if (match.Success)
|
||||
{
|
||||
// The permissive pattern can swallow sentence punctuation trailing the URI
|
||||
// ("...Security."). OMA-DM URI segments never end in punctuation — trim it.
|
||||
var candidate = match.Value.TrimEnd('.', ',', ';', ':', '\'', '"', ')');
|
||||
if (candidate.Length > 2) return candidate;
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Extracts <c><Data Name="...">value</Data></c> pairs from an event's rendered
|
||||
/// XML, keyed case-insensitively. Also collects UNNAMED Data elements under positional keys
|
||||
/// (<c>Data0</c>, <c>Data1</c>, …) — DM-Enterprise events frequently carry positional-only
|
||||
/// payloads. Namespace-agnostic; returns an empty map on null/malformed XML — never throws.
|
||||
/// </summary>
|
||||
internal static Dictionary<string, string> ParseEventData(string xml)
|
||||
{
|
||||
var result = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
|
||||
if (string.IsNullOrEmpty(xml)) return result;
|
||||
|
||||
try
|
||||
{
|
||||
var doc = XDocument.Parse(xml);
|
||||
var positionalIndex = 0;
|
||||
foreach (var el in doc.Descendants().Where(e => e.Name.LocalName == "Data"))
|
||||
{
|
||||
var nameAttr = el.Attribute("Name");
|
||||
var key = nameAttr != null && !string.IsNullOrEmpty(nameAttr.Value)
|
||||
? nameAttr.Value
|
||||
: "Data" + positionalIndex++;
|
||||
if (!result.ContainsKey(key))
|
||||
result[key] = el.Value;
|
||||
}
|
||||
}
|
||||
catch
|
||||
{
|
||||
// Malformed / unexpected XML — best effort, return what we have.
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// OMA-DM RebootRequired flag probe (fail-soft corroboration)
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
private bool? SafeProbeRebootRequiredFlag()
|
||||
{
|
||||
try
|
||||
{
|
||||
return RebootRequiredFlagProbe?.Invoke();
|
||||
}
|
||||
catch
|
||||
{
|
||||
return null; // corroboration only — never let the probe break the emit
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads the pending-reboot flag under <see cref="OmadmSyncMlKeyPath"/>. The exact shape is
|
||||
/// unverified (PMPC research names the key, not the value type), so both a value and a
|
||||
/// subkey named <c>RebootRequired</c> count as "pending". Fail-soft: null = unknown
|
||||
/// (key absent / unreadable), matching the EspSkipConfigurationProbe convention.
|
||||
/// </summary>
|
||||
internal static bool? ReadOmadmRebootRequiredFlag()
|
||||
{
|
||||
try
|
||||
{
|
||||
using (var key = Microsoft.Win32.Registry.LocalMachine.OpenSubKey(OmadmSyncMlKeyPath))
|
||||
{
|
||||
if (key == null) return null;
|
||||
if (key.GetValue("RebootRequired") != null) return true;
|
||||
using (var sub = key.OpenSubKey("RebootRequired"))
|
||||
{
|
||||
return sub != null ? (bool?)true : false;
|
||||
}
|
||||
}
|
||||
}
|
||||
catch
|
||||
{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Backfill
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
private void BackfillRecentEvents()
|
||||
{
|
||||
try
|
||||
{
|
||||
var lookbackMs = (long)_backfillLookbackMinutes * 60 * 1000;
|
||||
var xpath = $"*[System[(EventID={EventId_PolicyRebootRequired}) and TimeCreated[timediff(@SystemTime) <= {lookbackMs}]]]";
|
||||
|
||||
_logger.Info($"MdmRebootPolicy backfill: scanning {Channel} " +
|
||||
$"(lookback={_backfillLookbackMinutes}min, restartWatermark={_restartWatermark})");
|
||||
|
||||
var query = new EventLogQuery(Channel, PathType.LogName, xpath);
|
||||
|
||||
int processed = 0;
|
||||
using (var reader = new EventLogReader(query))
|
||||
{
|
||||
EventRecord record;
|
||||
while ((record = reader.ReadEvent()) != null)
|
||||
{
|
||||
using (record)
|
||||
{
|
||||
ProcessRecord(record, isBackfill: true);
|
||||
processed++;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (processed == 0)
|
||||
_logger.Debug($"MdmRebootPolicy backfill: no EventID {EventId_PolicyRebootRequired} records in last {_backfillLookbackMinutes} minutes");
|
||||
else
|
||||
_logger.Info($"MdmRebootPolicy backfill: scanned {processed} record(s)");
|
||||
}
|
||||
catch (EventLogNotFoundException)
|
||||
{
|
||||
_logger.Warning($"MdmRebootPolicy event log not found during backfill: {Channel} (normal on non-MDM-enrolled test environments)");
|
||||
}
|
||||
catch (UnauthorizedAccessException ex)
|
||||
{
|
||||
_logger.Warning($"MdmRebootPolicy backfill access denied: {ex.Message}");
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.Warning($"MdmRebootPolicy backfill failed: {ex.Message}");
|
||||
}
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Watermark dedup (cross-restart) — WindowsUpdateTracker pattern verbatim
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
private bool IsAlreadyProcessed(long recordId)
|
||||
{
|
||||
if (recordId < 0) return false; // no RecordId → cannot dedup, let it through
|
||||
lock (_watermarkLock)
|
||||
{
|
||||
return recordId <= _restartWatermark || _seenThisRun.Contains(recordId);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Atomically claims <paramref name="recordId"/>. Returns true if this is the first time we
|
||||
/// see it (caller should emit), false if it was already processed — either in a PRIOR run
|
||||
/// (at/below the restart watermark) or already this run (in the seen-set). Records without
|
||||
/// a RecordId (-1) are always emitted, never tracked or persisted.
|
||||
/// </summary>
|
||||
private bool MarkProcessed(long recordId)
|
||||
{
|
||||
if (recordId < 0) return true;
|
||||
|
||||
long toPersist = -1;
|
||||
lock (_watermarkLock)
|
||||
{
|
||||
if (recordId <= _restartWatermark || !_seenThisRun.Add(recordId))
|
||||
return false;
|
||||
|
||||
if (recordId > _maxEmittedRecordId)
|
||||
{
|
||||
_maxEmittedRecordId = recordId;
|
||||
toPersist = _maxEmittedRecordId;
|
||||
}
|
||||
}
|
||||
if (toPersist >= 0) PersistWatermark(toPersist);
|
||||
return true;
|
||||
}
|
||||
|
||||
internal void LoadWatermark()
|
||||
{
|
||||
if (string.IsNullOrEmpty(_stateDirectory)) return;
|
||||
|
||||
var filePath = Path.Combine(_stateDirectory, WatermarkStateFileName);
|
||||
if (!File.Exists(filePath)) return;
|
||||
|
||||
try
|
||||
{
|
||||
var json = File.ReadAllText(filePath);
|
||||
var state = JsonConvert.DeserializeObject<WatermarkState>(json);
|
||||
if (state != null)
|
||||
{
|
||||
lock (_watermarkLock)
|
||||
{
|
||||
_restartWatermark = state.LastRecordId;
|
||||
_maxEmittedRecordId = state.LastRecordId;
|
||||
}
|
||||
_logger.Info($"MdmRebootPolicy watermark loaded: lastRecordId={state.LastRecordId} (persisted {state.PersistedUtc:O})");
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.Warning($"Failed to load MdmRebootPolicy watermark: {ex.Message}");
|
||||
}
|
||||
}
|
||||
|
||||
private void PersistWatermark(long recordId)
|
||||
{
|
||||
if (string.IsNullOrEmpty(_stateDirectory)) return;
|
||||
|
||||
try
|
||||
{
|
||||
Directory.CreateDirectory(_stateDirectory);
|
||||
var filePath = Path.Combine(_stateDirectory, WatermarkStateFileName);
|
||||
var state = new WatermarkState { LastRecordId = recordId, PersistedUtc = DateTime.UtcNow };
|
||||
var json = JsonConvert.SerializeObject(state, Formatting.Indented);
|
||||
var tempPath = filePath + ".tmp";
|
||||
File.WriteAllText(tempPath, json);
|
||||
File.Copy(tempPath, filePath, overwrite: true);
|
||||
try { File.Delete(tempPath); } catch { }
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.Warning($"Failed to persist MdmRebootPolicy watermark: {ex.Message}");
|
||||
}
|
||||
}
|
||||
|
||||
internal sealed class WatermarkState
|
||||
{
|
||||
public long LastRecordId { get; set; }
|
||||
public DateTime PersistedUtc { get; set; }
|
||||
}
|
||||
}
|
||||
}
|
||||
+81
@@ -16,6 +16,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
/// CommercialOOBE_ESPProgress_Page_Exiting — normal ESP exit
|
||||
/// CommercialOOBE_ESPProgress_WhiteGlove_Success — WhiteGlove complete
|
||||
/// CommercialOOBE_ESPProgress_Failure/_Timeout/_Abort/WhiteGlove_Failed — ESP failure
|
||||
/// "RebootCoalescing" (DeviceSetup) — ESP-initiated coalesced reboot
|
||||
/// (MDM policies forced a mid-ESP restart → second sign-in; corroborates the
|
||||
/// MdmRebootPolicyTracker's per-URI 2800 attribution events)
|
||||
///
|
||||
/// Raises <see cref="FinalizingSetupPhaseTriggered"/>, <see cref="WhiteGloveCompleted"/>,
|
||||
/// and <see cref="EspFailureDetected"/>. Cross-notifies the <see cref="HelloTracker"/>
|
||||
@@ -47,6 +50,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
private bool _espExitDetected;
|
||||
private bool _whiteGloveDetected;
|
||||
private bool _helloWizardStartDetected;
|
||||
private bool _rebootCoalescingDetected;
|
||||
private readonly object _stateLock = new object();
|
||||
|
||||
/// <summary>
|
||||
@@ -247,6 +251,27 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
|
||||
_logger.Info("ESP phase exit detected - detected via Shell-Core event 62407");
|
||||
}
|
||||
else if (description.IndexOf("RebootCoalescing", StringComparison.OrdinalIgnoreCase) >= 0)
|
||||
{
|
||||
// ESP-initiated coalesced reboot (end of DeviceSetup): MDM policies forced a
|
||||
// restart, the user will see a second sign-in screen. Checked AFTER the
|
||||
// WhiteGlove/failure/exiting patterns so their semantics stay untouched even
|
||||
// if a description ever carried both tokens. Fire-once per process run — the
|
||||
// reboot restarts the agent, and the fresh run re-observes it via backfill.
|
||||
lock (_stateLock)
|
||||
{
|
||||
if (_rebootCoalescingDetected) return;
|
||||
_rebootCoalescingDetected = true;
|
||||
}
|
||||
|
||||
eventType = Constants.EventTypes.EspRebootCoalescing;
|
||||
severity = EventSeverity.Warning;
|
||||
message = "ESP initiated a coalesced reboot (RebootCoalescing) — device will restart and show a second sign-in";
|
||||
// No FinalizingSetup transition, no C# event raise — pure informational
|
||||
// corroboration for the MdmRebootPolicyTracker's per-URI attribution.
|
||||
|
||||
_logger.Warning("ESP coalesced reboot detected via Shell-Core event 62407 (RebootCoalescing)");
|
||||
}
|
||||
else
|
||||
{
|
||||
return;
|
||||
@@ -421,6 +446,29 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
return;
|
||||
}
|
||||
|
||||
if (description.IndexOf("RebootCoalescing", StringComparison.OrdinalIgnoreCase) >= 0)
|
||||
{
|
||||
// DELIBERATE deviation from the other 62407 backfill branches (which only raise C#
|
||||
// events): this is a pure informational type with no reducer/subscriber — the
|
||||
// emission IS the signal, and the live emit races the very reboot that kills the
|
||||
// process. Re-emitting after the restart is the only reliable delivery path. A
|
||||
// live+backfill cross-process duplicate is possible (no watermark here); accepted:
|
||||
// the analyze rule uses exists-semantics, both copies carry the identical historical
|
||||
// timestamp, and the backfill copy is marked backfilled:true.
|
||||
bool shouldEmit;
|
||||
lock (_stateLock)
|
||||
{
|
||||
shouldEmit = !_rebootCoalescingDetected;
|
||||
_rebootCoalescingDetected = true;
|
||||
}
|
||||
if (shouldEmit)
|
||||
{
|
||||
_logger.Info($"Backfill: ESP coalesced reboot (RebootCoalescing) found in recent Shell-Core logs (originalAt={occurredAtUtc:o})");
|
||||
EmitRebootCoalescing(description, occurredAtUtc, isBackfill: true);
|
||||
}
|
||||
return; // a RebootCoalescing record is not an ESP exit/failure — stop here
|
||||
}
|
||||
|
||||
if (EspExitingPattern.IsMatch(description))
|
||||
{
|
||||
bool shouldNotify = false;
|
||||
@@ -459,6 +507,39 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Emits the <c>esp_reboot_coalescing</c> event with the standard Shell-Core eventData
|
||||
/// shape. Used by the backfill path (the live path flows through the shared emit tail in
|
||||
/// <see cref="ProcessEvent"/>); <paramref name="isBackfill"/> adds the
|
||||
/// <c>backfilled</c> marker that distinguishes an accepted post-restart re-emission.
|
||||
/// </summary>
|
||||
private void EmitRebootCoalescing(string description, DateTime occurredAtUtc, bool isBackfill)
|
||||
{
|
||||
var eventData = new Dictionary<string, object>
|
||||
{
|
||||
{ "windowsEventId", EventId_ShellCore_WebAppEvent },
|
||||
{ "providerName", "" },
|
||||
{ "description", description },
|
||||
{ "eventLogChannel", ShellCoreEventLogChannel },
|
||||
{ "eventTime", occurredAtUtc.ToString("o") },
|
||||
{ "backfilled", isBackfill }
|
||||
};
|
||||
|
||||
_post.Emit(new EnrollmentEvent
|
||||
{
|
||||
SessionId = _sessionId,
|
||||
TenantId = _tenantId,
|
||||
Timestamp = occurredAtUtc,
|
||||
EventType = Constants.EventTypes.EspRebootCoalescing,
|
||||
Severity = EventSeverity.Warning,
|
||||
Source = "EspAndHelloTracker",
|
||||
Phase = EnrollmentPhase.Unknown,
|
||||
Message = "ESP initiated a coalesced reboot (RebootCoalescing) — device will restart and show a second sign-in",
|
||||
Data = eventData,
|
||||
ImmediateUpload = true
|
||||
});
|
||||
}
|
||||
|
||||
// =====================================================================
|
||||
// Pattern helpers
|
||||
// =====================================================================
|
||||
|
||||
@@ -335,6 +335,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
|
||||
osBuildChangedProvider: () => osBuildChangeHost.BuildChanged));
|
||||
}
|
||||
|
||||
// MDM reboot-policy watcher — subscribes to DeviceManagement-Enterprise-Diagnostics-
|
||||
// Provider/Admin EventID 2800 and attributes the "unexpected reboot + second sign-in"
|
||||
// pattern to the device-assigned policy URIs that forced the coalesced reboot.
|
||||
// Agent-native collector (NOT an IAgentAnalyzer); the config flag is an internal
|
||||
// steering lever only.
|
||||
if (collectors.MdmRebootPolicyWatcherEnabled)
|
||||
{
|
||||
hosts.Add(new MdmRebootPolicyWatcherHost(
|
||||
sessionId: sessionId,
|
||||
tenantId: tenantId,
|
||||
logger: logger,
|
||||
ingress: ingress,
|
||||
clock: clock,
|
||||
backfillLookbackMinutes: collectors.MdmRebootPolicyBackfillLookbackMinutes,
|
||||
stateDirectory: _stateDirectory));
|
||||
}
|
||||
|
||||
// ----- Peripheral hosts (event-only; driven by remote-config toggles) --------------
|
||||
|
||||
// V1 parity (PeriodicCollectorManager) — combine Performance + AgentSelfMetrics under
|
||||
|
||||
+68
@@ -0,0 +1,68 @@
|
||||
#nullable enable
|
||||
using System;
|
||||
using System.Threading;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
|
||||
using AutopilotMonitor.DecisionCore.Engine;
|
||||
using AutopilotMonitor.DecisionCore.Signals;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Orchestration
|
||||
{
|
||||
/// <summary>
|
||||
/// Standalone host for the <see cref="MdmRebootPolicyTracker"/> — watches
|
||||
/// <c>Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin</c> (EventID
|
||||
/// 2800) for MDM policies that force a coalesced reboot during ESP DeviceSetup and forwards
|
||||
/// them via the shared single-rail ingress. Mirrors <see cref="WindowsUpdateWatcherHost"/>:
|
||||
/// builds its own <see cref="InformationalEventPost"/> and delegates Start/Stop to the tracker.
|
||||
/// </summary>
|
||||
internal sealed class MdmRebootPolicyWatcherHost : ICollectorHost
|
||||
{
|
||||
public string Name => "MdmRebootPolicyTracker";
|
||||
|
||||
private readonly MdmRebootPolicyTracker _tracker;
|
||||
private readonly AgentLogger _logger;
|
||||
private int _disposed;
|
||||
|
||||
public MdmRebootPolicyWatcherHost(
|
||||
string sessionId,
|
||||
string tenantId,
|
||||
AgentLogger logger,
|
||||
ISignalIngressSink ingress,
|
||||
IClock clock,
|
||||
int backfillLookbackMinutes,
|
||||
string stateDirectory)
|
||||
{
|
||||
if (ingress == null) throw new ArgumentNullException(nameof(ingress));
|
||||
if (clock == null) throw new ArgumentNullException(nameof(clock));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
|
||||
var post = new InformationalEventPost(ingress, clock, logger);
|
||||
_tracker = new MdmRebootPolicyTracker(
|
||||
sessionId: sessionId,
|
||||
tenantId: tenantId,
|
||||
post: post,
|
||||
logger: logger,
|
||||
backfillEnabled: backfillLookbackMinutes > 0,
|
||||
backfillLookbackMinutes: backfillLookbackMinutes,
|
||||
stateDirectory: stateDirectory);
|
||||
}
|
||||
|
||||
public void Start()
|
||||
{
|
||||
_tracker.Start();
|
||||
_logger.Info("MdmRebootPolicyWatcherHost: started.");
|
||||
}
|
||||
|
||||
public void Stop()
|
||||
{
|
||||
try { _tracker.Stop(); }
|
||||
catch (Exception ex) { _logger.Warning($"MdmRebootPolicyWatcherHost: stop failed: {ex.Message}"); }
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
if (Interlocked.Exchange(ref _disposed, 1) == 1) return;
|
||||
Stop();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -270,7 +270,7 @@ namespace AutopilotMonitor.Functions.Functions.Config
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
await response.WriteAsJsonAsync(new AgentConfigResponse
|
||||
{
|
||||
ConfigVersion = 34, // Gather rule phase scoping (ActivePhases/ActiveFromPhase) + EmitMode on_change anti-spam
|
||||
ConfigVersion = 35, // MDM reboot-policy watcher (mdm_policy_reboot_required / esp_reboot_coalescing)
|
||||
UploadIntervalSeconds = 10,
|
||||
SelfDestructOnComplete = tenantConfig.SelfDestructOnComplete ?? true,
|
||||
KeepLogFile = tenantConfig.KeepLogFile ?? false,
|
||||
|
||||
@@ -24,6 +24,8 @@ export const EVENT_TYPES_CATALOG = {
|
||||
'esp_failure_settle_started',
|
||||
'esp_failure_settle_recovered',
|
||||
'esp_exiting',
|
||||
'esp_reboot_coalescing',
|
||||
'mdm_policy_reboot_required',
|
||||
'esp_config_detected',
|
||||
'completion_check',
|
||||
'completion_waiting',
|
||||
|
||||
@@ -405,6 +405,18 @@ namespace AutopilotMonitor.Shared
|
||||
public const string OsBuildChanged = "os_build_changed"; // OS build (CurrentBuild.UBR) differs across an agent restart — an update was installed during enrollment
|
||||
public const string WindowsUpdateChannelCensus = "windows_update_channel_census"; // One-shot EventID histogram of WU Client + UpdateOrchestrator channels when os_build_changed but zero targeted WU events captured
|
||||
|
||||
// MDM-policy-forced coalesced reboot during ESP (MdmRebootPolicyTracker + ShellCoreTracker;
|
||||
// PMPC "unexpected reboot / second sign-in" research). Device-assigned policies whose CSP
|
||||
// URIs match the OS reboot-required catalog force a coalesced reboot at the end of
|
||||
// DeviceSetup — the user sees an unexpected restart and a second sign-in screen before
|
||||
// AccountSetup. 2800 (DeviceManagement-Enterprise-Diagnostics-Provider/Admin) fires once
|
||||
// per reboot-forcing URI; Shell-Core 62407 "RebootCoalescing" records ESP initiating the
|
||||
// reboot. Deliberately DISTINCT from system_reboot_detected: the backend RebootCount
|
||||
// matches that string exactly (EventIngestProcessor.Classification + terminal recount) —
|
||||
// these attribution events must never inflate the reboot count.
|
||||
public const string MdmPolicyRebootRequired = "mdm_policy_reboot_required"; // DM-Enterprise EventID 2800 — Data.rebootUri = CSP URI that demands the reboot (one event per URI)
|
||||
public const string EspRebootCoalescing = "esp_reboot_coalescing"; // Shell-Core 62407 "RebootCoalescing" — ESP-initiated coalesced reboot imminent (second sign-in follows)
|
||||
|
||||
// TEMPORARY: shadow SM rollout verification — remove when CompletionStateMachine is promoted to primary
|
||||
public const string ShadowDiscrepancy = "shadow_discrepancy";
|
||||
|
||||
|
||||
@@ -482,6 +482,28 @@ namespace AutopilotMonitor.Shared.Models
|
||||
/// </summary>
|
||||
public bool WindowsUpdateChannelCensusEnabled { get; set; } = true;
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// MDM reboot-policy watcher (DeviceManagement-Enterprise-Diagnostics-Provider/Admin)
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
/// <summary>
|
||||
/// Master switch for the MDM reboot-policy watcher. When enabled the agent subscribes to
|
||||
/// <c>Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin</c> EventID
|
||||
/// 2800 and emits one <c>mdm_policy_reboot_required</c> event per device-assigned policy
|
||||
/// URI that forces a coalesced reboot during ESP DeviceSetup (the "unexpected reboot +
|
||||
/// second sign-in" pattern). Internal steering lever only — not surfaced in any admin UI;
|
||||
/// disable by pushing config server-side. Default: true.
|
||||
/// </summary>
|
||||
public bool MdmRebootPolicyWatcherEnabled { get; set; } = true;
|
||||
|
||||
/// <summary>
|
||||
/// Lookback window in minutes for the MDM reboot-policy backfill scan on startup. Device
|
||||
/// policies can apply before the agent starts, and after the coalesced reboot the restarted
|
||||
/// agent must re-scan past the reboot gap (the cross-restart watermark keeps re-reads from
|
||||
/// re-emitting). 0 = backfill disabled. Default: 60 minutes.
|
||||
/// </summary>
|
||||
public int MdmRebootPolicyBackfillLookbackMinutes { get; set; } = 60;
|
||||
|
||||
/// <summary>
|
||||
/// Creates default collector configuration
|
||||
/// </summary>
|
||||
|
||||
Reference in New Issue
Block a user