mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-24 04:32:32 +00:00
fix(agent): neutral aggregated 2800 observations; reboot claim gated on observed reboot
Rework after the first real enrollment (session b2e890c1) exposed three
defects in a654c730:
- Shell-Core "RebootCoalescing" match was a false positive by construction:
the token appears in the ROUTINE SubcategoryProcessing_Started bootstrap
marker ("Starting subcategory DeviceSetup.RebootCoalescing...") on EVERY
enrollment. esp_reboot_coalescing removed entirely (branch, constant,
catalog, tests).
- 2800 reboot-required flags fire at every policy sync; only DeviceSetup-time
flags cause the coalesced reboot. The session's flags fired during
AccountSetup - no reboot - yet our Warning events claimed "device will
restart". Raw event is now a neutral Info observation; the reboot CLAIM
moved to ANALYZE-ESP-005 with a REQUIRED system_reboot_detected >= 1
condition (no observed reboot -> advisory silent).
- One event per 2800 record was spam: records are now buffered and debounced
(10 s) into ONE aggregated event per burst (rebootUris[]/uriCount/
firstRebootUri/recordCount/unparsedCount). Watermark is persisted at FLUSH,
not claim - a pre-emit reboot kill loses nothing, the post-restart backfill
re-emits the burst at its historical time.
Verified 2800 description shape recorded in code, tests, and concept doc:
"The following URI has triggered a reboot: (./Device/...)."; extraction
covered by a real-text regression test. Lesson added to tasks/lessons.md
(observation-level events vs outcome-gated claims; substring tokens appear
in framework bookkeeping records).
Tests: Agent.V2.Core 1867/1867, Functions 3099/3099, MCP 400 passed,
solution build 0 errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
+1
-1
@@ -7,6 +7,6 @@ How the on-device agent works:
|
||||
* [Logs, Persistence & Crash Recovery](logs-and-persistence.md) - every persisted file and why it exists; signal log vs spool; snapshot/journal; the replay-based recovery flow and quarantine.
|
||||
* [Hello Wizard Un-Skip](hello-wizard-unskip.md) - why a single "WHfB disabled" policy read never decides completion: the HelloWizardStarted signal (prevention + cure) and the HelloTracker confirmation second read.
|
||||
* [Autopilot ZTD Diagnostics](autopilot-ztd-diagnostics.md) - Windows' own diagnostic surfaces for the profile-download flow: ModernDeployment event IDs (807/815/164/...), the Diagnostics\Autopilot registry key, deployment-service endpoints, and the known-issue error-code map.
|
||||
* [MDM Reboot Coalescing](mdm-reboot-coalescing.md) - attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: 2800 watcher with cross-restart watermark, Shell-Core RebootCoalescing corroboration, ANALYZE-ESP-005 advisory.
|
||||
* [MDM Reboot Coalescing](mdm-reboot-coalescing.md) - attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: aggregated 2800 observations (neutral, flush-time watermark) + ANALYZE-ESP-005 gated on an actually-observed reboot.
|
||||
|
||||
Reading order for newcomers: overview → decision engine → persistence.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: Concept
|
||||
title: MDM Reboot Coalescing — attributing the mid-ESP reboot and second sign-in to policy URIs
|
||||
description: How the agent detects device-assigned MDM policies that force a coalesced reboot during ESP DeviceSetup (DM-Enterprise EventID 2800 + Shell-Core 62407 RebootCoalescing), and the ANALYZE-ESP-005 advisory that makes the invisible "second sign-in" fixable.
|
||||
description: How the agent observes reboot-required MDM policy flags (DM-Enterprise EventID 2800, aggregated) and how ANALYZE-ESP-005 attributes an actually-observed enrollment reboot to them — including why the raw events stay neutral and why the Shell-Core RebootCoalescing match was removed.
|
||||
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals
|
||||
tags:
|
||||
- agent
|
||||
@@ -17,91 +17,82 @@ timestamp: 2026-07-20T00:00:00+02:00
|
||||
Device-assigned MDM policies whose CSP URIs match the OS reboot-required catalog
|
||||
(`HKLM\SOFTWARE\Microsoft\Provisioning\SyncML\RebootRequiredURIs`) force a **coalesced
|
||||
reboot at the end of ESP DeviceSetup** — the user sees an unexpected restart and a
|
||||
**second sign-in screen** before AccountSetup. This is expected OS behavior, invisible in
|
||||
the Intune console; most admins simply live with it, not knowing that reassigning the
|
||||
profiles to user groups removes the reboot entirely. Source research: Patch My PC,
|
||||
**second sign-in screen** before AccountSetup. Invisible in the Intune console; most
|
||||
admins live with it, not knowing that reassigning the profiles to user groups removes the
|
||||
reboot entirely. Source research: Patch My PC,
|
||||
["Avoid the second sign-in screen after Autopilot"](https://patchmypc.com/blog/autopilot-unexpected-reboot-autopilot-second-login-screen/).
|
||||
|
||||
The session already counts reboots (`system_reboot_detected` → `session.rebootCount`);
|
||||
this feature adds the **cause attribution**.
|
||||
|
||||
# Signal chain
|
||||
# Verified signal facts (session b2e890c1, 2026-07-20)
|
||||
|
||||
1. **EventID 2800**, `Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin`
|
||||
— logged by the SyncML engine once per policy URI that matched the reboot-required
|
||||
catalog, during device-policy application in DeviceSetup.
|
||||
2. **EventID 62407**, `Microsoft-Windows-Shell-Core/Operational`, description containing
|
||||
`RebootCoalescing` — ESP/CloudExperienceHost initiating the coalesced reboot at the end
|
||||
of DeviceSetup.
|
||||
3. The reboot kills the agent; the post-reboot restart re-observes both channels via
|
||||
backfill (the .evtx persists across the reboot).
|
||||
* **EventID 2800**, `Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin`,
|
||||
one record per URI, verified description shape:
|
||||
`The following URI has triggered a reboot: (./Device/Vendor/MSFT/Policy/Config/ServiceControlManager/SvchostProcessMitigation).`
|
||||
Records arrive in sub-second bursts during policy sync.
|
||||
* **Timing semantics (the critical nuance)**: 2800 fires at EVERY policy sync. Only URIs
|
||||
applied **during ESP DeviceSetup** are coalesced into the forced mid-ESP restart. In the
|
||||
verified session the flags fired during AccountSetup (after DeviceSetup completed) —
|
||||
**no reboot occurred**, and `omadmRebootRequiredFlag` read `false`. A detector that
|
||||
claims "reboot incoming" on every 2800 is therefore wrong.
|
||||
* **Shell-Core 62407 "RebootCoalescing" — REMOVED as a signal.** The token appears in the
|
||||
ROUTINE bootstrap marker on every enrollment:
|
||||
`Name: 'CommercialOOBE_BootstrapStatusCategory_SubcategoryProcessing_Started', Value: '{"message":"BootstrapStatus: Starting subcategory DeviceSetup.RebootCoalescing...","errorCode":0}'`
|
||||
— the ESP merely *processes* the RebootCoalescing subcategory (which checks whether a
|
||||
reboot is needed). A substring match on it is a false positive by construction. If a
|
||||
real coalesced-reboot record is ever captured on the E2E VM, a precise pattern can be
|
||||
added then.
|
||||
|
||||
# Detectors
|
||||
# Detector — `MdmRebootPolicyTracker` (`MdmRebootPolicyWatcherHost`)
|
||||
|
||||
## `MdmRebootPolicyTracker` (new collector, `MdmRebootPolicyWatcherHost`)
|
||||
Live `EventLogWatcher` on the DM-Enterprise Admin channel (EventID 2800) + startup
|
||||
backfill (default 60 min lookback). Records are **buffered and debounced (10 s)**, then
|
||||
flushed as **ONE aggregated `mdm_policy_reboot_required` event** per burst:
|
||||
|
||||
Structural clone of `WindowsUpdateTracker`: live `EventLogWatcher` on the DM-Enterprise
|
||||
Admin channel (EventID 2800) + startup backfill (default 60 min lookback) + **cross-restart
|
||||
RecordId watermark** (`mdm-reboot-watermark.json`). The watermark is mandatory here: the
|
||||
coalesced reboot restarts the agent, and without it every reboot-forcing URI would be
|
||||
re-emitted after the very reboot it caused.
|
||||
* `Severity=Info`, neutral message ("Windows flagged N MDM policy URIs as
|
||||
reboot-required: … (+N more)") — the raw event never claims a reboot.
|
||||
* `Data`: `rebootUris` (distinct, sorted, capped 20), `uriCount` (uncapped distinct),
|
||||
`firstRebootUri` (record-time order, drives rule interpolation), `recordCount`,
|
||||
`unparsedCount`/`sampleDescription` (extraction diagnostics), `backfilled` (all-backfill),
|
||||
fail-soft `omadmRebootRequiredFlag` probe of
|
||||
`HKLM\SOFTWARE\Microsoft\Provisioning\OMADM\SyncML\RebootRequired` (null = unknown).
|
||||
* `Timestamp` = earliest record time (backfilled bursts land at their historical instant);
|
||||
`ImmediateUpload=true` (one event per burst is cheap, and a DeviceSetup burst precedes a
|
||||
reboot that kills the agent).
|
||||
* **Watermark is persisted at FLUSH, not at claim** (`mdm-reboot-watermark.json`): if the
|
||||
coalesced reboot kills the process before the debounced emit, nothing is marked — the
|
||||
post-restart backfill re-reads and emits the burst. A flushed burst is never repeated.
|
||||
|
||||
Per 2800 record it emits one `mdm_policy_reboot_required` event (Warning,
|
||||
`ImmediateUpload=true` — the announced reboot will kill the agent, flush now) with
|
||||
`Data.rebootUri` (the CSP URI), `recordId`, `backfilled`, `description` (truncated 1000),
|
||||
and a fail-soft `omadmRebootRequiredFlag` corroboration probe of
|
||||
`HKLM\SOFTWARE\Microsoft\Provisioning\OMADM\SyncML\RebootRequired` (null = unknown, never
|
||||
blocks the emit). Per-URI emission (not aggregated) is deliberate: an aggregate would need
|
||||
a flush point that races the reboot kill.
|
||||
URI extraction: EventData values starting with `./` preferred, permissive regex over the
|
||||
description as fallback (trailing punctuation trimmed — the verified text wraps the URI in
|
||||
parens and ends with a period). Unparsed records still count into the aggregate.
|
||||
|
||||
**Tolerant matching**: the exact 2800 message text is unverified — emission is gated on
|
||||
the EventID only. URI extraction prefers structured EventData values starting with `./`
|
||||
and falls back to a permissive regex over the description (trailing punctuation trimmed).
|
||||
Events with an unparseable URI are still emitted (timeline-visible, rule stays silent, and
|
||||
the captured description tells us how to fix the extraction). Exact texts are validated
|
||||
post-deploy on the E2E VM and recorded here.
|
||||
|
||||
Config (internal steering only, no admin UI — this is an agent-native collector, NOT an
|
||||
Config (internal steering only, no admin UI — agent-native collector, NOT an
|
||||
`IAgentAnalyzer`): `Collectors.MdmRebootPolicyWatcherEnabled` (default true),
|
||||
`Collectors.MdmRebootPolicyBackfillLookbackMinutes` (default 60). ConfigVersion 35.
|
||||
|
||||
## `ShellCoreTracker` — `RebootCoalescing` branch
|
||||
|
||||
The already-watched 62407 stream gets a fourth branch (checked AFTER
|
||||
WhiteGlove/failure/exiting so their semantics stay untouched): description containing
|
||||
`RebootCoalescing` → one `esp_reboot_coalescing` event (Warning, ImmediateUpload), pure
|
||||
informational — no `FinalizingSetup` transition, no C# event raise, no DecisionCore
|
||||
involvement.
|
||||
|
||||
**Deliberate deviation**: unlike the other 62407 branches, the backfill path DOES emit
|
||||
this event. The live emit races the very reboot that kills the process; the post-restart
|
||||
5-min backfill re-emission is the only reliable delivery. A live+backfill cross-process
|
||||
duplicate is possible and accepted (fire-once bool resets per process; both copies carry
|
||||
the identical historical timestamp; the backfill copy is marked `backfilled:true`; the
|
||||
analyze rule uses exists-semantics). The shared 5-min lookback is NOT enlarged — the
|
||||
rule's required condition is the watermarked 2800 event; coalescing is optional
|
||||
corroboration only.
|
||||
|
||||
# Advisory — ANALYZE-ESP-005
|
||||
|
||||
"MDM Policy Forced a Mid-ESP Reboot (Second Sign-In)", severity `warning`,
|
||||
`markSessionAsFailedDefault: false`. Required: `mdm_policy_reboot_required.rebootUri`
|
||||
exists (first URI interpolates as `{{rebootUri}}`). Optional corroboration:
|
||||
`esp_reboot_coalescing` observed (+20 confidence) and ≥2 reboot URIs (+10) on base 70 /
|
||||
threshold 70. Remediation: map each URI to its Intune profile and reassign device → user
|
||||
groups (or exclude the Autopilot device group), or accept + communicate the restart.
|
||||
Because `{{token}}` interpolation is scalar-only, the explanation points at the timeline
|
||||
for the complete URI list — each URI is its own event.
|
||||
The reboot **claim** lives here, and only here. Required conditions (BOTH):
|
||||
|
||||
1. `mdm_policy_reboot_required.firstRebootUri` exists (→ `{{firstRebootUri}}` interpolation)
|
||||
2. `system_reboot_detected` count ≥ 1 — **an actually-observed reboot**
|
||||
|
||||
Policy flags alone (e.g. AccountSetup user-policy sync) stay advisory-silent. Optional
|
||||
confidence: `uriCount ≥ 2` (+10), `omadmRebootRequiredFlag == true` (+15) on base 70 /
|
||||
threshold 70. Remediation: map URIs to Intune profiles, reassign device → user groups (or
|
||||
exclude the Autopilot device group), or accept + communicate the restart.
|
||||
|
||||
# Non-goals / invariants
|
||||
|
||||
* The new event types are **deliberately distinct from `system_reboot_detected`** — the
|
||||
backend RebootCount matches that string exactly in two places
|
||||
(`EventIngestProcessor.Classification` + the terminal recount); attribution events must
|
||||
never inflate the reboot count.
|
||||
* Event type **deliberately distinct from `system_reboot_detected`** — the backend
|
||||
RebootCount matches that string exactly in two places
|
||||
(`EventIngestProcessor.Classification` + terminal recount); attribution must never
|
||||
inflate the reboot count.
|
||||
* No session-level field / SessionInfoCard change in this iteration (follow-up candidate).
|
||||
* The `RebootRequiredURIs` catalog itself is not read (format unverified); only the 2800
|
||||
events (which carry the actually-triggered URIs) are consumed.
|
||||
* The `RebootRequiredURIs` catalog itself is not read; only the 2800 events (which carry
|
||||
the actually-triggered URIs) are consumed.
|
||||
|
||||
# Citations
|
||||
|
||||
|
||||
+1
-1
@@ -18,7 +18,7 @@ https://docs.autopilotmonitor.com (separate repository).
|
||||
* [V2 Agent](agent/index.md) - How the on-device agent works: [runtime overview](agent/overview.md), [decision engine](agent/decision-engine.md) (reducer, signals, completion arms), [logs & persistence](agent/logs-and-persistence.md) (signal log, snapshot, crash recovery), [Hello wizard un-skip](agent/hello-wizard-unskip.md) (why a single "WHfB disabled" read never decides completion), [Autopilot ZTD diagnostics](agent/autopilot-ztd-diagnostics.md) (event IDs, diagnostic registry, endpoints, error-code map).
|
||||
* [Build Counter Blob](agent/build-counter-blob.md) - How agent build numbers are minted: shared counter blob with ETag-CAS so local and CI builds never collide; CI additionally attests build provenance (Sigstore).
|
||||
* [Agent Endpoint Migration](agent/endpoint-migration.md) - How the backend re-homes agents to a new API base URL via the config channel (backend move or per-tenant region move): live-fetch-only, allowlist-validated on both sides, one hop, kill wins.
|
||||
* [MDM Reboot Coalescing](agent/mdm-reboot-coalescing.md) - Attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: DM-Enterprise 2800 watcher with cross-restart watermark, Shell-Core RebootCoalescing corroboration, ANALYZE-ESP-005 advisory.
|
||||
* [MDM Reboot Coalescing](agent/mdm-reboot-coalescing.md) - Attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: aggregated DM-Enterprise 2800 observations (neutral, flush-time watermark) + the ANALYZE-ESP-005 advisory gated on an actually-observed reboot.
|
||||
|
||||
# Rules
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
## 2026-07-20
|
||||
|
||||
* **Update**: Reworked `agent/mdm-reboot-coalescing.md` after the first real enrollment (session b2e890c1): 2800 records are now **aggregated per burst** into ONE neutral Info event (`rebootUris[]`/`uriCount`/`firstRebootUri`; watermark persisted at flush so a pre-emit reboot kill loses nothing); the raw event no longer claims a reboot — ANALYZE-ESP-005 is gated on an actually-observed `system_reboot_detected`. **Removed** the Shell-Core `esp_reboot_coalescing` event entirely: the "RebootCoalescing" token appears in the routine `SubcategoryProcessing_Started` bootstrap marker on every enrollment (false positive by construction). Verified 2800 description shape recorded.
|
||||
* **Creation**: Added `agent/mdm-reboot-coalescing.md` — the agent now attributes the "unexpected mid-ESP reboot + second sign-in" pattern (PMPC research) to the device-assigned policy URIs that forced it: new `MdmRebootPolicyTracker` (DM-Enterprise EventID 2800, per-URI `mdm_policy_reboot_required` events, cross-restart RecordId watermark), a `RebootCoalescing` branch in `ShellCoreTracker` (`esp_reboot_coalescing`, backfill-emitting by design), and the `ANALYZE-ESP-005` advisory (reassign device → user groups). ConfigVersion 35; event types deliberately distinct from `system_reboot_detected` so RebootCount stays uninflated.
|
||||
|
||||
* **Creation**: Added `backend/business-timestamps.md` (new `backend/` section) — the 2026-07-18 migration reset every row's system Timestamp, exposing that AuditLogs/OpsEvents/Events had (unknowingly) displayed and filtered on it all along; documents the `OccurredUtc` column, the RowKey decoders and RowKey-range date/retention clauses in `BusinessTimestamp`, the GUID-guard trap, the audit/ops backfill endpoint, and the "system Timestamp is never authoritative" migration rule.
|
||||
|
||||
@@ -2,10 +2,10 @@
|
||||
"$schema": "../schema/analyze-rule.schema.json",
|
||||
"ruleId": "ANALYZE-ESP-005",
|
||||
"title": "MDM Policy Forced a Mid-ESP Reboot (Second Sign-In)",
|
||||
"description": "Detects the 'unexpected reboot + second sign-in screen' pattern: one or more device-assigned MDM policies matched the OS reboot-required catalog during ESP DeviceSetup, and Windows coalesced them into a restart at the end of DeviceSetup — forcing the user through a second sign-in before AccountSetup. Attributes the reboot to the specific CSP policy URIs so the profiles can be reassigned.",
|
||||
"description": "Attributes an observed enrollment reboot to device-assigned MDM policies that Windows flagged as reboot-required. Fires ONLY when both facts are present in the session: at least one actually-observed reboot (system_reboot_detected) AND reboot-required policy flags (mdm_policy_reboot_required). Policy flags alone — e.g. from a user-policy sync during AccountSetup — force no reboot and stay advisory-silent.",
|
||||
"severity": "warning",
|
||||
"category": "esp",
|
||||
"version": "1.0.0",
|
||||
"version": "1.1.0",
|
||||
"author": "Autopilot Monitor",
|
||||
"enabled": true,
|
||||
"trigger": "single",
|
||||
@@ -15,37 +15,48 @@
|
||||
"signal": "reboot_forcing_policy_uri",
|
||||
"source": "event_data",
|
||||
"eventType": "mdm_policy_reboot_required",
|
||||
"dataField": "rebootUri",
|
||||
"dataField": "firstRebootUri",
|
||||
"operator": "exists",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"signal": "reboot_uri_count",
|
||||
"signal": "reboot_observed",
|
||||
"source": "event_count",
|
||||
"eventType": "mdm_policy_reboot_required",
|
||||
"eventType": "system_reboot_detected",
|
||||
"operator": "count_gte",
|
||||
"value": "1",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"signal": "multiple_reboot_uris",
|
||||
"source": "event_data",
|
||||
"eventType": "mdm_policy_reboot_required",
|
||||
"dataField": "uriCount",
|
||||
"operator": "gte",
|
||||
"value": "2",
|
||||
"required": false
|
||||
},
|
||||
{
|
||||
"signal": "esp_coalesced_reboot_observed",
|
||||
"source": "event_type",
|
||||
"eventType": "esp_reboot_coalescing",
|
||||
"operator": "exists",
|
||||
"signal": "omadm_reboot_flag_set",
|
||||
"source": "event_data",
|
||||
"eventType": "mdm_policy_reboot_required",
|
||||
"dataField": "omadmRebootRequiredFlag",
|
||||
"operator": "equals",
|
||||
"value": "true",
|
||||
"required": false
|
||||
}
|
||||
],
|
||||
"confidenceFactors": [
|
||||
{ "signal": "esp_coalesced_reboot_observed", "condition": "exists", "weight": 20 },
|
||||
{ "signal": "mdm_policy_reboot_required", "condition": "count >= 2", "weight": 10 }
|
||||
{ "signal": "multiple_reboot_uris", "condition": "exists", "weight": 10 },
|
||||
{ "signal": "omadm_reboot_flag_set", "condition": "exists", "weight": 15 }
|
||||
],
|
||||
"confidenceThreshold": 70,
|
||||
"explanation": "During ESP DeviceSetup, a device-assigned MDM policy matched the OS reboot-required catalog and Windows scheduled a **coalesced reboot** at the end of DeviceSetup — the user saw an unexpected restart and a **second sign-in screen** before AccountSetup continued.\n\nFirst reboot-forcing policy URI: `{{rebootUri}}`\n\nWhen several policies force a reboot they are coalesced into one restart; **each** reboot-forcing policy appears as its own `mdm_policy_reboot_required` event on the session timeline — check the timeline for the complete URI list.\n\nThis is expected OS behavior, not a failure — but it degrades the enrollment experience, confuses users, and is invisible in the Intune console.",
|
||||
"explanation": "This session **rebooted during enrollment**, and Windows flagged one or more device-assigned MDM policies as **reboot-required** — the classic cause of the coalesced mid-ESP restart and the unexpected **second sign-in screen** before AccountSetup.\n\nFirst reboot-forcing policy URI: `{{firstRebootUri}}`\n\nThe complete URI list is in the `mdm_policy_reboot_required` timeline event (`rebootUris`). Policies applied DURING ESP DeviceSetup are coalesced into one forced restart at the end of DeviceSetup; the same flags raised later (e.g. user-policy sync in AccountSetup) have no immediate effect — this advisory only fires when a reboot was actually observed.\n\nThis is expected OS behavior, not a failure — but it degrades the enrollment experience, confuses users, and is invisible in the Intune console.",
|
||||
"remediation": [
|
||||
{
|
||||
"title": "Reassign the reboot-forcing policies to user groups",
|
||||
"steps": [
|
||||
"Open each `mdm_policy_reboot_required` timeline event and note the CSP URI (e.g. `./Device/Vendor/MSFT/...`)",
|
||||
"Open the `mdm_policy_reboot_required` timeline event and note the CSP URIs in `rebootUris` (e.g. `./Device/Vendor/MSFT/...`)",
|
||||
"Map each URI to its Intune profile (Settings Catalog / configuration profile) — the URI path names the CSP area",
|
||||
"Reassign those profiles from device groups to user groups so they apply during/after AccountSetup instead of DeviceSetup",
|
||||
"Alternatively, exclude the Autopilot device group from the profile and deliver the setting post-enrollment"
|
||||
|
||||
+27
-16
@@ -1788,10 +1788,10 @@
|
||||
{
|
||||
"ruleId": "ANALYZE-ESP-005",
|
||||
"title": "MDM Policy Forced a Mid-ESP Reboot (Second Sign-In)",
|
||||
"description": "Detects the 'unexpected reboot + second sign-in screen' pattern: one or more device-assigned MDM policies matched the OS reboot-required catalog during ESP DeviceSetup, and Windows coalesced them into a restart at the end of DeviceSetup — forcing the user through a second sign-in before AccountSetup. Attributes the reboot to the specific CSP policy URIs so the profiles can be reassigned.",
|
||||
"description": "Attributes an observed enrollment reboot to device-assigned MDM policies that Windows flagged as reboot-required. Fires ONLY when both facts are present in the session: at least one actually-observed reboot (system_reboot_detected) AND reboot-required policy flags (mdm_policy_reboot_required). Policy flags alone — e.g. from a user-policy sync during AccountSetup — force no reboot and stay advisory-silent.",
|
||||
"severity": "warning",
|
||||
"category": "esp",
|
||||
"version": "1.0.0",
|
||||
"version": "1.1.0",
|
||||
"author": "Autopilot Monitor",
|
||||
"enabled": true,
|
||||
"trigger": "single",
|
||||
@@ -1801,45 +1801,56 @@
|
||||
"signal": "reboot_forcing_policy_uri",
|
||||
"source": "event_data",
|
||||
"eventType": "mdm_policy_reboot_required",
|
||||
"dataField": "rebootUri",
|
||||
"dataField": "firstRebootUri",
|
||||
"operator": "exists",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"signal": "reboot_uri_count",
|
||||
"signal": "reboot_observed",
|
||||
"source": "event_count",
|
||||
"eventType": "mdm_policy_reboot_required",
|
||||
"eventType": "system_reboot_detected",
|
||||
"operator": "count_gte",
|
||||
"value": "1",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"signal": "multiple_reboot_uris",
|
||||
"source": "event_data",
|
||||
"eventType": "mdm_policy_reboot_required",
|
||||
"dataField": "uriCount",
|
||||
"operator": "gte",
|
||||
"value": "2",
|
||||
"required": false
|
||||
},
|
||||
{
|
||||
"signal": "esp_coalesced_reboot_observed",
|
||||
"source": "event_type",
|
||||
"eventType": "esp_reboot_coalescing",
|
||||
"operator": "exists",
|
||||
"signal": "omadm_reboot_flag_set",
|
||||
"source": "event_data",
|
||||
"eventType": "mdm_policy_reboot_required",
|
||||
"dataField": "omadmRebootRequiredFlag",
|
||||
"operator": "equals",
|
||||
"value": "true",
|
||||
"required": false
|
||||
}
|
||||
],
|
||||
"confidenceFactors": [
|
||||
{
|
||||
"signal": "esp_coalesced_reboot_observed",
|
||||
"signal": "multiple_reboot_uris",
|
||||
"condition": "exists",
|
||||
"weight": 20
|
||||
"weight": 10
|
||||
},
|
||||
{
|
||||
"signal": "mdm_policy_reboot_required",
|
||||
"condition": "count >= 2",
|
||||
"weight": 10
|
||||
"signal": "omadm_reboot_flag_set",
|
||||
"condition": "exists",
|
||||
"weight": 15
|
||||
}
|
||||
],
|
||||
"confidenceThreshold": 70,
|
||||
"explanation": "During ESP DeviceSetup, a device-assigned MDM policy matched the OS reboot-required catalog and Windows scheduled a **coalesced reboot** at the end of DeviceSetup — the user saw an unexpected restart and a **second sign-in screen** before AccountSetup continued.\n\nFirst reboot-forcing policy URI: `{{rebootUri}}`\n\nWhen several policies force a reboot they are coalesced into one restart; **each** reboot-forcing policy appears as its own `mdm_policy_reboot_required` event on the session timeline — check the timeline for the complete URI list.\n\nThis is expected OS behavior, not a failure — but it degrades the enrollment experience, confuses users, and is invisible in the Intune console.",
|
||||
"explanation": "This session **rebooted during enrollment**, and Windows flagged one or more device-assigned MDM policies as **reboot-required** — the classic cause of the coalesced mid-ESP restart and the unexpected **second sign-in screen** before AccountSetup.\n\nFirst reboot-forcing policy URI: `{{firstRebootUri}}`\n\nThe complete URI list is in the `mdm_policy_reboot_required` timeline event (`rebootUris`). Policies applied DURING ESP DeviceSetup are coalesced into one forced restart at the end of DeviceSetup; the same flags raised later (e.g. user-policy sync in AccountSetup) have no immediate effect — this advisory only fires when a reboot was actually observed.\n\nThis is expected OS behavior, not a failure — but it degrades the enrollment experience, confuses users, and is invisible in the Intune console.",
|
||||
"remediation": [
|
||||
{
|
||||
"title": "Reassign the reboot-forcing policies to user groups",
|
||||
"steps": [
|
||||
"Open each `mdm_policy_reboot_required` timeline event and note the CSP URI (e.g. `./Device/Vendor/MSFT/...`)",
|
||||
"Open the `mdm_policy_reboot_required` timeline event and note the CSP URIs in `rebootUris` (e.g. `./Device/Vendor/MSFT/...`)",
|
||||
"Map each URI to its Intune profile (Settings Catalog / configuration profile) — the URI path names the CSP area",
|
||||
"Reassign those profiles from device groups to user groups so they apply during/after AccountSetup instead of DeviceSetup",
|
||||
"Alternatively, exclude the Autopilot device group from the profile and deliver the setting post-enrollment"
|
||||
|
||||
-195
@@ -1,195 +0,0 @@
|
||||
#nullable enable
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
|
||||
using AutopilotMonitor.Agent.V2.Core.Orchestration;
|
||||
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
|
||||
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
|
||||
using AutopilotMonitor.DecisionCore.Engine;
|
||||
using AutopilotMonitor.DecisionCore.Signals;
|
||||
using AutopilotMonitor.Shared;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
|
||||
{
|
||||
/// <summary>
|
||||
/// Shell-Core 62407 "RebootCoalescing" branch — ESP-initiated coalesced reboot (MDM policies
|
||||
/// forced a mid-ESP restart → second sign-in). Pure informational corroboration for the
|
||||
/// MdmRebootPolicyTracker's per-URI 2800 attribution: no FinalizingSetup transition, no C#
|
||||
/// event raise, and — as a DELIBERATE deviation from the other 62407 backfill branches — the
|
||||
/// backfill path DOES emit (the live emit races the very reboot that kills the process).
|
||||
/// </summary>
|
||||
public sealed class ShellCoreTrackerRebootCoalescingTests
|
||||
{
|
||||
private static readonly DateTime ClockNow = new DateTime(2026, 7, 20, 10, 5, 0, DateTimeKind.Utc);
|
||||
private const string CoalescingDescription =
|
||||
"CloudExperienceHost Web App Event 2. Name: 'CommercialOOBE_DeviceSetup_RebootCoalescing', Value: '...'.";
|
||||
|
||||
private static (ShellCoreTracker tracker, FakeSignalIngressSink sink) MakeTracker(TempDirectory tmp)
|
||||
{
|
||||
var sink = new FakeSignalIngressSink();
|
||||
var logger = new AgentLogger(tmp.Path, AgentLogLevel.Info);
|
||||
var tracker = new ShellCoreTracker(
|
||||
sessionId: "S1",
|
||||
tenantId: "T1",
|
||||
post: new InformationalEventPost(sink, new VirtualClock(ClockNow)),
|
||||
logger: logger,
|
||||
helloTracker: null);
|
||||
return (tracker, sink);
|
||||
}
|
||||
|
||||
private static IReadOnlyList<FakeSignalIngressSink.PostedSignal> ByType(FakeSignalIngressSink sink, string eventType) =>
|
||||
sink.Posted.Where(p =>
|
||||
p.Kind == DecisionSignalKind.InformationalEvent
|
||||
&& p.Payload != null
|
||||
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
|
||||
&& et == eventType).ToList();
|
||||
|
||||
private static IReadOnlyDictionary<string, object> Data(FakeSignalIngressSink.PostedSignal s) =>
|
||||
(IReadOnlyDictionary<string, object>)s.TypedPayload!;
|
||||
|
||||
[Fact]
|
||||
public void ProcessEvent_62407_RebootCoalescing_EmitsWarning_ImmediateUpload()
|
||||
{
|
||||
using var tmp = new TempDirectory();
|
||||
var (tracker, sink) = MakeTracker(tmp);
|
||||
using (tracker)
|
||||
{
|
||||
var sourceTime = ClockNow.AddSeconds(-5);
|
||||
tracker.ProcessEvent(
|
||||
eventId: ShellCoreTracker.EventId_ShellCore_WebAppEvent,
|
||||
description: CoalescingDescription,
|
||||
timestamp: sourceTime,
|
||||
providerName: "Microsoft-Windows-Shell-Core",
|
||||
isBackfill: false);
|
||||
|
||||
var s = Assert.Single(ByType(sink, Constants.EventTypes.EspRebootCoalescing));
|
||||
Assert.Equal("Warning", s.Payload![SignalPayloadKeys.Severity]);
|
||||
Assert.Equal("true", s.Payload![SignalPayloadKeys.ImmediateUpload]);
|
||||
Assert.Equal(sourceTime, s.OccurredAtUtc);
|
||||
Assert.Equal(CoalescingDescription, Data(s)["description"]);
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ProcessEvent_62407_RebootCoalescing_FiresOncePerRun()
|
||||
{
|
||||
using var tmp = new TempDirectory();
|
||||
var (tracker, sink) = MakeTracker(tmp);
|
||||
using (tracker)
|
||||
{
|
||||
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
|
||||
CoalescingDescription, ClockNow, "Microsoft-Windows-Shell-Core", isBackfill: false);
|
||||
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
|
||||
CoalescingDescription, ClockNow.AddSeconds(1), "Microsoft-Windows-Shell-Core", isBackfill: false);
|
||||
|
||||
Assert.Single(ByType(sink, Constants.EventTypes.EspRebootCoalescing));
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ProcessEvent_62407_RebootCoalescing_DoesNotTriggerEspSemantics()
|
||||
{
|
||||
// The coalescing record is corroboration only — it must not trip the ESP-exit /
|
||||
// FinalizingSetup / failure machinery.
|
||||
using var tmp = new TempDirectory();
|
||||
var (tracker, sink) = MakeTracker(tmp);
|
||||
using (tracker)
|
||||
{
|
||||
var raised = new List<string>();
|
||||
tracker.FinalizingSetupPhaseTriggered += (_, reason) => raised.Add($"finalizing:{reason}");
|
||||
tracker.EspExited += (_, _) => raised.Add("espExited");
|
||||
tracker.EspFailureDetected += (_, type) => raised.Add($"failure:{type}");
|
||||
tracker.WhiteGloveCompleted += (_, _) => raised.Add("whiteGlove");
|
||||
|
||||
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
|
||||
CoalescingDescription, ClockNow, "Microsoft-Windows-Shell-Core", isBackfill: false);
|
||||
|
||||
Assert.Empty(raised);
|
||||
Assert.False(tracker.IsEspExitedForTest);
|
||||
Assert.Empty(ByType(sink, Constants.EventTypes.EspExiting));
|
||||
Assert.Empty(ByType(sink, Constants.EventTypes.EspFailure));
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ProcessEvent_62407_ExistingBranches_Unaffected()
|
||||
{
|
||||
// Ordering regression guard: WhiteGlove/failure/exiting still classify as before —
|
||||
// the RebootCoalescing check comes AFTER them.
|
||||
using var tmp = new TempDirectory();
|
||||
var (tracker, sink) = MakeTracker(tmp);
|
||||
using (tracker)
|
||||
{
|
||||
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
|
||||
"Name: 'CommercialOOBE_ESPProgress_Page_Exiting'", ClockNow, "p", isBackfill: false);
|
||||
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
|
||||
"Name: 'CommercialOOBE_ESPProgress_WhiteGlove_Success'", ClockNow, "p", isBackfill: false);
|
||||
|
||||
Assert.Single(ByType(sink, Constants.EventTypes.EspExiting));
|
||||
Assert.Single(ByType(sink, Constants.EventTypes.WhiteGloveComplete));
|
||||
Assert.Empty(ByType(sink, Constants.EventTypes.EspRebootCoalescing));
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Backfill_RebootCoalescing_Emits_WithBackfilledTrue_AndHistoricalTimestamp()
|
||||
{
|
||||
// Deliberate deviation from the other 62407 backfill branches: after the coalesced
|
||||
// reboot restarts the agent, the backfill re-emission is the only reliable delivery.
|
||||
using var tmp = new TempDirectory();
|
||||
var (tracker, sink) = MakeTracker(tmp);
|
||||
using (tracker)
|
||||
{
|
||||
var historical = ClockNow.AddMinutes(-3);
|
||||
tracker.HandleBackfillRecord(
|
||||
ShellCoreTracker.EventId_ShellCore_WebAppEvent, CoalescingDescription, historical);
|
||||
|
||||
var s = Assert.Single(ByType(sink, Constants.EventTypes.EspRebootCoalescing));
|
||||
Assert.Equal(historical, s.OccurredAtUtc);
|
||||
Assert.Equal(true, Data(s)["backfilled"]);
|
||||
Assert.Equal("true", s.Payload![SignalPayloadKeys.ImmediateUpload]);
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Backfill_RebootCoalescing_AfterLiveFire_DoesNotDoubleEmit_SameProcess()
|
||||
{
|
||||
// Same-process dedup via the fire-once flag. (Cross-process live+backfill duplication
|
||||
// is possible and accepted — exists-semantics rule, identical historical timestamp.)
|
||||
using var tmp = new TempDirectory();
|
||||
var (tracker, sink) = MakeTracker(tmp);
|
||||
using (tracker)
|
||||
{
|
||||
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
|
||||
CoalescingDescription, ClockNow, "p", isBackfill: false);
|
||||
tracker.HandleBackfillRecord(
|
||||
ShellCoreTracker.EventId_ShellCore_WebAppEvent, CoalescingDescription, ClockNow.AddMinutes(-1));
|
||||
|
||||
Assert.Single(ByType(sink, Constants.EventTypes.EspRebootCoalescing));
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Backfill_RebootCoalescing_DoesNotFallThroughToExitOrFailureHandling()
|
||||
{
|
||||
using var tmp = new TempDirectory();
|
||||
var (tracker, sink) = MakeTracker(tmp);
|
||||
using (tracker)
|
||||
{
|
||||
var raised = new List<string>();
|
||||
tracker.FinalizingSetupPhaseTriggered += (_, reason) => raised.Add($"finalizing:{reason}");
|
||||
tracker.EspExited += (_, _) => raised.Add("espExited");
|
||||
tracker.EspFailureDetected += (_, type) => raised.Add($"failure:{type}");
|
||||
|
||||
tracker.HandleBackfillRecord(
|
||||
ShellCoreTracker.EventId_ShellCore_WebAppEvent, CoalescingDescription, ClockNow.AddMinutes(-2));
|
||||
|
||||
Assert.Empty(raised);
|
||||
Assert.False(tracker.IsEspExitedForTest);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
+176
-75
@@ -15,16 +15,22 @@ using Xunit;
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
|
||||
{
|
||||
/// <summary>
|
||||
/// MdmRebootPolicyTracker — attributes the "unexpected reboot + second sign-in" pattern to the
|
||||
/// device-assigned policy URIs that forced the coalesced reboot (DM-Enterprise EventID 2800).
|
||||
/// Drives the primitive <see cref="MdmRebootPolicyTracker.ProcessEvent"/> test seam (no real
|
||||
/// EventRecord, which is abstract + Windows-only), mirroring the WindowsUpdateTracker pattern.
|
||||
/// MdmRebootPolicyTracker — aggregates DM-Enterprise EventID 2800 bursts into ONE neutral
|
||||
/// <c>mdm_policy_reboot_required</c> event (session b2e890c1 rework: per-record events were
|
||||
/// spam, and the raw event must not CLAIM a reboot — ANALYZE-ESP-005 does, gated on an
|
||||
/// observed system_reboot_detected). Drives the primitive ProcessEvent seam + explicit
|
||||
/// FlushPending (debounce 0 = manual-flush mode; no timers, no SerialThreading needed).
|
||||
/// The registry probe is always overridden — tests never touch the live registry.
|
||||
/// </summary>
|
||||
public sealed class MdmRebootPolicyTrackerTests : IDisposable
|
||||
{
|
||||
private static readonly DateTime At = new DateTime(2026, 7, 20, 9, 30, 0, DateTimeKind.Utc);
|
||||
private const string SampleUri = "./Device/Vendor/MSFT/Policy/Config/DeviceGuard/EnableVirtualizationBasedSecurity";
|
||||
private static readonly DateTime At = new DateTime(2026, 7, 20, 16, 20, 57, DateTimeKind.Utc);
|
||||
private const string UriSvchost = "./Device/Vendor/MSFT/Policy/Config/ServiceControlManager/SvchostProcessMitigation";
|
||||
private const string UriSystemGuard = "./Device/Vendor/MSFT/Policy/Config/DeviceGuard/ConfigureSystemGuardLaunch";
|
||||
|
||||
// Verified real 2800 description shape (session b2e890c1, 2026-07-20).
|
||||
private const string RealDescription =
|
||||
"The following URI has triggered a reboot: (./Device/Vendor/MSFT/Policy/Config/ServiceControlManager/SvchostProcessMitigation).";
|
||||
|
||||
private readonly TempDirectory _tmp = new TempDirectory();
|
||||
private readonly FakeSignalIngressSink _sink;
|
||||
@@ -34,21 +40,28 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
|
||||
public MdmRebootPolicyTrackerTests()
|
||||
{
|
||||
_sink = new FakeSignalIngressSink();
|
||||
_tracker = MakeTracker(stateDirectory: null);
|
||||
}
|
||||
|
||||
public void Dispose() => _tmp.Dispose();
|
||||
|
||||
private MdmRebootPolicyTracker MakeTracker(string? stateDirectory)
|
||||
{
|
||||
var post = new InformationalEventPost(_sink, new VirtualClock(At));
|
||||
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
|
||||
_tracker = new MdmRebootPolicyTracker(
|
||||
var tracker = new MdmRebootPolicyTracker(
|
||||
sessionId: "sess-mdm",
|
||||
tenantId: "tenant-mdm",
|
||||
post: post,
|
||||
logger: logger,
|
||||
backfillEnabled: false,
|
||||
stateDirectory: null); // in-memory dedup for most tests
|
||||
_tracker.RebootRequiredFlagProbe = () => null; // registry untouched by default
|
||||
stateDirectory: stateDirectory,
|
||||
debounceMilliseconds: 0); // manual-flush mode — tests call FlushPending explicitly
|
||||
tracker.RebootRequiredFlagProbe = () => null; // registry untouched by default
|
||||
return tracker;
|
||||
}
|
||||
|
||||
public void Dispose() => _tmp.Dispose();
|
||||
|
||||
private void Emit(string? rebootUri = SampleUri, string? description = null, long? recordId = null, bool isBackfill = false, DateTime? timeCreatedUtc = null)
|
||||
private void Buffer(string? rebootUri, long? recordId = null, bool isBackfill = false, DateTime? timeCreatedUtc = null, string? description = null)
|
||||
{
|
||||
_tracker.ProcessEvent(
|
||||
eventId: MdmRebootPolicyTracker.EventId_PolicyRebootRequired,
|
||||
@@ -70,104 +83,192 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
|
||||
(IReadOnlyDictionary<string, object>)s.TypedPayload!;
|
||||
|
||||
[Fact]
|
||||
public void ProcessEvent_EmitsWarning_WithUri_AndImmediateUpload()
|
||||
public void Burst_FlushesAsOneAggregatedEvent_InfoSeverity_ImmediateUpload()
|
||||
{
|
||||
Emit();
|
||||
Buffer(UriSvchost, timeCreatedUtc: At);
|
||||
Buffer(UriSystemGuard, timeCreatedUtc: At.AddMilliseconds(120));
|
||||
_tracker.FlushPending();
|
||||
|
||||
var s = Assert.Single(Emitted());
|
||||
Assert.Equal("Warning", s.Payload![SignalPayloadKeys.Severity]);
|
||||
Assert.Equal("Info", s.Payload![SignalPayloadKeys.Severity]);
|
||||
Assert.Equal("true", s.Payload![SignalPayloadKeys.ImmediateUpload]);
|
||||
|
||||
var data = Data(s);
|
||||
Assert.Equal(SampleUri, data["rebootUri"]);
|
||||
Assert.Equal(MdmRebootPolicyTracker.EventId_PolicyRebootRequired, data["windowsEventId"]);
|
||||
Assert.Equal(MdmRebootPolicyTracker.Channel, data["eventLogChannel"]);
|
||||
Assert.Equal(2, data["uriCount"]);
|
||||
Assert.Equal(2, data["recordCount"]);
|
||||
Assert.Equal(UriSvchost, data["firstRebootUri"]);
|
||||
var uris = Assert.IsType<List<string>>(data["rebootUris"]);
|
||||
Assert.Equal(2, uris.Count);
|
||||
Assert.Contains(UriSvchost, uris);
|
||||
Assert.Contains(UriSystemGuard, uris);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ProcessEvent_NoUri_StillEmits_WithoutRebootUriField()
|
||||
public void Message_IsNeutralObservation_NoRebootClaim()
|
||||
{
|
||||
// Tolerant contract: an unparsed URI must stay timeline-visible (with the captured
|
||||
// description telling us how to fix the extraction) — the analyze rule keys on
|
||||
// "rebootUri exists" and simply stays silent.
|
||||
Emit(rebootUri: null, description: "Some unexpected 2800 wording");
|
||||
// Session b2e890c1: flags raised during AccountSetup have no effect — the raw event
|
||||
// must state the observation only, never a future consequence.
|
||||
Buffer(UriSvchost);
|
||||
_tracker.FlushPending();
|
||||
|
||||
var s = Assert.Single(Emitted());
|
||||
var message = s.Payload![SignalPayloadKeys.Message];
|
||||
Assert.Contains("flagged", message);
|
||||
Assert.DoesNotContain("will restart", message);
|
||||
Assert.DoesNotContain("second sign-in", message);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FlushPending_Empty_EmitsNothing()
|
||||
{
|
||||
_tracker.FlushPending();
|
||||
Assert.Empty(Emitted());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AggregateTimestamp_IsEarliestRecordTime()
|
||||
{
|
||||
var early = At.AddMinutes(-30);
|
||||
Buffer(UriSystemGuard, timeCreatedUtc: At, isBackfill: true);
|
||||
Buffer(UriSvchost, timeCreatedUtc: early, isBackfill: true);
|
||||
_tracker.FlushPending();
|
||||
|
||||
var s = Assert.Single(Emitted());
|
||||
Assert.Equal(early, s.OccurredAtUtc);
|
||||
Assert.Equal(true, Data(s)["backfilled"]);
|
||||
// firstRebootUri follows record-time order, not buffer order.
|
||||
Assert.Equal(UriSvchost, Data(s)["firstRebootUri"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Backfilled_False_WhenAnyRecordIsLive()
|
||||
{
|
||||
Buffer(UriSvchost, isBackfill: true);
|
||||
Buffer(UriSystemGuard, isBackfill: false);
|
||||
_tracker.FlushPending();
|
||||
|
||||
Assert.Equal(false, Data(Assert.Single(Emitted()))["backfilled"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DuplicateUris_AreDistinctInAggregate()
|
||||
{
|
||||
Buffer(UriSvchost);
|
||||
Buffer(UriSvchost);
|
||||
_tracker.FlushPending();
|
||||
|
||||
var data = Data(Assert.Single(Emitted()));
|
||||
Assert.Equal(1, data["uriCount"]);
|
||||
Assert.Equal(2, data["recordCount"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void UnparsedRecords_CountedWithSampleDescription_StillEmitted()
|
||||
{
|
||||
Buffer(rebootUri: null, description: "Some unexpected 2800 wording");
|
||||
_tracker.FlushPending();
|
||||
|
||||
var s = Assert.Single(Emitted());
|
||||
var data = Data(s);
|
||||
Assert.False(data.ContainsKey("rebootUri"));
|
||||
Assert.Equal("Some unexpected 2800 wording", data["description"]);
|
||||
Assert.False(data.ContainsKey("rebootUris"));
|
||||
Assert.False(data.ContainsKey("firstRebootUri"));
|
||||
Assert.Equal(0, data["uriCount"]);
|
||||
Assert.Equal(1, data["unparsedCount"]);
|
||||
Assert.Equal("Some unexpected 2800 wording", data["sampleDescription"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BackfilledEvent_UsesEventTimeForTimelineTimestamp_AndMarksBackfilled()
|
||||
public void SameRecordId_BufferedTwice_CountsOnce()
|
||||
{
|
||||
var eventTime = At.AddMinutes(-25); // policy applied well before agent start
|
||||
Emit(isBackfill: true, timeCreatedUtc: eventTime);
|
||||
Buffer(UriSvchost, recordId: 42);
|
||||
Buffer(UriSvchost, recordId: 42);
|
||||
_tracker.FlushPending();
|
||||
|
||||
var s = Assert.Single(Emitted());
|
||||
Assert.Equal(eventTime, s.OccurredAtUtc);
|
||||
Assert.Equal(true, Data(s)["backfilled"]);
|
||||
Assert.Equal(1, Data(Assert.Single(Emitted()))["recordCount"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SameRecordId_ProcessedTwice_EmitsOnce()
|
||||
public void RecordAfterFlush_GoesIntoNextAggregate()
|
||||
{
|
||||
Emit(recordId: 42);
|
||||
Emit(recordId: 42);
|
||||
|
||||
Assert.Single(Emitted());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BackfillEventWithLowerRecordId_AfterLiveEvent_IsStillEmitted()
|
||||
{
|
||||
// Same contract as WindowsUpdateTracker: the live watcher is armed BEFORE the backfill
|
||||
// scan, so a live record with a higher RecordId can arrive first — the older,
|
||||
// never-emitted backfill record must not be suppressed by a high-water mark.
|
||||
Emit(recordId: 100);
|
||||
Emit(recordId: 50, isBackfill: true);
|
||||
Buffer(UriSvchost, recordId: 10);
|
||||
_tracker.FlushPending();
|
||||
Buffer(UriSystemGuard, recordId: 11);
|
||||
_tracker.FlushPending();
|
||||
|
||||
Assert.Equal(2, Emitted().Count);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BackfillRecordWithLowerRecordId_AfterLiveRecord_IsStillBuffered()
|
||||
{
|
||||
// Live watcher is armed BEFORE the backfill scan — an older, never-seen backfill
|
||||
// record must not be suppressed by a high-water mark.
|
||||
Buffer(UriSvchost, recordId: 100);
|
||||
Buffer(UriSystemGuard, recordId: 50, isBackfill: true);
|
||||
_tracker.FlushPending();
|
||||
|
||||
Assert.Equal(2, Data(Assert.Single(Emitted()))["recordCount"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NegativeRecordId_IsNeverDeduped()
|
||||
{
|
||||
Emit(recordId: -1);
|
||||
Emit(recordId: -1);
|
||||
Buffer(UriSvchost, recordId: -1);
|
||||
Buffer(UriSvchost, recordId: -1);
|
||||
_tracker.FlushPending();
|
||||
|
||||
Assert.Equal(2, Emitted().Count);
|
||||
Assert.Equal(2, Data(Assert.Single(Emitted()))["recordCount"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Watermark_PersistsAcrossTrackerInstances()
|
||||
public void Watermark_PersistedAtFlush_SkipsReReadAcrossInstances()
|
||||
{
|
||||
// The coalesced reboot restarts the agent and the post-reboot backfill re-reads the
|
||||
// same 2800 records — the persisted watermark must make that re-read a no-op.
|
||||
var post = new InformationalEventPost(_sink, new VirtualClock(At));
|
||||
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
|
||||
|
||||
var first = new MdmRebootPolicyTracker("s", "t", post, logger, backfillEnabled: false, stateDirectory: _tmp.Path);
|
||||
first.RebootRequiredFlagProbe = () => null;
|
||||
// The coalesced reboot restarts the agent; the post-reboot backfill re-reads the same
|
||||
// records — a FLUSHED record must be skipped by the next instance.
|
||||
var first = MakeTracker(stateDirectory: _tmp.Path);
|
||||
first.ProcessEvent(MdmRebootPolicyTracker.EventId_PolicyRebootRequired, recordId: 500,
|
||||
timeCreatedUtc: At, rebootUri: SampleUri, formattedDescription: null, isBackfill: false);
|
||||
timeCreatedUtc: At, rebootUri: UriSvchost, formattedDescription: null, isBackfill: false);
|
||||
first.FlushPending();
|
||||
|
||||
var second = new MdmRebootPolicyTracker("s", "t", post, logger, backfillEnabled: false, stateDirectory: _tmp.Path);
|
||||
second.RebootRequiredFlagProbe = () => null;
|
||||
var second = MakeTracker(stateDirectory: _tmp.Path);
|
||||
second.LoadWatermark();
|
||||
second.ProcessEvent(MdmRebootPolicyTracker.EventId_PolicyRebootRequired, recordId: 500,
|
||||
timeCreatedUtc: At, rebootUri: SampleUri, formattedDescription: null, isBackfill: true);
|
||||
timeCreatedUtc: At, rebootUri: UriSvchost, formattedDescription: null, isBackfill: true);
|
||||
second.FlushPending();
|
||||
|
||||
Assert.Single(Emitted());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Watermark_NotPersisted_WhenBufferedButNeverFlushed()
|
||||
{
|
||||
// Process killed by the very reboot the records announce: nothing was flushed, so the
|
||||
// next instance must re-read and emit the records.
|
||||
var first = MakeTracker(stateDirectory: _tmp.Path);
|
||||
first.ProcessEvent(MdmRebootPolicyTracker.EventId_PolicyRebootRequired, recordId: 500,
|
||||
timeCreatedUtc: At, rebootUri: UriSvchost, formattedDescription: null, isBackfill: false);
|
||||
// no FlushPending — simulated reboot kill
|
||||
|
||||
var second = MakeTracker(stateDirectory: _tmp.Path);
|
||||
second.LoadWatermark();
|
||||
second.ProcessEvent(MdmRebootPolicyTracker.EventId_PolicyRebootRequired, recordId: 500,
|
||||
timeCreatedUtc: At, rebootUri: UriSvchost, formattedDescription: null, isBackfill: true);
|
||||
second.FlushPending();
|
||||
|
||||
var s = Assert.Single(Emitted());
|
||||
Assert.Equal(true, Data(s)["backfilled"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RebootRequiredFlagProbe_True_AddsField_Null_OmitsIt()
|
||||
{
|
||||
_tracker.RebootRequiredFlagProbe = () => true;
|
||||
Emit(recordId: 1);
|
||||
Buffer(UriSvchost, recordId: 1);
|
||||
_tracker.FlushPending();
|
||||
|
||||
_tracker.RebootRequiredFlagProbe = () => null;
|
||||
Emit(recordId: 2);
|
||||
Buffer(UriSystemGuard, recordId: 2);
|
||||
_tracker.FlushPending();
|
||||
|
||||
var emitted = Emitted();
|
||||
Assert.Equal(2, emitted.Count);
|
||||
@@ -179,10 +280,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
|
||||
public void RebootRequiredFlagProbe_Throwing_IsFailSoft()
|
||||
{
|
||||
_tracker.RebootRequiredFlagProbe = () => throw new InvalidOperationException("boom");
|
||||
Emit();
|
||||
Buffer(UriSvchost);
|
||||
_tracker.FlushPending();
|
||||
|
||||
var s = Assert.Single(Emitted());
|
||||
Assert.False(Data(s).ContainsKey("omadmRebootRequiredFlag"));
|
||||
Assert.False(Data(Assert.Single(Emitted())).ContainsKey("omadmRebootRequiredFlag"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -194,20 +295,20 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
|
||||
// ------------------------------------------------------------------ URI extraction
|
||||
|
||||
[Fact]
|
||||
public void ExtractRebootUri_PrefersEventDataValue_OverDescription()
|
||||
public void ExtractRebootUri_FromVerifiedRealDescription()
|
||||
{
|
||||
var eventData = new Dictionary<string, string> { ["Data0"] = " " + SampleUri + " " };
|
||||
var uri = MdmRebootPolicyTracker.ExtractRebootUri(eventData, "unrelated ./Other/Uri text");
|
||||
Assert.Equal(SampleUri, uri);
|
||||
// Real 2800 text wraps the URI in parentheses and ends with a period — both must be
|
||||
// excluded from the extracted URI.
|
||||
var uri = MdmRebootPolicyTracker.ExtractRebootUri(null, RealDescription);
|
||||
Assert.Equal(UriSvchost, uri);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ExtractRebootUri_RegexFallback_FromDescription()
|
||||
public void ExtractRebootUri_PrefersEventDataValue_OverDescription()
|
||||
{
|
||||
var uri = MdmRebootPolicyTracker.ExtractRebootUri(
|
||||
new Dictionary<string, string> { ["Data0"] = "not a uri" },
|
||||
$"The following URI has triggered a reboot: {SampleUri}.");
|
||||
Assert.Equal(SampleUri, uri);
|
||||
var eventData = new Dictionary<string, string> { ["Data0"] = " " + UriSystemGuard + " " };
|
||||
var uri = MdmRebootPolicyTracker.ExtractRebootUri(eventData, RealDescription);
|
||||
Assert.Equal(UriSystemGuard, uri);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
|
||||
+200
-91
@@ -4,6 +4,7 @@ using System.Diagnostics.Eventing.Reader;
|
||||
using System.IO;
|
||||
using System.Linq;
|
||||
using System.Text.RegularExpressions;
|
||||
using System.Threading;
|
||||
using System.Xml.Linq;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using AutopilotMonitor.Agent.V2.Core.Orchestration;
|
||||
@@ -15,31 +16,25 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
{
|
||||
/// <summary>
|
||||
/// Watches <c>Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin</c> for
|
||||
/// EventID 2800 — logged once per device-assigned MDM policy whose CSP URI matches the OS
|
||||
/// reboot-required catalog — and forwards each as a <c>mdm_policy_reboot_required</c> event
|
||||
/// carrying the reboot-forcing URI.
|
||||
/// EventID 2800 — logged once per MDM policy URI that matches the OS reboot-required catalog —
|
||||
/// and emits ONE aggregated <c>mdm_policy_reboot_required</c> event per burst carrying the URI
|
||||
/// list (<c>rebootUris</c>, <c>uriCount</c>, <c>firstRebootUri</c>).
|
||||
/// <para>
|
||||
/// This attributes the "unexpected reboot + second sign-in screen" pattern (PMPC research):
|
||||
/// device-assigned policies applied during ESP DeviceSetup can demand a reboot; Windows
|
||||
/// coalesces them into one restart at the end of DeviceSetup, forcing the user through a second
|
||||
/// sign-in before AccountSetup. Admins are blind to this in the Intune console — the attributed
|
||||
/// URIs make it actionable (reassign the profiles to user groups and the reboot disappears).
|
||||
/// Verified against session b2e890c1 (2026-07-20): the 2800 description reads
|
||||
/// <c>The following URI has triggered a reboot: (./Device/...).</c> and records arrive in
|
||||
/// sub-second bursts during policy sync — hence the debounce aggregation instead of per-record
|
||||
/// events. CRITICAL semantics: only URIs applied DURING ESP DeviceSetup cause the coalesced
|
||||
/// mid-ESP reboot ("second sign-in", PMPC research); later syncs (AccountSetup and beyond)
|
||||
/// merely flag the requirement with no forced reboot. The event is therefore a NEUTRAL
|
||||
/// observation (Info) — the reboot claim lives in ANALYZE-ESP-005, which requires an
|
||||
/// actually-observed <c>system_reboot_detected</c> before attributing anything.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Backfill.</b> Device policies can apply before the agent starts, so a startup backfill
|
||||
/// scans the lookback window. The .evtx persists across the coalesced reboot, so the
|
||||
/// post-reboot agent restart re-reads the same records — which is why the
|
||||
/// <b>cross-restart RecordId watermark</b> (WindowsUpdateTracker pattern) is mandatory here:
|
||||
/// without it every reboot-forcing URI would be re-emitted after the very reboot it caused.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Tolerant matching.</b> The exact 2800 message text is unverified (validated post-deploy on
|
||||
/// the E2E VM): emission is gated on the EventID only, never on message wording. URI extraction
|
||||
/// prefers structured EventData values that look like an OMA-DM URI (<c>./…</c>) and falls back
|
||||
/// to a permissive regex over the formatted description. Events with an unparseable URI are
|
||||
/// still emitted (with the captured description) so the timeline shows them and the captured
|
||||
/// text tells us how to fix the extraction — the analyze rule keys on <c>rebootUri exists</c>
|
||||
/// and simply stays silent for those.
|
||||
/// <b>Backfill + watermark.</b> Device policies can apply before the agent starts, and a
|
||||
/// genuine coalesced reboot kills the agent BEFORE the debounced emit — in both cases the
|
||||
/// post-(re)start backfill re-reads the .evtx and emits the aggregate then (historical
|
||||
/// timestamps). The cross-restart RecordId watermark is persisted only at successful flush, so
|
||||
/// a pre-emit death loses nothing and a completed emit is never repeated.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
internal sealed class MdmRebootPolicyTracker : IDisposable
|
||||
@@ -49,12 +44,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
|
||||
internal const string WatermarkStateFileName = "mdm-reboot-watermark.json";
|
||||
|
||||
/// <summary>Cap for the emitted <c>rebootUris</c> list — <c>uriCount</c> always carries the uncapped distinct total.</summary>
|
||||
internal const int MaxUrisPerEvent = 20;
|
||||
|
||||
/// <summary>Default quiet-period before a burst of 2800 records is flushed as one event.</summary>
|
||||
internal const int DefaultDebounceMilliseconds = 10000;
|
||||
|
||||
/// <summary>Registry key whose <c>RebootRequired</c> value/subkey flags a pending OMA-DM coalesced reboot.</summary>
|
||||
internal const string OmadmSyncMlKeyPath = @"SOFTWARE\Microsoft\Provisioning\OMADM\SyncML";
|
||||
|
||||
// Permissive OMA-DM URI shape: "./Device/Vendor/MSFT/...", "./User/...", "./Vendor/...".
|
||||
// Kept deliberately broad — the exact 2800 text is unverified; post-deploy E2E validates
|
||||
// and tightens this against real records if needed.
|
||||
// Permissive OMA-DM URI shape ("./Device/Vendor/MSFT/...", "./User/..."). The verified 2800
|
||||
// text wraps the URI in parentheses — ')' is deliberately outside the class so the match
|
||||
// stops cleanly; trailing sentence punctuation is trimmed after the match.
|
||||
private static readonly Regex UriPattern = new Regex(
|
||||
@"\./[A-Za-z0-9_\-./\[\]{}%~+]+", RegexOptions.Compiled);
|
||||
|
||||
@@ -65,16 +66,30 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
private readonly bool _backfillEnabled;
|
||||
private readonly int _backfillLookbackMinutes;
|
||||
private readonly string _stateDirectory;
|
||||
private readonly int _debounceMilliseconds;
|
||||
|
||||
private EventLogWatcher _watcher;
|
||||
private Timer _debounceTimer;
|
||||
|
||||
// Dedup — same contract as WindowsUpdateTracker (see its field docs): immutable
|
||||
// cross-restart boundary from disk + intra-run HashSet (NOT a high-water mark, because the
|
||||
// live watcher is armed before the backfill scan and can deliver higher RecordIds first).
|
||||
private sealed class PendingRecord
|
||||
{
|
||||
public long RecordId;
|
||||
public DateTime TimeUtc;
|
||||
public string Uri; // null when extraction failed
|
||||
public string Description; // may be null
|
||||
public bool IsBackfill;
|
||||
}
|
||||
|
||||
// Pending burst buffer + dedup state, all guarded by _stateLock. Same claim contract as
|
||||
// WindowsUpdateTracker (immutable cross-restart boundary + intra-run HashSet), with ONE
|
||||
// deliberate difference: the watermark is persisted at FLUSH time, not at claim time —
|
||||
// records buffered but never flushed (process killed by the very reboot they announce)
|
||||
// must be re-read and emitted by the post-restart backfill.
|
||||
private readonly List<PendingRecord> _pending = new List<PendingRecord>();
|
||||
private long _restartWatermark = -1;
|
||||
private readonly HashSet<long> _seenThisRun = new HashSet<long>();
|
||||
private long _maxEmittedRecordId = -1;
|
||||
private readonly object _watermarkLock = new object();
|
||||
private long _maxPersistedRecordId = -1;
|
||||
private readonly object _stateLock = new object();
|
||||
|
||||
/// <summary>
|
||||
/// Fail-soft corroboration probe for the OMA-DM RebootRequired flag — null = unknown
|
||||
@@ -89,7 +104,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
AgentLogger logger,
|
||||
bool backfillEnabled = true,
|
||||
int backfillLookbackMinutes = 60,
|
||||
string stateDirectory = null)
|
||||
string stateDirectory = null,
|
||||
int debounceMilliseconds = DefaultDebounceMilliseconds)
|
||||
{
|
||||
_sessionId = sessionId ?? throw new ArgumentNullException(nameof(sessionId));
|
||||
_tenantId = tenantId ?? throw new ArgumentNullException(nameof(tenantId));
|
||||
@@ -98,6 +114,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
_backfillEnabled = backfillEnabled;
|
||||
_backfillLookbackMinutes = backfillLookbackMinutes;
|
||||
_stateDirectory = stateDirectory != null ? Environment.ExpandEnvironmentVariables(stateDirectory) : null;
|
||||
_debounceMilliseconds = debounceMilliseconds;
|
||||
}
|
||||
|
||||
public void Start()
|
||||
@@ -106,13 +123,30 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
StartWatcher();
|
||||
|
||||
if (_backfillEnabled && _backfillLookbackMinutes > 0)
|
||||
{
|
||||
BackfillRecentEvents();
|
||||
// The backfill batch is complete — flush immediately instead of waiting out the
|
||||
// debounce (live records that raced in are simply included).
|
||||
FlushPending();
|
||||
}
|
||||
else
|
||||
{
|
||||
_logger.Info("MdmRebootPolicy backfill disabled by config");
|
||||
}
|
||||
}
|
||||
|
||||
public void Stop()
|
||||
{
|
||||
// Flush any un-emitted burst before tearing down (normal agent shutdown path).
|
||||
try { FlushPending(); }
|
||||
catch (Exception ex) { _logger.Warning($"MdmRebootPolicy flush on stop failed: {ex.Message}"); }
|
||||
|
||||
var timer = Interlocked.Exchange(ref _debounceTimer, null);
|
||||
if (timer != null)
|
||||
{
|
||||
try { timer.Dispose(); } catch { }
|
||||
}
|
||||
|
||||
if (_watcher == null) return;
|
||||
try
|
||||
{
|
||||
@@ -168,7 +202,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
$"*[System[(EventID={EventId_PolicyRebootRequired})]]";
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Event processing
|
||||
// Event processing — buffer + debounce
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
private void OnEventRecordWritten(object sender, EventRecordWrittenEventArgs e)
|
||||
@@ -212,9 +246,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Core processing extracted to primitive inputs so tests can drive it without synthesizing
|
||||
/// an abstract, Windows-only <see cref="EventRecord"/>. Mirrors the WindowsUpdateTracker
|
||||
/// test-seam pattern.
|
||||
/// Buffers one 2800 record into the pending burst (primitive-input test seam, mirroring the
|
||||
/// WindowsUpdateTracker pattern). The aggregated event is emitted by <see cref="FlushPending"/>
|
||||
/// — after the debounce quiet-period, at backfill completion, or at <see cref="Stop"/>.
|
||||
/// </summary>
|
||||
internal void ProcessEvent(
|
||||
int eventId,
|
||||
@@ -224,53 +258,155 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
string formattedDescription,
|
||||
bool isBackfill)
|
||||
{
|
||||
if (!MarkProcessed(recordId))
|
||||
return; // already emitted (cross-restart or duplicate delivery)
|
||||
lock (_stateLock)
|
||||
{
|
||||
if (recordId >= 0)
|
||||
{
|
||||
// Already emitted in a prior run, or already buffered/emitted this run.
|
||||
if (recordId <= _restartWatermark || !_seenThisRun.Add(recordId))
|
||||
return;
|
||||
}
|
||||
|
||||
_pending.Add(new PendingRecord
|
||||
{
|
||||
RecordId = recordId,
|
||||
TimeUtc = timeCreatedUtc ?? DateTime.UtcNow,
|
||||
Uri = string.IsNullOrEmpty(rebootUri) ? null : rebootUri,
|
||||
Description = formattedDescription,
|
||||
IsBackfill = isBackfill,
|
||||
});
|
||||
}
|
||||
|
||||
ArmDebounce();
|
||||
}
|
||||
|
||||
private void ArmDebounce()
|
||||
{
|
||||
if (_debounceMilliseconds <= 0) return; // manual-flush mode (tests; backfill/Stop flush explicitly)
|
||||
|
||||
var timer = _debounceTimer;
|
||||
if (timer == null)
|
||||
{
|
||||
var created = new Timer(_ =>
|
||||
{
|
||||
try { FlushPending(); }
|
||||
catch (Exception ex) { _logger.Warning($"MdmRebootPolicy debounce flush failed: {ex.Message}"); }
|
||||
}, null, Timeout.Infinite, Timeout.Infinite);
|
||||
|
||||
timer = Interlocked.CompareExchange(ref _debounceTimer, created, null) ?? created;
|
||||
if (!ReferenceEquals(timer, created))
|
||||
{
|
||||
try { created.Dispose(); } catch { }
|
||||
}
|
||||
}
|
||||
|
||||
try { timer.Change(_debounceMilliseconds, Timeout.Infinite); }
|
||||
catch (ObjectDisposedException) { /* raced Stop() — Stop's flush covers the buffer */ }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Emits the pending burst as ONE aggregated <c>mdm_policy_reboot_required</c> event and
|
||||
/// persists the watermark. No-op when nothing is pending. Internal for tests.
|
||||
/// </summary>
|
||||
internal void FlushPending()
|
||||
{
|
||||
List<PendingRecord> batch;
|
||||
long watermarkToPersist = -1;
|
||||
|
||||
lock (_stateLock)
|
||||
{
|
||||
if (_pending.Count == 0) return;
|
||||
batch = new List<PendingRecord>(_pending);
|
||||
_pending.Clear();
|
||||
|
||||
foreach (var r in batch)
|
||||
{
|
||||
if (r.RecordId > _maxPersistedRecordId)
|
||||
_maxPersistedRecordId = r.RecordId;
|
||||
}
|
||||
watermarkToPersist = _maxPersistedRecordId;
|
||||
}
|
||||
|
||||
if (watermarkToPersist >= 0) PersistWatermark(watermarkToPersist);
|
||||
|
||||
batch.Sort((a, b) => a.TimeUtc.CompareTo(b.TimeUtc));
|
||||
|
||||
var distinctUris = batch
|
||||
.Where(r => r.Uri != null)
|
||||
.Select(r => r.Uri)
|
||||
.Distinct(StringComparer.OrdinalIgnoreCase)
|
||||
.ToList();
|
||||
var firstUri = batch.FirstOrDefault(r => r.Uri != null)?.Uri;
|
||||
var unparsedCount = batch.Count(r => r.Uri == null);
|
||||
var allBackfill = batch.All(r => r.IsBackfill);
|
||||
var earliest = batch[0].TimeUtc;
|
||||
|
||||
var data = new Dictionary<string, object>
|
||||
{
|
||||
{ "windowsEventId", eventId },
|
||||
{ "backfilled", isBackfill },
|
||||
{ "windowsEventId", EventId_PolicyRebootRequired },
|
||||
{ "eventLogChannel", Channel },
|
||||
{ "recordCount", batch.Count },
|
||||
{ "uriCount", distinctUris.Count },
|
||||
{ "backfilled", allBackfill },
|
||||
};
|
||||
if (recordId >= 0) data["recordId"] = recordId;
|
||||
if (timeCreatedUtc.HasValue) data["eventTime"] = timeCreatedUtc.Value.ToString("o");
|
||||
if (!string.IsNullOrEmpty(rebootUri)) data["rebootUri"] = rebootUri;
|
||||
if (!string.IsNullOrEmpty(formattedDescription))
|
||||
if (distinctUris.Count > 0)
|
||||
{
|
||||
data["description"] = formattedDescription.Length > 1000
|
||||
? formattedDescription.Substring(0, 1000) + "…"
|
||||
: formattedDescription;
|
||||
data["rebootUris"] = distinctUris
|
||||
.OrderBy(u => u, StringComparer.OrdinalIgnoreCase)
|
||||
.Take(MaxUrisPerEvent)
|
||||
.ToList();
|
||||
data["firstRebootUri"] = firstUri;
|
||||
}
|
||||
if (unparsedCount > 0)
|
||||
{
|
||||
data["unparsedCount"] = unparsedCount;
|
||||
var sample = batch.FirstOrDefault(r => r.Uri == null && !string.IsNullOrEmpty(r.Description))?.Description;
|
||||
if (sample != null)
|
||||
data["sampleDescription"] = sample.Length > 500 ? sample.Substring(0, 500) + "…" : sample;
|
||||
}
|
||||
|
||||
var flag = SafeProbeRebootRequiredFlag();
|
||||
if (flag.HasValue) data["omadmRebootRequiredFlag"] = flag.Value;
|
||||
|
||||
string message;
|
||||
if (distinctUris.Count == 1)
|
||||
message = $"Windows flagged an MDM policy URI as reboot-required: {firstUri}";
|
||||
else if (distinctUris.Count > 1)
|
||||
message = $"Windows flagged {distinctUris.Count} MDM policy URIs as reboot-required: {firstUri} (+{distinctUris.Count - 1} more)";
|
||||
else
|
||||
message = $"Windows flagged {batch.Count} MDM policy record(s) as reboot-required (URI not parsed — see sampleDescription)";
|
||||
|
||||
_post.Emit(new EnrollmentEvent
|
||||
{
|
||||
SessionId = _sessionId,
|
||||
TenantId = _tenantId,
|
||||
// The record's own time, not UtcNow: backfilled pre-agent (and post-reboot re-read)
|
||||
// records must land on the timeline at when the policy actually demanded the reboot.
|
||||
Timestamp = timeCreatedUtc ?? DateTime.UtcNow,
|
||||
// Earliest record time, not UtcNow: backfilled (and post-reboot re-read) bursts
|
||||
// must land on the timeline at when the policies were actually flagged.
|
||||
Timestamp = earliest,
|
||||
EventType = Constants.EventTypes.MdmPolicyRebootRequired,
|
||||
Severity = EventSeverity.Warning,
|
||||
// Neutral observation — reboot-required flags outside ESP DeviceSetup have no
|
||||
// immediate effect. The reboot CLAIM is ANALYZE-ESP-005's job, gated on an
|
||||
// actually-observed system_reboot_detected.
|
||||
Severity = EventSeverity.Info,
|
||||
Source = "MdmRebootPolicyWatcher",
|
||||
Phase = EnrollmentPhase.Unknown,
|
||||
Message = !string.IsNullOrEmpty(rebootUri)
|
||||
? $"MDM policy requires a reboot during ESP: {rebootUri}"
|
||||
: "MDM policy requires a reboot during ESP (URI not parsed from event — see description)",
|
||||
Message = message,
|
||||
Data = data,
|
||||
// The reboot this event announces will kill the agent — flush now.
|
||||
// Cheap (one event per burst) and still time-critical: a burst DURING DeviceSetup
|
||||
// precedes a coalesced reboot that kills the agent.
|
||||
ImmediateUpload = true,
|
||||
});
|
||||
|
||||
_logger.Info($"MdmRebootPolicy: flushed {batch.Count} record(s) as one event " +
|
||||
$"(uris={distinctUris.Count}, unparsed={unparsedCount}, backfilled={allBackfill})");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Extracts the reboot-forcing OMA-DM URI. Preference order: (1) a structured EventData
|
||||
/// value that starts with <c>./</c> (the CSP-URI shape), (2) a permissive regex over the
|
||||
/// formatted description. Returns null when nothing URI-like is found — the caller still
|
||||
/// emits, just without the <c>rebootUri</c> field.
|
||||
/// value that starts with <c>./</c>, (2) a permissive regex over the formatted description
|
||||
/// (verified shape: <c>The following URI has triggered a reboot: (./Device/...).</c>).
|
||||
/// Returns null when nothing URI-like is found — the record still counts into the
|
||||
/// aggregate (<c>unparsedCount</c> + <c>sampleDescription</c>).
|
||||
/// </summary>
|
||||
internal static string ExtractRebootUri(Dictionary<string, string> eventData, string description)
|
||||
{
|
||||
@@ -289,8 +425,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
var match = UriPattern.Match(description);
|
||||
if (match.Success)
|
||||
{
|
||||
// The permissive pattern can swallow sentence punctuation trailing the URI
|
||||
// ("...Security."). OMA-DM URI segments never end in punctuation — trim it.
|
||||
// The permissive pattern can swallow sentence punctuation trailing the URI.
|
||||
// OMA-DM URI segments never end in punctuation — trim it.
|
||||
var candidate = match.Value.TrimEnd('.', ',', ';', ':', '\'', '"', ')');
|
||||
if (candidate.Length > 2) return candidate;
|
||||
}
|
||||
@@ -349,8 +485,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads the pending-reboot flag under <see cref="OmadmSyncMlKeyPath"/>. The exact shape is
|
||||
/// unverified (PMPC research names the key, not the value type), so both a value and a
|
||||
/// Reads the pending-reboot flag under <see cref="OmadmSyncMlKeyPath"/>. Both a value and a
|
||||
/// subkey named <c>RebootRequired</c> count as "pending". Fail-soft: null = unknown
|
||||
/// (key absent / unreadable), matching the EspSkipConfigurationProbe convention.
|
||||
/// </summary>
|
||||
@@ -424,44 +559,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Watermark dedup (cross-restart) — WindowsUpdateTracker pattern verbatim
|
||||
// Watermark dedup (cross-restart) — persisted at flush, not at claim
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
private bool IsAlreadyProcessed(long recordId)
|
||||
{
|
||||
if (recordId < 0) return false; // no RecordId → cannot dedup, let it through
|
||||
lock (_watermarkLock)
|
||||
lock (_stateLock)
|
||||
{
|
||||
return recordId <= _restartWatermark || _seenThisRun.Contains(recordId);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Atomically claims <paramref name="recordId"/>. Returns true if this is the first time we
|
||||
/// see it (caller should emit), false if it was already processed — either in a PRIOR run
|
||||
/// (at/below the restart watermark) or already this run (in the seen-set). Records without
|
||||
/// a RecordId (-1) are always emitted, never tracked or persisted.
|
||||
/// </summary>
|
||||
private bool MarkProcessed(long recordId)
|
||||
{
|
||||
if (recordId < 0) return true;
|
||||
|
||||
long toPersist = -1;
|
||||
lock (_watermarkLock)
|
||||
{
|
||||
if (recordId <= _restartWatermark || !_seenThisRun.Add(recordId))
|
||||
return false;
|
||||
|
||||
if (recordId > _maxEmittedRecordId)
|
||||
{
|
||||
_maxEmittedRecordId = recordId;
|
||||
toPersist = _maxEmittedRecordId;
|
||||
}
|
||||
}
|
||||
if (toPersist >= 0) PersistWatermark(toPersist);
|
||||
return true;
|
||||
}
|
||||
|
||||
internal void LoadWatermark()
|
||||
{
|
||||
if (string.IsNullOrEmpty(_stateDirectory)) return;
|
||||
@@ -475,10 +584,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
var state = JsonConvert.DeserializeObject<WatermarkState>(json);
|
||||
if (state != null)
|
||||
{
|
||||
lock (_watermarkLock)
|
||||
lock (_stateLock)
|
||||
{
|
||||
_restartWatermark = state.LastRecordId;
|
||||
_maxEmittedRecordId = state.LastRecordId;
|
||||
_maxPersistedRecordId = state.LastRecordId;
|
||||
}
|
||||
_logger.Info($"MdmRebootPolicy watermark loaded: lastRecordId={state.LastRecordId} (persisted {state.PersistedUtc:O})");
|
||||
}
|
||||
|
||||
+6
-81
@@ -16,9 +16,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
/// CommercialOOBE_ESPProgress_Page_Exiting — normal ESP exit
|
||||
/// CommercialOOBE_ESPProgress_WhiteGlove_Success — WhiteGlove complete
|
||||
/// CommercialOOBE_ESPProgress_Failure/_Timeout/_Abort/WhiteGlove_Failed — ESP failure
|
||||
/// "RebootCoalescing" (DeviceSetup) — ESP-initiated coalesced reboot
|
||||
/// (MDM policies forced a mid-ESP restart → second sign-in; corroborates the
|
||||
/// MdmRebootPolicyTracker's per-URI 2800 attribution events)
|
||||
///
|
||||
/// NOTE (session b2e890c1, 2026-07-20): a bare "RebootCoalescing" substring match on 62407 was
|
||||
/// tried and REMOVED — the token appears in the ROUTINE SubcategoryProcessing_Started marker
|
||||
/// ("Starting subcategory DeviceSetup.RebootCoalescing...") on every enrollment, so it proves
|
||||
/// nothing about an actual coalesced reboot. Policy-reboot attribution lives entirely in
|
||||
/// MdmRebootPolicyTracker (EventID 2800) + the ANALYZE-ESP-005 rule gated on an observed reboot.
|
||||
///
|
||||
/// Raises <see cref="FinalizingSetupPhaseTriggered"/>, <see cref="WhiteGloveCompleted"/>,
|
||||
/// and <see cref="EspFailureDetected"/>. Cross-notifies the <see cref="HelloTracker"/>
|
||||
@@ -50,7 +53,6 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
private bool _espExitDetected;
|
||||
private bool _whiteGloveDetected;
|
||||
private bool _helloWizardStartDetected;
|
||||
private bool _rebootCoalescingDetected;
|
||||
private readonly object _stateLock = new object();
|
||||
|
||||
/// <summary>
|
||||
@@ -251,27 +253,6 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
|
||||
_logger.Info("ESP phase exit detected - detected via Shell-Core event 62407");
|
||||
}
|
||||
else if (description.IndexOf("RebootCoalescing", StringComparison.OrdinalIgnoreCase) >= 0)
|
||||
{
|
||||
// ESP-initiated coalesced reboot (end of DeviceSetup): MDM policies forced a
|
||||
// restart, the user will see a second sign-in screen. Checked AFTER the
|
||||
// WhiteGlove/failure/exiting patterns so their semantics stay untouched even
|
||||
// if a description ever carried both tokens. Fire-once per process run — the
|
||||
// reboot restarts the agent, and the fresh run re-observes it via backfill.
|
||||
lock (_stateLock)
|
||||
{
|
||||
if (_rebootCoalescingDetected) return;
|
||||
_rebootCoalescingDetected = true;
|
||||
}
|
||||
|
||||
eventType = Constants.EventTypes.EspRebootCoalescing;
|
||||
severity = EventSeverity.Warning;
|
||||
message = "ESP initiated a coalesced reboot (RebootCoalescing) — device will restart and show a second sign-in";
|
||||
// No FinalizingSetup transition, no C# event raise — pure informational
|
||||
// corroboration for the MdmRebootPolicyTracker's per-URI attribution.
|
||||
|
||||
_logger.Warning("ESP coalesced reboot detected via Shell-Core event 62407 (RebootCoalescing)");
|
||||
}
|
||||
else
|
||||
{
|
||||
return;
|
||||
@@ -446,29 +427,6 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
return;
|
||||
}
|
||||
|
||||
if (description.IndexOf("RebootCoalescing", StringComparison.OrdinalIgnoreCase) >= 0)
|
||||
{
|
||||
// DELIBERATE deviation from the other 62407 backfill branches (which only raise C#
|
||||
// events): this is a pure informational type with no reducer/subscriber — the
|
||||
// emission IS the signal, and the live emit races the very reboot that kills the
|
||||
// process. Re-emitting after the restart is the only reliable delivery path. A
|
||||
// live+backfill cross-process duplicate is possible (no watermark here); accepted:
|
||||
// the analyze rule uses exists-semantics, both copies carry the identical historical
|
||||
// timestamp, and the backfill copy is marked backfilled:true.
|
||||
bool shouldEmit;
|
||||
lock (_stateLock)
|
||||
{
|
||||
shouldEmit = !_rebootCoalescingDetected;
|
||||
_rebootCoalescingDetected = true;
|
||||
}
|
||||
if (shouldEmit)
|
||||
{
|
||||
_logger.Info($"Backfill: ESP coalesced reboot (RebootCoalescing) found in recent Shell-Core logs (originalAt={occurredAtUtc:o})");
|
||||
EmitRebootCoalescing(description, occurredAtUtc, isBackfill: true);
|
||||
}
|
||||
return; // a RebootCoalescing record is not an ESP exit/failure — stop here
|
||||
}
|
||||
|
||||
if (EspExitingPattern.IsMatch(description))
|
||||
{
|
||||
bool shouldNotify = false;
|
||||
@@ -507,39 +465,6 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Emits the <c>esp_reboot_coalescing</c> event with the standard Shell-Core eventData
|
||||
/// shape. Used by the backfill path (the live path flows through the shared emit tail in
|
||||
/// <see cref="ProcessEvent"/>); <paramref name="isBackfill"/> adds the
|
||||
/// <c>backfilled</c> marker that distinguishes an accepted post-restart re-emission.
|
||||
/// </summary>
|
||||
private void EmitRebootCoalescing(string description, DateTime occurredAtUtc, bool isBackfill)
|
||||
{
|
||||
var eventData = new Dictionary<string, object>
|
||||
{
|
||||
{ "windowsEventId", EventId_ShellCore_WebAppEvent },
|
||||
{ "providerName", "" },
|
||||
{ "description", description },
|
||||
{ "eventLogChannel", ShellCoreEventLogChannel },
|
||||
{ "eventTime", occurredAtUtc.ToString("o") },
|
||||
{ "backfilled", isBackfill }
|
||||
};
|
||||
|
||||
_post.Emit(new EnrollmentEvent
|
||||
{
|
||||
SessionId = _sessionId,
|
||||
TenantId = _tenantId,
|
||||
Timestamp = occurredAtUtc,
|
||||
EventType = Constants.EventTypes.EspRebootCoalescing,
|
||||
Severity = EventSeverity.Warning,
|
||||
Source = "EspAndHelloTracker",
|
||||
Phase = EnrollmentPhase.Unknown,
|
||||
Message = "ESP initiated a coalesced reboot (RebootCoalescing) — device will restart and show a second sign-in",
|
||||
Data = eventData,
|
||||
ImmediateUpload = true
|
||||
});
|
||||
}
|
||||
|
||||
// =====================================================================
|
||||
// Pattern helpers
|
||||
// =====================================================================
|
||||
|
||||
@@ -24,7 +24,6 @@ export const EVENT_TYPES_CATALOG = {
|
||||
'esp_failure_settle_started',
|
||||
'esp_failure_settle_recovered',
|
||||
'esp_exiting',
|
||||
'esp_reboot_coalescing',
|
||||
'mdm_policy_reboot_required',
|
||||
'esp_config_detected',
|
||||
'completion_check',
|
||||
|
||||
@@ -405,17 +405,17 @@ namespace AutopilotMonitor.Shared
|
||||
public const string OsBuildChanged = "os_build_changed"; // OS build (CurrentBuild.UBR) differs across an agent restart — an update was installed during enrollment
|
||||
public const string WindowsUpdateChannelCensus = "windows_update_channel_census"; // One-shot EventID histogram of WU Client + UpdateOrchestrator channels when os_build_changed but zero targeted WU events captured
|
||||
|
||||
// MDM-policy-forced coalesced reboot during ESP (MdmRebootPolicyTracker + ShellCoreTracker;
|
||||
// PMPC "unexpected reboot / second sign-in" research). Device-assigned policies whose CSP
|
||||
// URIs match the OS reboot-required catalog force a coalesced reboot at the end of
|
||||
// DeviceSetup — the user sees an unexpected restart and a second sign-in screen before
|
||||
// AccountSetup. 2800 (DeviceManagement-Enterprise-Diagnostics-Provider/Admin) fires once
|
||||
// per reboot-forcing URI; Shell-Core 62407 "RebootCoalescing" records ESP initiating the
|
||||
// reboot. Deliberately DISTINCT from system_reboot_detected: the backend RebootCount
|
||||
// matches that string exactly (EventIngestProcessor.Classification + terminal recount) —
|
||||
// these attribution events must never inflate the reboot count.
|
||||
public const string MdmPolicyRebootRequired = "mdm_policy_reboot_required"; // DM-Enterprise EventID 2800 — Data.rebootUri = CSP URI that demands the reboot (one event per URI)
|
||||
public const string EspRebootCoalescing = "esp_reboot_coalescing"; // Shell-Core 62407 "RebootCoalescing" — ESP-initiated coalesced reboot imminent (second sign-in follows)
|
||||
// MDM reboot-required policy flags (MdmRebootPolicyTracker; PMPC "unexpected reboot /
|
||||
// second sign-in" research). DM-Enterprise EventID 2800 fires once per policy URI that
|
||||
// matches the OS reboot-required catalog ("The following URI has triggered a reboot:
|
||||
// (./Device/...)."; verified session b2e890c1). Only URIs applied DURING ESP DeviceSetup
|
||||
// cause the coalesced mid-ESP reboot — later syncs (AccountSetup+) merely flag, no forced
|
||||
// reboot. The tracker therefore emits ONE aggregated Info event per burst with the URI
|
||||
// list; the reboot CLAIM lives in ANALYZE-ESP-005, gated on an actually-observed
|
||||
// system_reboot_detected. Deliberately DISTINCT from system_reboot_detected: the backend
|
||||
// RebootCount matches that string exactly (EventIngestProcessor.Classification + terminal
|
||||
// recount) — this attribution event must never inflate the reboot count.
|
||||
public const string MdmPolicyRebootRequired = "mdm_policy_reboot_required"; // Aggregated per burst — Data.rebootUris[] + uriCount + firstRebootUri
|
||||
|
||||
// TEMPORARY: shadow SM rollout verification — remove when CompletionStateMachine is promoted to primary
|
||||
public const string ShadowDiscrepancy = "shadow_discrepancy";
|
||||
|
||||
Reference in New Issue
Block a user