Compare commits

...

2 Commits

Author SHA1 Message Date
Oliver Kieselbach 08455cc5f9 OOBE quality update: install events from the System log, servicing steps, a bounded engine phase in Shadow mode (D-310)
Report 3a2207978e4c: a session was failed 30 min after the ESP exit while Windows installed its
OOBE quality update. The update watcher had been blind since July and the engine had no notion
of the phase.

- Windows Update client: 19/20/43/44 come from the System log (provider-filtered XPath), as the
  provider manifest routes them; 25/26/31/41 from Operational via their own config list.
  Updates are classified os/defender/store/other: only Windows/.NET updates become rows, the
  rest is counted into windows_update_activity_summary. Per-channel watermarks persist a record
  only after its row and engine signal were handed to the ingress.
- Servicing: CBS steps from the Setup log (windows_update_servicing), client UpdateAgentLCU.
- Engine: OsUpdateActivity and OsUpdateFacts (event-time ordered) anchor an OOBE update phase
  after the ESP exit. Each state has its own bound: the update runs at most 3 h (then Failed
  oobe_update_not_finished), after it the user has 60 min to sign in (then the engine stops
  waiting without a verdict), after the sign-in one window of grace.
- The phase acts only in OobeUpdatePhaseMode=Active. Shadow, the default, reports what it would
  decide as oobe_update_phase_shadow right before today's verdict. The agent stamps the mode on
  EnrollmentFactsObserved at every start; a later stamp can take Active away, never grant it.
- Agent: a stop without a verdict ends the session like the max-lifetime watchdog
  (agent_shutting_down reason=stopped_waiting); the backend classifies it honestly (verdict
  origin stopwait), never Failed.
- Measurement: oobe_update_page (the update page and its ExpeditedUpdate_* names; a 62407 value
  only as boolean, integer or HRESULT) and oobe_update_state (NDUP and ESP-policy registry
  values), both Debug and bounded per run.
- ANALYZE-DEV-004 1.2.0 counts install and download failures only, never scan failures.
- Event types in Constants, MCP catalog and web; shared manifests regenerated; privacy page
  names Windows updates installed during the enrollment.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-02 15:09:50 +02:00
Oliver Kieselbach 87c5bf018c bump pointer: internal (gather-rule state released, B-hx3 field verify)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 22:26:18 +02:00
70 changed files with 5633 additions and 484 deletions
+16 -3
View File
@@ -2,10 +2,10 @@
"$schema": "../schema/analyze-rule.schema.json",
"ruleId": "ANALYZE-DEV-004",
"title": "Windows Update Failed During Enrollment",
"description": "Detects a Windows quality/cumulative update that FAILED to install during the OOBE / ESP enrollment window (WindowsUpdateClient EventID 20). A cumulative update failing mid-OOBE is a known cause of enrollment errors and is otherwise invisible in the Intune console.",
"description": "Detects a Windows quality/cumulative update that FAILED to install or download during the OOBE / ESP enrollment window (WindowsUpdateClient EventID 20 from the System log, 31 from the Operational channel). A cumulative update failing mid-OOBE is a known cause of enrollment errors and is otherwise invisible in the Intune console.",
"severity": "high",
"category": "device",
"version": "1.1.0",
"version": "1.2.0",
"enabled": true,
"evaluateOn": [
"enrollment_end",
@@ -18,10 +18,14 @@
"conditions": [
{
"signal": "wu_install_failed",
"source": "event_type",
"source": "event_data",
"eventType": "windows_update_failed",
"dataField": "wuEventId",
"operator": "exists",
"value": "",
"filterField": "wuPhase",
"filterOperator": "in",
"filterValue": "install,download",
"required": true
},
{
@@ -31,6 +35,9 @@
"dataField": "updateTitle",
"operator": "exists",
"value": "",
"filterField": "wuPhase",
"filterOperator": "in",
"filterValue": "install,download",
"required": false
},
{
@@ -40,6 +47,9 @@
"dataField": "hresult",
"operator": "exists",
"value": "",
"filterField": "wuPhase",
"filterOperator": "in",
"filterValue": "install,download",
"required": false
},
{
@@ -49,6 +59,9 @@
"dataField": "hresultSymbol",
"operator": "exists",
"value": "",
"filterField": "wuPhase",
"filterOperator": "in",
"filterValue": "install,download",
"required": false
},
{
+16 -3
View File
@@ -1385,10 +1385,10 @@
{
"ruleId": "ANALYZE-DEV-004",
"title": "Windows Update Failed During Enrollment",
"description": "Detects a Windows quality/cumulative update that FAILED to install during the OOBE / ESP enrollment window (WindowsUpdateClient EventID 20). A cumulative update failing mid-OOBE is a known cause of enrollment errors and is otherwise invisible in the Intune console.",
"description": "Detects a Windows quality/cumulative update that FAILED to install or download during the OOBE / ESP enrollment window (WindowsUpdateClient EventID 20 from the System log, 31 from the Operational channel). A cumulative update failing mid-OOBE is a known cause of enrollment errors and is otherwise invisible in the Intune console.",
"severity": "high",
"category": "device",
"version": "1.1.0",
"version": "1.2.0",
"enabled": true,
"evaluateOn": [
"enrollment_end",
@@ -1401,10 +1401,14 @@
"conditions": [
{
"signal": "wu_install_failed",
"source": "event_type",
"source": "event_data",
"eventType": "windows_update_failed",
"dataField": "wuEventId",
"operator": "exists",
"value": "",
"filterField": "wuPhase",
"filterOperator": "in",
"filterValue": "install,download",
"required": true
},
{
@@ -1414,6 +1418,9 @@
"dataField": "updateTitle",
"operator": "exists",
"value": "",
"filterField": "wuPhase",
"filterOperator": "in",
"filterValue": "install,download",
"required": false
},
{
@@ -1423,6 +1430,9 @@
"dataField": "hresult",
"operator": "exists",
"value": "",
"filterField": "wuPhase",
"filterOperator": "in",
"filterValue": "install,download",
"required": false
},
{
@@ -1432,6 +1442,9 @@
"dataField": "hresultSymbol",
"operator": "exists",
"value": "",
"filterField": "wuPhase",
"filterOperator": "in",
"filterValue": "install,download",
"required": false
},
{
@@ -0,0 +1,304 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.Shared;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
{
/// <summary>
/// CloudExperienceHost navigation (D-310): one agent.log line per page start/stop and per
/// update-page event name, and for the OOBE update page itself (<c>OobeNDUP</c>,
/// <c>RebootNDUP</c>) the <c>oobe_update_page</c> telemetry plus the registry state at its live
/// start and stop. Descriptions follow the Shell-Core manifest templates; a 62407 value
/// (URLs with tenant ids) leaves the tracker only as a boolean, an integer or an HRESULT.
/// </summary>
public sealed class ShellCoreTrackerBreadcrumbTests : IDisposable
{
private static readonly DateTime At = new DateTime(2026, 10, 2, 9, 0, 0, DateTimeKind.Utc);
private const string NdupStarted = "CloudExperienceHost Web App Activity started. CXID: 'OobeNDUP'.";
private const string EspPageStarted = "CloudExperienceHost Web App Activity started. CXID: 'MDMProgressAfterAADRefactored'.";
private const string PageStopped = "CloudExperienceHost Web App Activity stopped. Result: 'success'.";
private const string ScanStartedName = "CloudExperienceHost Web App Event 1. Name: 'ExpeditedUpdate_startWUScanStarted'.";
private const string NavigateWithTenantUrl =
"CloudExperienceHost Web App Event 2. Name: 'Navigate', Value: '{\"httpMethod\":\"GET\",\"url\":\"https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/oauth2/authorize\"}'.";
private const string DownloadSucceededWithValue =
"CloudExperienceHost Web App Event 2. Name: 'ExpeditedUpdate_commitExpeditionDownloadInstallAsyncSucceeded', Value: '{\"url\":\"https://login.microsoftonline.com/x\"}'.";
private const string CspGateTrue =
"CloudExperienceHost Web App Event 2. Name: 'ExpeditedUpdate_isNDUPAllowedByCSPSucceeded', Value: 'true'.";
private readonly TempDirectory _tmp = new TempDirectory();
private readonly FakeSignalIngressSink _sink = new FakeSignalIngressSink();
private IReadOnlyList<OobeUpdateRegistrySnapshot.KeyState> _registry = Array.Empty<OobeUpdateRegistrySnapshot.KeyState>();
public void Dispose() => _tmp.Dispose();
private ShellCoreTracker Build() =>
new ShellCoreTracker("S1", "T1",
new InformationalEventPost(_sink, new VirtualClock(At)),
new AgentLogger(_tmp.Path, AgentLogLevel.Info),
helloTracker: null,
oobeUpdateRegistryReader: () => _registry);
private IReadOnlyList<FakeSignalIngressSink.PostedSignal> ByType(string eventType) =>
_sink.Posted.Where(p =>
p.Kind == DecisionSignalKind.InformationalEvent
&& p.Payload != null
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
&& et == eventType).ToList();
private static IReadOnlyDictionary<string, object> Data(FakeSignalIngressSink.PostedSignal posted) =>
(IReadOnlyDictionary<string, object>)posted.TypedPayload!;
private static OobeUpdateRegistrySnapshot.KeyState NdupKey(string line) =>
new OobeUpdateRegistrySnapshot.KeyState("setupOobeNdup", @"SOFTWARE\Microsoft\Windows\CurrentVersion\Setup\OOBE\NDUP",
exists: true, hasValues: true, line: line);
// ------------------------------------------------------------- breadcrumbs ----
[Fact]
public void PageStart_NamesTheCxid()
{
Assert.Equal("CXH page started: OobeNDUP",
ShellCoreTracker.FormatCxhBreadcrumb(ShellCoreTracker.EventId_ShellCore_WebAppStarted, NdupStarted, lastPage: null));
}
[Fact]
public void PageStop_IsAttributedToTheLastStartedPage()
{
// 62405 names no page — it ends the one started last.
Assert.Equal("CXH page stopped: OobeNDUP (result=success)",
ShellCoreTracker.FormatCxhBreadcrumb(ShellCoreTracker.EventId_ShellCore_WebAppStopped, PageStopped, lastPage: "OobeNDUP"));
Assert.Equal("CXH page stopped: ? (result=success)",
ShellCoreTracker.FormatCxhBreadcrumb(ShellCoreTracker.EventId_ShellCore_WebAppStopped, PageStopped, lastPage: null));
}
[Fact]
public void UpdatePageEventNames_AreLogged_FromBothEventIds()
{
Assert.Equal("CXH event: ExpeditedUpdate_startWUScanStarted",
ShellCoreTracker.FormatCxhBreadcrumb(ShellCoreTracker.EventId_ShellCore_WebAppEventName, ScanStartedName, lastPage: null));
Assert.Equal("CXH event: ExpeditedUpdate_commitExpeditionDownloadInstallAsyncSucceeded",
ShellCoreTracker.FormatCxhBreadcrumb(ShellCoreTracker.EventId_ShellCore_WebAppEvent, DownloadSucceededWithValue, lastPage: null));
}
[Fact]
public void UnsafeEventValues_NeverReachTheLog()
{
var crumb = ShellCoreTracker.FormatCxhBreadcrumb(
ShellCoreTracker.EventId_ShellCore_WebAppEvent, DownloadSucceededWithValue, lastPage: null);
Assert.NotNull(crumb);
Assert.DoesNotContain("microsoftonline", crumb);
Assert.DoesNotContain("url", crumb!, StringComparison.OrdinalIgnoreCase);
}
[Fact]
public void SafeEventValues_AreLogged()
{
Assert.Equal("CXH event: ExpeditedUpdate_isNDUPAllowedByCSPSucceeded (value=true)",
ShellCoreTracker.FormatCxhBreadcrumb(ShellCoreTracker.EventId_ShellCore_WebAppEvent, CspGateTrue, lastPage: null));
}
[Fact]
public void OtherEventNames_AreNotLogged()
{
// Ordinary page traffic (Navigate, Visible, ESP progress) stays out of the log.
Assert.Null(ShellCoreTracker.FormatCxhBreadcrumb(
ShellCoreTracker.EventId_ShellCore_WebAppEvent, NavigateWithTenantUrl, lastPage: "OobeNDUP"));
Assert.Null(ShellCoreTracker.FormatCxhBreadcrumb(
ShellCoreTracker.EventId_ShellCore_WebAppEvent, "CommercialOOBE_ESPProgress_Page_Exiting", lastPage: null));
}
[Theory]
[InlineData("")]
[InlineData("CloudExperienceHost Web App Activity started.")]
public void PageStartWithoutCxid_IsNoBreadcrumb(string description)
{
Assert.Null(ShellCoreTracker.FormatCxhBreadcrumb(ShellCoreTracker.EventId_ShellCore_WebAppStarted, description, lastPage: null));
}
[Fact]
public void LongTokens_AreBounded()
{
var longCxid = new string('X', 500);
var crumb = ShellCoreTracker.FormatCxhBreadcrumb(
ShellCoreTracker.EventId_ShellCore_WebAppStarted, $"CloudExperienceHost Web App Activity started. CXID: '{longCxid}'.", lastPage: null);
Assert.NotNull(crumb);
Assert.True(crumb!.Length < 200, crumb);
Assert.EndsWith("…", crumb);
}
[Theory]
[InlineData("OobeNDUP", true)]
[InlineData("RebootNDUP", true)]
[InlineData("EarlyRebootNDUP", true)]
[InlineData("ExpeditedUpdate_startWUScanStarted", true)]
[InlineData("AADHello", false)]
[InlineData("MDMProgressAfterAADRefactored", false)]
[InlineData(null, false)]
public void UpdatePageTokens_AreRecognized(string? token, bool expected)
{
Assert.Equal(expected, ShellCoreTracker.IsUpdatePageToken(token!));
}
// ------------------------------------------------------ update page records ----
[Fact]
public void UpdatePageRecords_ArePageStartsStopsAndNames_OfTheUpdatePageOnly()
{
var start = ShellCoreTracker.ParseUpdatePageRecord(ShellCoreTracker.EventId_ShellCore_WebAppStarted, NdupStarted, lastPage: null);
Assert.Equal(OobeUpdatePageRecord.PageStarted, start!.CxhEvent);
Assert.Equal("OobeNDUP", start.Page);
var stop = ShellCoreTracker.ParseUpdatePageRecord(ShellCoreTracker.EventId_ShellCore_WebAppStopped, PageStopped, lastPage: "RebootNDUP");
Assert.Equal(OobeUpdatePageRecord.PageStopped, stop!.CxhEvent);
Assert.Equal("RebootNDUP", stop.Page);
Assert.Equal("success", stop.Result);
var name = ShellCoreTracker.ParseUpdatePageRecord(ShellCoreTracker.EventId_ShellCore_WebAppEventName, ScanStartedName, lastPage: null);
Assert.Equal(OobeUpdatePageRecord.EventName, name!.CxhEvent);
Assert.Equal("ExpeditedUpdate_startWUScanStarted", name.Name);
Assert.Null(name.Value);
// Other pages and names are no update-page records.
Assert.Null(ShellCoreTracker.ParseUpdatePageRecord(ShellCoreTracker.EventId_ShellCore_WebAppStarted, EspPageStarted, lastPage: null));
Assert.Null(ShellCoreTracker.ParseUpdatePageRecord(ShellCoreTracker.EventId_ShellCore_WebAppStopped, PageStopped, lastPage: "MDMProgressAfterAADRefactored"));
Assert.Null(ShellCoreTracker.ParseUpdatePageRecord(ShellCoreTracker.EventId_ShellCore_WebAppEvent, NavigateWithTenantUrl, lastPage: "OobeNDUP"));
}
[Fact]
public void UpdatePageRecord_CarriesOnlyASafeValue()
{
var unsafeValue = ShellCoreTracker.ParseUpdatePageRecord(ShellCoreTracker.EventId_ShellCore_WebAppEvent, DownloadSucceededWithValue, lastPage: null);
Assert.Null(unsafeValue!.Value);
var gate = ShellCoreTracker.ParseUpdatePageRecord(ShellCoreTracker.EventId_ShellCore_WebAppEvent, CspGateTrue, lastPage: null);
Assert.Equal("true", gate!.Value);
}
[Theory]
[InlineData("true", "true")]
[InlineData("FALSE", "FALSE")]
[InlineData("0", "0")]
[InlineData("-2147024891", "-2147024891")]
[InlineData("0x80240022", "0x80240022")]
[InlineData("12345678901", null)] // more than ten digits
[InlineData("0x123456789", null)] // more than eight hex digits
[InlineData("https://login.microsoftonline.com/x", null)]
[InlineData("{\"result\":0}", null)]
[InlineData("true and more", null)]
[InlineData("", null)]
public void SafeValues_AreBooleansIntegersAndHresults(string value, string? expected)
{
var description = $"CloudExperienceHost Web App Event 2. Name: 'ExpeditedUpdate_x', Value: '{value}'.";
Assert.Equal(expected, ShellCoreTracker.ExtractSafeValue(description));
}
[Fact]
public void LiveUpdatePage_IsReported_WithTheRegistryState_AndRaisesNothing()
{
_registry = new[] { NdupKey("EnableExpeditedUpdate=1") };
using var tracker = Build();
var raised = 0;
tracker.EspExited += (_, _) => raised++;
tracker.EspFailureDetected += (_, _) => raised++;
tracker.FinalizingSetupPhaseTriggered += (_, _) => raised++;
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppStarted, NdupStarted, At, "Microsoft-Windows-Shell-Core", isBackfill: false);
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEventName, ScanStartedName, At.AddMinutes(1), "Microsoft-Windows-Shell-Core", isBackfill: false);
_registry = new[] { NdupKey("EnableExpeditedUpdate=1, EnableExpeditedUpdateSyncInstallCompleted=1") };
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppStopped, PageStopped, At.AddMinutes(30), "Microsoft-Windows-Shell-Core", isBackfill: false);
var pages = ByType(Constants.EventTypes.OobeUpdatePage);
Assert.Equal(new object[] { "page_started", "event_name", "page_stopped" }, pages.Select(p => Data(p)["cxhEvent"]).ToArray());
Assert.All(pages, p => Assert.Equal(false, Data(p)["backfill"]));
var states = ByType(Constants.EventTypes.OobeUpdateState);
Assert.Equal(new object[] { "update_page_started", "update_page_stopped" }, states.Select(s => Data(s)["moment"]).ToArray());
Assert.All(states, s => Assert.Equal("OobeNDUP", Data(s)["page"]));
// Telemetry only — no ESP signal, no other event.
Assert.Equal(0, raised);
Assert.False(tracker.IsEspExitedForTest);
Assert.Equal(pages.Count + states.Count, _sink.Posted.Count);
}
[Fact]
public void OtherPages_AreNotReported()
{
_registry = new[] { NdupKey("EnableExpeditedUpdate=1") };
using var tracker = Build();
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppStarted, EspPageStarted, At, "Microsoft-Windows-Shell-Core", isBackfill: false);
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppStopped, PageStopped, At.AddMinutes(1), "Microsoft-Windows-Shell-Core", isBackfill: false);
Assert.Empty(_sink.Posted);
}
[Fact]
public void Backfill_ReportsTheUpdatePage_WithItsEventTimes_ButReadsNoRegistry()
{
// The registry shows the state now, not at the record's time.
var reads = 0;
using var tracker = new ShellCoreTracker("S1", "T1",
new InformationalEventPost(_sink, new VirtualClock(At)),
new AgentLogger(_tmp.Path, AgentLogLevel.Info),
helloTracker: null,
oobeUpdateRegistryReader: () => { reads++; return new[] { NdupKey("EnableExpeditedUpdate=1") }; });
tracker.ReplayBackfillRecords(new[]
{
(ShellCoreTracker.EventId_ShellCore_WebAppStarted, NdupStarted, At.AddMinutes(-4)),
(ShellCoreTracker.EventId_ShellCore_WebAppEventName, ScanStartedName, At.AddMinutes(-3)),
(ShellCoreTracker.EventId_ShellCore_WebAppStopped, PageStopped, At.AddMinutes(-2)),
});
var pages = ByType(Constants.EventTypes.OobeUpdatePage);
Assert.Equal(3, pages.Count);
Assert.All(pages, p => Assert.Equal(true, Data(p)["backfill"]));
Assert.Equal(At.AddMinutes(-4), pages[0].OccurredAtUtc);
Assert.Equal(0, reads);
// Nothing else: no ESP exit or failure was in the window, so no skipped-records trace.
Assert.Equal(3, _sink.Posted.Count);
}
[Fact]
public void AgentStartAndStop_ReportTheRegistryState_StopOnce()
{
_registry = new[] { NdupKey("EnableExpeditedUpdate=1") };
var tracker = Build();
tracker.Start();
_registry = new[] { NdupKey("EnableExpeditedUpdate=1, ExpeditedUpdateStatus=2") };
tracker.Stop();
tracker.Stop();
tracker.Dispose();
var states = ByType(Constants.EventTypes.OobeUpdateState);
Assert.Equal(new object[] { "agent_start", "agent_stop" }, states.Select(s => Data(s)["moment"]).ToArray());
}
[Fact]
public void TrackerThatNeverStarted_ReportsNothingAtStop()
{
_registry = new[] { NdupKey("EnableExpeditedUpdate=1") };
var tracker = Build();
tracker.Dispose();
Assert.Empty(_sink.Posted);
}
}
}
@@ -0,0 +1,128 @@
#nullable enable
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
{
/// <summary>
/// The OOBE update registry snapshot (D-310, S4/S5): one readable, bounded line per key, for
/// agent.log and the <c>oobe_update_state</c> event. The registry read itself is a thin wrapper.
/// </summary>
public sealed class OobeUpdateRegistrySnapshotTests
{
private static KeyValuePair<string, object> Entry(string name, object value) => new KeyValuePair<string, object>(name, value);
[Fact]
public void Values_AreFormattedByType()
{
Assert.Equal("1", OobeUpdateRegistrySnapshot.FormatValue(1));
Assert.Equal("5000000000", OobeUpdateRegistrySnapshot.FormatValue(5000000000L));
Assert.Equal("Succeeded", OobeUpdateRegistrySnapshot.FormatValue("Succeeded"));
Assert.Equal("a|b", OobeUpdateRegistrySnapshot.FormatValue(new[] { "a", "b" }));
Assert.Equal("(binary 3 bytes)", OobeUpdateRegistrySnapshot.FormatValue(new byte[] { 1, 2, 3 }));
Assert.Equal("(null)", OobeUpdateRegistrySnapshot.FormatValue(null!));
}
[Fact]
public void LongValues_AreTruncated()
{
var formatted = OobeUpdateRegistrySnapshot.FormatValue(new string('x', 500));
Assert.Equal(OobeUpdateRegistrySnapshot.MaxValueLength + 1, formatted.Length);
Assert.EndsWith("…", formatted);
}
[Fact]
public void Entries_AreOrderedByName()
{
var line = OobeUpdateRegistrySnapshot.FormatEntries(new[]
{
Entry("ExpeditedUpdateStatus", 2),
Entry("EnableExpeditedUpdate", 1),
Entry("", "default"),
}, oobeOnly: false);
Assert.Equal("(default)=default, EnableExpeditedUpdate=1, ExpeditedUpdateStatus=2", line);
}
[Fact]
public void PolicyManagerSystemKey_KeepsOnlyOobeValues()
{
// That key holds every System policy of the device; only the OOBE ones matter here.
var line = OobeUpdateRegistrySnapshot.FormatEntries(new[]
{
Entry("AllowTelemetry", 1),
Entry("AllowOOBEUpdates", 1),
Entry("AllowOOBEUpdates_ProviderSet", 1),
}, oobeOnly: true);
Assert.Equal("AllowOOBEUpdates=1, AllowOOBEUpdates_ProviderSet=1", line);
Assert.Equal("(no OOBE values)", OobeUpdateRegistrySnapshot.FormatEntries(new[] { Entry("AllowTelemetry", 1) }, oobeOnly: true));
Assert.Equal("(no values)", OobeUpdateRegistrySnapshot.FormatEntries(new KeyValuePair<string, object>[0], oobeOnly: false));
}
[Fact]
public void ManyEntries_AreCapped_AndSaySo()
{
var entries = Enumerable.Range(0, OobeUpdateRegistrySnapshot.MaxEntries + 5)
.Select(i => Entry($"Value{i:000}", i));
var line = OobeUpdateRegistrySnapshot.FormatEntries(entries, oobeOnly: false);
Assert.EndsWith("… (+5 more)", line);
}
[Fact]
public void Describe_CountsOnlyOobeValues_OfThePolicyManagerSystemKey()
{
var policy = OobeUpdateRegistrySnapshot.Describe("policyManagerSystem", OobeUpdateRegistrySnapshot.PolicyManagerSystemKey,
new[] { Entry("AllowTelemetry", 1) }, new string[0]);
Assert.True(policy.Exists);
Assert.False(policy.HasValues);
var oobe = OobeUpdateRegistrySnapshot.Describe("policyManagerSystem", OobeUpdateRegistrySnapshot.PolicyManagerSystemKey,
new[] { Entry("AllowTelemetry", 1), Entry("AllowOOBEUpdates", 1) }, new string[0]);
Assert.True(oobe.HasValues);
Assert.Equal("AllowOOBEUpdates=1", oobe.Line);
}
[Fact]
public void Describe_SubkeysAlone_AreNoValues()
{
var state = OobeUpdateRegistrySnapshot.Describe("ndup", @"SOFTWARE\Microsoft\Windows\CurrentVersion\NDUP",
new KeyValuePair<string, object>[0], new[] { "Updates" });
Assert.False(state.HasValues);
Assert.Equal("(no values) | subkeys: Updates", state.Line);
}
[Fact]
public void Keys_HaveDistinctAliases()
{
// The aliases are the data keys of oobe_update_state.
Assert.Equal(OobeUpdateRegistrySnapshot.Keys.Length,
OobeUpdateRegistrySnapshot.Keys.Select(k => k.Alias).Distinct().Count());
}
[Fact]
public void Read_ReturnsEveryKey_FromTheRealRegistry()
{
var states = OobeUpdateRegistrySnapshot.Read();
Assert.Equal(OobeUpdateRegistrySnapshot.Keys.Select(k => k.Path), states.Select(s => s.Path));
Assert.All(states.Where(s => !s.Exists), s => Assert.False(s.HasValues));
}
[Fact]
public void Log_ToleratesMissingLoggerOrStates()
{
using var tmp = new TempDirectory();
OobeUpdateRegistrySnapshot.Log(new AgentLogger(tmp.Path, AgentLogLevel.Info), "test", OobeUpdateRegistrySnapshot.Read());
OobeUpdateRegistrySnapshot.Log(null!, "test", OobeUpdateRegistrySnapshot.Read());
OobeUpdateRegistrySnapshot.Log(new AgentLogger(tmp.Path, AgentLogLevel.Info), "test", null!);
}
}
}
@@ -0,0 +1,201 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.Shared;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
{
/// <summary>
/// Measurement telemetry of the OOBE update page (D-310): <c>oobe_update_page</c> per
/// update-page record and <c>oobe_update_state</c> for the registry behind it — Debug, bounded
/// per run, the state only when a key holds values and the content changed.
/// </summary>
public sealed class OobeUpdateTelemetryTests : IDisposable
{
private static readonly DateTime At = new DateTime(2026, 10, 2, 9, 0, 0, DateTimeKind.Utc);
private readonly TempDirectory _tmp = new TempDirectory();
private readonly FakeSignalIngressSink _sink = new FakeSignalIngressSink();
private IReadOnlyList<OobeUpdateRegistrySnapshot.KeyState> _registry = Array.Empty<OobeUpdateRegistrySnapshot.KeyState>();
public void Dispose() => _tmp.Dispose();
private OobeUpdateTelemetry Build(Func<IReadOnlyList<OobeUpdateRegistrySnapshot.KeyState>>? reader = null) =>
new OobeUpdateTelemetry("sess-oobe", "tenant-oobe",
new InformationalEventPost(_sink, new VirtualClock(At)),
new AgentLogger(_tmp.Path, AgentLogLevel.Info),
reader ?? (() => _registry));
private IReadOnlyList<FakeSignalIngressSink.PostedSignal> ByType(string eventType) =>
_sink.Posted.Where(p =>
p.Kind == DecisionSignalKind.InformationalEvent
&& p.Payload != null
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
&& et == eventType).ToList();
private static IReadOnlyDictionary<string, object> Data(FakeSignalIngressSink.PostedSignal posted) =>
(IReadOnlyDictionary<string, object>)posted.TypedPayload!;
private static OobeUpdateRegistrySnapshot.KeyState Key(string alias, string? line) =>
line == null
? new OobeUpdateRegistrySnapshot.KeyState(alias, $@"SOFTWARE\{alias}", exists: false, hasValues: false, line: null)
: new OobeUpdateRegistrySnapshot.KeyState(alias, $@"SOFTWARE\{alias}", exists: true, hasValues: true, line: line);
// ------------------------------------------------------------- oobe_update_page ----
[Fact]
public void Page_IsReported_WithItsEventTime_AndFields()
{
Build().ReportPage(
new OobeUpdatePageRecord(OobeUpdatePageRecord.PageStopped, 62405, page: "OobeNDUP", result: "success"),
At.AddMinutes(-3), isBackfill: true);
var page = Assert.Single(ByType(Constants.EventTypes.OobeUpdatePage));
Assert.Equal(At.AddMinutes(-3), page.OccurredAtUtc);
Assert.Equal("Debug", page.Payload![SignalPayloadKeys.Severity]);
var data = Data(page);
Assert.Equal("page_stopped", data["cxhEvent"]);
Assert.Equal("OobeNDUP", data["page"]);
Assert.Equal("success", data["result"]);
Assert.Equal(true, data["backfill"]);
Assert.Equal(1, data["occurrence"]);
Assert.False(data.ContainsKey("value"));
}
[Fact]
public void EventName_CarriesItsSafeValue()
{
Build().ReportPage(
new OobeUpdatePageRecord(OobeUpdatePageRecord.EventName, 62407, name: "ExpeditedUpdate_isNDUPAllowedByCSPSucceeded", value: "true"),
At, isBackfill: false);
var data = Data(Assert.Single(ByType(Constants.EventTypes.OobeUpdatePage)));
Assert.Equal("ExpeditedUpdate_isNDUPAllowedByCSPSucceeded", data["name"]);
Assert.Equal("true", data["value"]);
}
[Fact]
public void SamePageOrName_IsReportedAtMostThreeTimes()
{
var telemetry = Build();
for (var i = 0; i < 10; i++)
{
telemetry.ReportPage(new OobeUpdatePageRecord(OobeUpdatePageRecord.EventName, 62406, name: "ExpeditedUpdate_progress"), At.AddSeconds(i), isBackfill: false);
}
telemetry.ReportPage(new OobeUpdatePageRecord(OobeUpdatePageRecord.PageStopped, 62405, page: "OobeNDUP", result: "success"), At.AddMinutes(1), isBackfill: false);
var pages = ByType(Constants.EventTypes.OobeUpdatePage);
Assert.Equal(OobeUpdateTelemetry.MaxPageEventsPerKey + 1, pages.Count);
Assert.Equal(new object[] { 1, 2, 3 }, pages.Take(3).Select(p => Data(p)["occurrence"]).ToArray());
// The repeats do not crowd out the page end.
Assert.Equal("page_stopped", Data(pages.Last())["cxhEvent"]);
}
[Fact]
public void PageEvents_AreCappedPerRun()
{
var telemetry = Build();
for (var i = 0; i < OobeUpdateTelemetry.MaxPageEventsPerRun + 20; i++)
{
telemetry.ReportPage(new OobeUpdatePageRecord(OobeUpdatePageRecord.EventName, 62406, name: $"ExpeditedUpdate_step{i}"), At, isBackfill: false);
}
Assert.Equal(OobeUpdateTelemetry.MaxPageEventsPerRun, ByType(Constants.EventTypes.OobeUpdatePage).Count);
}
// ------------------------------------------------------------ oobe_update_state ----
[Fact]
public void State_WithoutValues_IsNotReported()
{
_registry = new[] { Key("setupOobeNdup", null), Key("policiesOobe", null) };
Build().ReportState(OobeUpdateTelemetry.Moments.AgentStart);
Assert.Empty(ByType(Constants.EventTypes.OobeUpdateState));
}
[Fact]
public void State_ReportsOnlyKeysWithValues_UnderTheirAlias()
{
_registry = new[]
{
Key("setupOobeNdup", "EnableExpeditedUpdate=1, EnableExpeditedUpdateSyncDownloadCompleted=1"),
Key("ndup", null),
new OobeUpdateRegistrySnapshot.KeyState("policyManagerSystem", "x", exists: true, hasValues: false, line: "(no OOBE values)"),
Key("espSetupPolicy", "InstallQualityUpdates=1"),
};
Build().ReportState(OobeUpdateTelemetry.Moments.UpdatePageStopped, "OobeNDUP");
var state = Assert.Single(ByType(Constants.EventTypes.OobeUpdateState));
Assert.Equal("Debug", state.Payload![SignalPayloadKeys.Severity]);
var data = Data(state);
Assert.Equal("update_page_stopped", data["moment"]);
Assert.Equal("OobeNDUP", data["page"]);
Assert.Equal(2, data["keysWithValues"]);
Assert.Equal("EnableExpeditedUpdate=1, EnableExpeditedUpdateSyncDownloadCompleted=1", data["setupOobeNdup"]);
Assert.Equal("InstallQualityUpdates=1", data["espSetupPolicy"]);
Assert.False(data.ContainsKey("ndup"));
Assert.False(data.ContainsKey("policyManagerSystem"));
}
[Fact]
public void UnchangedState_IsReportedOnce_AChangeAgain()
{
var telemetry = Build();
_registry = new[] { Key("espSetupPolicy", "InstallQualityUpdates=1") };
telemetry.ReportState(OobeUpdateTelemetry.Moments.AgentStart);
telemetry.ReportState(OobeUpdateTelemetry.Moments.UpdatePageStarted, "OobeNDUP");
_registry = new[] { Key("espSetupPolicy", "InstallQualityUpdates=1"), Key("setupOobeNdup", "EnableExpeditedUpdate=1") };
telemetry.ReportState(OobeUpdateTelemetry.Moments.UpdatePageStopped, "OobeNDUP");
telemetry.ReportState(OobeUpdateTelemetry.Moments.AgentStop);
var states = ByType(Constants.EventTypes.OobeUpdateState);
Assert.Equal(new object[] { "agent_start", "update_page_stopped" }, states.Select(s => Data(s)["moment"]).ToArray());
}
[Fact]
public void StateEvents_AreCappedPerRun()
{
var telemetry = Build();
for (var i = 0; i < OobeUpdateTelemetry.MaxStateEventsPerRun + 4; i++)
{
_registry = new[] { Key("setupOobeNdup", $"ExpeditedUpdateStatus={i}") };
telemetry.ReportState(OobeUpdateTelemetry.Moments.UpdatePageStarted, "OobeNDUP");
}
Assert.Equal(OobeUpdateTelemetry.MaxStateEventsPerRun, ByType(Constants.EventTypes.OobeUpdateState).Count);
}
[Fact]
public void LongLines_AreTruncated()
{
_registry = new[] { Key("ndupUpdates", new string('x', 3000)) };
Build().ReportState(OobeUpdateTelemetry.Moments.AgentStart);
var line = (string)Data(Assert.Single(ByType(Constants.EventTypes.OobeUpdateState)))["ndupUpdates"];
Assert.Equal(OobeUpdateTelemetry.MaxStateLineLength + 1, line.Length);
Assert.EndsWith("…", line);
}
[Fact]
public void FailingRegistryRead_ReportsNothing_AndDoesNotThrow()
{
Build(() => throw new UnauthorizedAccessException("denied")).ReportState(OobeUpdateTelemetry.Moments.AgentStart);
Assert.Empty(_sink.Posted);
}
}
}
@@ -0,0 +1,289 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.DecisionCore.State;
using AutopilotMonitor.Shared;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
{
/// <summary>
/// ServicingTracker (D-310) — the CBS servicing steps of OS updates from the Setup log. The
/// sequence and XML shapes are those Windows 11 writes for a cumulative update installed by
/// <c>UpdateAgentLCU</c>: initiate Absent → Staged, Staged reached, initiate Staged → Installed,
/// restart required, Installed reached after the restart.
/// </summary>
public sealed class ServicingTrackerTests : IDisposable
{
private static readonly DateTime T0 = new DateTime(2026, 9, 25, 18, 52, 37, DateTimeKind.Utc);
private const string Kb = "KB5099999";
private readonly TempDirectory _tmp = new TempDirectory();
private readonly FakeSignalIngressSink _sink = new FakeSignalIngressSink();
private readonly UpdateActivityTally _tally = new UpdateActivityTally();
private readonly List<(string Step, string Update, DateTime At)> _osActivity = new List<(string, string, DateTime)>();
private readonly ServicingTracker _tracker;
private long _nextRecordId = 1200;
public ServicingTrackerTests()
{
_tracker = NewTracker(stateDirectory: null);
}
public void Dispose() => _tmp.Dispose();
private ServicingTracker NewTracker(string? stateDirectory) =>
new ServicingTracker(
sessionId: "sess-cbs",
tenantId: "tenant-cbs",
post: new InformationalEventPost(_sink, new VirtualClock(T0)),
logger: new AgentLogger(_tmp.Path, AgentLogLevel.Info),
backfillEnabled: false,
stateDirectory: stateDirectory,
tally: _tally,
onOsUpdateActivity: (step, update, at) => _osActivity.Add((step, update, at)));
private void Process(
int eventId,
DateTime at,
string? initialState = null,
string? targetState = "Installed",
string? errorCode = null,
string? client = "UpdateAgentLCU",
string? package = Kb,
long? recordId = null,
ServicingTracker? tracker = null,
bool isBackfill = false)
{
(tracker ?? _tracker).ProcessEvent(
eventId: eventId,
recordId: recordId ?? _nextRecordId++,
timeCreatedUtc: at,
package: package,
initialState: initialState,
targetState: targetState,
errorCode: errorCode,
client: client,
isBackfill: isBackfill);
}
private IReadOnlyList<FakeSignalIngressSink.PostedSignal> Servicing() =>
_sink.Posted.Where(p =>
p.Kind == DecisionSignalKind.InformationalEvent
&& p.Payload != null
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
&& et == Constants.EventTypes.WindowsUpdateServicing).ToList();
private static IReadOnlyDictionary<string, object> Data(FakeSignalIngressSink.PostedSignal s) =>
(IReadOnlyDictionary<string, object>)s.TypedPayload!;
[Fact]
public void CumulativeUpdateSequence_ListsEveryStep_AndReportsTheEngineSteps()
{
Process(ServicingTracker.EventId_InitiatingChanges, T0, initialState: "Absent", targetState: "Staged");
Process(ServicingTracker.EventId_StateReached, T0.AddMinutes(4), targetState: "Staged", errorCode: "0x0");
Process(ServicingTracker.EventId_InitiatingChanges, T0.AddMinutes(29), initialState: "Staged", targetState: "Installed");
Process(ServicingTracker.EventId_RebootRequired, T0.AddMinutes(33), targetState: "Installed", errorCode: "0x0");
// Written after the restart.
Process(ServicingTracker.EventId_StateReached, T0.AddMinutes(41), targetState: "Installed", errorCode: "0x0");
var rows = Servicing();
Assert.Equal(
new[] { "initiating", "state_reached", "initiating", "reboot_required", "state_reached" },
rows.Select(s => (string)Data(s)["step"]).ToArray());
Assert.All(rows, s => Assert.Equal("Info", s.Payload![SignalPayloadKeys.Severity]));
Assert.All(rows, s => Assert.Equal("ServicingWatcher", s.Payload![SignalPayloadKeys.Source]));
Assert.All(rows, s => Assert.Equal(Kb, Data(s)["package"]));
Assert.All(rows, s => Assert.Equal("UpdateAgentLCU", Data(s)["client"]));
// ErrorCode 0x0 is success, not a decoded HRESULT.
Assert.All(rows, s => Assert.False(Data(s).ContainsKey("hresult")));
Assert.Equal("Absent", Data(rows[0])["initialState"]);
Assert.Equal("Staged", Data(rows[0])["targetState"]);
Assert.Equal(T0.AddMinutes(33), rows[3].OccurredAtUtc);
Assert.Contains("needs a restart", rows[3].Payload![SignalPayloadKeys.Message]);
Assert.Equal(
new[]
{
OsUpdateSteps.StagingStarted, OsUpdateSteps.Staged, OsUpdateSteps.InstallStarted,
OsUpdateSteps.RebootRequired, OsUpdateSteps.Installed,
},
_osActivity.Select(a => a.Step).ToArray());
Assert.All(_osActivity, a => Assert.Equal(Kb, a.Update));
Assert.Equal(T0.AddMinutes(29), _osActivity[2].At);
}
[Theory]
[InlineData("CbsTask")]
[InlineData("DISM Package Manager Provider")]
[InlineData("")]
[InlineData(null)]
public void NonOsClients_AreCountedNotListed(string? client)
{
// CBS maintenance writes hundreds of these in seconds; language packs and features on
// demand installed by apps are not the OOBE update either.
Process(ServicingTracker.EventId_InitiatingChanges, T0, initialState: "Staged", targetState: "Installed",
client: client, package: "Microsoft-Windows-Telnet-Client-FOD-en-US");
Process(ServicingTracker.EventId_StateReached, T0, targetState: "Installed", errorCode: "0x0",
client: client, package: "Microsoft-Windows-Telnet-Client-FOD-en-US");
Assert.Empty(_sink.Posted);
Assert.Empty(_osActivity);
var counts = _tally.Snapshot().ToDictionary(kv => kv.Key, kv => kv.Value);
Assert.Equal(1, counts["servicing_other_1"]);
Assert.Equal(1, counts["servicing_other_2"]);
}
[Fact]
public void WindowsUpdateAgentClient_IsAnOsClient()
{
Process(ServicingTracker.EventId_RebootRequired, T0, client: "WindowsUpdateAgent", errorCode: "0x0");
Assert.Single(Servicing());
Assert.Equal(OsUpdateSteps.RebootRequired, Assert.Single(_osActivity).Step);
}
[Fact]
public void ChangeFailed_EmitsWarning_WithDecodedHResult()
{
Process(ServicingTracker.EventId_ChangeFailed, T0, targetState: "Installed", errorCode: "0x800f0922");
var s = Assert.Single(Servicing());
Assert.Equal("Warning", s.Payload![SignalPayloadKeys.Severity]);
var data = Data(s);
Assert.Equal("failed", data["step"]);
Assert.Equal("0x800F0922", data["hresult"]);
Assert.Equal("CBS_E_INSTALLERS_FAILED", data["hresultSymbol"]);
Assert.Contains("FAILED", s.Payload![SignalPayloadKeys.Message]);
Assert.Equal(OsUpdateSteps.Failed, Assert.Single(_osActivity).Step);
}
[Fact]
public void PartiallyInstalled_IsAFailure_AndSaysSo()
{
Process(ServicingTracker.EventId_PartiallyInstalled, T0, targetState: "Installed", errorCode: "0x800f0922");
var s = Assert.Single(Servicing());
Assert.Equal("failed", Data(s)["step"]);
Assert.Contains("partially installed", s.Payload![SignalPayloadKeys.Message]);
Assert.Equal(OsUpdateSteps.Failed, Assert.Single(_osActivity).Step);
}
[Fact]
public void ChangeTowardsAnotherState_IsListed_ButIsNoEngineStep()
{
// Removing a superseded package is servicing, not progress of the update install.
Process(ServicingTracker.EventId_InitiatingChanges, T0, initialState: "Installed", targetState: "Absent");
Assert.Single(Servicing());
Assert.Empty(_osActivity);
}
[Fact]
public void BackfilledRecord_KeepsItsEventTime()
{
var earlier = T0.AddMinutes(-20);
Process(ServicingTracker.EventId_InitiatingChanges, earlier, initialState: "Absent", targetState: "Staged", isBackfill: true);
var s = Assert.Single(Servicing());
Assert.Equal(earlier, s.OccurredAtUtc);
Assert.Equal(true, Data(s)["backfilled"]);
Assert.Equal(earlier, Assert.Single(_osActivity).At);
}
[Fact]
public void Watermark_SkipsEmittedRecordsAfterRestart()
{
var first = NewTracker(_tmp.Path);
Process(ServicingTracker.EventId_RebootRequired, T0, errorCode: "0x0", recordId: 1205, tracker: first);
var second = NewTracker(_tmp.Path);
second.LoadWatermark();
Process(ServicingTracker.EventId_RebootRequired, T0, errorCode: "0x0", recordId: 1205, tracker: second, isBackfill: true);
Process(ServicingTracker.EventId_StateReached, T0.AddMinutes(8), errorCode: "0x0", recordId: 1207, tracker: second);
Assert.Equal(new[] { "reboot_required", "state_reached" }, Servicing().Select(s => (string)Data(s)["step"]).ToArray());
}
[Fact]
public void SameRecordDeliveredTwice_IsListedOnce()
{
Process(ServicingTracker.EventId_RebootRequired, T0, errorCode: "0x0", recordId: 1205);
Process(ServicingTracker.EventId_RebootRequired, T0, errorCode: "0x0", recordId: 1205);
Assert.Single(Servicing());
Assert.Single(_osActivity);
}
[Fact]
public void ParseUserData_ReadsBothCbsPayloadShapes()
{
const string initiate =
"<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Microsoft-Windows-Servicing' Guid='{bd12f3b8-fc40-4a61-a307-b7a013a069c1}'/>" +
"<EventID>1</EventID><Version>0</Version><Level>0</Level><Task>1</Task><Opcode>0</Opcode><Keywords>0x8000000000000000</Keywords>" +
"<TimeCreated SystemTime='2026-09-25T19:21:42.4486782Z'/><EventRecordID>1203</EventRecordID><Channel>Setup</Channel><Computer>DESKTOP-TEST</Computer>" +
"<Security UserID='S-1-5-18'/></System><UserData><CbsPackageInitiateChanges xmlns='http://manifests.microsoft.com/win/2004/08/windows/setup_provider'>" +
"<PackageIdentifier>KB5099999</PackageIdentifier><InitialPackageState>5064</InitialPackageState><InitialPackageStateTextized>Staged</InitialPackageStateTextized>" +
"<IntendedPackageState>5112</IntendedPackageState><IntendedPackageStateTextized>Installed</IntendedPackageStateTextized><Client>UpdateAgentLCU</Client>" +
"</CbsPackageInitiateChanges></UserData></Event>";
const string changeState =
"<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Microsoft-Windows-Servicing' Guid='{bd12f3b8-fc40-4a61-a307-b7a013a069c1}'/>" +
"<EventID>4</EventID><TimeCreated SystemTime='2026-09-25T19:26:07.1381462Z'/><EventRecordID>1205</EventRecordID><Channel>Setup</Channel></System>" +
"<UserData><CbsPackageChangeState xmlns='http://manifests.microsoft.com/win/2004/08/windows/setup_provider'><PackageIdentifier>KB5099999</PackageIdentifier>" +
"<IntendedPackageState>5112</IntendedPackageState><IntendedPackageStateTextized>Installed</IntendedPackageStateTextized><ErrorCode>0x0</ErrorCode>" +
"<Client>UpdateAgentLCU</Client></CbsPackageChangeState></UserData></Event>";
var a = ServicingTracker.ParseUserData(initiate);
Assert.Equal("KB5099999", a["PackageIdentifier"]);
Assert.Equal("Staged", a["InitialPackageStateTextized"]);
Assert.Equal("Installed", a["IntendedPackageStateTextized"]);
Assert.Equal("UpdateAgentLCU", a["client"]); // case-insensitive
Assert.False(a.ContainsKey("ErrorCode"));
var b = ServicingTracker.ParseUserData(changeState);
Assert.Equal("Installed", b["IntendedPackageStateTextized"]);
Assert.Equal("0x0", b["ErrorCode"]);
Assert.False(b.ContainsKey("InitialPackageStateTextized"));
}
[Fact]
public void ParseUserData_MalformedOrWithoutUserData_ReturnsEmpty_NeverThrows()
{
Assert.Empty(ServicingTracker.ParseUserData(null));
Assert.Empty(ServicingTracker.ParseUserData(""));
Assert.Empty(ServicingTracker.ParseUserData("<not-xml"));
Assert.Empty(ServicingTracker.ParseUserData("<Event><System><EventID>1</EventID></System></Event>"));
}
[Fact]
public void Watermark_PersistsOnlyAfterTheEngineSignalWasHandedOver()
{
// "Reboot required" is written right before the update's restart: its engine signal
// must be on its way before the watermark tells the next run's backfill to skip it.
var path = System.IO.Path.Combine(_tmp.Path, ServicingTracker.WatermarkStateFileName);
bool? watermarkOnDiskAtSignal = null;
var tracker = new ServicingTracker(
sessionId: "sess-cbs",
tenantId: "tenant-cbs",
post: new InformationalEventPost(_sink, new VirtualClock(T0)),
logger: new AgentLogger(_tmp.Path, AgentLogLevel.Info),
backfillEnabled: false,
stateDirectory: _tmp.Path,
tally: _tally,
onOsUpdateActivity: (step, update, at) => watermarkOnDiskAtSignal = System.IO.File.Exists(path));
Process(ServicingTracker.EventId_RebootRequired, T0, errorCode: "0x0", tracker: tracker);
Assert.False(watermarkOnDiskAtSignal);
Assert.True(System.IO.File.Exists(path));
}
}
}
@@ -0,0 +1,85 @@
#nullable enable
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
{
/// <summary>
/// WindowsUpdateClassifier (D-310) — decides which Windows Update client events are listed one
/// by one (OS updates) and which are only counted (Store, Defender, everything else). The
/// titles are the shapes the WU client writes on Windows 11 GA and Insider builds.
/// </summary>
public sealed class WindowsUpdateClassifierTests
{
[Theory]
[InlineData("2026-09 Cumulative Update for Windows 11, version 25H2 for x64-based Systems (KB5099999)")]
[InlineData("2026-09 Cumulative Update Preview for Windows 11, version 24H2 for x64-based Systems (KB5099998)")]
[InlineData("2026-09 Dynamic Update for Windows 11, version 25H2 for x64-based Systems (KB5099997)")]
[InlineData("2026-09 Servicing Stack Update for Windows 11, version 25H2 for x64-based Systems (KB5099996)")]
[InlineData("Windows 11 Insider Preview Quality Update (26220.9568)")]
[InlineData("Windows 11 Insider Preview Feature Update (26220.9022)")]
[InlineData("Windows 11, version 25H2 Upgrade")]
[InlineData("2026-09 .NET Framework Security Update (KB5099995)")]
[InlineData("2026-08 .NET Framework Preview Update (KB5099994)")]
[InlineData("Windows Malicious Software Removal Tool x64 - v5.145 (KB890830)")]
public void WindowsAndDotNetFrameworkUpdates_AreOs(string title)
{
Assert.Equal(WindowsUpdateClassifier.Os, WindowsUpdateClassifier.Classify(title, serviceGuid: null));
}
[Theory]
[InlineData("Security Intelligence Update for Microsoft Defender Antivirus - KB2267602 (Version 1.459.505.0) - Current Channel (Broad)")]
[InlineData("Update for Microsoft Defender Antivirus antimalware platform - KB4052623 (Version 4.18.26080.4) - Current Channel (Staged)")]
// Names "Windows" and "Update" — Defender must still win.
[InlineData("Windows 10/11 - Update for Microsoft Defender for Endpoint - KB5005292 (Version 10.8838.26060.15013) - Current Channel (Staged)")]
public void DefenderUpdates_AreDefender(string title)
{
Assert.Equal(WindowsUpdateClassifier.Defender, WindowsUpdateClassifier.Classify(title, serviceGuid: null));
}
[Theory]
[InlineData("9NSTH9KHZDLQ-Microsoft.UI.Xaml.2.8")]
[InlineData("9NBLGGH4NNS1-Microsoft.DesktopAppInstaller")]
// Upper-case "WINDOWS" inside the package name must not make it an OS update.
[InlineData("9WZDNCRFHVN5-MICROSOFT.WINDOWSCALCULATOR")]
[InlineData("9WZDNCRFHWKN-MICROSOFT.WINDOWSSOUNDRECORDER")]
[InlineData("ApplicationSet-9WZDNCRFJ3Q2-Microsoft.BingWeather")]
public void StoreProductTitles_AreStore(string title)
{
Assert.Equal(WindowsUpdateClassifier.Store, WindowsUpdateClassifier.Classify(title, serviceGuid: null));
}
[Theory]
[InlineData("{855e8a7c-ecb4-4ca3-b045-1dfa50104289}")]
[InlineData("{855E8A7C-ECB4-4CA3-B045-1DFA50104289}")]
[InlineData("855e8a7c-ecb4-4ca3-b045-1dfa50104289")]
public void StoreServiceGuid_IsStore_WhateverTheTitle(string serviceGuid)
{
Assert.Equal(WindowsUpdateClassifier.Store,
WindowsUpdateClassifier.Classify("2026-09 Cumulative Update for Windows 11 (KB5099999)", serviceGuid));
}
[Fact]
public void OtherServiceGuid_LeavesTheTitleInCharge()
{
Assert.Equal(WindowsUpdateClassifier.Os,
WindowsUpdateClassifier.Classify("2026-09 Cumulative Update for Windows 11 (KB5099999)", "{7971f918-a847-4430-9279-4a52d1efe18d}"));
}
[Theory]
[InlineData("Intel net Driver Update (24.50.0.4)")]
[InlineData("PowerShell v7.6.6 (x64)")]
[InlineData("Remote help 5.2.1040.0")]
[InlineData("Microsoft Edge-Stable Channel Version 140 Update for x64 based Edge-Chromium (Build 140.0.3485.54)")]
[InlineData("Microsoft .NET 8.0.20 Security Update for x64 Client (KB5099993)")]
[InlineData("Windows Feature Experience Pack 1000.26100.234.0")]
[InlineData("KB5099999")]
[InlineData("")]
[InlineData(" ")]
[InlineData(null)]
public void EverythingElse_IsOther(string? title)
{
Assert.Equal(WindowsUpdateClassifier.Other, WindowsUpdateClassifier.Classify(title!, serviceGuid: null));
}
}
}
@@ -1,6 +1,7 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
@@ -9,55 +10,81 @@ using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.DecisionCore.State;
using AutopilotMonitor.Shared;
using Newtonsoft.Json.Linq;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
{
/// <summary>
/// WindowsUpdateTracker — surfaces quality/cumulative update activity during OOBE from the
/// WindowsUpdateClient/Operational channel. Drives the primitive <see cref="WindowsUpdateTracker.ProcessEvent"/>
/// WindowsUpdateClient provider: install events from the System log, scan/download events from
/// the Operational channel (D-310). Drives the primitive <see cref="WindowsUpdateTracker.ProcessEvent"/>
/// test seam (no real EventRecord, which is abstract + Windows-only), mirroring the
/// ModernDeploymentTracker test pattern.
/// ModernDeploymentTracker test pattern. Update titles are the shapes the provider writes.
/// </summary>
public sealed class WindowsUpdateTrackerTests : IDisposable
{
private static readonly DateTime At = new DateTime(2026, 7, 8, 10, 15, 0, DateTimeKind.Utc);
private const string Lcu = "2026-09 Cumulative Update for Windows 11, version 25H2 for x64-based Systems (KB5099999)";
private const string StoreApp = "9WZDNCRFHVN5-MICROSOFT.WINDOWSCALCULATOR";
private const string DefenderIntelligence = "Security Intelligence Update for Microsoft Defender Antivirus - KB2267602 (Version 1.459.505.0) - Current Channel (Broad)";
private readonly TempDirectory _tmp = new TempDirectory();
private readonly FakeSignalIngressSink _sink;
private readonly WindowsUpdateTracker _tracker;
private readonly FakeSignalIngressSink _sink = new FakeSignalIngressSink();
private readonly UpdateActivityTally _tally = new UpdateActivityTally();
private readonly List<(string Step, string Update, DateTime At)> _osActivity = new List<(string, string, DateTime)>();
private readonly WindowsUpdateTracker _system;
private readonly WindowsUpdateTracker _operational;
private long _nextRecordId = 1;
public WindowsUpdateTrackerTests()
{
_sink = new FakeSignalIngressSink();
var post = new InformationalEventPost(_sink, new VirtualClock(At));
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
_tracker = new WindowsUpdateTracker(
sessionId: "sess-wu",
tenantId: "tenant-wu",
post: post,
logger: logger,
backfillEnabled: false,
stateDirectory: null); // in-memory dedup for most tests
// In-memory dedup for most tests.
_system = NewTracker(WindowsUpdateTracker.SystemChannel, stateDirectory: null);
_operational = NewTracker(channel: null, stateDirectory: null);
}
public void Dispose() => _tmp.Dispose();
private void Emit(int eventId, string? errorCode = null, string? updateTitle = "Cumulative Update KB5099999", long? recordId = null)
private WindowsUpdateTracker NewTracker(string? channel, string? stateDirectory) =>
new WindowsUpdateTracker(
sessionId: "sess-wu",
tenantId: "tenant-wu",
post: new InformationalEventPost(_sink, new VirtualClock(At)),
logger: new AgentLogger(_tmp.Path, AgentLogLevel.Info),
backfillEnabled: false,
stateDirectory: stateDirectory,
channel: channel,
tally: _tally,
onOsUpdateActivity: (step, update, at) => _osActivity.Add((step, update, at)));
private void Emit(
WindowsUpdateTracker tracker,
int eventId,
string? updateTitle = Lcu,
string? errorCode = null,
long? recordId = null,
string? serviceGuid = null,
string? updateCount = null,
DateTime? at = null,
bool isBackfill = false)
{
_tracker.ProcessEvent(
tracker.ProcessEvent(
eventId: eventId,
level: 4,
recordId: recordId ?? _nextRecordId++,
timeCreatedUtc: At,
timeCreatedUtc: at ?? At,
updateTitle: updateTitle,
updateGuid: "{8b1c8726-1111-2222-3333-444455556666}",
updateRevisionNumber: "200",
errorCode: errorCode,
formattedDescription: null,
isBackfill: false);
isBackfill: isBackfill,
serviceGuid: serviceGuid,
updateCount: updateCount);
}
private IReadOnlyList<FakeSignalIngressSink.PostedSignal> ByType(string eventType) =>
@@ -70,10 +97,17 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
private static IReadOnlyDictionary<string, object> Data(FakeSignalIngressSink.PostedSignal s) =>
(IReadOnlyDictionary<string, object>)s.TypedPayload!;
private int Counted(string key) =>
_tally.Snapshot().Where(kv => kv.Key == key).Select(kv => kv.Value).SingleOrDefault();
// ---------------------------------------------------------------------------
// Install events (System log)
// ---------------------------------------------------------------------------
[Fact]
public void FailureEvent_EmitsWindowsUpdateFailed_WithDecodedHResult_AndImmediateUpload()
{
Emit(WindowsUpdateTracker.EventId_InstallFailure, errorCode: "0x80240022");
Emit(_system, WindowsUpdateTracker.EventId_InstallFailure, errorCode: "0x80240022");
var failed = ByType(Constants.EventTypes.WindowsUpdateFailed);
var s = Assert.Single(failed);
@@ -83,37 +117,44 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
var data = Data(s);
Assert.Equal("0x80240022", data["hresult"]);
Assert.Equal("WU_E_ALL_UPDATES_FAILED", data["hresultSymbol"]);
Assert.Equal("Cumulative Update KB5099999", data["updateTitle"]);
Assert.Equal(Lcu, data["updateTitle"]);
Assert.Equal(20, data["wuEventId"]);
Assert.Equal("system", data["wuChannel"]);
Assert.Equal("install", data["wuPhase"]);
Assert.Equal(WindowsUpdateClassifier.Os, data["updateClass"]);
}
[Fact]
public void SuccessEvent_EmitsWindowsUpdateSucceeded_InfoSeverity()
{
Emit(WindowsUpdateTracker.EventId_InstallSuccess);
Emit(_system, WindowsUpdateTracker.EventId_InstallSuccess);
var ok = ByType(Constants.EventTypes.WindowsUpdateSucceeded);
var s = Assert.Single(ok);
Assert.Equal("Info", s.Payload![SignalPayloadKeys.Severity]);
Assert.Equal("false", s.Payload![SignalPayloadKeys.ImmediateUpload]);
Assert.Equal("installed", Data(s)["wuPhase"]);
}
[Fact]
public void StartedAndDownloadEvents_EmitDebugStarted()
public void StartedAndDownloadEvents_EmitInfoStarted_WithTheirPhase()
{
Emit(WindowsUpdateTracker.EventId_InstallStarted);
Emit(WindowsUpdateTracker.EventId_DownloadStarted);
// Info, not Debug: with only OS updates listed they are the visible progress of a
// long OOBE update.
Emit(_system, WindowsUpdateTracker.EventId_InstallStarted);
Emit(_system, WindowsUpdateTracker.EventId_DownloadStarted);
var started = ByType(Constants.EventTypes.WindowsUpdateStarted);
Assert.Equal(2, started.Count);
Assert.All(started, s => Assert.Equal("Debug", s.Payload![SignalPayloadKeys.Severity]));
Assert.All(started, s => Assert.Equal("Info", s.Payload![SignalPayloadKeys.Severity]));
Assert.Equal(new[] { "install", "download" }, started.Select(s => (string)Data(s)["wuPhase"]).ToArray());
}
[Fact]
public void SameRecordId_ProcessedTwice_EmitsOnce()
{
Emit(WindowsUpdateTracker.EventId_InstallSuccess, recordId: 42);
Emit(WindowsUpdateTracker.EventId_InstallSuccess, recordId: 42);
Emit(_system, WindowsUpdateTracker.EventId_InstallSuccess, recordId: 42);
Emit(_system, WindowsUpdateTracker.EventId_InstallSuccess, recordId: 42);
Assert.Single(ByType(Constants.EventTypes.WindowsUpdateSucceeded));
}
@@ -125,8 +166,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
// with a higher RecordId can be processed first. An older, never-emitted backfill event
// (lower RecordId) must NOT be suppressed as "already processed" — these early pre-agent
// OOBE updates are the whole point of the feature. A high-water-mark dedup dropped them.
Emit(WindowsUpdateTracker.EventId_InstallSuccess, recordId: 100); // live, high RecordId
Emit(WindowsUpdateTracker.EventId_InstallSuccess, recordId: 50); // backfill, older, never seen
Emit(_system, WindowsUpdateTracker.EventId_InstallSuccess, recordId: 100); // live, high RecordId
Emit(_system, WindowsUpdateTracker.EventId_InstallSuccess, recordId: 50); // backfill, older, never seen
Assert.Equal(2, ByType(Constants.EventTypes.WindowsUpdateSucceeded).Count);
}
@@ -138,53 +179,212 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
// time, not at agent start (the ctor's default UtcNow). InformationalEventPost forwards
// EnrollmentEvent.Timestamp as the signal's OccurredAtUtc, which drives the timeline entry.
var eventTime = new DateTime(2026, 7, 8, 7, 5, 0, DateTimeKind.Utc); // well before `At`
_tracker.ProcessEvent(
eventId: WindowsUpdateTracker.EventId_InstallFailure,
level: 4,
recordId: 7,
timeCreatedUtc: eventTime,
updateTitle: "KB5099999",
updateGuid: null,
updateRevisionNumber: null,
errorCode: "0x80240022",
formattedDescription: null,
isBackfill: true);
Emit(_system, WindowsUpdateTracker.EventId_InstallFailure, errorCode: "0x80240022",
recordId: 7, at: eventTime, isBackfill: true);
var s = Assert.Single(ByType(Constants.EventTypes.WindowsUpdateFailed));
Assert.Equal(eventTime, s.OccurredAtUtc);
Assert.Equal(true, Data(s)["backfilled"]);
}
[Fact]
public void NegativeRecordId_IsNeverDeduped()
{
Emit(WindowsUpdateTracker.EventId_InstallSuccess, recordId: -1);
Emit(WindowsUpdateTracker.EventId_InstallSuccess, recordId: -1);
Emit(_system, WindowsUpdateTracker.EventId_InstallSuccess, recordId: -1);
Emit(_system, WindowsUpdateTracker.EventId_InstallSuccess, recordId: -1);
Assert.Equal(2, ByType(Constants.EventTypes.WindowsUpdateSucceeded).Count);
}
// ---------------------------------------------------------------------------
// Scan/download events (Operational channel)
// ---------------------------------------------------------------------------
[Fact]
public void Watermark_PersistsAcrossTrackerInstances()
public void Downloaded_EmitsInfoStarted_WithDownloadedPhase()
{
var post = new InformationalEventPost(_sink, new VirtualClock(At));
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
Emit(_operational, WindowsUpdateTracker.EventId_Downloaded);
var first = new WindowsUpdateTracker("s", "t", post, logger, backfillEnabled: false, stateDirectory: _tmp.Path);
first.ProcessEvent(WindowsUpdateTracker.EventId_InstallSuccess, 4, recordId: 500, timeCreatedUtc: At,
updateTitle: "KB1", updateGuid: null, updateRevisionNumber: null, errorCode: null,
formattedDescription: null, isBackfill: false);
var s = Assert.Single(ByType(Constants.EventTypes.WindowsUpdateStarted));
Assert.Equal("Info", s.Payload![SignalPayloadKeys.Severity]);
Assert.Equal("downloaded", Data(s)["wuPhase"]);
Assert.Equal("operational", Data(s)["wuChannel"]);
}
// A fresh tracker (simulating an agent restart) must load the watermark and skip the
// already-emitted record when the OOBE backfill re-reads it.
var second = new WindowsUpdateTracker("s", "t", post, logger, backfillEnabled: false, stateDirectory: _tmp.Path);
second.LoadWatermark();
second.ProcessEvent(WindowsUpdateTracker.EventId_InstallSuccess, 4, recordId: 500, timeCreatedUtc: At,
updateTitle: "KB1", updateGuid: null, updateRevisionNumber: null, errorCode: null,
formattedDescription: null, isBackfill: true);
[Fact]
public void DownloadFailure_OfAnOsUpdate_EmitsWarning_WithDownloadPhase()
{
Emit(_operational, WindowsUpdateTracker.EventId_DownloadFailed, errorCode: "0x80248007");
var s = Assert.Single(ByType(Constants.EventTypes.WindowsUpdateFailed));
Assert.Equal("Warning", s.Payload![SignalPayloadKeys.Severity]);
Assert.Equal("false", s.Payload![SignalPayloadKeys.ImmediateUpload]);
var data = Data(s);
Assert.Equal("download", data["wuPhase"]);
Assert.Equal("0x80248007", data["hresult"]);
}
[Fact]
public void ScanFound_EmitsOnlyChangedCounts_AndCountsTheRepeats()
{
foreach (var count in new[] { "3", "3", "5", "5", "0" })
Emit(_operational, WindowsUpdateTracker.EventId_ScanFound, updateTitle: null, updateCount: count);
var found = ByType(Constants.EventTypes.WindowsUpdateStarted);
Assert.Equal(new[] { "3", "5", "0" }, found.Select(s => (string)Data(s)["updateCount"]).ToArray());
Assert.All(found, s => Assert.Equal("Debug", s.Payload![SignalPayloadKeys.Severity]));
Assert.All(found, s => Assert.Equal("scan", Data(s)["wuPhase"]));
Assert.All(found, s => Assert.False(Data(s).ContainsKey("updateClass")));
Assert.Contains("scan found 3 update(s)", found[0].Payload![SignalPayloadKeys.Message]);
Assert.Equal(2, Counted("scan_found_repeat"));
Assert.Empty(_osActivity); // a scan result is not an update step
}
[Fact]
public void ScanFound_IsCappedPerRun()
{
for (var i = 0; i < WindowsUpdateTracker.MaxScanFoundEmissions + 2; i++)
Emit(_operational, WindowsUpdateTracker.EventId_ScanFound, updateTitle: null, updateCount: i.ToString());
Assert.Equal(WindowsUpdateTracker.MaxScanFoundEmissions, ByType(Constants.EventTypes.WindowsUpdateStarted).Count);
Assert.Equal(2, Counted("scan_found_repeat"));
}
[Fact]
public void ScanFailed_EmitsEachErrorCodeOnce()
{
Emit(_operational, WindowsUpdateTracker.EventId_ScanFailed, updateTitle: null, errorCode: "0x8024401c");
Emit(_operational, WindowsUpdateTracker.EventId_ScanFailed, updateTitle: null, errorCode: "0x8024401c");
Emit(_operational, WindowsUpdateTracker.EventId_ScanFailed, updateTitle: null, errorCode: "0x80072EE2");
Emit(_operational, WindowsUpdateTracker.EventId_ScanFailed, updateTitle: null, errorCode: "0x8024401C");
var failed = ByType(Constants.EventTypes.WindowsUpdateFailed);
Assert.Equal(2, failed.Count);
Assert.All(failed, s => Assert.Equal("Warning", s.Payload![SignalPayloadKeys.Severity]));
Assert.All(failed, s => Assert.Equal("scan", Data(s)["wuPhase"]));
Assert.Equal(2, Counted("scan_failed_repeat"));
Assert.Empty(_osActivity);
}
[Fact]
public void ScanFailed_IsCappedPerRun()
{
for (var i = 0; i < WindowsUpdateTracker.MaxScanFailedEmissions + 2; i++)
Emit(_operational, WindowsUpdateTracker.EventId_ScanFailed, updateTitle: null, errorCode: $"0x8024000{i}");
Assert.Equal(WindowsUpdateTracker.MaxScanFailedEmissions, ByType(Constants.EventTypes.WindowsUpdateFailed).Count);
Assert.Equal(2, Counted("scan_failed_repeat"));
}
// ---------------------------------------------------------------------------
// Classification — only OS updates are listed, the rest is counted
// ---------------------------------------------------------------------------
[Theory]
[InlineData(StoreApp, null, WindowsUpdateTracker.EventId_InstallSuccess, "store_19")]
[InlineData("Some App", "{855e8a7c-ecb4-4ca3-b045-1dfa50104289}", WindowsUpdateTracker.EventId_Downloaded, "store_41")]
[InlineData(DefenderIntelligence, null, WindowsUpdateTracker.EventId_InstallStarted, "defender_43")]
[InlineData("Intel net Driver Update (24.50.0.4)", null, WindowsUpdateTracker.EventId_DownloadStarted, "other_44")]
// A real download failure names no update at all; it must not reach ANALYZE-DEV-004 as
// a failed OS update.
[InlineData("", null, WindowsUpdateTracker.EventId_DownloadFailed, "other_31")]
[InlineData("Some App", "{855e8a7c-ecb4-4ca3-b045-1dfa50104289}", WindowsUpdateTracker.EventId_InstallFailure, "store_20")]
public void NonOsUpdates_AreCountedNotEmitted(string title, string? serviceGuid, int eventId, string expectedKey)
{
var tracker = WindowsUpdateTracker.IsSystemChannelEventId(eventId) ? _system : _operational;
Emit(tracker, eventId, updateTitle: title, serviceGuid: serviceGuid, errorCode: "0x80248007");
Assert.Empty(_sink.Posted);
Assert.Empty(_osActivity);
Assert.Equal(1, Counted(expectedKey));
// The census still sees the channel working.
Assert.Equal(1, tracker.EmittedThisRun);
}
[Fact]
public void CountedRecord_DeliveredTwice_IsCountedOnce()
{
// Live watcher and backfill overlap: the seen-set dedups counted records too.
Emit(_system, WindowsUpdateTracker.EventId_InstallSuccess, updateTitle: StoreApp, recordId: 7);
Emit(_system, WindowsUpdateTracker.EventId_InstallSuccess, updateTitle: StoreApp, recordId: 7);
Assert.Equal(1, Counted("store_19"));
}
[Fact]
public void OsUpdateSteps_AreReportedToTheEngineCallback()
{
var t0 = At;
Emit(_system, WindowsUpdateTracker.EventId_DownloadStarted, at: t0);
Emit(_operational, WindowsUpdateTracker.EventId_Downloaded, at: t0.AddMinutes(4));
Emit(_system, WindowsUpdateTracker.EventId_InstallStarted, at: t0.AddMinutes(5));
Emit(_system, WindowsUpdateTracker.EventId_InstallSuccess, at: t0.AddMinutes(31));
Emit(_system, WindowsUpdateTracker.EventId_InstallFailure, errorCode: "0x800F0922", at: t0.AddMinutes(32));
Emit(_operational, WindowsUpdateTracker.EventId_DownloadFailed, errorCode: "0x80248007", at: t0.AddMinutes(33));
Assert.Equal(
new[]
{
OsUpdateSteps.DownloadStarted, OsUpdateSteps.Downloaded, OsUpdateSteps.InstallStarted,
OsUpdateSteps.Installed, OsUpdateSteps.Failed, OsUpdateSteps.Failed,
},
_osActivity.Select(a => a.Step).ToArray());
Assert.All(_osActivity, a => Assert.Equal(Lcu, a.Update));
Assert.Equal(t0.AddMinutes(31), _osActivity[3].At);
}
[Fact]
public void ThrowingCallback_NeverCostsTheEvent()
{
var tracker = new WindowsUpdateTracker(
"s", "t", new InformationalEventPost(_sink, new VirtualClock(At)),
new AgentLogger(_tmp.Path, AgentLogLevel.Info),
backfillEnabled: false,
channel: WindowsUpdateTracker.SystemChannel,
onOsUpdateActivity: (step, update, at) => throw new InvalidOperationException("boom"));
Emit(tracker, WindowsUpdateTracker.EventId_InstallSuccess);
Assert.Single(ByType(Constants.EventTypes.WindowsUpdateSucceeded));
}
// ---------------------------------------------------------------------------
// Channels and XPath
// ---------------------------------------------------------------------------
[Fact]
public void DefaultEventIds_FollowTheProviderManifest()
{
Assert.Equal(WindowsUpdateTracker.SystemChannel, _system.WatchedChannel);
Assert.Equal(new[] { 19, 20, 43, 44 }, _system.TargetedEventIds.OrderBy(id => id).ToArray());
Assert.Equal(WindowsUpdateTracker.Channel, _operational.WatchedChannel);
Assert.Equal(new[] { 25, 26, 31, 41 }, _operational.TargetedEventIds.OrderBy(id => id).ToArray());
}
[Theory]
[InlineData(16, true)]
[InlineData(19, true)]
[InlineData(20, true)]
[InlineData(24, true)]
[InlineData(27, true)]
[InlineData(28, true)]
[InlineData(32, true)]
[InlineData(33, true)]
[InlineData(43, true)]
[InlineData(44, true)]
[InlineData(212, true)]
[InlineData(218, true)]
[InlineData(25, false)]
[InlineData(26, false)]
[InlineData(31, false)]
[InlineData(41, false)]
[InlineData(15, false)]
[InlineData(219, false)]
public void IsSystemChannelEventId_MatchesTheManifest(int eventId, bool expected)
{
Assert.Equal(expected, WindowsUpdateTracker.IsSystemChannelEventId(eventId));
}
[Fact]
public void BuildXPath_ContainsAllTargetedIds_Ordered()
{
@@ -192,6 +392,113 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
Assert.Equal("*[System[(EventID=19 or EventID=20 or EventID=43 or EventID=44)]]", xpath);
}
[Fact]
public void BuildXPath_SharedSystemLog_FiltersTheProvider_AndTheLookback()
{
// Verified against a real System log: the provider clause keeps the other providers'
// EventIDs 19/20/43/44 out of the shared channel.
var ids = new HashSet<int> { 44, 19, 20, 43 };
Assert.Equal(
"*[System[Provider[@Name='Microsoft-Windows-WindowsUpdateClient'] and (EventID=19 or EventID=20 or EventID=43 or EventID=44)]]",
WindowsUpdateTracker.BuildXPath(ids, WindowsUpdateTracker.ProviderName));
Assert.Equal(
"*[System[Provider[@Name='Microsoft-Windows-WindowsUpdateClient'] and (EventID=19 or EventID=20 or EventID=43 or EventID=44) and TimeCreated[timediff(@SystemTime) <= 3600000]]]",
WindowsUpdateTracker.BuildXPath(ids, WindowsUpdateTracker.ProviderName, lookbackMs: 3600000));
Assert.Equal(
"*[System[(EventID=25 or EventID=26) and TimeCreated[timediff(@SystemTime) <= 60000]]]",
WindowsUpdateTracker.BuildXPath(new HashSet<int> { 26, 25 }, providerName: null, lookbackMs: 60000));
}
// ---------------------------------------------------------------------------
// Watermark
// ---------------------------------------------------------------------------
[Fact]
public void Watermark_PersistsAcrossTrackerInstances()
{
var first = NewTracker(WindowsUpdateTracker.SystemChannel, _tmp.Path);
Emit(first, WindowsUpdateTracker.EventId_InstallSuccess, recordId: 500);
// A fresh tracker (simulating an agent restart) must load the watermark and skip the
// already-emitted record when the OOBE backfill re-reads it.
var second = NewTracker(WindowsUpdateTracker.SystemChannel, _tmp.Path);
second.LoadWatermark();
Emit(second, WindowsUpdateTracker.EventId_InstallSuccess, recordId: 500, isBackfill: true);
Assert.Single(ByType(Constants.EventTypes.WindowsUpdateSucceeded));
}
[Fact]
public void Watermark_IsPerChannel()
{
// RecordIds are per channel: System record 500 and Operational record 500 are two
// different events and must not dedup each other — before or after a restart.
Emit(NewTracker(WindowsUpdateTracker.SystemChannel, _tmp.Path), WindowsUpdateTracker.EventId_InstallSuccess, recordId: 500);
Emit(NewTracker(null, _tmp.Path), WindowsUpdateTracker.EventId_Downloaded, recordId: 500);
Assert.Single(ByType(Constants.EventTypes.WindowsUpdateSucceeded));
Assert.Single(ByType(Constants.EventTypes.WindowsUpdateStarted));
Assert.Equal(500L, ReadWatermark(WindowsUpdateTracker.SystemWatermarkStateFileName));
Assert.Equal(500L, ReadWatermark(WindowsUpdateTracker.WatermarkStateFileName));
var system = NewTracker(WindowsUpdateTracker.SystemChannel, _tmp.Path);
var operational = NewTracker(null, _tmp.Path);
system.LoadWatermark();
operational.LoadWatermark();
Emit(system, WindowsUpdateTracker.EventId_InstallSuccess, recordId: 500, isBackfill: true);
Emit(operational, WindowsUpdateTracker.EventId_Downloaded, recordId: 500, isBackfill: true);
Assert.Single(ByType(Constants.EventTypes.WindowsUpdateSucceeded));
Assert.Single(ByType(Constants.EventTypes.WindowsUpdateStarted));
}
[Fact]
public void CountedRecords_WriteNoStateFile()
{
// A Store/Defender burst must not cost a state-file write per record; only emitted
// records move the persisted watermark.
var tracker = NewTracker(WindowsUpdateTracker.SystemChannel, _tmp.Path);
Emit(tracker, WindowsUpdateTracker.EventId_InstallSuccess, updateTitle: StoreApp, recordId: 600);
Emit(tracker, WindowsUpdateTracker.EventId_InstallStarted, updateTitle: DefenderIntelligence, recordId: 601);
Assert.False(File.Exists(Path.Combine(_tmp.Path, WindowsUpdateTracker.SystemWatermarkStateFileName)));
Emit(tracker, WindowsUpdateTracker.EventId_InstallStarted, recordId: 602);
Assert.Equal(602L, ReadWatermark(WindowsUpdateTracker.SystemWatermarkStateFileName));
}
[Fact]
public void Watermark_PersistsOnlyAfterTheEngineSignalWasHandedOver()
{
// A restart's backfill skips a persisted record, so the record's engine signal must be
// on its way before the watermark reaches the disk — never the other way round.
var path = Path.Combine(_tmp.Path, WindowsUpdateTracker.SystemWatermarkStateFileName);
bool? watermarkOnDiskAtSignal = null;
var tracker = new WindowsUpdateTracker(
sessionId: "sess-wu",
tenantId: "tenant-wu",
post: new InformationalEventPost(_sink, new VirtualClock(At)),
logger: new AgentLogger(_tmp.Path, AgentLogLevel.Info),
backfillEnabled: false,
stateDirectory: _tmp.Path,
channel: WindowsUpdateTracker.SystemChannel,
tally: _tally,
onOsUpdateActivity: (step, update, at) => watermarkOnDiskAtSignal = File.Exists(path));
Emit(tracker, WindowsUpdateTracker.EventId_InstallStarted, recordId: 700);
Assert.False(watermarkOnDiskAtSignal);
Assert.Equal(700L, ReadWatermark(WindowsUpdateTracker.SystemWatermarkStateFileName));
}
private long ReadWatermark(string fileName) =>
JObject.Parse(File.ReadAllText(Path.Combine(_tmp.Path, fileName)))["LastRecordId"]!.Value<long>();
// ---------------------------------------------------------------------------
// Parsing and HRESULT decoding
// ---------------------------------------------------------------------------
[Fact]
public void ParseEventData_ExtractsNamedFields_NamespaceAgnostic()
{
@@ -212,6 +519,24 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
Assert.True(data.ContainsKey("UPDATETITLE"));
}
[Fact]
public void ParseEventData_ReadsTheScanFields()
{
// Event 26 (scan found N) and 25 (scan failed) carry no update title.
const string xml =
"<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'>" +
"<System><EventID>26</EventID></System>" +
"<EventData>" +
"<Data Name='updateCount'>3</Data>" +
"<Data Name='serviceGuid'>{8b24b027-1dee-babb-9a95-3517dfb9c552}</Data>" +
"</EventData></Event>";
var data = WindowsUpdateTracker.ParseEventData(xml);
Assert.Equal("3", data["updateCount"]);
Assert.Equal("{8b24b027-1dee-babb-9a95-3517dfb9c552}", data["serviceGuid"]);
Assert.False(data.ContainsKey("updateTitle"));
}
[Fact]
public void ParseEventData_MalformedXml_ReturnsEmpty_NeverThrows()
{
@@ -303,10 +628,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
var data = Data(s);
Assert.Equal("21=1,25=3", data["wuClientCensus"]);
Assert.Equal("200=2", data["updateOrchestratorCensus"]);
Assert.Equal("19,20,43,44", data["targetedEventIds"]);
Assert.Equal("25,26,31,41", data["targetedEventIds"]);
Assert.False(data.ContainsKey("censusTruncated"));
}
[Fact]
public void Census_ListsTheTargetedIdsOfEveryChannel()
{
// The host runs the census on the Operational instance; the evidence must still say
// the System-log install IDs were watched too.
var tracker = BuildCensusTracker(censusEnabled: true, osBuildChanged: true);
tracker.EmitChannelCensusIfBlind(otherChannelsEmitted: 0, otherTargetedEventIds: new[] { 44, 19, 20, 43 });
var s = Assert.Single(ByType(Constants.EventTypes.WindowsUpdateChannelCensus));
Assert.Equal("19,20,25,26,31,41,43,44", Data(s)["targetedEventIds"]);
}
[Fact]
public void Census_BuildUnchanged_EmitsNothing()
{
@@ -320,17 +658,27 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
[Fact]
public void Census_TargetedEventsWereEmittedThisRun_EmitsNothing()
{
// The watcher captured the update through its normal path — no blind spot to report.
// The watcher captured activity through its normal path — no blind spot to report.
// Any class counts: a counted Store update proves the channel is read as well.
var tracker = BuildCensusTracker(censusEnabled: true, osBuildChanged: true);
tracker.ProcessEvent(WindowsUpdateTracker.EventId_InstallSuccess, 4, recordId: 9, timeCreatedUtc: At,
updateTitle: "KB1", updateGuid: null, updateRevisionNumber: null, errorCode: null,
formattedDescription: null, isBackfill: false);
Emit(tracker, WindowsUpdateTracker.EventId_Downloaded, updateTitle: StoreApp, recordId: 9);
tracker.EmitChannelCensusIfBlind();
Assert.Empty(ByType(Constants.EventTypes.WindowsUpdateChannelCensus));
}
[Fact]
public void Census_AnotherChannelEmitted_EmitsNothing()
{
// The System-log watcher saw the install — the update was not missed.
var tracker = BuildCensusTracker(censusEnabled: true, osBuildChanged: true);
tracker.EmitChannelCensusIfBlind(otherChannelsEmitted: 1);
Assert.Empty(ByType(Constants.EventTypes.WindowsUpdateChannelCensus));
}
[Fact]
public void Census_DisabledByConfig_EmitsNothing()
{
@@ -153,6 +153,62 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
Assert.Equal(0, callCount);
}
[Fact]
public void Stopped_waiting_fires_callback_once_with_the_stage_still_non_terminal()
{
// D-310: the engine stopped waiting for the user without a verdict. The stage stays
// non-terminal, yet the agent must end the session — the callback carries the state.
using var rig = new Rig();
DecisionState? captured = null;
var callCount = 0;
var sut = rig.Build(DecisionState.CreateInitial("S1", "T1"),
onTerminal: state =>
{
Interlocked.Increment(ref callCount);
captured = state;
});
var builder = new DecisionStateBuilder(sut.CurrentState)
{
Stage = SessionStage.EspAccountSetup,
StepIndex = sut.CurrentState.StepIndex + 1,
StoppedWaitingReason = new SignalFact<string>(StoppedWaitingReasons.OobeUpdateNoSignIn, 1),
};
var stopped = builder.Build();
var step = new DecisionStep(stopped, new DecisionTransition(
stepIndex: stopped.StepIndex, sessionTraceOrdinal: 1, signalOrdinalRef: 1, occurredAtUtc: At,
trigger: "DeadlineFired:advisory_completion:OobeUpdateSignInWaitExpired",
fromStage: SessionStage.SessionStarted, toStage: SessionStage.EspAccountSetup,
taken: true, deadEndReason: null, reducerVersion: "2.0.0.0"), Array.Empty<DecisionEffect>());
sut.ApplyStep(step, Sig(1, At));
// A follow-up step must not fire it again.
sut.ApplyStep(StepToStage(sut.CurrentState, SessionStage.EspAccountSetup, signalOrdinal: 2, at: At.AddSeconds(1)),
Sig(2, At.AddSeconds(1)));
Assert.Equal(1, callCount);
Assert.Equal(SessionStage.EspAccountSetup, captured!.Stage);
Assert.Equal(StoppedWaitingReasons.OobeUpdateNoSignIn, captured.StoppedWaitingReason!.Value);
}
[Fact]
public void Stopped_waiting_on_load_does_not_refire_the_callback()
{
// The orchestrator ends a recovered stopped-waiting session itself at start.
using var rig = new Rig();
var callCount = 0;
var recovered = new DecisionStateBuilder(DecisionState.CreateInitial("S1", "T1"))
{
Stage = SessionStage.EspAccountSetup,
StoppedWaitingReason = new SignalFact<string>(StoppedWaitingReasons.OobeUpdateNoSignIn, 7),
}.Build();
var sut = rig.Build(recovered, onTerminal: _ => Interlocked.Increment(ref callCount));
sut.ApplyStep(StepToStage(sut.CurrentState, SessionStage.EspAccountSetup, signalOrdinal: 8, at: At), Sig(8, At));
Assert.Equal(0, callCount);
}
[Fact]
public void Terminal_callback_null_is_supported()
{
@@ -174,6 +174,40 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
sut.Stop();
}
[Fact]
public void Recovered_stopped_waiting_state_ends_the_session_at_start()
{
// D-310: a crash between "the engine stopped waiting" and the termination must not
// leave the session running — the restart ends it the same way, with the engine's reason.
using var rig = new Rig();
Directory.CreateDirectory(rig.StateDir);
var stoppedState = DecisionState.CreateInitial("S1", "T1")
.ToBuilder()
.WithStage(SessionStage.EspAccountSetup)
.WithStepIndex(9)
.Build()
.ToBuilder();
stoppedState.StoppedWaitingReason = new SignalFact<string>(StoppedWaitingReasons.OobeUpdateNoSignIn, 42);
new SnapshotPersistence(Path.Combine(rig.StateDir, "snapshot.json")).Save(stoppedState.Build());
var sut = rig.Build();
using var fired = new ManualResetEventSlim(false);
EnrollmentTerminatedEventArgs? captured = null;
sut.Terminated += (_, e) => { captured = e; fired.Set(); };
sut.Start();
Assert.True(fired.Wait(5000), "Terminated did not fire for the recovered stopped-waiting state.");
Assert.Equal(EnrollmentTerminationReason.StoppedWaiting, captured!.Reason);
Assert.Equal(EnrollmentTerminationOutcome.TimedOut, captured.Outcome);
Assert.Equal(StoppedWaitingReasons.OobeUpdateNoSignIn, captured.StopReason);
Assert.Equal(SessionStage.EspAccountSetup.ToString(), captured.StageName);
Assert.Contains("60 minutes", captured.Details);
sut.Stop();
}
// ================================================================= Sonderfall 2: Segment-Quarantine
[Fact]
@@ -0,0 +1,209 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.DecisionCore.State;
using AutopilotMonitor.Shared;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
{
/// <summary>
/// WindowsUpdateWatcherHost (D-310) — routes every configured WU EventID to the channel the
/// provider manifest writes it to, reports counted activity once, and feeds Windows/.NET update
/// steps to the decision engine unless <c>OobeUpdatePhaseMode</c> is Off. The event-log
/// watchers themselves are not armed here; the trackers' ProcessEvent seams drive the host.
/// </summary>
public sealed class WindowsUpdateWatcherHostTests : IDisposable
{
private static readonly DateTime At = new DateTime(2026, 10, 1, 9, 30, 0, DateTimeKind.Utc);
private const string Lcu = "2026-09 Cumulative Update for Windows 11, version 25H2 for x64-based Systems (KB5099999)";
private readonly TempDirectory _tmp = new TempDirectory();
private readonly FakeSignalIngressSink _sink = new FakeSignalIngressSink();
private long _nextRecordId = 1;
public void Dispose() => _tmp.Dispose();
private WindowsUpdateWatcherHost Build(bool osUpdateSignalsEnabled, bool servicingWatcherEnabled = true) =>
new WindowsUpdateWatcherHost(
sessionId: "sess-wuhost",
tenantId: "tenant-wuhost",
logger: new AgentLogger(_tmp.Path, AgentLogLevel.Info),
ingress: _sink,
clock: new VirtualClock(At),
targetedEventIds: new[] { 19, 20, 43, 44 },
backfillLookbackMinutes: 0,
stateDirectory: _tmp.Path,
operationalEventIds: new[] { 25, 26, 31, 41 },
servicingWatcherEnabled: servicingWatcherEnabled,
osUpdateSignalsEnabled: osUpdateSignalsEnabled);
private void Wu(WindowsUpdateTracker tracker, int eventId, string title, DateTime at) =>
tracker.ProcessEvent(eventId, 4, _nextRecordId++, at, title, updateGuid: null, updateRevisionNumber: null,
errorCode: null, formattedDescription: null, isBackfill: false);
private void Cbs(ServicingTracker tracker, int eventId, string client, DateTime at) =>
tracker.ProcessEvent(eventId, _nextRecordId++, at, "KB5099999", initialState: null, targetState: "Installed",
errorCode: "0x0", client: client, isBackfill: false);
private IReadOnlyList<FakeSignalIngressSink.PostedSignal> OsUpdateActivity() =>
_sink.Posted.Where(p => p.Kind == DecisionSignalKind.OsUpdateActivity).ToList();
private IReadOnlyList<FakeSignalIngressSink.PostedSignal> ByType(string eventType) =>
_sink.Posted.Where(p =>
p.Kind == DecisionSignalKind.InformationalEvent
&& p.Payload != null
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
&& et == eventType).ToList();
// ---------------------------------------------------------------- routing ----
[Fact]
public void RouteEventIds_Defaults_SplitByManifest()
{
var (system, operational) = WindowsUpdateWatcherHost.RouteEventIds(new[] { 19, 20, 43, 44 }, new[] { 25, 26, 31, 41 });
Assert.Equal(new[] { 19, 20, 43, 44 }, system);
Assert.Equal(new[] { 25, 26, 31, 41 }, operational);
}
[Fact]
public void RouteEventIds_MisplacedIds_EndUpOnTheirRealChannel()
{
// An old config that lists 41 under the targeted IDs, or 20 under the operational
// ones, must still watch the channel that carries them.
var (system, operational) = WindowsUpdateWatcherHost.RouteEventIds(new[] { 41, 19 }, new[] { 20, 41 });
Assert.Equal(new[] { 19, 20 }, system);
Assert.Equal(new[] { 41 }, operational);
}
[Fact]
public void RouteEventIds_NothingConfigured_WatchesNothing()
{
var (system, operational) = WindowsUpdateWatcherHost.RouteEventIds(null, Array.Empty<int>());
Assert.Empty(system);
Assert.Empty(operational);
}
[Fact]
public void Host_BuildsOneTrackerPerChannel_AndTheServicingTracker()
{
var host = Build(osUpdateSignalsEnabled: false);
Assert.Equal(WindowsUpdateTracker.SystemChannel, host.SystemTrackerForTest!.WatchedChannel);
Assert.Equal(new[] { 19, 20, 43, 44 }, host.SystemTrackerForTest.TargetedEventIds.OrderBy(id => id).ToArray());
Assert.Equal(WindowsUpdateTracker.Channel, host.OperationalTrackerForTest!.WatchedChannel);
Assert.Equal(new[] { 25, 26, 31, 41 }, host.OperationalTrackerForTest.TargetedEventIds.OrderBy(id => id).ToArray());
Assert.NotNull(host.ServicingTrackerForTest);
}
[Fact]
public void ServicingWatcherDisabled_BuildsNoServicingTracker()
{
Assert.Null(Build(osUpdateSignalsEnabled: false, servicingWatcherEnabled: false).ServicingTrackerForTest);
}
// ------------------------------------------------- engine signal behind flag ----
[Fact]
public void OsUpdateSignalsDisabled_ListsTheUpdate_ButPostsNoEngineSignal()
{
var host = Build(osUpdateSignalsEnabled: false);
Wu(host.SystemTrackerForTest!, WindowsUpdateTracker.EventId_InstallStarted, Lcu, At);
Cbs(host.ServicingTrackerForTest!, ServicingTracker.EventId_RebootRequired, "UpdateAgentLCU", At.AddMinutes(20));
Assert.Single(ByType(Constants.EventTypes.WindowsUpdateStarted));
Assert.Single(ByType(Constants.EventTypes.WindowsUpdateServicing));
Assert.Empty(OsUpdateActivity());
}
[Fact]
public void OsUpdateSignalsEnabled_PostsOsUpdateActivity_FromBothSources()
{
var host = Build(osUpdateSignalsEnabled: true);
var wuAt = DateTime.SpecifyKind(At, DateTimeKind.Unspecified);
Wu(host.SystemTrackerForTest!, WindowsUpdateTracker.EventId_InstallStarted, Lcu, wuAt);
Cbs(host.ServicingTrackerForTest!, ServicingTracker.EventId_RebootRequired, "UpdateAgentLCU", At.AddMinutes(20));
var signals = OsUpdateActivity();
Assert.Equal(2, signals.Count);
var wu = signals[0];
Assert.Equal(OsUpdateSteps.InstallStarted, wu.Payload![OsUpdatePayloadKeys.Step]);
Assert.Equal("wu", wu.Payload![OsUpdatePayloadKeys.Source]);
Assert.Equal(Lcu, wu.Payload![OsUpdatePayloadKeys.Update]);
Assert.Equal("WindowsUpdateWatcher", wu.SourceOrigin);
Assert.Equal(EvidenceKind.Raw, wu.Evidence.Kind);
Assert.Equal("os-update-activity:wu", wu.Evidence.Identifier);
Assert.Equal(At, wu.OccurredAtUtc);
Assert.Equal(DateTimeKind.Utc, wu.OccurredAtUtc.Kind);
Assert.Equal(1, wu.KindSchemaVersion);
var cbs = signals[1];
Assert.Equal(OsUpdateSteps.RebootRequired, cbs.Payload![OsUpdatePayloadKeys.Step]);
Assert.Equal("servicing", cbs.Payload![OsUpdatePayloadKeys.Source]);
Assert.Equal("KB5099999", cbs.Payload![OsUpdatePayloadKeys.Update]);
Assert.Equal("ServicingWatcher", cbs.SourceOrigin);
Assert.Equal("os-update-activity:servicing", cbs.Evidence.Identifier);
Assert.Equal(At.AddMinutes(20), cbs.OccurredAtUtc);
}
[Fact]
public void OsUpdateSignalsEnabled_NonOsActivity_NeverReachesTheEngine()
{
var host = Build(osUpdateSignalsEnabled: true);
Wu(host.SystemTrackerForTest!, WindowsUpdateTracker.EventId_InstallSuccess, "9NSTH9KHZDLQ-Microsoft.UI.Xaml.2.8", At);
Wu(host.OperationalTrackerForTest!, WindowsUpdateTracker.EventId_Downloaded,
"Security Intelligence Update for Microsoft Defender Antivirus - KB2267602 (Version 1.459.505.0) - Current Channel (Broad)", At);
Cbs(host.ServicingTrackerForTest!, ServicingTracker.EventId_StateReached, "CbsTask", At);
Assert.Empty(OsUpdateActivity());
Assert.Empty(_sink.Posted);
}
// ------------------------------------------------------- activity summary ----
[Fact]
public void ActivitySummary_IsEmittedOnce_WhenTheHostStops()
{
var host = Build(osUpdateSignalsEnabled: false);
Wu(host.SystemTrackerForTest!, WindowsUpdateTracker.EventId_InstallSuccess, "9NSTH9KHZDLQ-Microsoft.UI.Xaml.2.8", At);
Wu(host.SystemTrackerForTest!, WindowsUpdateTracker.EventId_InstallSuccess, "9NBLGGH4NNS1-Microsoft.DesktopAppInstaller", At);
Cbs(host.ServicingTrackerForTest!, ServicingTracker.EventId_InitiatingChanges, "CbsTask", At);
// The termination path stops the host explicitly and disposes it afterwards.
host.Stop();
host.Dispose();
var summary = Assert.Single(ByType(Constants.EventTypes.WindowsUpdateActivitySummary));
Assert.Equal("Debug", summary.Payload![SignalPayloadKeys.Severity]);
Assert.Contains("3 event(s)", summary.Payload![SignalPayloadKeys.Message]);
var data = (IReadOnlyDictionary<string, object>)summary.TypedPayload!;
Assert.Equal(2, data["store_19"]);
Assert.Equal(1, data["servicing_other_1"]);
}
[Fact]
public void ActivitySummary_NothingCounted_EmitsNothing()
{
var host = Build(osUpdateSignalsEnabled: false);
Wu(host.SystemTrackerForTest!, WindowsUpdateTracker.EventId_InstallSuccess, Lcu, At);
host.Stop();
Assert.Empty(ByType(Constants.EventTypes.WindowsUpdateActivitySummary));
}
}
}
@@ -102,6 +102,24 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Program
Assert.Equal(expected, sink.Posted[0].Payload![SignalPayloadKeys.HelloWaitTimeoutSeconds]);
}
[Theory]
[InlineData(null, "Shadow")] // no config value — the default mode
[InlineData("Shadow", "Shadow")]
[InlineData("active", "Active")] // case-insensitive
[InlineData("Off", "Off")]
[InlineData("Enforce", "Shadow")] // unknown — changes no decision, still measures
public void Oobe_update_phase_mode_is_stamped_at_every_start_normalized(string? configured, string expected)
{
// D-310: stamped every time, so a later Off/Shadow can take Active away mid-session.
using var tmp = new TempDirectory();
var sink = new FakeSignalIngressSink();
AutopilotMonitor.Agent.V2.Runtime.LifecycleEmitters.PostEnrollmentFactsObserved(
sink, HelloWaitTimeout.DefaultSeconds, NewLogger(tmp.Path), configured);
Assert.Equal(expected, sink.Posted[0].Payload![SignalPayloadKeys.OobeUpdatePhaseMode]);
}
[Fact]
public void Swallows_sink_exceptions()
{
@@ -933,6 +933,43 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Termination
Assert.Equal(EnrollmentTerminationOutcome.Failed.ToString(), (string)data["outcome"]);
}
[Fact]
public void Handle_propagates_stopped_waiting_reason_and_cause_in_agent_shutting_down_data()
{
// D-310: the engine stopped waiting for the user (no sign-in after the OOBE update).
// The backend classifies this shutdown honestly — it keys on the reason tag.
using var rig = new Rig();
rig.State = new DecisionStateBuilder(DecisionState.CreateInitial("S1", "T1")) { Stage = SessionStage.EspAccountSetup }.Build();
rig.Build().Handle(sender: null!, new EnrollmentTerminatedEventArgs(
EnrollmentTerminationReason.StoppedWaiting,
EnrollmentTerminationOutcome.TimedOut,
SessionStage.EspAccountSetup.ToString(),
EndUtc,
details: StoppedWaitingReasons.Describe(StoppedWaitingReasons.OobeUpdateNoSignIn),
stopReason: StoppedWaitingReasons.OobeUpdateNoSignIn));
var data = rig.DataOf(Constants.EventTypes.AgentShuttingDown);
Assert.NotNull(data);
Assert.Equal("stopped_waiting", (string)data!["reason"]);
Assert.Equal(StoppedWaitingReasons.OobeUpdateNoSignIn, (string)data["stopReason"]);
Assert.Equal(EnrollmentTerminationOutcome.TimedOut.ToString(), (string)data["outcome"]);
}
[Fact]
public void Handle_omits_stop_reason_for_other_shutdowns()
{
using var rig = new Rig();
rig.State = new DecisionStateBuilder(DecisionState.CreateInitial("S1", "T1")) { Stage = SessionStage.AwaitingHello }.Build();
rig.Build().Handle(sender: null!,
Args(EnrollmentTerminationReason.MaxLifetimeExceeded, EnrollmentTerminationOutcome.TimedOut, SessionStage.AwaitingHello));
var data = rig.DataOf(Constants.EventTypes.AgentShuttingDown);
Assert.NotNull(data);
Assert.False(data!.ContainsKey("stopReason"));
}
[Fact]
public void Handle_includes_uptime_and_agent_version_in_agent_shutting_down()
{
@@ -0,0 +1,148 @@
using System;
using System.Collections.Generic;
using System.IO;
using AutopilotMonitor.Agent.V2.Core.Logging;
using Newtonsoft.Json;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
/// <summary>
/// Cross-restart dedup for event-log watchers that backfill a channel on start. RecordIds are
/// per channel, so every watched channel owns one instance with its own state file.
/// <para>
/// Two boundaries, deliberately separate: the restart watermark is the highest RecordId a
/// PRIOR run emitted (loaded once, immutable for this run); the seen-set holds what THIS run
/// processed. A single high-water mark would be wrong because the live watcher is armed before
/// the backfill runs: a live record with a higher RecordId can arrive first and would then
/// make the backfill skip every older, never-emitted record.
/// </para>
/// </summary>
internal sealed class EventRecordWatermark
{
private readonly string _stateFilePath;
private readonly AgentLogger _logger;
private readonly string _label;
private readonly object _lock = new object();
private readonly HashSet<long> _seenThisRun = new HashSet<long>();
private long _restartWatermark = -1;
private long _maxEmittedRecordId = -1;
/// <param name="stateDirectory">Directory for the state file; null keeps the dedup in memory only.</param>
/// <param name="stateFileName">File name inside <paramref name="stateDirectory"/>.</param>
/// <param name="label">Watcher name for log lines.</param>
public EventRecordWatermark(string stateDirectory, string stateFileName, AgentLogger logger, string label)
{
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_label = label ?? "EventLog";
_stateFilePath = string.IsNullOrEmpty(stateDirectory) || string.IsNullOrEmpty(stateFileName)
? null
: Path.Combine(Environment.ExpandEnvironmentVariables(stateDirectory), stateFileName);
}
/// <summary>Highest RecordId emitted by a prior run; -1 when none was persisted.</summary>
public long RestartWatermark
{
get { lock (_lock) return _restartWatermark; }
}
/// <summary>True when the record was already emitted, by a prior run or by this one.</summary>
public bool IsAlreadyProcessed(long recordId)
{
if (recordId < 0) return false; // no RecordId → cannot dedup, let it through
lock (_lock)
{
return recordId <= _restartWatermark || _seenThisRun.Contains(recordId);
}
}
/// <summary>
/// Atomically claims <paramref name="recordId"/> for this run. True on first sight (the
/// caller processes it), false when it was already processed. Records without a RecordId
/// (-1) are always claimed, never tracked. Claiming persists nothing — only
/// <see cref="MarkEmitted"/> does, so a record that is counted instead of emitted costs no
/// state-file write (CBS maintenance writes hundreds of servicing records in seconds).
/// </summary>
public bool TryClaim(long recordId)
{
if (recordId < 0) return true;
lock (_lock)
{
return recordId > _restartWatermark && _seenThisRun.Add(recordId);
}
}
/// <summary>
/// Persists <paramref name="recordId"/> when it raises the emitted maximum, so a restarted
/// agent's backfill skips it. Called after the record's event and engine signal were
/// handed to the ingress: a crash before that re-emits the record instead of losing it.
/// The ingress makes them durable only when its worker appends them to the SignalLog —
/// normally milliseconds later; a process killed inside that window (e.g. by a reboot)
/// loses the record, and the backfill does not bring it back. Counted records above the
/// persisted maximum are counted again after a restart, which only touches the activity
/// summary.
/// </summary>
public void MarkEmitted(long recordId)
{
if (recordId < 0) return;
long toPersist = -1;
lock (_lock)
{
if (recordId > _maxEmittedRecordId)
{
_maxEmittedRecordId = recordId;
toPersist = recordId;
}
}
if (toPersist >= 0) Persist(toPersist);
}
public void Load()
{
if (_stateFilePath == null || !File.Exists(_stateFilePath)) return;
try
{
var state = JsonConvert.DeserializeObject<WatermarkState>(File.ReadAllText(_stateFilePath));
if (state == null) return;
lock (_lock)
{
_restartWatermark = state.LastRecordId;
_maxEmittedRecordId = state.LastRecordId;
}
_logger.Info($"{_label} watermark loaded: lastRecordId={state.LastRecordId} (persisted {state.PersistedUtc:O})");
}
catch (Exception ex)
{
_logger.Warning($"Failed to load {_label} watermark: {ex.Message}");
}
}
private void Persist(long recordId)
{
if (_stateFilePath == null) return;
try
{
Directory.CreateDirectory(Path.GetDirectoryName(_stateFilePath));
var json = JsonConvert.SerializeObject(
new WatermarkState { LastRecordId = recordId, PersistedUtc = DateTime.UtcNow },
Formatting.Indented);
var tempPath = _stateFilePath + ".tmp";
File.WriteAllText(tempPath, json);
File.Copy(tempPath, _stateFilePath, overwrite: true);
try { File.Delete(tempPath); } catch { }
}
catch (Exception ex)
{
_logger.Warning($"Failed to persist {_label} watermark: {ex.Message}");
}
}
internal sealed class WatermarkState
{
public long LastRecordId { get; set; }
public DateTime PersistedUtc { get; set; }
}
}
}
@@ -0,0 +1,157 @@
using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using Microsoft.Win32;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
/// <summary>
/// Reads the registry state behind the OOBE quality update (D-310, S4/S5): the NDUP state
/// Windows keeps while the update page runs, and both policy paths of the ESP setting "Install
/// Windows quality updates". <see cref="OobeUpdateTelemetry"/> logs it and reports it as
/// <c>oobe_update_state</c>. Read-only; the values are update-state flags and policy values,
/// no user data.
/// </summary>
internal static class OobeUpdateRegistrySnapshot
{
/// <summary>The snapshot's keys, each with the alias it carries in <c>oobe_update_state</c>.</summary>
internal static readonly (string Alias, string Path)[] Keys =
{
("setupOobeNdup", @"SOFTWARE\Microsoft\Windows\CurrentVersion\Setup\OOBE\NDUP"),
("setupOobeNdupUpdates", @"SOFTWARE\Microsoft\Windows\CurrentVersion\Setup\OOBE\NDUP\Updates"),
("ndup", @"SOFTWARE\Microsoft\Windows\CurrentVersion\NDUP"),
("ndupUpdates", @"SOFTWARE\Microsoft\Windows\CurrentVersion\NDUP\Updates"),
("espSetupPolicy", @"SOFTWARE\Microsoft\Windows\Autopilot\EnrollmentStatusTracking\Device\Setup\Policy"),
("policyManagerSystem", PolicyManagerSystemKey),
("policiesOobe", @"SOFTWARE\Policies\Microsoft\Windows\OOBE"),
};
// Carries every System policy of the device; only the OOBE ones belong in this snapshot.
internal const string PolicyManagerSystemKey = @"SOFTWARE\Microsoft\PolicyManager\current\device\System";
internal const int MaxEntries = 30;
internal const int MaxValueLength = 200;
/// <summary>One key of the snapshot: whether it exists, whether it holds values, and its line.</summary>
internal sealed class KeyState
{
public KeyState(string alias, string path, bool exists, bool hasValues, string line)
{
Alias = alias;
Path = path;
Exists = exists;
HasValues = hasValues;
Line = line;
}
public string Alias { get; }
public string Path { get; }
public bool Exists { get; }
/// <summary>
/// True when the key holds at least one value — for <see cref="PolicyManagerSystemKey"/>
/// one with OOBE in its name. Subkeys alone are no values.
/// </summary>
public bool HasValues { get; }
/// <summary><see cref="FormatEntries"/> plus the subkey names; null when the key is absent.</summary>
public string Line { get; }
}
/// <summary>Reads every key from the 64-bit view. Throws on a registry failure; the caller is fail-soft.</summary>
public static IReadOnlyList<KeyState> Read()
{
var states = new List<KeyState>(Keys.Length);
using (var hklm = RegistryKey.OpenBaseKey(RegistryHive.LocalMachine, RegistryView.Registry64))
{
foreach (var (alias, path) in Keys)
{
using (var key = hklm.OpenSubKey(path))
{
if (key == null)
{
states.Add(new KeyState(alias, path, exists: false, hasValues: false, line: null));
continue;
}
var entries = key.GetValueNames()
.Select(name => new KeyValuePair<string, object>(name, key.GetValue(name)))
.ToList();
states.Add(Describe(alias, path, entries, key.GetSubKeyNames()));
}
}
}
return states;
}
/// <summary>The state of a present key from its values and subkey names.</summary>
internal static KeyState Describe(string alias, string path, IReadOnlyList<KeyValuePair<string, object>> entries, string[] subKeys)
{
var oobeOnly = string.Equals(path, PolicyManagerSystemKey, StringComparison.OrdinalIgnoreCase);
var line = FormatEntries(entries, oobeOnly);
if (subKeys != null && subKeys.Length > 0)
line += $" | subkeys: {string.Join(", ", subKeys.Take(MaxEntries))}";
return new KeyState(alias, path, exists: true,
hasValues: entries.Any(e => !oobeOnly || IsOobeName(e.Key)), line: line);
}
/// <summary>One agent.log line per present key, one naming the absent keys.</summary>
public static void Log(AgentLogger logger, string moment, IReadOnlyList<KeyState> states)
{
if (logger == null || states == null) return;
var absent = new List<string>();
foreach (var state in states)
{
if (!state.Exists)
{
absent.Add(state.Path);
continue;
}
logger.Info($"OOBE update registry ({moment}): HKLM\\{state.Path}: {state.Line}");
}
if (absent.Count > 0)
logger.Info($"OOBE update registry ({moment}): absent: {string.Join(", ", absent)}");
}
/// <summary>
/// <c>name=value</c> pairs ordered by name, bounded in count and length.
/// <paramref name="oobeOnly"/> keeps only names that mention OOBE.
/// </summary>
internal static string FormatEntries(IEnumerable<KeyValuePair<string, object>> entries, bool oobeOnly)
{
var selected = entries
.Where(e => !oobeOnly || IsOobeName(e.Key))
.OrderBy(e => e.Key, StringComparer.OrdinalIgnoreCase)
.ToList();
if (selected.Count == 0) return oobeOnly ? "(no OOBE values)" : "(no values)";
var shown = selected.Take(MaxEntries)
.Select(e => $"{(e.Key.Length == 0 ? "(default)" : e.Key)}={FormatValue(e.Value)}");
var line = string.Join(", ", shown);
return selected.Count > MaxEntries ? $"{line}, … (+{selected.Count - MaxEntries} more)" : line;
}
internal static string FormatValue(object value)
{
string text;
switch (value)
{
case null: text = "(null)"; break;
case int i: text = i.ToString(CultureInfo.InvariantCulture); break;
case long l: text = l.ToString(CultureInfo.InvariantCulture); break;
case string s: text = s; break;
case string[] multi: text = string.Join("|", multi); break;
case byte[] bytes: text = $"(binary {bytes.Length} bytes)"; break;
default: text = Convert.ToString(value, CultureInfo.InvariantCulture) ?? string.Empty; break;
}
return text.Length > MaxValueLength ? text.Substring(0, MaxValueLength) + "…" : text;
}
private static bool IsOobeName(string name) =>
name != null && name.IndexOf("OOBE", StringComparison.OrdinalIgnoreCase) >= 0;
}
}
@@ -0,0 +1,245 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
/// <summary>
/// One CloudExperienceHost record of the OOBE update page, parsed by
/// <see cref="ShellCoreTracker.ParseUpdatePageRecord"/>.
/// </summary>
internal sealed class OobeUpdatePageRecord
{
public const string PageStarted = "page_started";
public const string PageStopped = "page_stopped";
public const string EventName = "event_name";
public OobeUpdatePageRecord(string cxhEvent, int windowsEventId, string page = null, string result = null, string name = null, string value = null)
{
CxhEvent = cxhEvent;
WindowsEventId = windowsEventId;
Page = page;
Result = result;
Name = name;
Value = value;
}
/// <summary><see cref="PageStarted"/>, <see cref="PageStopped"/> or <see cref="EventName"/>.</summary>
public string CxhEvent { get; }
public int WindowsEventId { get; }
/// <summary>The page's CXID (start), or the page a stop is attributed to.</summary>
public string Page { get; }
/// <summary>The page result of a stop.</summary>
public string Result { get; }
/// <summary>The event name of a 62406/62407 record.</summary>
public string Name { get; }
/// <summary>A 62407 value of a safe form (boolean, integer, HRESULT); null otherwise.</summary>
public string Value { get; }
}
/// <summary>
/// Measurement telemetry of the OOBE quality update (D-310): the update page's
/// CloudExperienceHost records as <c>oobe_update_page</c> and the registry state behind it as
/// <c>oobe_update_state</c>. Debug, bounded per agent run, no decisions. Owned by
/// <see cref="ShellCoreTracker"/>, which observes the page.
/// </summary>
internal sealed class OobeUpdateTelemetry
{
internal const int MaxPageEventsPerRun = 50;
internal const int MaxPageEventsPerKey = 3;
internal const int MaxStateEventsPerRun = 6;
internal const int MaxStateLineLength = 1000;
/// <summary>When a registry snapshot is taken (<c>moment</c> of <c>oobe_update_state</c>).</summary>
internal static class Moments
{
public const string AgentStart = "agent_start";
public const string UpdatePageStarted = "update_page_started";
public const string UpdatePageStopped = "update_page_stopped";
public const string AgentStop = "agent_stop";
}
private const string Source = "ShellCoreTracker";
private readonly string _sessionId;
private readonly string _tenantId;
private readonly InformationalEventPost _post;
private readonly AgentLogger _logger;
private readonly Func<IReadOnlyList<OobeUpdateRegistrySnapshot.KeyState>> _readRegistry;
private readonly object _lock = new object();
private readonly Dictionary<string, int> _pageEventsPerKey = new Dictionary<string, int>(StringComparer.Ordinal);
private int _pageEvents;
private bool _pageCapLogged;
private int _stateEvents;
private string _lastStateContent;
public OobeUpdateTelemetry(
string sessionId,
string tenantId,
InformationalEventPost post,
AgentLogger logger,
Func<IReadOnlyList<OobeUpdateRegistrySnapshot.KeyState>> readRegistry)
{
_sessionId = sessionId ?? throw new ArgumentNullException(nameof(sessionId));
_tenantId = tenantId ?? throw new ArgumentNullException(nameof(tenantId));
_post = post ?? throw new ArgumentNullException(nameof(post));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_readRegistry = readRegistry ?? throw new ArgumentNullException(nameof(readRegistry));
}
/// <summary>
/// One <c>oobe_update_page</c> per update-page record, at most
/// <see cref="MaxPageEventsPerKey"/> per page/name and <see cref="MaxPageEventsPerRun"/>
/// per run — a page that logs the same name in a loop cannot flood the session.
/// </summary>
public void ReportPage(OobeUpdatePageRecord record, DateTime occurredAtUtc, bool isBackfill)
{
if (record == null) return;
var key = $"{record.CxhEvent}:{record.Page ?? record.Name}";
int occurrence;
lock (_lock)
{
if (_pageEvents >= MaxPageEventsPerRun)
{
if (!_pageCapLogged)
{
_pageCapLogged = true;
_logger.Info($"OOBE update page telemetry: {MaxPageEventsPerRun} events reached, the rest of this run stays in agent.log");
}
return;
}
_pageEventsPerKey.TryGetValue(key, out occurrence);
if (occurrence >= MaxPageEventsPerKey) return;
occurrence++;
_pageEventsPerKey[key] = occurrence;
_pageEvents++;
}
var data = new Dictionary<string, object>
{
{ "cxhEvent", record.CxhEvent },
{ "windowsEventId", record.WindowsEventId },
{ "eventTime", occurredAtUtc.ToString("o") },
{ "backfill", isBackfill },
{ "occurrence", occurrence },
};
if (record.Page != null) data["page"] = record.Page;
if (record.Result != null) data["result"] = record.Result;
if (record.Name != null) data["name"] = record.Name;
if (record.Value != null) data["value"] = record.Value;
try
{
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
Timestamp = occurredAtUtc,
EventType = Constants.EventTypes.OobeUpdatePage,
Severity = EventSeverity.Debug,
Source = Source,
Phase = EnrollmentPhase.Unknown,
Message = DescribePage(record),
Data = data,
ImmediateUpload = false,
});
}
catch (Exception ex)
{
_logger.Debug($"OOBE update page telemetry failed: {ex.Message}");
}
}
/// <summary>
/// Reads the OOBE update registry state, logs it and reports it as <c>oobe_update_state</c>
/// — only when at least one key holds values and the content changed since the last report
/// of this run, at most <see cref="MaxStateEventsPerRun"/> times. Fail-soft.
/// </summary>
public void ReportState(string moment, string page = null)
{
var label = page == null ? moment : $"{moment} {page}";
IReadOnlyList<OobeUpdateRegistrySnapshot.KeyState> states;
try
{
states = _readRegistry() ?? Array.Empty<OobeUpdateRegistrySnapshot.KeyState>();
}
catch (Exception ex)
{
_logger.Debug($"OOBE update registry ({label}): snapshot failed: {ex.Message}");
return;
}
OobeUpdateRegistrySnapshot.Log(_logger, label, states);
var withValues = states.Where(s => s.HasValues).ToList();
if (withValues.Count == 0) return;
var content = string.Join("\n", withValues.Select(s => $"{s.Alias}={s.Line}"));
lock (_lock)
{
if (string.Equals(content, _lastStateContent, StringComparison.Ordinal)) return;
if (_stateEvents >= MaxStateEventsPerRun) return;
_lastStateContent = content;
_stateEvents++;
}
var data = new Dictionary<string, object>
{
{ "moment", moment },
{ "keysWithValues", withValues.Count },
};
if (page != null) data["page"] = page;
foreach (var state in withValues)
{
data[state.Alias] = state.Line.Length > MaxStateLineLength
? state.Line.Substring(0, MaxStateLineLength) + "…"
: state.Line;
}
try
{
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
EventType = Constants.EventTypes.OobeUpdateState,
Severity = EventSeverity.Debug,
Source = Source,
Phase = EnrollmentPhase.Unknown,
Message = $"OOBE update registry state ({label}): {withValues.Count} key(s) with values",
Data = data,
ImmediateUpload = false,
});
}
catch (Exception ex)
{
_logger.Debug($"OOBE update state telemetry failed: {ex.Message}");
}
}
private static string DescribePage(OobeUpdatePageRecord record)
{
switch (record.CxhEvent)
{
case OobeUpdatePageRecord.PageStarted:
return $"OOBE update page started: {record.Page}";
case OobeUpdatePageRecord.PageStopped:
return $"OOBE update page stopped: {record.Page} (result: {record.Result})";
default:
return record.Value == null
? $"OOBE update page event: {record.Name}"
: $"OOBE update page event: {record.Name} = {record.Value}";
}
}
}
}
@@ -0,0 +1,373 @@
using System;
using System.Collections.Generic;
using System.Diagnostics.Eventing.Reader;
using System.Linq;
using System.Xml.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.DecisionCore.State;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
/// <summary>
/// Watches the CBS servicing steps of OS updates in the <c>Setup</c> log
/// (<c>Microsoft-Windows-Servicing</c>, D-310): 1 initiating changes (current → target state),
/// 2 target state reached, 3 failed, 4 a restart is needed to finish, 6 failed and partially
/// installed. The Windows Update client logs only the start of an installation; these steps
/// show the long part of it — staging, installing, "reboot required" right before the restart.
/// <para>
/// Only packages of the OS update clients (<c>UpdateAgentLCU</c> for the cumulative update,
/// <c>WindowsUpdateAgent</c>) are emitted as <c>windows_update_servicing</c> and reported to
/// the decision engine; language packs, features on demand and DISM operations started by
/// apps are counted into the host's <see cref="UpdateActivityTally"/>.
/// </para>
/// </summary>
internal sealed class ServicingTracker : IDisposable
{
internal const string Channel = "Setup";
internal const string ProviderName = "Microsoft-Windows-Servicing";
internal const int EventId_InitiatingChanges = 1;
internal const int EventId_StateReached = 2;
internal const int EventId_ChangeFailed = 3;
internal const int EventId_RebootRequired = 4;
internal const int EventId_PartiallyInstalled = 6;
internal static readonly int[] TargetedEventIds =
{
EventId_InitiatingChanges, EventId_StateReached, EventId_ChangeFailed, EventId_RebootRequired, EventId_PartiallyInstalled,
};
internal const string WatermarkStateFileName = "servicing-watermark.json";
// CBS client ids of the Windows Update installers (seen on 24H2/25H2: UpdateAgentLCU).
private static readonly HashSet<string> OsUpdateClients = new HashSet<string>(StringComparer.OrdinalIgnoreCase)
{
"UpdateAgentLCU",
"WindowsUpdateAgent",
};
private readonly AgentLogger _logger;
private readonly string _sessionId;
private readonly string _tenantId;
private readonly InformationalEventPost _post;
private readonly bool _backfillEnabled;
private readonly int _backfillLookbackMinutes;
private readonly UpdateActivityTally _tally;
private readonly Action<string, string, DateTime> _onOsUpdateActivity;
private readonly EventRecordWatermark _watermark;
private EventLogWatcher _watcher;
public ServicingTracker(
string sessionId,
string tenantId,
InformationalEventPost post,
AgentLogger logger,
bool backfillEnabled = true,
int backfillLookbackMinutes = 60,
string stateDirectory = null,
UpdateActivityTally tally = null,
Action<string, string, DateTime> onOsUpdateActivity = null)
{
_sessionId = sessionId ?? throw new ArgumentNullException(nameof(sessionId));
_tenantId = tenantId ?? throw new ArgumentNullException(nameof(tenantId));
_post = post ?? throw new ArgumentNullException(nameof(post));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_backfillEnabled = backfillEnabled;
_backfillLookbackMinutes = backfillLookbackMinutes;
_tally = tally;
_onOsUpdateActivity = onOsUpdateActivity;
_watermark = new EventRecordWatermark(stateDirectory, WatermarkStateFileName, logger, "Servicing");
}
public void Start()
{
_watermark.Load();
StartWatcher();
if (_backfillEnabled && _backfillLookbackMinutes > 0)
BackfillRecentEvents();
else
_logger.Info("Servicing backfill disabled by config");
}
public void Stop()
{
if (_watcher == null) return;
try
{
_watcher.Enabled = false;
_watcher.Dispose();
_logger.Info("Servicing watcher stopped");
}
catch (Exception ex)
{
_logger.Error("Error stopping Servicing watcher", ex);
}
finally
{
_watcher = null;
}
}
public void Dispose() => Stop();
internal void LoadWatermark() => _watermark.Load();
private static HashSet<int> TargetedSet => new HashSet<int>(TargetedEventIds);
private void StartWatcher()
{
try
{
var query = new EventLogQuery(Channel, PathType.LogName,
WindowsUpdateTracker.BuildXPath(TargetedSet, ProviderName));
_watcher = new EventLogWatcher(query);
_watcher.EventRecordWritten += OnEventRecordWritten;
_watcher.Enabled = true;
_logger.Info($"Servicing watcher started: {Channel} ({ProviderName}, targetedIds={string.Join(",", TargetedEventIds)})");
}
catch (EventLogNotFoundException)
{
_logger.Warning($"Servicing event log not found: {Channel}");
}
catch (UnauthorizedAccessException ex)
{
_logger.Warning($"Servicing watcher access denied for {Channel}: {ex.Message}");
}
catch (Exception ex)
{
_logger.Error($"Failed to start Servicing watcher for {Channel}", ex);
CollectorDegradationReporter.Report(_post, _sessionId, _tenantId,
collectorName: "ServicingTracker", reason: $"watcher_arm_failed:{Channel}", ex: ex);
}
}
private void OnEventRecordWritten(object sender, EventRecordWrittenEventArgs e)
{
if (e.EventRecord == null) return;
try
{
ProcessRecord(e.EventRecord, isBackfill: false);
}
catch (Exception ex)
{
_logger.Error("Error processing Servicing event", ex);
}
}
private void ProcessRecord(EventRecord record, bool isBackfill)
{
var recordId = record.RecordId ?? -1;
if (_watermark.IsAlreadyProcessed(recordId)) return;
string xml = null;
try { xml = record.ToXml(); }
catch { }
var fields = ParseUserData(xml);
fields.TryGetValue("PackageIdentifier", out var package);
fields.TryGetValue("InitialPackageStateTextized", out var initialState);
fields.TryGetValue("IntendedPackageStateTextized", out var targetState);
fields.TryGetValue("ErrorCode", out var errorCode);
fields.TryGetValue("Client", out var client);
ProcessEvent(
eventId: record.Id,
recordId: recordId,
timeCreatedUtc: record.TimeCreated?.ToUniversalTime(),
package: package,
initialState: initialState,
targetState: targetState,
errorCode: errorCode,
client: client,
isBackfill: isBackfill);
}
/// <summary>Test seam — drives the processing without a Windows-only <see cref="EventRecord"/>.</summary>
internal void ProcessEvent(
int eventId,
long recordId,
DateTime? timeCreatedUtc,
string package,
string initialState,
string targetState,
string errorCode,
string client,
bool isBackfill)
{
if (!_watermark.TryClaim(recordId)) return;
if (string.IsNullOrEmpty(client) || !OsUpdateClients.Contains(client))
{
_tally?.Increment($"servicing_other_{eventId}");
return;
}
var pkg = string.IsNullOrEmpty(package) ? "(unknown package)" : package;
var target = string.IsNullOrEmpty(targetState) ? "?" : targetState;
string step;
string osStep;
string message;
var severity = EventSeverity.Info;
string hresultHex = null;
string hresultSymbol = null;
if (WindowsUpdateTracker.TryNormalizeHResult(errorCode, out var hresultValue, out var hex) && hresultValue != 0)
{
hresultHex = hex;
hresultSymbol = WindowsUpdateTracker.ResolveHResultSymbol(hresultValue);
}
switch (eventId)
{
case EventId_InitiatingChanges:
step = "initiating";
osStep = IsState(targetState, "Installed") ? OsUpdateSteps.InstallStarted
: IsState(targetState, "Staged") ? OsUpdateSteps.StagingStarted
: null;
message = $"Servicing started for {pkg}: {(string.IsNullOrEmpty(initialState) ? "?" : initialState)} → {target}";
break;
case EventId_StateReached:
step = "state_reached";
osStep = IsState(targetState, "Installed") ? OsUpdateSteps.Installed
: IsState(targetState, "Staged") ? OsUpdateSteps.Staged
: null;
message = $"Servicing finished for {pkg}: now {target}";
break;
case EventId_RebootRequired:
step = "reboot_required";
osStep = OsUpdateSteps.RebootRequired;
message = $"Servicing needs a restart to finish {pkg} → {target}";
break;
case EventId_ChangeFailed:
case EventId_PartiallyInstalled:
step = "failed";
osStep = OsUpdateSteps.Failed;
severity = EventSeverity.Warning;
message = $"Servicing FAILED for {pkg} → {target}" +
(hresultHex != null ? $" ({hresultHex} {hresultSymbol})" : string.Empty) +
(eventId == EventId_PartiallyInstalled ? ", package is partially installed" : string.Empty);
break;
default:
return;
}
var data = new Dictionary<string, object>
{
{ "servicingEventId", eventId },
{ "step", step },
{ "package", pkg },
{ "client", client },
{ "backfilled", isBackfill },
};
if (!string.IsNullOrEmpty(initialState)) data["initialState"] = initialState;
if (!string.IsNullOrEmpty(targetState)) data["targetState"] = targetState;
if (hresultHex != null)
{
data["hresult"] = hresultHex;
data["hresultSymbol"] = hresultSymbol;
}
if (recordId >= 0) data["recordId"] = recordId;
if (timeCreatedUtc.HasValue) data["timeCreated"] = timeCreatedUtc.Value.ToString("o");
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
Timestamp = timeCreatedUtc ?? DateTime.UtcNow,
EventType = Constants.EventTypes.WindowsUpdateServicing,
Severity = severity,
Source = "ServicingWatcher",
Phase = EnrollmentPhase.Unknown,
Message = message,
Data = data,
ImmediateUpload = false,
});
if (osStep != null && _onOsUpdateActivity != null)
{
try { _onOsUpdateActivity(osStep, pkg, timeCreatedUtc ?? DateTime.UtcNow); }
catch (Exception ex) { _logger.Warning($"Servicing: OS update activity callback failed: {ex.Message}"); }
}
// Last: everything this record produced has been handed to the ingress first.
_watermark.MarkEmitted(recordId);
}
private static bool IsState(string state, string expected) =>
string.Equals(state, expected, StringComparison.OrdinalIgnoreCase);
/// <summary>
/// Child elements of the first element under <c>UserData</c>
/// (<c>CbsPackageInitiateChanges</c> / <c>CbsPackageChangeState</c>), keyed by local name,
/// case-insensitively. Empty on null/malformed XML — never throws.
/// </summary>
internal static Dictionary<string, string> ParseUserData(string xml)
{
var result = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
if (string.IsNullOrEmpty(xml)) return result;
try
{
var doc = XDocument.Parse(xml);
var userData = doc.Descendants().FirstOrDefault(e => e.Name.LocalName == "UserData");
var payload = userData?.Elements().FirstOrDefault();
if (payload == null) return result;
foreach (var el in payload.Elements())
{
if (!result.ContainsKey(el.Name.LocalName))
result[el.Name.LocalName] = el.Value;
}
}
catch
{
// Malformed / unexpected XML — best effort.
}
return result;
}
private void BackfillRecentEvents()
{
try
{
var lookbackMs = (long)_backfillLookbackMinutes * 60 * 1000;
var xpath = WindowsUpdateTracker.BuildXPath(TargetedSet, ProviderName, lookbackMs);
_logger.Info($"Servicing backfill: scanning {Channel} (lookback={_backfillLookbackMinutes}min, restartWatermark={_watermark.RestartWatermark})");
int processed = 0;
using (var reader = new EventLogReader(new EventLogQuery(Channel, PathType.LogName, xpath)))
{
EventRecord record;
while ((record = reader.ReadEvent()) != null)
{
using (record)
{
ProcessRecord(record, isBackfill: true);
processed++;
}
}
}
if (processed > 0)
_logger.Info($"Servicing backfill: scanned {processed} event(s)");
}
catch (EventLogNotFoundException)
{
_logger.Warning($"Servicing event log not found during backfill: {Channel}");
}
catch (UnauthorizedAccessException ex)
{
_logger.Warning($"Servicing backfill access denied: {ex.Message}");
}
catch (Exception ex)
{
_logger.Warning($"Servicing backfill failed: {ex.Message}");
}
}
}
}
@@ -2,6 +2,7 @@ using System;
using System.Collections.Generic;
using System.Diagnostics.Eventing.Reader;
using System.Text.RegularExpressions;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Shared;
@@ -31,9 +32,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
internal const string ShellCoreEventLogChannel = "Microsoft-Windows-Shell-Core/Operational";
internal const int EventId_ShellCore_WebAppStarted = 62404;
internal const int EventId_ShellCore_WebAppStopped = 62405;
internal const int EventId_ShellCore_WebAppEventName = 62406;
internal const int EventId_ShellCore_WebAppEvent = 62407;
internal const int BackfillLookbackMinutes = 5;
// 62405 (page stopped) and 62406 (event with a name only) feed the CloudExperienceHost
// navigation breadcrumbs in agent.log and the OOBE update page telemetry (D-310), nothing else.
private const string WatchedEventIdsXPath =
"(EventID=62404 or EventID=62405 or EventID=62406 or EventID=62407)";
/// <summary>
/// Upper bound for the caller-supplied backfill lookback. Matches the agent's max
/// lifetime (360 min) — a window wider than the agent can ever live would only widen
@@ -45,22 +53,45 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
private static readonly HashSet<int> TrackedShellCoreEventIds = new HashSet<int>
{
EventId_ShellCore_WebAppStarted,
EventId_ShellCore_WebAppStopped,
EventId_ShellCore_WebAppEventName,
EventId_ShellCore_WebAppEvent
};
private static readonly Regex EspExitingPattern = new Regex(
@"OOBE_ESP.*Exiting", RegexOptions.IgnoreCase | RegexOptions.Compiled);
// Manifest templates: 62404 "... started. CXID: '%1'.", 62405 "... stopped. Result: '%1'.",
// 62406 "... Event 1. Name: '%1'.", 62407 "... Event 2. Name: '%1', Value: '%2'.". A 62407
// value can carry URLs with tenant ids — a value leaves the tracker only in a form that
// cannot (SafeValuePattern: a boolean, an integer, an HRESULT).
private static readonly Regex CxidPattern = new Regex(
@"CXID:\s*'([^']*)'", RegexOptions.CultureInvariant | RegexOptions.Compiled, TimeSpan.FromMilliseconds(100));
private static readonly Regex ResultPattern = new Regex(
@"Result:\s*'([^']*)'", RegexOptions.CultureInvariant | RegexOptions.Compiled, TimeSpan.FromMilliseconds(100));
private static readonly Regex NamePattern = new Regex(
@"Name:\s*'([^']*)'", RegexOptions.CultureInvariant | RegexOptions.Compiled, TimeSpan.FromMilliseconds(100));
private static readonly Regex ValuePattern = new Regex(
@"Value:\s*'([^']*)'", RegexOptions.CultureInvariant | RegexOptions.Compiled, TimeSpan.FromMilliseconds(100));
private static readonly Regex SafeValuePattern = new Regex(
@"^(?:true|false|-?[0-9]{1,10}|0x[0-9a-f]{1,8})$",
RegexOptions.CultureInvariant | RegexOptions.IgnoreCase | RegexOptions.Compiled, TimeSpan.FromMilliseconds(100));
private const int MaxBreadcrumbTokenLength = 128;
private readonly AgentLogger _logger;
private readonly string _sessionId;
private readonly string _tenantId;
private readonly InformationalEventPost _post;
private readonly HelloTracker _helloTracker;
private readonly OobeUpdateTelemetry _oobeUpdate;
private EventLogWatcher _watcher;
private int _started;
private int _stopReported;
private bool _espExitDetected;
private bool _whiteGloveDetected;
private bool _helloWizardStartDetected;
private string _lastCxhPage; // 62405 names no page, it ends the last one started
private readonly object _stateLock = new object();
/// <summary>
@@ -99,13 +130,17 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
string tenantId,
InformationalEventPost post,
AgentLogger logger,
HelloTracker helloTracker)
HelloTracker helloTracker,
Func<IReadOnlyList<OobeUpdateRegistrySnapshot.KeyState>> oobeUpdateRegistryReader = null)
{
_sessionId = sessionId ?? throw new ArgumentNullException(nameof(sessionId));
_tenantId = tenantId ?? throw new ArgumentNullException(nameof(tenantId));
_post = post ?? throw new ArgumentNullException(nameof(post));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_helloTracker = helloTracker; // nullable — HelloTracker may be unavailable in some test setups
// Test seam: a fake registry reader; production reads the real registry.
_oobeUpdate = new OobeUpdateTelemetry(sessionId, tenantId, post, logger,
oobeUpdateRegistryReader ?? OobeUpdateRegistrySnapshot.Read);
}
internal bool IsEspExitedForTest { get { lock (_stateLock) { return _espExitDetected; } } }
@@ -113,12 +148,17 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
public void Start()
{
// The OOBE update registry state at every agent start — so also right after the
// update's restart (D-310).
if (Interlocked.Exchange(ref _started, 1) == 0)
_oobeUpdate.ReportState(OobeUpdateTelemetry.Moments.AgentStart);
try
{
var query = new EventLogQuery(
ShellCoreEventLogChannel,
PathType.LogName,
"*[System[(EventID=62404 or EventID=62407)]]");
$"*[System[{WatchedEventIdsXPath}]]");
_watcher = new EventLogWatcher(query);
_watcher.EventRecordWritten += OnEventRecordWritten;
@@ -143,6 +183,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
public void Stop()
{
// Stop runs again from Dispose — one snapshot, and only for a tracker that started.
if (Volatile.Read(ref _started) == 1 && Interlocked.Exchange(ref _stopReported, 1) == 0)
_oobeUpdate.ReportState(OobeUpdateTelemetry.Moments.AgentStop);
if (_watcher == null) return;
try
{
@@ -186,6 +230,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
/// </summary>
internal void ProcessEvent(int eventId, string description, DateTime timestamp, string providerName, bool isBackfill)
{
ObserveCxhRecord(eventId, description, timestamp, isBackfill);
if (eventId == EventId_ShellCore_WebAppStopped || eventId == EventId_ShellCore_WebAppEventName) return;
string eventType;
EventSeverity severity = EventSeverity.Info;
string message;
@@ -388,7 +435,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
var query = new EventLogQuery(
ShellCoreEventLogChannel,
PathType.LogName,
$"*[System[(EventID=62404 or EventID=62407) and TimeCreated[timediff(@SystemTime) <= {lookbackMs}]]]");
$"*[System[{WatchedEventIdsXPath} and TimeCreated[timediff(@SystemTime) <= {lookbackMs}]]]");
var records = new List<(int Id, string Description, DateTime OccurredAtUtc)>();
using (var reader = new EventLogReader(query))
@@ -466,6 +513,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
foreach (var record in records)
{
var description = record.Description ?? string.Empty;
ObserveCxhRecord(record.Id, description, record.OccurredAtUtc, isBackfill: true);
if (record.Id == EventId_ShellCore_WebAppStarted)
{
@@ -473,6 +521,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
continue;
}
// 62405/62406 only feed the breadcrumbs and the update page telemetry above.
if (record.Id != EventId_ShellCore_WebAppEvent) continue;
var isFailure = HasEspFailurePattern(description);
var isExit = !isFailure && EspExitingPattern.IsMatch(description);
if (!isFailure && !isExit) continue;
@@ -569,6 +620,159 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
}
// =====================================================================
// CloudExperienceHost navigation: breadcrumbs and OOBE update page telemetry (D-310)
// =====================================================================
/// <summary>
/// Writes one agent.log line per CloudExperienceHost page start/stop and per update-page
/// event name, so a diagnostics package shows the OOBE navigation. Records of the update
/// page itself (<c>OobeNDUP</c>, <c>RebootNDUP</c> and its <c>ExpeditedUpdate_*</c> names)
/// are also reported as <c>oobe_update_page</c>, and at the page's live start and stop the
/// registry state behind it as <c>oobe_update_state</c>. No decision.
/// </summary>
private void ObserveCxhRecord(int eventId, string description, DateTime occurredAtUtc, bool isBackfill)
{
try
{
string page;
lock (_stateLock)
{
if (eventId == EventId_ShellCore_WebAppStarted)
_lastCxhPage = ExtractToken(CxidPattern, description) ?? _lastCxhPage;
page = _lastCxhPage;
}
var crumb = FormatCxhBreadcrumb(eventId, description, page);
if (crumb == null) return;
_logger.Info(isBackfill ? $"{crumb} (backfill, at {occurredAtUtc:o})" : crumb);
var record = ParseUpdatePageRecord(eventId, description, page);
if (record == null) return;
_oobeUpdate.ReportPage(record, occurredAtUtc, isBackfill);
// The registry is read now — only a live page boundary is the page's moment.
if (!isBackfill && record.CxhEvent != OobeUpdatePageRecord.EventName)
{
_oobeUpdate.ReportState(
record.CxhEvent == OobeUpdatePageRecord.PageStarted
? OobeUpdateTelemetry.Moments.UpdatePageStarted
: OobeUpdateTelemetry.Moments.UpdatePageStopped,
record.Page);
}
}
catch (Exception ex)
{
_logger.Debug($"ShellCoreTracker: CXH record observation failed: {ex.Message}");
}
}
/// <summary>
/// The breadcrumb for one Shell-Core record, or null when it is not one: a page start
/// (62404, its CXID), a page stop (62405, its result, attributed to the last started
/// page), or an update-page event name (62406/62407 — a 62407 value only in a safe form).
/// </summary>
internal static string FormatCxhBreadcrumb(int eventId, string description, string lastPage)
{
switch (eventId)
{
case EventId_ShellCore_WebAppStarted:
{
var cxid = ExtractToken(CxidPattern, description);
return cxid == null ? null : $"CXH page started: {cxid}";
}
case EventId_ShellCore_WebAppStopped:
{
var result = ExtractToken(ResultPattern, description) ?? "?";
return $"CXH page stopped: {lastPage ?? "?"} (result={result})";
}
case EventId_ShellCore_WebAppEventName:
case EventId_ShellCore_WebAppEvent:
{
var name = ExtractToken(NamePattern, description);
if (!IsUpdatePageToken(name)) return null;
var value = eventId == EventId_ShellCore_WebAppEvent ? ExtractSafeValue(description) : null;
return value == null ? $"CXH event: {name}" : $"CXH event: {name} (value={value})";
}
default:
return null;
}
}
/// <summary>
/// The record of the OOBE update page, or null for any other CloudExperienceHost record:
/// the start of a page whose CXID names NDUP, the stop of such a page (62405, attributed
/// to the last started page), or an update-page event name from 62406/62407.
/// </summary>
internal static OobeUpdatePageRecord ParseUpdatePageRecord(int eventId, string description, string lastPage)
{
switch (eventId)
{
case EventId_ShellCore_WebAppStarted:
{
var cxid = ExtractToken(CxidPattern, description);
return IsUpdatePageToken(cxid)
? new OobeUpdatePageRecord(OobeUpdatePageRecord.PageStarted, eventId, page: cxid)
: null;
}
case EventId_ShellCore_WebAppStopped:
return IsUpdatePageToken(lastPage)
? new OobeUpdatePageRecord(OobeUpdatePageRecord.PageStopped, eventId,
page: lastPage, result: ExtractToken(ResultPattern, description) ?? "?")
: null;
case EventId_ShellCore_WebAppEventName:
case EventId_ShellCore_WebAppEvent:
{
var name = ExtractToken(NamePattern, description);
if (!IsUpdatePageToken(name)) return null;
var value = eventId == EventId_ShellCore_WebAppEvent ? ExtractSafeValue(description) : null;
return new OobeUpdatePageRecord(OobeUpdatePageRecord.EventName, eventId, name: name, value: value);
}
default:
return null;
}
}
internal static bool IsUpdatePageToken(string token) =>
token != null
&& (token.IndexOf("NDUP", StringComparison.OrdinalIgnoreCase) >= 0
|| token.IndexOf("ExpeditedUpdate", StringComparison.OrdinalIgnoreCase) >= 0);
/// <summary>
/// The 62407 value when it is a boolean, an integer of up to ten digits or an HRESULT —
/// forms that cannot carry a URL or a tenant id (the CSP gate, error codes). Null for
/// anything else.
/// </summary>
internal static string ExtractSafeValue(string description)
{
var value = ExtractToken(ValuePattern, description);
if (string.IsNullOrEmpty(value)) return null;
try
{
return SafeValuePattern.IsMatch(value) ? value : null;
}
catch (RegexMatchTimeoutException)
{
return null;
}
}
private static string ExtractToken(Regex pattern, string description)
{
if (string.IsNullOrEmpty(description)) return null;
try
{
var match = pattern.Match(description);
if (!match.Success) return null;
var token = match.Groups[1].Value;
return token.Length > MaxBreadcrumbTokenLength ? token.Substring(0, MaxBreadcrumbTokenLength) + "…" : token;
}
catch (RegexMatchTimeoutException)
{
return null;
}
}
// =====================================================================
// Pattern helpers
// =====================================================================
@@ -0,0 +1,42 @@
using System;
using System.Collections.Generic;
using System.Linq;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
/// <summary>
/// Thread-safe counters for update activity that is counted instead of emitted (D-310): Store,
/// Defender and other non-OS update events, repeated scan results, servicing steps of non-OS
/// packages. Shared by the update watchers of one host, which reports the totals once as
/// <c>windows_update_activity_summary</c> when it stops.
/// </summary>
internal sealed class UpdateActivityTally
{
private readonly object _lock = new object();
private readonly Dictionary<string, int> _counts = new Dictionary<string, int>(StringComparer.Ordinal);
public void Increment(string key)
{
if (string.IsNullOrEmpty(key)) return;
lock (_lock)
{
_counts.TryGetValue(key, out var current);
_counts[key] = current + 1;
}
}
public bool IsEmpty
{
get { lock (_lock) return _counts.Count == 0; }
}
/// <summary>Counts ordered by key, for a stable event payload.</summary>
public IReadOnlyList<KeyValuePair<string, int>> Snapshot()
{
lock (_lock)
{
return _counts.OrderBy(kv => kv.Key, StringComparer.Ordinal).ToList();
}
}
}
}
@@ -0,0 +1,65 @@
using System;
using System.Text.RegularExpressions;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
/// <summary>
/// Sorts a Windows Update client event into the update class that decides how it is reported
/// (D-310). Only <see cref="Os"/> updates — Windows quality/cumulative/servicing-stack/feature
/// updates, .NET Framework and the Malicious Software Removal Tool — are enrollment-relevant and
/// emitted one by one. Defender intelligence and Store app updates arrive every few minutes on a
/// fresh device (up to 171 downloads per hour in the field census) and are only counted.
/// </summary>
internal static class WindowsUpdateClassifier
{
public const string Os = "os";
public const string Defender = "defender";
public const string Store = "store";
public const string Other = "other";
/// <summary>The Microsoft Store service; WU events carry it as <c>serviceGuid</c>.</summary>
internal const string StoreServiceGuid = "855e8a7c-ecb4-4ca3-b045-1dfa50104289";
// Store packages are titled "<12-char product id>-<Publisher.App>", e.g. "9NBLGGH4NNS1-Microsoft.DesktopAppInstaller";
// bundles add an "ApplicationSet-" prefix.
private static readonly Regex StoreProductTitle =
new Regex(@"^(ApplicationSet-)?[0-9A-Z]{12}-\S", RegexOptions.CultureInvariant, TimeSpan.FromMilliseconds(100));
private static readonly Regex DefenderTitle =
new Regex(@"Microsoft Defender|Windows Defender", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant, TimeSpan.FromMilliseconds(100));
// "Windows" plus "Update"/"Upgrade" covers cumulative, quality, security, servicing-stack,
// dynamic and feature updates in both the GA and the Insider title forms.
private static readonly Regex WindowsUpdateTitle =
new Regex(@"\bWindows\b.*\b(Update|Upgrade)\b|\b(Update|Upgrade)\b.*\bWindows\b",
RegexOptions.IgnoreCase | RegexOptions.CultureInvariant, TimeSpan.FromMilliseconds(100));
private static readonly Regex OsTitleExtras =
new Regex(@"\.NET Framework|Servicing Stack|Malicious Software Removal Tool",
RegexOptions.IgnoreCase | RegexOptions.CultureInvariant, TimeSpan.FromMilliseconds(100));
public static string Classify(string updateTitle, string serviceGuid)
{
if (!string.IsNullOrEmpty(serviceGuid)
&& serviceGuid.Trim('{', '}', ' ').Equals(StoreServiceGuid, StringComparison.OrdinalIgnoreCase))
{
return Store;
}
if (string.IsNullOrWhiteSpace(updateTitle)) return Other;
try
{
if (StoreProductTitle.IsMatch(updateTitle)) return Store;
if (DefenderTitle.IsMatch(updateTitle)) return Defender;
if (OsTitleExtras.IsMatch(updateTitle) || WindowsUpdateTitle.IsMatch(updateTitle)) return Os;
}
catch (RegexMatchTimeoutException)
{
// A pathological title must never cost the event — it is reported as "other".
}
return Other;
}
}
}
@@ -2,100 +2,106 @@ using System;
using System.Collections.Generic;
using System.Diagnostics.Eventing.Reader;
using System.Globalization;
using System.IO;
using System.Linq;
using System.Threading;
using System.Xml.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.DecisionCore.State;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
using AutopilotMonitor.Shared.Services;
using Newtonsoft.Json;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
/// <summary>
/// Watches <c>Microsoft-Windows-WindowsUpdateClient/Operational</c> for quality/cumulative
/// update install activity that happens DURING Autopilot OOBE / ESP and forwards it as
/// structured <c>windows_update_succeeded</c> / <c>windows_update_failed</c> /
/// <c>windows_update_started</c> events.
/// Watches ONE channel of the <c>Microsoft-Windows-WindowsUpdateClient</c> provider for update
/// activity during OOBE / ESP and forwards it as <c>windows_update_started</c> /
/// <c>windows_update_succeeded</c> / <c>windows_update_failed</c> events.
/// <para>
/// This is a blind spot no other tool surfaces well: a cumulative update installing mid-OOBE
/// can break the enrollment (r/Intune KB5095189), and the ESP "Install Windows quality updates
/// during OOBE" feature makes it increasingly common. The Intune console shows none of it — it
/// is otherwise deep manual digging on the device.
/// <b>Two channels (D-310).</b> The provider manifest writes the install events — 19 success,
/// 20 failure (HRESULT), 43 install started, 44 download started — to the <b>System</b> log and
/// the scan/download events — 25 scan failed, 26 scan found N, 31 download failed, 41
/// downloaded — to the Operational channel. The host runs one instance per channel; RecordIds
/// are per channel, so each instance keeps its own watermark. The System log is shared with
/// every other provider, hence the provider filter in its XPath.
/// </para>
/// <para>
/// <b>Backfill.</b> OOBE updates run at the early "Getting updates" screen, frequently BEFORE the
/// agent starts. The WindowsUpdateClient/Operational .evtx persists across the mid-OOBE reboot,
/// so a startup backfill with a generous lookback catches those pre-agent events.
/// <b>Classification.</b> Only Windows/.NET updates (<see cref="WindowsUpdateClassifier.Os"/>)
/// are emitted one by one and reported to the decision engine. Store, Defender and other
/// updates arrive every few minutes on a fresh device and are only counted into the host's
/// <see cref="UpdateActivityTally"/>. Repeated scan results (26) and scan failures (25) are
/// deduplicated; their repeats are counted the same way.
/// </para>
/// <para>
/// <b>Cross-restart dedup.</b> Unlike the single-latch WhiteGlove backfill, WU events are many
/// distinct records, so an agent restart would re-read (and re-emit) events already reported in
/// the prior run. We persist the highest processed <see cref="EventRecord.RecordId"/> as a
/// watermark and skip records at/below it. Live records always carry a higher RecordId and pass.
/// </para>
/// <para>
/// <b>Verified EventIDs (build against these):</b> 19=install success, 20=install failure
/// (carries HRESULT in <c>errorCode</c>), 43=install started, 44=download started. IDs whose
/// semantics are unverified across sources (25/26/31, restart-required 21 vs 41) are excluded by
/// default and can be added via <c>CollectorConfiguration.WindowsUpdateTargetedEventIds</c>
/// after validation on a real device trace — no redeploy needed.
/// <b>Backfill.</b> Updates can start before the agent does; the .evtx survives the reboot, so
/// a startup backfill with a generous lookback catches them. Backfilled events keep their own
/// event time on the timeline.
/// </para>
/// </summary>
internal sealed class WindowsUpdateTracker : IDisposable
{
internal const string Channel = "Microsoft-Windows-WindowsUpdateClient/Operational";
internal const string SystemChannel = "System";
internal const string ProviderName = "Microsoft-Windows-WindowsUpdateClient";
internal const string OrchestratorChannel = "Microsoft-Windows-UpdateOrchestrator/Operational";
internal const int EventId_InstallSuccess = 19;
internal const int EventId_InstallFailure = 20;
// System channel (provider manifest).
internal const int EventId_InstallSuccess = 19;
internal const int EventId_InstallFailure = 20;
internal const int EventId_InstallStarted = 43;
internal const int EventId_DownloadStarted = 44;
// Operational channel.
internal const int EventId_ScanFailed = 25;
internal const int EventId_ScanFound = 26;
internal const int EventId_DownloadFailed = 31;
internal const int EventId_Downloaded = 41;
internal const string WatermarkStateFileName = "windows-update-watermark.json";
internal const string SystemWatermarkStateFileName = "windows-update-system-watermark.json";
// Hard bound for the unfiltered census scan — during OOBE both channels carry at most a
// few hundred records in the lookback window; the cap only guards against a pathological
// log. A truncated census says so in its payload (no silent caps).
internal const int CensusRecordCap = 5000;
// Scan results repeat every few minutes; only changes are worth a timeline row.
internal const int MaxScanFoundEmissions = 10;
internal const int MaxScanFailedEmissions = 5;
private static readonly int[] DefaultSystemEventIds = { EventId_InstallSuccess, EventId_InstallFailure, EventId_InstallStarted, EventId_DownloadStarted };
private static readonly int[] DefaultOperationalEventIds = { EventId_ScanFailed, EventId_ScanFound, EventId_DownloadFailed, EventId_Downloaded };
private readonly AgentLogger _logger;
private readonly string _sessionId;
private readonly string _tenantId;
private readonly InformationalEventPost _post;
private readonly string _channel;
private readonly bool _isSystemChannel;
private readonly HashSet<int> _targetedEventIds;
private readonly bool _backfillEnabled;
private readonly int _backfillLookbackMinutes;
private readonly string _stateDirectory;
private readonly bool _channelCensusEnabled;
private readonly Func<bool> _osBuildChangedProvider;
private readonly UpdateActivityTally _tally;
private readonly Action<string, string, DateTime> _onOsUpdateActivity;
private readonly EventRecordWatermark _watermark;
// Count of targeted WU events emitted THIS run (live + backfill). Zero after the backfill
// while the OS build provably changed = the watcher is blind to the update's channel/IDs.
// Targeted events processed THIS run (any class). Zero after the backfill while the OS
// build provably changed = the watcher is blind to the update's channel/IDs.
private int _emittedThisRun;
private readonly object _throttleLock = new object();
private readonly HashSet<string> _scanFailureCodesReported = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
private string _lastScanFoundCount;
private int _scanFoundEmitted;
private EventLogWatcher _watcher;
// Dedup. Guarded because EventRecordWritten callbacks run concurrently with the backfill scan.
//
// Cross-restart boundary loaded from disk: the highest RecordId emitted in a PRIOR run.
// Immutable during this run — events at/below it were already emitted before this process
// started and must not be re-emitted by the backfill scan.
private long _restartWatermark = -1;
// RecordIds emitted in THIS run — the intra-run dedup set. Deliberately a HashSet, NOT a
// high-water mark: the live watcher is armed before the backfill runs, so a live event with a
// HIGHER RecordId can be processed before/during backfill. A high-water mark would then make
// the dedup skip every older, never-emitted backfill event (recordId <= max) — silently
// dropping exactly the early pre-agent OOBE updates that are this feature's whole point.
// WU install events are low-volume (a handful per enrollment), so the set stays tiny.
private readonly HashSet<long> _seenThisRun = new HashSet<long>();
// Highest RecordId emitted so far (max of restart watermark + this run). Persisted so the next
// run's restart watermark is correct.
private long _maxEmittedRecordId = -1;
private readonly object _watermarkLock = new object();
/// <param name="channel">Event log channel; defaults to the Operational channel.</param>
/// <param name="targetedEventIds">IDs to watch on <paramref name="channel"/>; defaults to the manifest set of that channel.</param>
/// <param name="tally">Counter for activity that is counted instead of emitted; null drops the counts.</param>
/// <param name="onOsUpdateActivity">Called for every Windows/.NET update step with (step, update title, event time).</param>
public WindowsUpdateTracker(
string sessionId,
string tenantId,
@@ -106,25 +112,60 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
int backfillLookbackMinutes = 60,
string stateDirectory = null,
bool channelCensusEnabled = true,
Func<bool> osBuildChangedProvider = null)
Func<bool> osBuildChangedProvider = null,
string channel = null,
UpdateActivityTally tally = null,
Action<string, string, DateTime> onOsUpdateActivity = null)
{
_sessionId = sessionId ?? throw new ArgumentNullException(nameof(sessionId));
_tenantId = tenantId ?? throw new ArgumentNullException(nameof(tenantId));
_post = post ?? throw new ArgumentNullException(nameof(post));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_channel = string.IsNullOrEmpty(channel) ? Channel : channel;
_isSystemChannel = string.Equals(_channel, SystemChannel, StringComparison.OrdinalIgnoreCase);
_targetedEventIds = targetedEventIds != null && targetedEventIds.Length > 0
? new HashSet<int>(targetedEventIds)
: new HashSet<int> { EventId_InstallSuccess, EventId_InstallFailure, EventId_InstallStarted, EventId_DownloadStarted };
: new HashSet<int>(_isSystemChannel ? DefaultSystemEventIds : DefaultOperationalEventIds);
_backfillEnabled = backfillEnabled;
_backfillLookbackMinutes = backfillLookbackMinutes;
_stateDirectory = stateDirectory != null ? Environment.ExpandEnvironmentVariables(stateDirectory) : null;
_channelCensusEnabled = channelCensusEnabled;
_osBuildChangedProvider = osBuildChangedProvider;
_tally = tally;
_onOsUpdateActivity = onOsUpdateActivity;
_watermark = new EventRecordWatermark(
stateDirectory,
_isSystemChannel ? SystemWatermarkStateFileName : WatermarkStateFileName,
logger,
_isSystemChannel ? "WindowsUpdate (System)" : "WindowsUpdate");
}
/// <summary>Channel this instance watches.</summary>
internal string WatchedChannel => _channel;
/// <summary>Targeted events processed this run, any class.</summary>
internal int EmittedThisRun => Volatile.Read(ref _emittedThisRun);
/// <summary>EventIDs this instance watches on <see cref="WatchedChannel"/>.</summary>
internal IReadOnlyCollection<int> TargetedEventIds => _targetedEventIds;
/// <summary>
/// True when the provider manifest writes <paramref name="eventId"/> to the System log
/// (16–24, 27, 28, 32, 33, 43, 44, 212–218); all other IDs live in the Operational channel.
/// </summary>
internal static bool IsSystemChannelEventId(int eventId) =>
(eventId >= 16 && eventId <= 24)
|| eventId == 27 || eventId == 28
|| eventId == 32 || eventId == 33
|| eventId == 43 || eventId == 44
|| (eventId >= 212 && eventId <= 218);
/// <summary>
/// Watches the channel and backfills it. The blind-spot census is NOT run here: it needs
/// the emissions of every channel, so the host runs it after all its trackers started.
/// </summary>
public void Start()
{
LoadWatermark();
_watermark.Load();
StartWatcher();
if (_backfillEnabled && _backfillLookbackMinutes > 0)
@@ -133,10 +174,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
}
else
{
_logger.Info("WindowsUpdate backfill disabled by config");
_logger.Info($"WindowsUpdate backfill disabled by config ({_channel})");
}
EmitChannelCensusIfBlind();
}
public void Stop()
@@ -146,11 +185,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
_watcher.Enabled = false;
_watcher.Dispose();
_logger.Info("WindowsUpdate watcher stopped");
_logger.Info($"WindowsUpdate watcher stopped ({_channel})");
}
catch (Exception ex)
{
_logger.Error("Error stopping WindowsUpdate watcher", ex);
_logger.Error($"Error stopping WindowsUpdate watcher ({_channel})", ex);
}
finally
{
@@ -160,52 +199,64 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
public void Dispose() => Stop();
internal void LoadWatermark() => _watermark.Load();
// -----------------------------------------------------------------------
// Watcher lifecycle
// -----------------------------------------------------------------------
private string ProviderFilter => _isSystemChannel ? ProviderName : null;
private void StartWatcher()
{
if (_targetedEventIds.Count == 0)
{
_logger.Warning("WindowsUpdate watcher not started: no targeted EventIDs configured");
_logger.Warning($"WindowsUpdate watcher not started: no targeted EventIDs configured ({_channel})");
return;
}
try
{
var xpath = BuildXPath(_targetedEventIds);
var query = new EventLogQuery(Channel, PathType.LogName, xpath);
var query = new EventLogQuery(_channel, PathType.LogName, BuildXPath(_targetedEventIds, ProviderFilter));
_watcher = new EventLogWatcher(query);
_watcher.EventRecordWritten += OnEventRecordWritten;
_watcher.Enabled = true;
_logger.Info($"WindowsUpdate watcher started: {Channel} (targetedIds={_targetedEventIds.Count})");
_logger.Info($"WindowsUpdate watcher started: {_channel} (targetedIds={string.Join(",", _targetedEventIds.OrderBy(id => id))})");
}
catch (EventLogNotFoundException)
{
_logger.Warning($"WindowsUpdate event log not found: {Channel} (normal on non-Windows 10/11 test environments)");
_logger.Warning($"WindowsUpdate event log not found: {_channel} (normal on non-Windows 10/11 test environments)");
}
catch (UnauthorizedAccessException ex)
{
_logger.Warning($"WindowsUpdate watcher access denied for {Channel}: {ex.Message}");
_logger.Warning($"WindowsUpdate watcher access denied for {_channel}: {ex.Message}");
}
catch (Exception ex)
{
_logger.Error($"Failed to start WindowsUpdate watcher for {Channel}", ex);
_logger.Error($"Failed to start WindowsUpdate watcher for {_channel}", ex);
// MON-D1: surface a dead watcher as one-shot telemetry, not just a local log line.
CollectorDegradationReporter.Report(_post, _sessionId, _tenantId,
collectorName: "WindowsUpdateTracker", reason: $"watcher_arm_failed:{Channel}", ex: ex);
collectorName: "WindowsUpdateTracker", reason: $"watcher_arm_failed:{_channel}", ex: ex);
}
}
/// <summary>Targeted-EventID XPath filter without a provider clause. Exposed for tests.</summary>
internal static string BuildXPath(HashSet<int> targetedEventIds) =>
BuildXPath(targetedEventIds, providerName: null);
/// <summary>
/// Builds the targeted-EventID XPath filter. Exposed as internal for tests.
/// Targeted-EventID XPath filter, optionally restricted to <paramref name="providerName"/>
/// (mandatory on the shared System log) and to the last <paramref name="lookbackMs"/>.
/// </summary>
internal static string BuildXPath(HashSet<int> targetedEventIds)
internal static string BuildXPath(HashSet<int> targetedEventIds, string providerName, long? lookbackMs = null)
{
var idClauses = string.Join(" or ",
targetedEventIds.OrderBy(id => id).Select(id => $"EventID={id}"));
return $"*[System[({idClauses})]]";
var clauses = new List<string>(3);
if (!string.IsNullOrEmpty(providerName))
clauses.Add($"Provider[@Name='{providerName}']");
clauses.Add("(" + string.Join(" or ", targetedEventIds.OrderBy(id => id).Select(id => $"EventID={id}")) + ")");
if (lookbackMs.HasValue)
clauses.Add($"TimeCreated[timediff(@SystemTime) <= {lookbackMs.Value}]");
return $"*[System[{string.Join(" and ", clauses)}]]";
}
// -----------------------------------------------------------------------
@@ -230,7 +281,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
var recordId = record.RecordId ?? -1;
// Cheap dedup gate BEFORE the expensive XML/FormatDescription work.
if (IsAlreadyProcessed(recordId))
if (_watermark.IsAlreadyProcessed(recordId))
return;
string xml = null;
@@ -242,6 +293,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
eventData.TryGetValue("updateGuid", out var updateGuid);
eventData.TryGetValue("updateRevisionNumber", out var updateRevision);
eventData.TryGetValue("errorCode", out var errorCode);
eventData.TryGetValue("serviceGuid", out var serviceGuid);
eventData.TryGetValue("updateCount", out var updateCount);
string description = null;
try { description = record.FormatDescription(); }
@@ -257,7 +310,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
updateRevisionNumber: updateRevision,
errorCode: errorCode,
formattedDescription: description,
isBackfill: isBackfill);
isBackfill: isBackfill,
serviceGuid: serviceGuid,
updateCount: updateCount);
}
/// <summary>
@@ -275,66 +330,68 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
string updateRevisionNumber,
string errorCode,
string formattedDescription,
bool isBackfill)
bool isBackfill,
string serviceGuid = null,
string updateCount = null)
{
if (!MarkProcessed(recordId))
if (!_watermark.TryClaim(recordId))
return; // already emitted (cross-restart or duplicate delivery)
System.Threading.Interlocked.Increment(ref _emittedThisRun);
Interlocked.Increment(ref _emittedThisRun);
string eventType;
EventSeverity severity;
bool immediateUpload;
ClassifyEventId(eventId, out eventType, out severity, out immediateUpload);
var shape = ShapeFor(eventId);
var title = string.IsNullOrEmpty(updateTitle) ? "(unknown update)" : updateTitle;
// Scan-level events name no update: deduplicate instead of classifying. Every other
// event is classified; only Windows/.NET updates are listed, the rest is counted —
// so windows_update_succeeded/failed always mean an OS update (ANALYZE-DEV-004/005).
string updateClass = null;
if (eventId == EventId_ScanFound)
{
if (!ClaimScanFoundEmission(updateCount)) { _tally?.Increment("scan_found_repeat"); return; }
}
else if (eventId == EventId_ScanFailed)
{
if (!ClaimScanFailedEmission(errorCode)) { _tally?.Increment("scan_failed_repeat"); return; }
}
else
{
updateClass = WindowsUpdateClassifier.Classify(updateTitle, serviceGuid);
if (updateClass != WindowsUpdateClassifier.Os)
{
_tally?.Increment($"{updateClass}_{eventId}");
return;
}
}
var data = new Dictionary<string, object>
{
{ "wuEventId", eventId },
{ "wuChannel", _isSystemChannel ? "system" : "operational" },
{ "wuPhase", shape.Phase },
{ "backfilled", isBackfill },
};
if (updateClass != null) data["updateClass"] = updateClass;
if (recordId >= 0) data["recordId"] = recordId;
if (level.HasValue) data["level"] = level.Value;
if (!string.IsNullOrEmpty(updateTitle)) data["updateTitle"] = updateTitle;
if (!string.IsNullOrEmpty(updateGuid)) data["updateGuid"] = updateGuid;
if (!string.IsNullOrEmpty(updateRevisionNumber)) data["updateRevisionNumber"] = updateRevisionNumber;
if (!string.IsNullOrEmpty(updateCount)) data["updateCount"] = updateCount;
if (timeCreatedUtc.HasValue) data["timeCreated"] = timeCreatedUtc.Value.ToString("o");
// Decode the HRESULT for failure events (and any event that carries a non-empty errorCode).
string hresultHex;
if (TryNormalizeHResult(errorCode, out var hresultValue, out hresultHex))
if (TryNormalizeHResult(errorCode, out var hresultValue, out var hresultHex))
{
data["hresult"] = hresultHex;
data["hresultSymbol"] = ResolveHResultSymbol(hresultValue);
}
var title = string.IsNullOrEmpty(updateTitle) ? "(unknown update)" : updateTitle;
string message;
switch (eventId)
{
case EventId_InstallSuccess:
message = $"Windows Update installed during enrollment: {title}";
break;
case EventId_InstallFailure:
message = data.TryGetValue("hresultSymbol", out var sym)
? $"Windows Update FAILED during enrollment: {title} ({data["hresult"]} {sym})"
: $"Windows Update FAILED during enrollment: {title}";
break;
case EventId_DownloadStarted:
message = $"Windows Update download started during enrollment: {title}";
break;
case EventId_InstallStarted:
message = $"Windows Update install started during enrollment: {title}";
break;
default:
message = $"Windows Update activity (EventID {eventId}) during enrollment: {title}";
break;
}
if (!string.IsNullOrEmpty(formattedDescription))
{
var truncated = formattedDescription.Length > 1000
data["description"] = formattedDescription.Length > 1000
? formattedDescription.Substring(0, 1000) + "…"
: formattedDescription;
data["description"] = truncated;
}
_post.Emit(new EnrollmentEvent
@@ -343,46 +400,107 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
TenantId = _tenantId,
// Use the WU event's own time, not the default UtcNow the ctor stamps: backfilled
// pre-agent OOBE updates must land on the timeline at when the update actually
// installed, not at agent start. InformationalEventPost forwards this as occurredAtUtc,
// happened, not at agent start. InformationalEventPost forwards this as occurredAtUtc,
// which drives the timeline entry's timestamp (EventTimelineEmitter).
Timestamp = timeCreatedUtc ?? DateTime.UtcNow,
EventType = eventType,
Severity = severity,
EventType = shape.EventType,
Severity = shape.Severity,
Source = "WindowsUpdateWatcher",
Phase = EnrollmentPhase.Unknown,
Message = message,
Message = BuildMessage(eventId, title, updateCount, data),
Data = data,
ImmediateUpload = immediateUpload,
ImmediateUpload = shape.Immediate,
});
if (updateClass == WindowsUpdateClassifier.Os && shape.OsStep != null && _onOsUpdateActivity != null)
{
try { _onOsUpdateActivity(shape.OsStep, updateTitle, timeCreatedUtc ?? DateTime.UtcNow); }
catch (Exception ex) { _logger.Warning($"WindowsUpdate: OS update activity callback failed: {ex.Message}"); }
}
// Last: everything this record produced has been handed to the ingress first.
_watermark.MarkEmitted(recordId);
}
private static void ClassifyEventId(int eventId, out string eventType, out EventSeverity severity, out bool immediateUpload)
private struct EventShape
{
public string EventType;
public EventSeverity Severity;
public bool Immediate;
public string Phase;
/// <summary>Step reported to the decision engine for OS updates; null = not a step.</summary>
public string OsStep;
}
private static EventShape ShapeFor(int eventId)
{
switch (eventId)
{
case EventId_InstallFailure:
eventType = Constants.EventTypes.WindowsUpdateFailed;
severity = EventSeverity.Error;
immediateUpload = true; // a mid-OOBE update failure is high-signal — surface it fast
break;
case EventId_InstallSuccess:
eventType = Constants.EventTypes.WindowsUpdateSucceeded;
severity = EventSeverity.Info;
immediateUpload = false;
break;
return new EventShape { EventType = Constants.EventTypes.WindowsUpdateSucceeded, Severity = EventSeverity.Info, Phase = "installed", OsStep = OsUpdateSteps.Installed };
case EventId_InstallFailure:
// A mid-OOBE install failure is high-signal — surface it fast.
return new EventShape { EventType = Constants.EventTypes.WindowsUpdateFailed, Severity = EventSeverity.Error, Immediate = true, Phase = "install", OsStep = OsUpdateSteps.Failed };
case EventId_InstallStarted:
return new EventShape { EventType = Constants.EventTypes.WindowsUpdateStarted, Severity = EventSeverity.Info, Phase = "install", OsStep = OsUpdateSteps.InstallStarted };
case EventId_DownloadStarted:
eventType = Constants.EventTypes.WindowsUpdateStarted;
severity = EventSeverity.Debug; // timeline context only
immediateUpload = false;
break;
return new EventShape { EventType = Constants.EventTypes.WindowsUpdateStarted, Severity = EventSeverity.Info, Phase = "download", OsStep = OsUpdateSteps.DownloadStarted };
case EventId_Downloaded:
return new EventShape { EventType = Constants.EventTypes.WindowsUpdateStarted, Severity = EventSeverity.Info, Phase = "downloaded", OsStep = OsUpdateSteps.Downloaded };
case EventId_DownloadFailed:
return new EventShape { EventType = Constants.EventTypes.WindowsUpdateFailed, Severity = EventSeverity.Warning, Phase = "download", OsStep = OsUpdateSteps.Failed };
case EventId_ScanFailed:
return new EventShape { EventType = Constants.EventTypes.WindowsUpdateFailed, Severity = EventSeverity.Warning, Phase = "scan" };
case EventId_ScanFound:
return new EventShape { EventType = Constants.EventTypes.WindowsUpdateStarted, Severity = EventSeverity.Debug, Phase = "scan" };
default:
// A newly-enabled (still-being-validated) EventID: keep it visible (Info, not Debug)
// so it can be assessed on real traces.
eventType = Constants.EventTypes.WindowsUpdateStarted;
severity = EventSeverity.Info;
immediateUpload = false;
break;
// An EventID enabled via config whose meaning has not been mapped: keep it
// visible (Info, not Debug) so it can be assessed on real traces.
return new EventShape { EventType = Constants.EventTypes.WindowsUpdateStarted, Severity = EventSeverity.Info, Phase = "activity" };
}
}
private static string BuildMessage(int eventId, string title, string updateCount, Dictionary<string, object> data)
{
var hresultSuffix = data.TryGetValue("hresultSymbol", out var sym)
? $" ({data["hresult"]} {sym})"
: string.Empty;
switch (eventId)
{
case EventId_InstallSuccess: return $"Windows Update installed during enrollment: {title}";
case EventId_InstallFailure: return $"Windows Update FAILED during enrollment: {title}{hresultSuffix}";
case EventId_InstallStarted: return $"Windows Update install started during enrollment: {title}";
case EventId_DownloadStarted: return $"Windows Update download started during enrollment: {title}";
case EventId_Downloaded: return $"Windows Update downloaded during enrollment: {title}";
case EventId_DownloadFailed: return $"Windows Update download FAILED during enrollment: {title}{hresultSuffix}";
case EventId_ScanFailed: return $"Windows Update scan failed during enrollment{hresultSuffix}";
case EventId_ScanFound:
return string.IsNullOrEmpty(updateCount)
? "Windows Update scan completed during enrollment"
: $"Windows Update scan found {updateCount} update(s) during enrollment";
default: return $"Windows Update activity (EventID {eventId}) during enrollment: {title}";
}
}
private bool ClaimScanFoundEmission(string updateCount)
{
lock (_throttleLock)
{
if (_scanFoundEmitted >= MaxScanFoundEmissions) return false;
if (string.Equals(_lastScanFoundCount, updateCount ?? string.Empty, StringComparison.Ordinal)) return false;
_lastScanFoundCount = updateCount ?? string.Empty;
_scanFoundEmitted++;
return true;
}
}
private bool ClaimScanFailedEmission(string errorCode)
{
lock (_throttleLock)
{
if (_scanFailureCodesReported.Count >= MaxScanFailedEmissions) return false;
return _scanFailureCodesReported.Add(errorCode ?? string.Empty);
}
}
@@ -496,14 +614,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
try
{
var lookbackMs = (long)_backfillLookbackMinutes * 60 * 1000;
var idClauses = string.Join(" or ",
_targetedEventIds.OrderBy(id => id).Select(id => $"EventID={id}"));
var xpath = $"*[System[({idClauses}) and TimeCreated[timediff(@SystemTime) <= {lookbackMs}]]]";
var xpath = BuildXPath(_targetedEventIds, ProviderFilter, lookbackMs);
_logger.Info($"WindowsUpdate backfill: scanning {Channel} " +
$"(lookback={_backfillLookbackMinutes}min, targetedIds={_targetedEventIds.Count}, restartWatermark={_restartWatermark})");
_logger.Info($"WindowsUpdate backfill: scanning {_channel} " +
$"(lookback={_backfillLookbackMinutes}min, targetedIds={_targetedEventIds.Count}, restartWatermark={_watermark.RestartWatermark})");
var query = new EventLogQuery(Channel, PathType.LogName, xpath);
var query = new EventLogQuery(_channel, PathType.LogName, xpath);
int processed = 0;
using (var reader = new EventLogReader(query))
@@ -520,21 +636,21 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
}
if (processed == 0)
_logger.Debug($"WindowsUpdate backfill: no targeted events in last {_backfillLookbackMinutes} minutes");
_logger.Debug($"WindowsUpdate backfill: no targeted events in last {_backfillLookbackMinutes} minutes ({_channel})");
else
_logger.Info($"WindowsUpdate backfill: scanned {processed} event(s)");
_logger.Info($"WindowsUpdate backfill: scanned {processed} event(s) ({_channel})");
}
catch (EventLogNotFoundException)
{
_logger.Warning($"WindowsUpdate event log not found during backfill: {Channel} (normal on non-Windows 10/11 test environments)");
_logger.Warning($"WindowsUpdate event log not found during backfill: {_channel} (normal on non-Windows 10/11 test environments)");
}
catch (UnauthorizedAccessException ex)
{
_logger.Warning($"WindowsUpdate backfill access denied: {ex.Message}");
_logger.Warning($"WindowsUpdate backfill access denied ({_channel}): {ex.Message}");
}
catch (Exception ex)
{
_logger.Warning($"WindowsUpdate backfill failed: {ex.Message}");
_logger.Warning($"WindowsUpdate backfill failed ({_channel}): {ex.Message}");
}
}
@@ -544,17 +660,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
/// <summary>
/// Self-evidencing blind spot diagnosis. Session 7443317c: an OOBE quality update
/// provably installed (OS build changed across the mid-enrollment reboot) yet neither
/// the live watcher nor two 60-min backfills captured a single targeted event — the
/// update logged via a channel/EventID set the watcher is blind to. With diagnostics
/// upload tenant-disabled there is no client-log path to find out which one; so when
/// the blind spot provably occurred (build changed AND zero targeted events emitted
/// this run), emit a one-shot unfiltered EventID histogram of both update channels
/// over the lookback window. The histogram arrives via normal event upload and tells
/// us which IDs to add to <c>WindowsUpdateTargetedEventIds</c> via remote config —
/// no agent redeploy, no diagnostics access needed.
/// provably installed (OS build changed across the mid-enrollment reboot) yet the watcher
/// captured nothing — it read the install IDs from the Operational channel, while the
/// manifest writes them to the System log (D-310). When the blind spot occurs again
/// (build changed AND zero targeted events on every channel this run), emit a one-shot
/// unfiltered EventID histogram of the Operational and UpdateOrchestrator channels.
/// </summary>
internal void EmitChannelCensusIfBlind()
/// <param name="otherChannelsEmitted">Targeted events the host's other update watchers processed this run.</param>
/// <param name="otherTargetedEventIds">EventIDs the host's other update watchers target, listed with this instance's own.</param>
internal void EmitChannelCensusIfBlind(int otherChannelsEmitted = 0, IEnumerable<int> otherTargetedEventIds = null)
{
if (!_channelCensusEnabled) return;
@@ -562,7 +676,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
if (_osBuildChangedProvider == null || !_osBuildChangedProvider())
return;
if (System.Threading.Volatile.Read(ref _emittedThisRun) > 0)
if (EmittedThisRun + otherChannelsEmitted > 0)
return;
var lookback = _backfillLookbackMinutes > 0 ? _backfillLookbackMinutes : 60;
@@ -581,8 +695,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{ "wuClientCensus", wuClientHistogram },
{ "updateOrchestratorCensus", orchestratorHistogram },
{ "lookbackMinutes", lookback },
{ "restartWatermark", _restartWatermark },
{ "targetedEventIds", string.Join(",", _targetedEventIds.OrderBy(id => id)) },
{ "restartWatermark", _watermark.RestartWatermark },
{ "targetedEventIds", string.Join(",", _targetedEventIds.Concat(otherTargetedEventIds ?? Enumerable.Empty<int>()).Distinct().OrderBy(id => id)) },
};
if (wuClient.Truncated || orchestrator.Truncated)
data["censusTruncated"] = true;
@@ -666,102 +780,5 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
/// <summary>Compact "id=count" form ordered by EventID, e.g. "19=2,25=1,43=2".</summary>
internal static string FormatHistogram(Dictionary<int, int> histogram) =>
string.Join(",", histogram.OrderBy(kv => kv.Key).Select(kv => $"{kv.Key}={kv.Value}"));
// -----------------------------------------------------------------------
// Watermark dedup (cross-restart)
// -----------------------------------------------------------------------
private bool IsAlreadyProcessed(long recordId)
{
if (recordId < 0) return false; // no RecordId → cannot dedup, let it through
lock (_watermarkLock)
{
return recordId <= _restartWatermark || _seenThisRun.Contains(recordId);
}
}
/// <summary>
/// Atomically claims <paramref name="recordId"/>. Returns true if this is the first time we
/// see it (caller should emit), false if it was already processed — either in a PRIOR run
/// (at/below the restart watermark) or already this run (in the seen-set). Only advances +
/// persists the max on a genuinely-new, higher RecordId. Records without a RecordId (-1) are
/// always emitted, never tracked or persisted.
/// </summary>
private bool MarkProcessed(long recordId)
{
if (recordId < 0) return true;
long toPersist = -1;
lock (_watermarkLock)
{
// Prior run, or already emitted this run. Note: a lower RecordId that was never
// emitted this run (e.g. a backfill record read after a higher live record) is NOT
// in _seenThisRun and is above the restart watermark, so it is correctly emitted.
if (recordId <= _restartWatermark || !_seenThisRun.Add(recordId))
return false;
if (recordId > _maxEmittedRecordId)
{
_maxEmittedRecordId = recordId;
toPersist = _maxEmittedRecordId;
}
}
if (toPersist >= 0) PersistWatermark(toPersist);
return true;
}
internal void LoadWatermark()
{
if (string.IsNullOrEmpty(_stateDirectory)) return;
var filePath = Path.Combine(_stateDirectory, WatermarkStateFileName);
if (!File.Exists(filePath)) return;
try
{
var json = File.ReadAllText(filePath);
var state = JsonConvert.DeserializeObject<WatermarkState>(json);
if (state != null)
{
lock (_watermarkLock)
{
_restartWatermark = state.LastRecordId;
_maxEmittedRecordId = state.LastRecordId;
}
_logger.Info($"WindowsUpdate watermark loaded: lastRecordId={state.LastRecordId} (persisted {state.PersistedUtc:O})");
}
}
catch (Exception ex)
{
_logger.Warning($"Failed to load WindowsUpdate watermark: {ex.Message}");
}
}
private void PersistWatermark(long recordId)
{
if (string.IsNullOrEmpty(_stateDirectory)) return;
try
{
Directory.CreateDirectory(_stateDirectory);
var filePath = Path.Combine(_stateDirectory, WatermarkStateFileName);
var state = new WatermarkState { LastRecordId = recordId, PersistedUtc = DateTime.UtcNow };
var json = JsonConvert.SerializeObject(state, Formatting.Indented);
var tempPath = filePath + ".tmp";
File.WriteAllText(tempPath, json);
File.Copy(tempPath, filePath, overwrite: true);
try { File.Delete(tempPath); } catch { }
}
catch (Exception ex)
{
_logger.Warning($"Failed to persist WindowsUpdate watermark: {ex.Message}");
}
}
internal sealed class WatermarkState
{
public long LastRecordId { get; set; }
public DateTime PersistedUtc { get; set; }
}
}
}
@@ -132,12 +132,21 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
_informationalEvents = informationalEvents;
_parkedTripwireDwell = parkedTripwireDwell ?? DefaultParkedTripwireDwell;
// If recovery loaded a state that already sits on a terminal stage (e.g. a crash
// after a success-path step but before Stop()), treat it as already-notified so we
// do not re-fire the hook.
_terminalNotified = initialState.Stage.IsTerminal();
// If recovery loaded a state that already ends the session (e.g. a crash after a
// success-path step but before Stop()), treat it as already-notified so we do not
// re-fire the hook — the orchestrator handles that state at start.
_terminalNotified = EndsSession(initialState);
}
/// <summary>
/// True when the state ends this agent's session: a terminal stage, or the engine stopped
/// waiting for the user without a verdict (<see cref="DecisionState.StoppedWaitingReason"/>,
/// D-310) — the stage stays non-terminal there, and the agent ends the session like its
/// max-lifetime watchdog.
/// </summary>
internal static bool EndsSession(DecisionState state) =>
state.Stage.IsTerminal() || state.StoppedWaitingReason != null;
public DecisionState CurrentState => _currentState;
/// <summary>Test-Observability — anzahl aufeinanderfolgender Journal-Failures.</summary>
@@ -284,7 +293,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
// orchestrator turns this into the public EnrollmentTerminated event so peripheral
// consumers (CleanupService, SummaryDialog, DiagnosticsPackageService) can react
// without touching the kernel state machine.
if (!_terminalNotified && _currentState.Stage.IsTerminal())
if (!_terminalNotified && EndsSession(_currentState))
{
_terminalNotified = true;
try { _onTerminalStageReached?.Invoke(_currentState); }
@@ -405,7 +414,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
/// </summary>
private static bool IsParkedWithoutDeadline(DecisionState state)
{
if (state.Stage.IsTerminal()) return false;
if (EndsSession(state)) return false;
if (state.Stage == SessionStage.Unknown || state.Stage == SessionStage.SessionStarted) return false;
// WDP (afee7ae0): Device Preparation has no ESP, so neither the AccountSetup
@@ -547,9 +547,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
stateDirectory: _stateDirectory);
hosts.Add(osBuildChangeHost);
// Windows Update during OOBE watcher — subscribes to WindowsUpdateClient/Operational and
// backfills recent events (OOBE quality updates run before the agent starts). Surfaces
// quality/cumulative updates installing/failing DURING enrollment — otherwise invisible.
// OS update watchers (D-310) — the Windows Update client on the System log (install
// events) and its Operational channel (scan/download), plus the CBS servicing steps on
// the Setup log; each backfills recent events because updates can start before the
// agent does. Surfaces Windows/.NET updates installing/failing DURING enrollment and,
// unless OobeUpdatePhaseMode is Off, feeds them into the engine's OOBE update phase.
if (collectors.WindowsUpdateWatcherEnabled)
{
hosts.Add(new WindowsUpdateWatcherHost(
@@ -562,7 +564,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
backfillLookbackMinutes: collectors.WindowsUpdateBackfillLookbackMinutes,
stateDirectory: _stateDirectory,
channelCensusEnabled: collectors.WindowsUpdateChannelCensusEnabled,
osBuildChangedProvider: () => osBuildChangeHost.BuildChanged));
osBuildChangedProvider: () => osBuildChangeHost.BuildChanged,
operationalEventIds: collectors.WindowsUpdateOperationalEventIds,
servicingWatcherEnabled: collectors.ServicingWatcherEnabled,
osUpdateSignalsEnabled: DecisionCore.State.OobeUpdatePhaseModes.PostsSignals(collectors.OobeUpdatePhaseMode)));
}
// MDM reboot-policy watcher — subscribes to DeviceManagement-Enterprise-Diagnostics-
@@ -740,7 +740,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
// itself is constructed inside the onIngressReady hook (step 13b), which has
// already fired by this point — so when our recovery-fire propagates through
// Task.Run, the handler is fully wired.
if (initialState.Stage == SessionStage.Completed || initialState.Stage == SessionStage.Failed)
// The same applies to a state in which the engine stopped waiting (D-310): a
// crash between that step and the termination must not leave the session running.
if (initialState.Stage == SessionStage.Completed || initialState.Stage == SessionStage.Failed
|| initialState.StoppedWaitingReason != null)
{
_logger.Warning(
$"EnrollmentOrchestrator: post-recovery initialState already terminal " +
@@ -869,25 +872,40 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
// Stop the max-lifetime watchdog — the real terminal arrived before it could trip.
try { _maxLifetimeTimer?.Dispose(); _maxLifetimeTimer = null; } catch { }
var outcome = terminalState.Stage switch
{
SessionStage.Completed or SessionStage.WhiteGloveSealed
=> EnrollmentTerminationOutcome.Succeeded,
SessionStage.Failed => EnrollmentTerminationOutcome.Failed,
_ => EnrollmentTerminationOutcome.TimedOut,
};
// D-310: the engine stopped waiting for the user without a verdict. The stage stays
// non-terminal; the session ends like the max-lifetime watchdog ends it (TimedOut), so
// the backend classifies it honestly — never as a failure.
var stoppedWaiting = !terminalState.Stage.IsTerminal() && terminalState.StoppedWaitingReason != null;
var outcome = stoppedWaiting
? EnrollmentTerminationOutcome.TimedOut
: terminalState.Stage switch
{
SessionStage.Completed or SessionStage.WhiteGloveSealed
=> EnrollmentTerminationOutcome.Succeeded,
SessionStage.Failed => EnrollmentTerminationOutcome.Failed,
_ => EnrollmentTerminationOutcome.TimedOut,
};
var reason = stoppedWaiting
? EnrollmentTerminationReason.StoppedWaiting
: EnrollmentTerminationReason.DecisionTerminalStage;
_logger.Info(
$"EnrollmentOrchestrator: decision terminal stage reached (stage={terminalState.Stage}, outcome={outcome}) — dispatching Terminated(DecisionTerminalStage) off worker.");
$"EnrollmentOrchestrator: decision ended the session (stage={terminalState.Stage}, outcome={outcome}, reason={reason}) — dispatching Terminated off worker.");
string? details = null;
if (stoppedWaiting)
details = StoppedWaitingReasons.Describe(terminalState.StoppedWaitingReason!.Value);
else if (terminalState.Stage.IsPauseBeforePart2())
details = "WhiteGlove Part 1 sealed — session will resume on Part 2 post-reboot; self-destruct suppressed.";
var terminatedArgs = new EnrollmentTerminatedEventArgs(
reason: EnrollmentTerminationReason.DecisionTerminalStage,
reason: reason,
outcome: outcome,
stageName: terminalState.Stage.ToString(),
terminatedAtUtc: _clock.UtcNow,
details: terminalState.Stage.IsPauseBeforePart2()
? "WhiteGlove Part 1 sealed — session will resume on Part 2 post-reboot; self-destruct suppressed."
: null);
details: details,
stopReason: stoppedWaiting ? terminalState.StoppedWaitingReason!.Value : null);
// Codex Finding 2 — off-worker dispatch. Task.Run unblocks the ingress worker
// immediately; the worker keeps draining the channel (including events the
@@ -16,6 +16,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
/// longer than the configured cap without reaching a terminal stage.
/// </summary>
MaxLifetimeExceeded,
/// <summary>
/// The DecisionEngine stopped waiting for the user without a verdict — a bounded wait
/// expired (<c>DecisionState.StoppedWaitingReason</c>, D-310). Ends the session like
/// <see cref="MaxLifetimeExceeded"/> (outcome TimedOut), with the engine's reason.
/// </summary>
StoppedWaiting,
}
/// <summary>
@@ -60,18 +67,26 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
public DateTime TerminatedAtUtc { get; }
public string? Details { get; }
/// <summary>
/// The engine's reason code for <see cref="EnrollmentTerminationReason.StoppedWaiting"/>
/// (e.g. <c>oobe_update_no_sign_in</c>); null for every other reason.
/// </summary>
public string? StopReason { get; }
public EnrollmentTerminatedEventArgs(
EnrollmentTerminationReason reason,
EnrollmentTerminationOutcome outcome,
string? stageName,
DateTime terminatedAtUtc,
string? details = null)
string? details = null,
string? stopReason = null)
{
Reason = reason;
Outcome = outcome;
StageName = stageName;
TerminatedAtUtc = terminatedAtUtc;
Details = details;
StopReason = stopReason;
}
}
}
@@ -1,26 +1,41 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.DecisionCore.State;
using AutopilotMonitor.Shared.Models;
using SharedConstants = AutopilotMonitor.Shared.Constants;
namespace AutopilotMonitor.Agent.V2.Core.Orchestration
{
/// <summary>
/// Standalone host for the <see cref="WindowsUpdateTracker"/> — watches
/// <c>Microsoft-Windows-WindowsUpdateClient/Operational</c> for quality/cumulative update
/// activity during OOBE / ESP and forwards it via the shared single-rail ingress.
/// Mirrors <see cref="StallProbeHost"/>: builds its own <see cref="InformationalEventPost"/>
/// and delegates Start/Stop to the tracker.
/// Hosts the OS-update watchers (D-310): the Windows Update client on the System log
/// (install events) and on its Operational channel (scan/download events), plus the CBS
/// servicing steps on the Setup log. Routes every configured WU EventID to the channel the
/// provider manifest assigns it, runs the blind-spot census across all channels, reports the
/// counted (non-OS) activity once on stop, and — unless <c>OobeUpdatePhaseMode</c> is Off —
/// feeds Windows/.NET update steps into the decision engine as
/// <see cref="DecisionSignalKind.OsUpdateActivity"/>.
/// </summary>
internal sealed class WindowsUpdateWatcherHost : ICollectorHost
{
public string Name => "WindowsUpdateTracker";
private readonly WindowsUpdateTracker _tracker;
private readonly WindowsUpdateTracker? _systemTracker;
private readonly WindowsUpdateTracker? _operationalTracker;
private readonly ServicingTracker? _servicingTracker;
private readonly UpdateActivityTally _tally = new UpdateActivityTally();
private readonly InformationalEventPost _post;
private readonly ISignalIngressSink _ingress;
private readonly AgentLogger _logger;
private readonly string _sessionId;
private readonly string _tenantId;
private int _summaryEmitted;
private int _disposed;
public WindowsUpdateWatcherHost(
@@ -33,36 +48,125 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
int backfillLookbackMinutes,
string stateDirectory,
bool channelCensusEnabled = true,
Func<bool>? osBuildChangedProvider = null)
Func<bool>? osBuildChangedProvider = null,
int[]? operationalEventIds = null,
bool servicingWatcherEnabled = true,
bool osUpdateSignalsEnabled = false)
{
if (ingress == null) throw new ArgumentNullException(nameof(ingress));
if (clock == null) throw new ArgumentNullException(nameof(clock));
_ingress = ingress ?? throw new ArgumentNullException(nameof(ingress));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_sessionId = sessionId ?? throw new ArgumentNullException(nameof(sessionId));
_tenantId = tenantId ?? throw new ArgumentNullException(nameof(tenantId));
_post = new InformationalEventPost(ingress, clock, logger);
var post = new InformationalEventPost(ingress, clock, logger);
_tracker = new WindowsUpdateTracker(
sessionId: sessionId,
tenantId: tenantId,
post: post,
logger: logger,
targetedEventIds: targetedEventIds,
backfillEnabled: backfillLookbackMinutes > 0,
backfillLookbackMinutes: backfillLookbackMinutes,
stateDirectory: stateDirectory,
channelCensusEnabled: channelCensusEnabled,
osBuildChangedProvider: osBuildChangedProvider);
var (systemIds, operationalIds) = RouteEventIds(targetedEventIds, operationalEventIds);
Action<string, string, DateTime>? wuActivity = null;
Action<string, string, DateTime>? servicingActivity = null;
if (osUpdateSignalsEnabled)
{
wuActivity = (step, update, at) => PostOsUpdateActivity(step, "wu", update, at);
servicingActivity = (step, update, at) => PostOsUpdateActivity(step, "servicing", update, at);
}
if (systemIds.Length > 0)
{
_systemTracker = new WindowsUpdateTracker(
sessionId: sessionId,
tenantId: tenantId,
post: _post,
logger: logger,
targetedEventIds: systemIds,
backfillEnabled: backfillLookbackMinutes > 0,
backfillLookbackMinutes: backfillLookbackMinutes,
stateDirectory: stateDirectory,
channelCensusEnabled: false,
channel: WindowsUpdateTracker.SystemChannel,
tally: _tally,
onOsUpdateActivity: wuActivity);
}
if (operationalIds.Length > 0)
{
_operationalTracker = new WindowsUpdateTracker(
sessionId: sessionId,
tenantId: tenantId,
post: _post,
logger: logger,
targetedEventIds: operationalIds,
backfillEnabled: backfillLookbackMinutes > 0,
backfillLookbackMinutes: backfillLookbackMinutes,
stateDirectory: stateDirectory,
channelCensusEnabled: channelCensusEnabled,
osBuildChangedProvider: osBuildChangedProvider,
channel: WindowsUpdateTracker.Channel,
tally: _tally,
onOsUpdateActivity: wuActivity);
}
if (servicingWatcherEnabled)
{
_servicingTracker = new ServicingTracker(
sessionId: sessionId,
tenantId: tenantId,
post: _post,
logger: logger,
backfillEnabled: backfillLookbackMinutes > 0,
backfillLookbackMinutes: backfillLookbackMinutes,
stateDirectory: stateDirectory,
tally: _tally,
onOsUpdateActivity: servicingActivity);
}
}
// Test seams — expose the trackers so unit tests can drive their ProcessEvent seams.
internal WindowsUpdateTracker? SystemTrackerForTest => _systemTracker;
internal WindowsUpdateTracker? OperationalTrackerForTest => _operationalTracker;
internal ServicingTracker? ServicingTrackerForTest => _servicingTracker;
/// <summary>
/// Splits the configured WU EventIDs by the channel the provider manifest writes them to.
/// An ID configured in either list ends up on its real channel, so a misplaced ID still
/// works instead of watching a channel that never carries it.
/// </summary>
internal static (int[] systemIds, int[] operationalIds) RouteEventIds(int[]? targetedEventIds, int[]? operationalEventIds)
{
var all = (targetedEventIds ?? Array.Empty<int>())
.Concat(operationalEventIds ?? Array.Empty<int>())
.Distinct()
.OrderBy(id => id)
.ToArray();
return (
all.Where(WindowsUpdateTracker.IsSystemChannelEventId).ToArray(),
all.Where(id => !WindowsUpdateTracker.IsSystemChannelEventId(id)).ToArray());
}
public void Start()
{
_tracker.Start();
_systemTracker?.Start();
_operationalTracker?.Start();
_servicingTracker?.Start();
// The census asks "did ANY channel see the update?" — only answerable after every
// backfill ran.
_operationalTracker?.EmitChannelCensusIfBlind(
otherChannelsEmitted: _systemTracker?.EmittedThisRun ?? 0,
otherTargetedEventIds: _systemTracker?.TargetedEventIds);
_logger.Info("WindowsUpdateWatcherHost: started.");
}
public void Stop()
{
try { _tracker.Stop(); }
catch (Exception ex) { _logger.Warning($"WindowsUpdateWatcherHost: stop failed: {ex.Message}"); }
try { _systemTracker?.Stop(); }
catch (Exception ex) { _logger.Warning($"WindowsUpdateWatcherHost: system watcher stop failed: {ex.Message}"); }
try { _operationalTracker?.Stop(); }
catch (Exception ex) { _logger.Warning($"WindowsUpdateWatcherHost: operational watcher stop failed: {ex.Message}"); }
try { _servicingTracker?.Stop(); }
catch (Exception ex) { _logger.Warning($"WindowsUpdateWatcherHost: servicing watcher stop failed: {ex.Message}"); }
EmitActivitySummaryOnce();
}
public void Dispose()
@@ -70,5 +174,61 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
if (Interlocked.Exchange(ref _disposed, 1) == 1) return;
Stop();
}
/// <summary>
/// One summary of everything that was counted instead of emitted. Stop runs twice on the
/// termination path (explicit stop, then dispose), hence the once-guard.
/// </summary>
internal void EmitActivitySummaryOnce()
{
if (_tally.IsEmpty) return;
if (Interlocked.Exchange(ref _summaryEmitted, 1) == 1) return;
try
{
var counts = _tally.Snapshot();
var data = new Dictionary<string, object>(StringComparer.Ordinal);
foreach (var kv in counts) data[kv.Key] = kv.Value;
var total = counts.Sum(kv => kv.Value);
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
EventType = SharedConstants.EventTypes.WindowsUpdateActivitySummary,
Severity = EventSeverity.Debug,
Source = "WindowsUpdateWatcher",
Phase = EnrollmentPhase.Unknown,
Message = $"Update activity counted instead of listed: {total} event(s) " +
"(Store, Defender and other non-OS updates, repeated scan results, servicing of non-OS packages)",
Data = data,
ImmediateUpload = false,
});
}
catch (Exception ex)
{
_logger.Warning($"WindowsUpdateWatcherHost: activity summary failed: {ex.Message}");
}
}
private void PostOsUpdateActivity(string step, string source, string update, DateTime occurredAtUtc)
{
var payload = new Dictionary<string, string>(StringComparer.Ordinal)
{
[OsUpdatePayloadKeys.Step] = step,
[OsUpdatePayloadKeys.Source] = source,
};
if (!string.IsNullOrEmpty(update)) payload[OsUpdatePayloadKeys.Update] = update;
_ingress.Post(
kind: DecisionSignalKind.OsUpdateActivity,
occurredAtUtc: DateTime.SpecifyKind(occurredAtUtc, DateTimeKind.Utc),
sourceOrigin: source == "servicing" ? "ServicingWatcher" : "WindowsUpdateWatcher",
evidence: new Evidence(
kind: EvidenceKind.Raw,
identifier: $"os-update-activity:{source}",
summary: string.IsNullOrEmpty(update) ? step : $"{step}: {update}"),
payload: payload);
}
}
}
@@ -92,6 +92,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
SharedConstants.EventTypes.ImePatternHits,
// Bookkeeping of runs that were deliberately not reported as activity.
SharedConstants.EventTypes.ScriptRecurrenceSummary,
// Same for the update watchers' tally of Store/Defender/other update events (D-310):
// emitted on stop, never progress. Windows/.NET update steps stay activity.
SharedConstants.EventTypes.WindowsUpdateActivitySummary,
// The OOBE update registry snapshot (D-310) is observation at agent start/stop and at
// the update page — the page's own records (oobe_update_page) stay activity.
SharedConstants.EventTypes.OobeUpdateState,
};
/// <summary>
@@ -1227,11 +1227,25 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
case EnrollmentTerminationReason.MaxLifetimeExceeded:
reasonTag = "max_lifetime";
break;
case EnrollmentTerminationReason.StoppedWaiting:
// The backend classifies this shutdown honestly, exactly like max_lifetime.
reasonTag = "stopped_waiting";
break;
default:
reasonTag = args.Reason.ToString();
break;
}
var data = new Dictionary<string, object>
{
{ "reason", reasonTag },
{ "outcome", args.Outcome.ToString() },
{ "stage", args.StageName ?? string.Empty },
{ "uptimeMinutes", uptimeMinutes },
{ "agentVersion", _agentVersion },
};
if (!string.IsNullOrEmpty(args.StopReason)) data["stopReason"] = args.StopReason;
EmitEventSafe(new EnrollmentEvent
{
SessionId = _configuration.SessionId,
@@ -1240,15 +1254,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
Severity = EventSeverity.Info,
Source = "EnrollmentTerminationHandler",
Phase = EnrollmentPhase.Unknown,
Message = $"Agent shutting down (reason={reasonTag}, outcome={args.Outcome}).",
Data = new Dictionary<string, object>
{
{ "reason", reasonTag },
{ "outcome", args.Outcome.ToString() },
{ "stage", args.StageName ?? string.Empty },
{ "uptimeMinutes", uptimeMinutes },
{ "agentVersion", _agentVersion },
},
Message = args.Reason == EnrollmentTerminationReason.StoppedWaiting && !string.IsNullOrEmpty(args.Details)
? $"Agent shutting down (reason={reasonTag}, outcome={args.Outcome}): {args.Details}"
: $"Agent shutting down (reason={reasonTag}, outcome={args.Outcome}).",
Data = data,
ImmediateUpload = true,
});
}
@@ -526,7 +526,11 @@ namespace AutopilotMonitor.Agent.V2.Runtime
// analytics. WhiteGlove Part-2 resume runs as a fresh Classic
// enrollment after PR-A's archive-and-reset, so the engine state is
// empty and needs the facts re-seeded — same as a first boot.
LifecycleEmitters.PostEnrollmentFactsObserved(orchestrator.IngressSink, agentConfig.HelloWaitTimeoutSeconds, logger);
LifecycleEmitters.PostEnrollmentFactsObserved(
orchestrator.IngressSink,
agentConfig.HelloWaitTimeoutSeconds,
logger,
remoteConfig.Collectors?.OobeUpdatePhaseMode);
// V2 parity — post SessionStarted so the reducer establishes the session
// anchor (HandleSessionStartedV1 in DecisionEngine.Shared.cs). Skipped only
@@ -315,7 +315,8 @@ namespace AutopilotMonitor.Agent.V2.Runtime
/// registry-derived enrollment facts (<c>enrollmentType</c> + <c>isHybridJoin</c>)
/// so the reducer can seed <see cref="State.EnrollmentScenarioProfile"/> via the
/// stage-agnostic <c>HandleEnrollmentFactsObservedV1</c> handler — plus the tenant's
/// Hello wait when it extends the engine's built-in hello_safety window.
/// Hello wait when it extends the engine's built-in hello_safety window, and the mode of
/// the OOBE update phase (D-310).
/// <para>
/// Posted immediately before <see cref="PostSessionStarted"/> so the Inspector
/// timeline reads naturally (facts → anchor) — but the reducer correctness does
@@ -331,7 +332,8 @@ namespace AutopilotMonitor.Agent.V2.Runtime
public static void PostEnrollmentFactsObserved(
ISignalIngressSink ingressSink,
int helloWaitTimeoutSeconds,
AgentLogger logger)
AgentLogger logger,
string oobeUpdatePhaseMode = null)
{
try
{
@@ -364,6 +366,12 @@ namespace AutopilotMonitor.Agent.V2.Runtime
helloWaitSeconds.ToString(System.Globalization.CultureInfo.InvariantCulture);
}
// OOBE update phase mode (remote config) — stamped at every start, the same value
// the update watchers were wired with: the engine lets the phase act only while
// every start of the session said Active, so Off or Shadow is a kill switch.
var phaseMode = OobeUpdatePhaseModes.Normalize(oobeUpdatePhaseMode);
payload[SignalPayloadKeys.OobeUpdatePhaseMode] = phaseMode;
var evidence = new Evidence(
kind: EvidenceKind.Synthetic,
identifier: "enrollment_registry_facts_read",
@@ -376,7 +384,7 @@ namespace AutopilotMonitor.Agent.V2.Runtime
evidence: evidence,
payload: payload);
logger.Debug($"EnrollmentFactsObserved signal posted (enrollmentType={enrollmentType}, isHybridJoin={isHybridJoin}, isSelfDeployingProfile={isSelfDeploying}, isCloudPc={isCloudPc}, helloWaitSeconds={helloWaitSeconds}).");
logger.Debug($"EnrollmentFactsObserved signal posted (enrollmentType={enrollmentType}, isHybridJoin={isHybridJoin}, isSelfDeployingProfile={isSelfDeploying}, isCloudPc={isCloudPc}, helloWaitSeconds={helloWaitSeconds}, oobeUpdatePhaseMode={phaseMode}).");
}
catch (Exception ex)
{
@@ -117,7 +117,9 @@ namespace AutopilotMonitor.DecisionCore.Tests
// Session 4910a5a5 (2026-07-23) added `espAdvisoryFailureResolvedUtc` (the failed
// category recovered — lifts the advisory's re-arm/rebase guard exemption) and
// `espAdvisoryFailureCategory` (which category the advisory was for) → 20.
Assert.Equal(20, facts.Count);
// D-310 (2026-10-02) added `stoppedWaitingReason` — the engine stopped waiting for
// the user without a verdict (no sign-in after the OOBE update) → 21.
Assert.Equal(21, facts.Count);
Assert.All(facts.Values, v => Assert.Null(v));
}
@@ -242,9 +244,11 @@ namespace AutopilotMonitor.DecisionCore.Tests
// that vetoes/retracts the policy-disabled Hello skip → 18. Session 4910a5a5
// (2026-07-23) added `espAdvisoryFailureResolvedUtc` + `espAdvisoryFailureCategory`
// — advisory-recovery facts that lift the AdvisoryCompletion guard exemption → 20.
// D-310 (2026-10-02) added `stoppedWaitingReason` — the engine stopped waiting for
// the user without a verdict → 21.
// If this number ever changes, both the count expectation AND the actual snapshot
// output need to evolve in lockstep.
Assert.Equal(20, expectedFactKeys.Count);
Assert.Equal(21, expectedFactKeys.Count);
var snapshot = DecisionStateSnapshotBuilder.Build(DecisionState.CreateInitial("s", "t"));
var facts = (Dictionary<string, object?>)snapshot["facts"]!;
@@ -0,0 +1,749 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Serialization;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.DecisionCore.State;
using Newtonsoft.Json.Linq;
using Xunit;
namespace AutopilotMonitor.DecisionCore.Tests
{
/// <summary>
/// Report 3a2207978e4c (2026-10-01, D-310) — the OOBE quality update. With the ESP setting
/// "Install Windows quality updates" Windows installs the update after the Device-ESP page
/// closes and before the user signs in: 20–40 minutes plus a restart to the lock screen. The
/// IME had logged its AccountSetup phase line before that handoff exit, so the exit armed the
/// 30-min <c>advisory_completion</c> window, and the session was failed
/// (<c>esp_exit_without_completion_evidence</c>) while the update was still installing.
/// <para>
/// The OOBE update phase holds the window, every state bounded on its own (user decision
/// 2026-10-02): the update runs at most 3 h, then fails as not finished; after the update the
/// user has 60 min to sign in, then the engine stops waiting — no verdict, the backend
/// classifies; after the sign-in the user's own setup gets one window.
/// </para>
/// <para>
/// Only the Active mode acts (measurement first, user decision 2026-10-02). In the Shadow mode
/// — the default, and what an unstamped session counts as — the fire reports what the phase
/// would decide and today's rule decides; Off evaluates nothing.
/// </para>
/// </summary>
public sealed class OobeUpdatePhaseTests
{
private static readonly DateTime T0 = new DateTime(2026, 10, 1, 11, 0, 0, DateTimeKind.Utc);
private const string Lcu = "2026-09 Cumulative Update for Windows 11, version 25H2 for x64-based Systems (KB5099999)";
private static DecisionSignal MakeSignal(
long ordinal,
DecisionSignalKind kind,
DateTime occurredAtUtc,
IReadOnlyDictionary<string, string>? payload = null)
{
return new DecisionSignal(
sessionSignalOrdinal: ordinal,
sessionTraceOrdinal: ordinal,
kind: kind,
kindSchemaVersion: 1,
occurredAtUtc: occurredAtUtc,
sourceOrigin: "test",
evidence: new Evidence(EvidenceKind.Synthetic, $"{kind}-{ordinal}", "test"),
payload: payload);
}
private static DecisionSignal OsUpdate(long ordinal, DateTime at, string step, string? update = Lcu)
{
var payload = new Dictionary<string, string>
{
[OsUpdatePayloadKeys.Step] = step,
[OsUpdatePayloadKeys.Source] = "wu",
};
if (update != null) payload[OsUpdatePayloadKeys.Update] = update;
return MakeSignal(ordinal, DecisionSignalKind.OsUpdateActivity, at, payload);
}
private static DecisionSignal AdvisoryFired(long ordinal, DateTime at) =>
MakeSignal(ordinal, DecisionSignalKind.DeadlineFired, at,
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.AdvisoryCompletion });
private static DecisionSignal Reboot(long ordinal, DateTime at) =>
MakeSignal(ordinal, DecisionSignalKind.SystemRebootObserved, at);
/// <summary>The agent's mode stamp on <c>EnrollmentFactsObserved</c>, posted at every start.</summary>
private static DecisionSignal PhaseMode(long ordinal, DateTime at, string mode) =>
MakeSignal(ordinal, DecisionSignalKind.EnrollmentFactsObserved, at,
new Dictionary<string, string> { [SignalPayloadKeys.OobeUpdatePhaseMode] = mode });
private static bool HasTimelineEffect(DecisionStep step, string eventType) =>
step.Effects.Any(e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry
&& e.Parameters != null
&& e.Parameters.TryGetValue("eventType", out var et)
&& et == eventType);
private static ActiveDeadline? FindDeadline(DecisionState state, string name) =>
state.Deadlines.FirstOrDefault(d => d.Name == name);
private static DecisionEffect SingleTimelineEffect(DecisionStep step, string eventType) =>
step.Effects.Single(e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry
&& e.Parameters != null
&& e.Parameters.TryGetValue("eventType", out var et)
&& et == eventType);
/// <summary>
/// The report's shape up to the arming exit: Classic user-driven, full ESP, DeviceSetup,
/// the IME's pre-sign-in AccountSetup line, then the Device-ESP handoff exit at T0+21,
/// which arms the window (due T0+51). <paramref name="beforeExit"/> runs before the exit;
/// <paramref name="mode"/> is the agent's mode stamp (null: none, an older agent).
/// </summary>
private static DecisionState HandoffExitArmsTheWindow(
DecisionEngine engine,
Func<DecisionState, DecisionState>? beforeExit = null,
string? mode = OobeUpdatePhaseModes.Active)
{
var state = DecisionState.CreateInitial("sess-3a220797", "tenant-3a220797", T0);
state = engine.Reduce(state, MakeSignal(0, DecisionSignalKind.SessionStarted, T0)).NewState;
if (mode != null) state = engine.Reduce(state, PhaseMode(1, T0, mode)).NewState;
state = engine.Reduce(state, MakeSignal(
5, DecisionSignalKind.EspConfigDetected, T0.AddMinutes(1),
new Dictionary<string, string>
{
[SignalPayloadKeys.SkipUserEsp] = "false",
[SignalPayloadKeys.SkipDeviceEsp] = "false",
[SignalPayloadKeys.EspAllowContinueAnyway] = "false",
})).NewState;
state = engine.Reduce(state, MakeSignal(
10, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(1),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "DeviceSetup" })).NewState;
state = engine.Reduce(state, MakeSignal(
20, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(20),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState;
if (beforeExit != null) state = beforeExit(state);
state = engine.Reduce(state, MakeSignal(30, DecisionSignalKind.EspExiting, T0.AddMinutes(21))).NewState;
var armed = FindDeadline(state, DeadlineNames.AdvisoryCompletion);
Assert.NotNull(armed);
Assert.Equal(T0.AddMinutes(51), armed!.DueAtUtc);
Assert.Null(state.Outcome);
return state;
}
/// <summary>
/// Fires the armed advisory deadline at its due time until a fire does not re-arm it;
/// returns every step. Proves each re-arm lands on time — the last one on the bound.
/// </summary>
private static List<DecisionStep> FireUntilResolved(DecisionEngine engine, DecisionState state, long firstOrdinal)
{
var steps = new List<DecisionStep>();
var ordinal = firstOrdinal;
while (true)
{
var deadline = FindDeadline(state, DeadlineNames.AdvisoryCompletion);
Assert.NotNull(deadline);
var step = engine.Reduce(state, AdvisoryFired(ordinal++, deadline!.DueAtUtc));
steps.Add(step);
state = step.NewState;
if (FindDeadline(state, DeadlineNames.AdvisoryCompletion) == null) return steps;
Assert.True(steps.Count < 50, "the phase never resolved");
}
}
// ================================================================= handler ====
[Fact]
public void ActivityAfterTheHandoffExit_AnchorsThePhase()
{
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine);
var first = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(22), OsUpdateSteps.DownloadStarted));
Assert.True(first.Transition.Taken);
Assert.Equal("OsUpdateActivity:OobeUpdatePhaseStarted", first.Transition.Trigger);
Assert.Equal(state.Stage, first.NewState.Stage);
Assert.Empty(first.Effects);
var facts = first.NewState.OsUpdateFacts;
Assert.Equal(T0.AddMinutes(22), facts.PostExitActivityUtc!.Value);
Assert.Equal(40, facts.PostExitActivityUtc.SourceSignalOrdinal);
Assert.True(DecisionEngine.IsInOobeUpdatePhase(first.NewState));
var second = engine.Reduce(first.NewState, OsUpdate(41, T0.AddMinutes(35), OsUpdateSteps.InstallStarted));
Assert.Equal("OsUpdateActivity", second.Transition.Trigger);
facts = second.NewState.OsUpdateFacts;
Assert.Equal(T0.AddMinutes(22), facts.PostExitActivityUtc!.Value);
Assert.Equal(T0.AddMinutes(35), facts.LastActivityUtc!.Value);
Assert.Equal(OsUpdateSteps.InstallStarted, facts.LastStep!.Value);
Assert.Equal(Lcu, facts.LastUpdate!.Value);
Assert.Equal(2, facts.ActivityCount);
}
[Fact]
public void RebootRequiredAndFailure_AreRecorded()
{
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine);
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(30), OsUpdateSteps.RebootRequired, "KB5099999")).NewState;
state = engine.Reduce(state, OsUpdate(41, T0.AddMinutes(31), OsUpdateSteps.RebootRequired, "KB5099999")).NewState;
state = engine.Reduce(state, OsUpdate(42, T0.AddMinutes(32), OsUpdateSteps.Failed, update: null)).NewState;
var facts = state.OsUpdateFacts;
Assert.Equal(T0.AddMinutes(30), facts.RebootRequiredUtc!.Value); // earliest
Assert.Equal(T0.AddMinutes(32), facts.FailedUtc!.Value);
Assert.Equal("KB5099999", facts.LastUpdate!.Value); // a step without an update keeps the last one
}
[Fact]
public void ActivityWithoutStep_IsADeadEnd()
{
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine);
var step = engine.Reduce(state, MakeSignal(40, DecisionSignalKind.OsUpdateActivity, T0.AddMinutes(22),
new Dictionary<string, string> { [OsUpdatePayloadKeys.Source] = "wu" }));
Assert.False(step.Transition.Taken);
Assert.Equal("os_update_activity_without_step", step.Transition.DeadEndReason);
Assert.Equal(0, step.NewState.OsUpdateFacts.ActivityCount);
}
[Fact]
public void ActivityBeforeTheExit_IsNoPhase()
{
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine, beforeExit: s =>
engine.Reduce(s, OsUpdate(25, T0.AddMinutes(15), OsUpdateSteps.Installed)).NewState);
Assert.NotNull(state.OsUpdateFacts.FirstActivityUtc);
Assert.Null(state.OsUpdateFacts.PostExitActivityUtc);
Assert.False(DecisionEngine.IsInOobeUpdatePhase(state));
}
[Fact]
public void BackfilledActivityThatPredatesTheExit_IsNoPhase()
{
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine);
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(18), OsUpdateSteps.InstallStarted)).NewState;
Assert.Null(state.OsUpdateFacts.PostExitActivityUtc);
Assert.Equal(1, state.OsUpdateFacts.ActivityCount);
}
[Fact]
public void ActivityAfterTheUserSignedIn_IsNoPhase()
{
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine);
state = engine.Reduce(state, MakeSignal(35, DecisionSignalKind.HelloWizardStarted, T0.AddMinutes(25))).NewState;
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(30), OsUpdateSteps.InstallStarted)).NewState;
Assert.Null(state.OsUpdateFacts.PostExitActivityUtc);
Assert.False(DecisionEngine.IsInOobeUpdatePhase(state));
}
// ============================================================ restart facts ====
[Fact]
public void RestartDuringThePhase_IsRecorded()
{
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine);
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(22), OsUpdateSteps.DownloadStarted)).NewState;
state = engine.Reduce(state, Reboot(50, T0.AddMinutes(53))).NewState;
Assert.Equal(T0.AddMinutes(53), state.OsUpdateFacts.RestartUtc!.Value);
Assert.Equal(50, state.OsUpdateFacts.RestartUtc.SourceSignalOrdinal);
}
[Fact]
public void RestartBeforeThePhase_OrAfterTheSignIn_IsNotTheUpdatesRestart()
{
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine);
state = engine.Reduce(state, Reboot(35, T0.AddMinutes(21.5))).NewState; // no anchor yet
Assert.Null(state.OsUpdateFacts.RestartUtc);
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(22), OsUpdateSteps.DownloadStarted)).NewState;
state = engine.Reduce(state, MakeSignal(45, DecisionSignalKind.HelloWizardStarted, T0.AddMinutes(30))).NewState;
state = engine.Reduce(state, Reboot(50, T0.AddMinutes(40))).NewState; // user already there
Assert.Null(state.OsUpdateFacts.RestartUtc);
}
// ======================================================= advisory window ====
[Fact]
public void Report3a2207978e4c_FireDuringTheUpdate_RearmsInsteadOfFailing()
{
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine);
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(22), OsUpdateSteps.DownloadStarted)).NewState;
state = engine.Reduce(state, OsUpdate(41, T0.AddMinutes(35), OsUpdateSteps.InstallStarted)).NewState;
var fireAt = T0.AddMinutes(51);
var step = engine.Reduce(state, AdvisoryFired(50, fireAt));
Assert.True(step.Transition.Taken);
Assert.Null(step.NewState.Outcome);
Assert.NotEqual(SessionStage.Failed, step.NewState.Stage);
Assert.Equal("DeadlineFired:advisory_completion:OobeUpdatePhase", step.Transition.Trigger);
var rearmed = FindDeadline(step.NewState, DeadlineNames.AdvisoryCompletion);
Assert.Equal(fireAt.AddMinutes(30), rearmed!.DueAtUtc);
var schedule = step.Effects.Single(e => e.Kind == DecisionEffectKind.ScheduleDeadline);
Assert.Equal(DeadlineNames.AdvisoryCompletion, schedule.Deadline!.Name);
var waiting = SingleTimelineEffect(step, "completion_waiting");
Assert.Contains("installing an update in OOBE", waiting.Parameters!["message"]);
Assert.Equal(OsUpdateSteps.InstallStarted, waiting.Parameters!["osUpdateLastStep"]);
Assert.Equal(Lcu, waiting.Parameters!["osUpdate"]);
Assert.Equal(rearmed.DueAtUtc.ToString("o"), waiting.Parameters!["resolutionDeadlineDueAtUtc"]);
// The update may run 3 h from its first activity.
Assert.Equal(T0.AddMinutes(22).AddHours(3).ToString("o"), waiting.Parameters!["oobeUpdateHoldsUntilUtc"]);
}
[Fact]
public void WithoutUpdateActivity_TheFireStillFails()
{
// Mutation proof: the guard keys on observed update activity, never on its absence.
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine);
var step = engine.Reduce(state, AdvisoryFired(50, T0.AddMinutes(51)));
Assert.Equal(SessionStage.Failed, step.NewState.Stage);
Assert.Equal("esp_exit_without_completion_evidence", SingleTimelineEffect(step, "enrollment_failed").Parameters!["reason"]);
}
[Fact]
public void ActivityOnlyBeforeTheExit_TheFireStillFails()
{
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine, beforeExit: s =>
engine.Reduce(s, OsUpdate(25, T0.AddMinutes(15), OsUpdateSteps.Installed)).NewState);
var step = engine.Reduce(state, AdvisoryFired(50, T0.AddMinutes(51)));
Assert.Equal(SessionStage.Failed, step.NewState.Stage);
}
[Fact]
public void UpdateStillRunningAtTheCap_FailsAsNotFinished()
{
// A hanging update: the last step says it still runs, and no restart ever comes.
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine);
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(22), OsUpdateSteps.DownloadStarted)).NewState;
state = engine.Reduce(state, OsUpdate(41, T0.AddMinutes(30), OsUpdateSteps.InstallStarted)).NewState;
var steps = FireUntilResolved(engine, state, firstOrdinal: 100);
Assert.All(steps.Take(steps.Count - 1), s =>
Assert.Equal("DeadlineFired:advisory_completion:OobeUpdatePhase", s.Transition.Trigger));
var cap = T0.AddMinutes(22).AddHours(3);
// The last re-arm lands on the cap, not up to 30 min after it.
Assert.Equal(cap, FindDeadline(steps[steps.Count - 2].NewState, DeadlineNames.AdvisoryCompletion)!.DueAtUtc);
var resolved = steps.Last();
Assert.Equal(SessionStage.Failed, resolved.NewState.Stage);
Assert.Equal(SessionOutcome.EnrollmentFailed, resolved.NewState.Outcome);
Assert.Equal("DeadlineFired:advisory_completion:OobeUpdateNotFinished", resolved.Transition.Trigger);
var failed = SingleTimelineEffect(resolved, "enrollment_failed");
Assert.Equal("oobe_update_not_finished", failed.Parameters!["reason"]);
Assert.Equal(OsUpdateSteps.InstallStarted, failed.Parameters!["osUpdateLastStep"]);
Assert.Equal("3", failed.Parameters!["oobeUpdateCapHours"]);
Assert.Null(resolved.NewState.StoppedWaitingReason);
}
[Fact]
public void UpdateWithRestart_WaitsSixtyMinutesForTheSignIn_ThenStopsWaiting()
{
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine);
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(22), OsUpdateSteps.DownloadStarted)).NewState;
state = engine.Reduce(state, OsUpdate(41, T0.AddMinutes(40), OsUpdateSteps.InstallStarted)).NewState;
state = engine.Reduce(state, OsUpdate(42, T0.AddMinutes(48), OsUpdateSteps.RebootRequired, "KB5099999")).NewState;
var beforeRestart = engine.Reduce(state, AdvisoryFired(50, T0.AddMinutes(51)));
Assert.Equal("DeadlineFired:advisory_completion:OobeUpdateAwaitingSignIn", beforeRestart.Transition.Trigger);
state = beforeRestart.NewState;
// The update page restarts the device; the lock screen waits for the user.
state = engine.Reduce(state, Reboot(60, T0.AddMinutes(53))).NewState;
Assert.Equal(T0.AddMinutes(83), FindDeadline(state, DeadlineNames.AdvisoryCompletion)!.DueAtUtc); // re-based
var steps = FireUntilResolved(engine, state, firstOrdinal: 100);
var waiting = steps.Take(steps.Count - 1).ToList();
Assert.All(waiting, s => Assert.Equal("DeadlineFired:advisory_completion:OobeUpdateAwaitingSignIn", s.Transition.Trigger));
Assert.Contains("waiting for the user to sign in",
SingleTimelineEffect(waiting[0], "completion_waiting").Parameters!["message"]);
var stopped = steps.Last();
Assert.Equal(T0.AddMinutes(53).AddMinutes(60), stopped.Transition.OccurredAtUtc);
Assert.Equal("DeadlineFired:advisory_completion:OobeUpdateSignInWaitExpired", stopped.Transition.Trigger);
Assert.True(stopped.Transition.Taken);
Assert.Equal(StoppedWaitingReasons.OobeUpdateNoSignIn, stopped.NewState.StoppedWaitingReason!.Value);
// No verdict: the stage stays, nothing is failed, nothing is armed any more.
Assert.Equal(state.Stage, stopped.NewState.Stage);
Assert.Null(stopped.NewState.Outcome);
Assert.Empty(stopped.NewState.Deadlines);
var expired = SingleTimelineEffect(stopped, "completion_wait_expired");
Assert.Equal("Warning", expired.Parameters!["severity"]);
Assert.Equal("true", expired.Parameters!["immediateUpload"]);
Assert.Equal(StoppedWaitingReasons.OobeUpdateNoSignIn, expired.Parameters!["reason"]);
Assert.Equal("60", expired.Parameters!["signInWindowMinutes"]);
Assert.Equal(T0.AddMinutes(53).ToString("o"), expired.Parameters!["oobeUpdateEndedUtc"]);
Assert.Contains("Not a failure", expired.Parameters!["message"]);
}
[Fact]
public void UpdateEndingWithoutRestart_SignInWindowStartsAtItsLastStep()
{
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine);
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(22), OsUpdateSteps.DownloadStarted)).NewState;
state = engine.Reduce(state, OsUpdate(41, T0.AddMinutes(40), OsUpdateSteps.Installed)).NewState;
var steps = FireUntilResolved(engine, state, firstOrdinal: 100);
var stopped = steps.Last();
Assert.Equal(T0.AddMinutes(100), stopped.Transition.OccurredAtUtc);
Assert.Equal(StoppedWaitingReasons.OobeUpdateNoSignIn, stopped.NewState.StoppedWaitingReason!.Value);
}
[Fact]
public void BackfilledOlderStepAfterTheRestart_DoesNotReopenTheUpdate()
{
// The watchers backfill after the restart: a record written before it arrives late.
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine);
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(22), OsUpdateSteps.DownloadStarted)).NewState;
state = engine.Reduce(state, OsUpdate(41, T0.AddMinutes(48), OsUpdateSteps.RebootRequired, "KB5099999")).NewState;
state = engine.Reduce(state, Reboot(60, T0.AddMinutes(53))).NewState;
state = engine.Reduce(state, OsUpdate(70, T0.AddMinutes(45), OsUpdateSteps.InstallStarted)).NewState;
Assert.Equal(OsUpdateSteps.RebootRequired, state.OsUpdateFacts.LastStep!.Value);
var phase = DecisionEngine.EvaluateOobeUpdatePhase(state, T0.AddMinutes(83));
Assert.Equal(DecisionEngine.OobeUpdatePhaseKind.AwaitingSignIn, phase.Kind);
Assert.Equal(T0.AddMinutes(113), phase.BoundUtc);
}
[Fact]
public void NewerUpdateStepAfterTheRestart_ReopensTheUpdate_BoundedByTheCap()
{
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine);
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(22), OsUpdateSteps.DownloadStarted)).NewState;
state = engine.Reduce(state, OsUpdate(41, T0.AddMinutes(48), OsUpdateSteps.RebootRequired, "KB5099999")).NewState;
state = engine.Reduce(state, Reboot(60, T0.AddMinutes(53))).NewState;
state = engine.Reduce(state, OsUpdate(70, T0.AddMinutes(60), OsUpdateSteps.DownloadStarted)).NewState;
var phase = DecisionEngine.EvaluateOobeUpdatePhase(state, T0.AddMinutes(83));
Assert.Equal(DecisionEngine.OobeUpdatePhaseKind.Updating, phase.Kind);
Assert.Equal(T0.AddMinutes(22).AddHours(3), phase.BoundUtc); // the cap counts from the first activity
}
[Fact]
public void UserSignIn_GetsOneFullWindow_ThenTheFireResolvesNormally()
{
// The transition to AwaitingHello does not cancel the window. Without the grace a fire
// right after the sign-in would fail the session inside the Hello wizard.
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine);
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(22), OsUpdateSteps.DownloadStarted)).NewState;
state = engine.Reduce(state, AdvisoryFired(50, T0.AddMinutes(51))).NewState; // due T0+81
state = engine.Reduce(state, MakeSignal(60, DecisionSignalKind.HelloWizardStarted, T0.AddMinutes(75))).NewState;
Assert.False(DecisionEngine.IsInOobeUpdatePhase(state));
// 6 minutes after the sign-in — inside the grace, which ends at T0+105.
var grace = engine.Reduce(state, AdvisoryFired(70, T0.AddMinutes(81)));
Assert.Null(grace.NewState.Outcome);
Assert.Equal("DeadlineFired:advisory_completion:OobeUpdatePhaseUserArrived", grace.Transition.Trigger);
Assert.Contains("signed in after the OOBE update", SingleTimelineEffect(grace, "completion_waiting").Parameters!["message"]);
Assert.Equal(T0.AddMinutes(105), FindDeadline(grace.NewState, DeadlineNames.AdvisoryCompletion)!.DueAtUtc);
// One full window after the sign-in, still no completion evidence — resolved the normal way.
var resolved = engine.Reduce(grace.NewState, AdvisoryFired(80, T0.AddMinutes(105)));
Assert.Equal(SessionStage.Failed, resolved.NewState.Stage);
Assert.Equal("esp_exit_without_completion_evidence", SingleTimelineEffect(resolved, "enrollment_failed").Parameters!["reason"]);
Assert.Null(resolved.NewState.StoppedWaitingReason);
// Active decides itself — no shadow report.
Assert.False(HasTimelineEffect(resolved, "oobe_update_phase_shadow"));
}
[Fact]
public void AdvisoryVariant_IsExempt()
{
// Mutation proof: an unresolved ESP terminal failure does not un-happen because
// Windows installs an update afterwards.
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine);
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(22), OsUpdateSteps.DownloadStarted)).NewState;
Assert.True(DecisionEngine.IsInOobeUpdatePhase(state));
var builder = state.ToBuilder();
builder.EspAdvisoryFailureRecordedUtc = new SignalFact<DateTime>(T0.AddMinutes(21), 30);
state = builder.Build();
var step = engine.Reduce(state, AdvisoryFired(50, T0.AddMinutes(51)));
Assert.Equal(SessionStage.Failed, step.NewState.Stage);
Assert.Equal("esp_terminal_failure", SingleTimelineEffect(step, "enrollment_failed").Parameters!["reason"]);
}
[Fact]
public void NoPhase_WithoutAnAnchor()
{
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine);
Assert.Equal(DecisionEngine.OobeUpdatePhaseKind.None,
DecisionEngine.EvaluateOobeUpdatePhase(state, T0.AddMinutes(51)).Kind);
}
// ================================================= serialization + census ====
[Fact]
public void SnapshotRoundTrip_PreservesTheUpdateFacts_AndTheStop()
{
// The update ends with a restart; the agent restores the engine state from its
// snapshot, so the phase — and a decided stop — must survive it.
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine);
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(22), OsUpdateSteps.DownloadStarted)).NewState;
state = engine.Reduce(state, OsUpdate(41, T0.AddMinutes(48), OsUpdateSteps.RebootRequired, "KB5099999")).NewState;
state = engine.Reduce(state, Reboot(60, T0.AddMinutes(53))).NewState;
var stopped = FireUntilResolved(engine, state, firstOrdinal: 100).Last().NewState;
var roundtripped = StateSerializer.Deserialize(StateSerializer.Serialize(stopped));
var facts = roundtripped.OsUpdateFacts;
Assert.Equal(T0.AddMinutes(22), facts.PostExitActivityUtc!.Value);
Assert.Equal(T0.AddMinutes(48), facts.RebootRequiredUtc!.Value);
Assert.Equal(T0.AddMinutes(53), facts.RestartUtc!.Value);
Assert.Equal(OsUpdateSteps.RebootRequired, facts.LastStep!.Value);
Assert.Equal(StoppedWaitingReasons.OobeUpdateNoSignIn, roundtripped.StoppedWaitingReason!.Value);
Assert.Equal(OobeUpdatePhaseModes.Active, roundtripped.ScenarioObservations.OobeUpdatePhaseMode!.Value);
}
[Fact]
public void LegacySnapshot_WithoutTheNewFacts_DeserializesToEmpty()
{
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine);
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(22), OsUpdateSteps.DownloadStarted)).NewState;
var legacy = JObject.Parse(StateSerializer.Serialize(state));
foreach (var prop in legacy.Properties()
.Where(p => p.Name.IndexOf("OsUpdateFacts", StringComparison.OrdinalIgnoreCase) >= 0
|| p.Name.IndexOf("StoppedWaitingReason", StringComparison.OrdinalIgnoreCase) >= 0)
.ToList())
{
prop.Remove();
}
var roundtripped = StateSerializer.Deserialize(legacy.ToString());
Assert.Same(OsUpdateFacts.Empty, roundtripped.OsUpdateFacts);
Assert.Null(roundtripped.StoppedWaitingReason);
Assert.False(DecisionEngine.IsInOobeUpdatePhase(roundtripped));
}
[Fact]
public void Census_SurfacesThePhase_AndTheStop()
{
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine);
Assert.DoesNotContain("oobe_update_phase", DecisionStateSignalCensus.Build(state).SignalsSeen);
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(22), OsUpdateSteps.Installed)).NewState;
var census = DecisionStateSignalCensus.Build(state);
Assert.Contains("oobe_update_phase", census.SignalsSeen);
Assert.Equal(T0.AddMinutes(22).ToString("o"), census.SignalTimestamps["oobeUpdatePhase"]);
Assert.DoesNotContain("stopped_waiting", census.SignalsSeen);
var stopped = FireUntilResolved(engine, state, firstOrdinal: 100).Last().NewState;
Assert.Contains("stopped_waiting", DecisionStateSignalCensus.Build(stopped).SignalsSeen);
}
[Theory]
[InlineData(OobeUpdatePhaseModes.Active, "Active")]
[InlineData(OobeUpdatePhaseModes.Shadow, "Shadow")]
[InlineData(null, "Shadow")] // never stamped
public void Census_NamesTheModeThePhaseRanIn(string? mode, string expected)
{
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine, mode: mode);
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(22), OsUpdateSteps.DownloadStarted)).NewState;
var evidence = (IDictionary<string, object>)DecisionStateSignalCensus.Build(state).SignalEvidence["oobeUpdatePhase"];
Assert.Equal(expected, evidence["mode"]);
}
// ======================================================= measurement mode ====
[Fact]
public void ShadowMode_ReportsWhatThePhaseWouldDecide_AndTodaysRuleDecides()
{
// Report 3a2207978e4c in the Shadow mode: the fire mid-update fails the session as
// before, and says that the phase would have kept waiting.
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine, mode: OobeUpdatePhaseModes.Shadow);
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(22), OsUpdateSteps.DownloadStarted)).NewState;
state = engine.Reduce(state, OsUpdate(41, T0.AddMinutes(35), OsUpdateSteps.InstallStarted)).NewState;
var step = engine.Reduce(state, AdvisoryFired(50, T0.AddMinutes(51)));
Assert.Equal(SessionStage.Failed, step.NewState.Stage);
Assert.Equal("DeadlineFired:advisory_completion", step.Transition.Trigger);
Assert.Equal("esp_exit_without_completion_evidence", SingleTimelineEffect(step, "enrollment_failed").Parameters!["reason"]);
var shadow = SingleTimelineEffect(step, "oobe_update_phase_shadow");
var parameters = shadow.Parameters!;
Assert.Equal("Debug", parameters["severity"]);
Assert.Equal(OobeUpdatePhaseModes.Shadow, parameters["phaseMode"]);
Assert.Equal("update_running", parameters["phaseState"]);
Assert.Equal("hold", parameters["wouldDecide"]);
Assert.Equal("esp_exit_without_completion_evidence", parameters["decidedReason"]);
Assert.Equal(T0.AddMinutes(22).AddHours(3).ToString("o"), parameters["phaseBoundUtc"]);
Assert.Equal(T0.AddMinutes(22).ToString("o"), parameters["oobeUpdatePhaseStartedUtc"]);
Assert.Equal(OsUpdateSteps.InstallStarted, parameters["osUpdateLastStep"]);
Assert.Equal(Lcu, parameters["osUpdate"]);
Assert.Contains("still installing an update", parameters["message"]);
// The report precedes the verdict on the timeline.
Assert.True(step.Effects.ToList().IndexOf(shadow) < step.Effects.ToList().IndexOf(SingleTimelineEffect(step, "enrollment_failed")));
}
[Fact]
public void ShadowMode_AfterTheUpdatesRestart_ReportsTheSignInWait()
{
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine, mode: OobeUpdatePhaseModes.Shadow);
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(22), OsUpdateSteps.DownloadStarted)).NewState;
state = engine.Reduce(state, OsUpdate(41, T0.AddMinutes(48), OsUpdateSteps.RebootRequired, "KB5099999")).NewState;
state = engine.Reduce(state, Reboot(60, T0.AddMinutes(53))).NewState; // re-bases the window to T0+83
var step = engine.Reduce(state, AdvisoryFired(70, T0.AddMinutes(83)));
Assert.Equal(SessionStage.Failed, step.NewState.Stage);
var parameters = SingleTimelineEffect(step, "oobe_update_phase_shadow").Parameters!;
Assert.Equal("awaiting_sign_in", parameters["phaseState"]);
Assert.Equal("hold", parameters["wouldDecide"]);
Assert.Equal(T0.AddMinutes(53).ToString("o"), parameters["oobeUpdateEndedUtc"]);
Assert.Equal(T0.AddMinutes(113).ToString("o"), parameters["phaseBoundUtc"]);
}
[Fact]
public void ShadowMode_HangingUpdate_ReportsTheNotFinishedFailure()
{
// A fire after the cap (a shadow session lives this long only when another guard held
// it, e.g. enforcement progress) reports the not-finished failure the phase would give.
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine, mode: OobeUpdatePhaseModes.Shadow);
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(22), OsUpdateSteps.InstallStarted)).NewState;
var afterCap = T0.AddMinutes(22).AddHours(3).AddMinutes(5);
var armed = FindDeadline(state, DeadlineNames.AdvisoryCompletion)!;
state = state.ToBuilder()
.CancelDeadline(DeadlineNames.AdvisoryCompletion)
.AddDeadline(new ActiveDeadline(armed.Name, afterCap, armed.FiresSignalKind, armed.FiresPayload))
.Build();
var step = engine.Reduce(state, AdvisoryFired(50, afterCap));
Assert.Equal("esp_exit_without_completion_evidence", SingleTimelineEffect(step, "enrollment_failed").Parameters!["reason"]);
var parameters = SingleTimelineEffect(step, "oobe_update_phase_shadow").Parameters!;
Assert.Equal("update_not_finished", parameters["phaseState"]);
Assert.Equal("fail_oobe_update_not_finished", parameters["wouldDecide"]);
}
[Fact]
public void UnstampedSession_CountsAsShadow()
{
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine, mode: null);
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(22), OsUpdateSteps.DownloadStarted)).NewState;
var step = engine.Reduce(state, AdvisoryFired(50, T0.AddMinutes(51)));
Assert.Equal(SessionStage.Failed, step.NewState.Stage);
Assert.Equal("update_running", SingleTimelineEffect(step, "oobe_update_phase_shadow").Parameters!["phaseState"]);
}
[Fact]
public void OffMode_EvaluatesNothing()
{
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine, mode: OobeUpdatePhaseModes.Off);
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(22), OsUpdateSteps.DownloadStarted)).NewState;
var step = engine.Reduce(state, AdvisoryFired(50, T0.AddMinutes(51)));
Assert.Equal(SessionStage.Failed, step.NewState.Stage);
Assert.False(HasTimelineEffect(step, "oobe_update_phase_shadow"));
}
[Fact]
public void ShadowMode_WithoutPhase_ReportsNothing()
{
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine, mode: OobeUpdatePhaseModes.Shadow);
var step = engine.Reduce(state, AdvisoryFired(50, T0.AddMinutes(51)));
Assert.Equal(SessionStage.Failed, step.NewState.Stage);
Assert.False(HasTimelineEffect(step, "oobe_update_phase_shadow"));
}
[Fact]
public void ModeStamp_ActiveCanOnlyBeTakenAway_NeverGranted()
{
var engine = new DecisionEngine();
var initial = engine.Reduce(DecisionState.CreateInitial("s", "t", T0), MakeSignal(0, DecisionSignalKind.SessionStarted, T0)).NewState;
// A later agent start in Shadow or Off takes Active away (kill switch) ...
var state = engine.Reduce(initial, PhaseMode(1, T0, "active")).NewState;
Assert.Equal(OobeUpdatePhaseModes.Active, state.ScenarioObservations.OobeUpdatePhaseMode!.Value);
state = engine.Reduce(state, PhaseMode(2, T0.AddMinutes(60), OobeUpdatePhaseModes.Active)).NewState;
Assert.Equal(1, state.ScenarioObservations.OobeUpdatePhaseMode!.SourceSignalOrdinal);
state = engine.Reduce(state, PhaseMode(3, T0.AddMinutes(90), OobeUpdatePhaseModes.Off)).NewState;
Assert.Equal(OobeUpdatePhaseModes.Off, state.ScenarioObservations.OobeUpdatePhaseMode!.Value);
Assert.Equal(3, state.ScenarioObservations.OobeUpdatePhaseMode.SourceSignalOrdinal);
// ... and switching it on mid-session applies from the next session only.
state = engine.Reduce(state, PhaseMode(4, T0.AddMinutes(120), OobeUpdatePhaseModes.Active)).NewState;
Assert.Equal(OobeUpdatePhaseModes.Off, state.ScenarioObservations.OobeUpdatePhaseMode!.Value);
var shadow = engine.Reduce(initial, PhaseMode(1, T0, "unknown-mode")).NewState;
Assert.Equal(OobeUpdatePhaseModes.Shadow, shadow.ScenarioObservations.OobeUpdatePhaseMode!.Value);
shadow = engine.Reduce(shadow, PhaseMode(2, T0.AddMinutes(60), OobeUpdatePhaseModes.Active)).NewState;
Assert.Equal(OobeUpdatePhaseModes.Shadow, shadow.ScenarioObservations.OobeUpdatePhaseMode!.Value);
}
[Fact]
public void ActiveTakenAwayAfterTheRestart_TheFireFallsBackToTodaysRule()
{
var engine = new DecisionEngine();
var state = HandoffExitArmsTheWindow(engine); // Active
state = engine.Reduce(state, OsUpdate(40, T0.AddMinutes(22), OsUpdateSteps.DownloadStarted)).NewState;
state = engine.Reduce(state, OsUpdate(41, T0.AddMinutes(48), OsUpdateSteps.RebootRequired, "KB5099999")).NewState;
state = engine.Reduce(state, Reboot(60, T0.AddMinutes(53))).NewState;
// The agent comes back after the update's restart with the phase switched back to Shadow.
state = engine.Reduce(state, PhaseMode(61, T0.AddMinutes(54), OobeUpdatePhaseModes.Shadow)).NewState;
var step = engine.Reduce(state, AdvisoryFired(70, T0.AddMinutes(83)));
Assert.Equal(SessionStage.Failed, step.NewState.Stage);
Assert.Equal("awaiting_sign_in", SingleTimelineEffect(step, "oobe_update_phase_shadow").Parameters!["phaseState"]);
}
}
}
@@ -90,6 +90,11 @@ namespace AutopilotMonitor.DecisionCore.Tests.State
lastClassifierVerdictId: "verdict-" + seed));
}
if (type == typeof(RealmJoinFacts)) return RealmJoinFacts.Empty.WithDetected(BaseUtc.AddMinutes(seed), ordinal);
if (type == typeof(OsUpdateFacts))
{
return OsUpdateFacts.Empty.WithActivity(
BaseUtc.AddMinutes(seed), ordinal, OsUpdateSteps.RebootRequired, "KB-" + seed, isPostExit: true);
}
throw new Xunit.Sdk.XunitException(
$"No value factory for builder property type {type}. A new DecisionState field " +
@@ -422,6 +422,31 @@ public class EnrollmentTimeoutClassifierTests
Assert.Equal(ClassifierRules.R1bWhiteGloveAwaiting, wgRule);
}
[Fact]
public void Classify_engine_stopped_waiting_after_the_oobe_update_is_never_failed()
{
// D-310: nobody signed in within the sign-in window after the OOBE update, the engine
// stopped waiting and the agent ended the session. Same honest classification as the
// watchdog — user-driven terminalizes as Incomplete without naming the watchdog, a
// WhiteGlove Part 2 stays AwaitingUser.
var stopped = Evt("agent_shutting_down", data: new Dictionary<string, object>
{
["reason"] = "stopped_waiting",
["stopReason"] = "oobe_update_no_sign_in",
});
var (status, reason) = Classify(new[] { Esp(DeviceSetup44), stopped }, hoursSinceStart: 2);
Assert.Equal(SessionStatus.Incomplete, status);
Assert.Contains("stopped waiting for the user", reason);
Assert.DoesNotContain("max-lifetime", reason);
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(new[] { Esp(DeviceSetup44), stopped });
var (wgStatus, _, wgRule) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, Start, Start.AddHours(2), 72, isPreProvisioned: true, resumedAt: Start.AddHours(1));
Assert.Equal(SessionStatus.AwaitingUser, wgStatus);
Assert.Equal(ClassifierRules.R1bWhiteGloveAwaiting, wgRule);
}
// -------- Rule 5a: completed (assumed) — calibration read 2026-08-27 --------
[Fact]
@@ -429,12 +454,21 @@ public class EnrollmentTimeoutClassifierTests
{
Assert.True(EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("app_install_failed") }).HasAppInstallFailure);
var timeout = Evt("enrollment_failed", data: new Dictionary<string, object> { ["failureType"] = "agent_timeout" });
Assert.True(EnrollmentTimeoutClassifier.ExtractRollup(new[] { timeout }).HasAgentMaxLifetimeTimeout);
Assert.True(EnrollmentTimeoutClassifier.ExtractRollup(new[] { timeout }).AgentStoppedForGood);
var shutdown = Evt("agent_shutting_down", data: new Dictionary<string, object> { ["reason"] = "max_lifetime" });
Assert.True(EnrollmentTimeoutClassifier.ExtractRollup(new[] { shutdown }).HasAgentMaxLifetimeTimeout);
// Other shutdown reasons say nothing about max lifetime.
Assert.True(EnrollmentTimeoutClassifier.ExtractRollup(new[] { shutdown }).AgentStoppedForGood);
Assert.False(EnrollmentTimeoutClassifier.ExtractRollup(new[] { shutdown }).AgentStoppedWaiting);
// D-310: the engine's bounded wait expired — the agent is gone just the same.
var stopped = Evt("agent_shutting_down", data: new Dictionary<string, object>
{
["reason"] = "stopped_waiting",
["stopReason"] = "oobe_update_no_sign_in",
});
Assert.True(EnrollmentTimeoutClassifier.ExtractRollup(new[] { stopped }).AgentStoppedForGood);
Assert.True(EnrollmentTimeoutClassifier.ExtractRollup(new[] { stopped }).AgentStoppedWaiting);
// Other shutdown reasons say nothing about the agent being gone for good.
var ctrlC = Evt("agent_shutting_down", data: new Dictionary<string, object> { ["reason"] = "ctrl_c" });
Assert.False(EnrollmentTimeoutClassifier.ExtractRollup(new[] { ctrlC }).HasAgentMaxLifetimeTimeout);
Assert.False(EnrollmentTimeoutClassifier.ExtractRollup(new[] { ctrlC }).AgentStoppedForGood);
// A real failure event does not raise the app flag.
Assert.False(EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("enrollment_failed") }).HasAppInstallFailure);
}
@@ -79,6 +79,37 @@ public class IngestMaxLifetimeShutdownTests
Assert.False(EventIngestProcessor.IsMaxLifetimeAgentShutdown(null));
}
// -------- stopped_waiting (D-310) --------
// The engine stopped waiting for the user (no sign-in after the OOBE update) and the agent
// ended the session like its watchdog: same contract, classified through the same path.
[Fact]
public void StoppedWaitingReason_IsDetected_AsGaveUp_ButNotAsMaxLifetime()
{
var stopped = Shutdown("stopped_waiting");
Assert.True(EventIngestProcessor.IsStoppedWaitingAgentShutdown(stopped));
Assert.True(EventIngestProcessor.IsAgentGaveUpShutdown(stopped));
Assert.False(EventIngestProcessor.IsMaxLifetimeAgentShutdown(stopped));
}
[Fact]
public void MaxLifetimeReason_IsAlsoGaveUp()
{
Assert.True(EventIngestProcessor.IsAgentGaveUpShutdown(Shutdown("max_lifetime")));
Assert.False(EventIngestProcessor.IsStoppedWaitingAgentShutdown(Shutdown("max_lifetime")));
}
[Theory]
[InlineData("decision_terminal")]
[InlineData("ctrl_c")]
[InlineData(null)]
public void OtherShutdowns_AreNotGaveUp(string? reason)
{
Assert.False(EventIngestProcessor.IsAgentGaveUpShutdown(Shutdown(reason)));
Assert.False(EventIngestProcessor.IsAgentGaveUpShutdown(null));
}
// -------- IsAgentTimeoutFailure (misclassification audit 2026-07-16) --------
// The V1-parity companion: enrollment_failed(failureType=agent_timeout) emitted by
// LifecycleEmitters.CreateMaxLifetimeEmitter. Same semantics as the shutdown shape —
@@ -283,6 +283,18 @@ public class VerdictCalibrationRadarTests
Assert.Equal((14, 21, 56, 84), gap);
}
[Fact]
public void StoppedWaiting_counts_as_a_classifier_verdict_but_never_as_silence()
{
// D-310: the engine deliberately stopped waiting for the user — not an agent that went
// quiet, so the agent-liveness share must not see it.
var rows = Horizon(w => new[] { ("sweep:r6", "Incomplete", 2), ("stopwait:r5_incomplete", "Incomplete", 1), ("agent:complete", "Succeeded", 17) });
var silence = VerdictCalibrationRadar.CurrentSums(new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.SilenceShareRegression }, rows, Target);
Assert.Equal((14, 140, 56, 560), silence);
var gap = VerdictCalibrationRadar.CurrentSums(new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.EvidenceGap }, rows, Target);
Assert.Equal((14, 21, 56, 84), gap);
}
[Fact]
public void Rows_outside_the_horizon_are_ignored()
{
@@ -62,6 +62,8 @@ public class VerdictPathTests
Assert.True(VerdictPaths.IsClassifierPath(path));
Assert.True(VerdictPaths.IsClassifierPath("sweep:r5_assumed"));
Assert.True(VerdictPaths.IsClassifierPath("maxlife:r1"));
Assert.True(VerdictPaths.IsClassifierPath("stopwait:r5_incomplete"));
Assert.True(VerdictPaths.IsClassifierPath("stopwait:r1b_awaiting"));
Assert.True(VerdictPaths.IsClassifierPath("late:r4"));
Assert.True(VerdictPaths.IsClassifierPath("retro:r6"));
Assert.False(VerdictPaths.IsClassifierPath(VerdictPaths.AgentComplete));
@@ -9,7 +9,8 @@ namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Windows Update during OOBE — pins the firing behaviour of the built-in analyze rules
/// that surface update activity captured by the agent's WindowsUpdateTracker:
/// - ANALYZE-DEV-004 (high): a quality/cumulative update FAILED during enrollment.
/// - ANALYZE-DEV-004 (high): a quality/cumulative update FAILED to install or download during
/// enrollment (wuPhase install/download — never an update-scan failure).
/// - ANALYZE-DEV-005 (info): a quality/cumulative update INSTALLED during enrollment.
/// - ANALYZE-DEV-006 (info): the OS build changed across a mid-enrollment reboot —
/// deterministic corroboration that works even when the WU channel showed nothing
@@ -51,6 +52,62 @@ public class WindowsUpdateRuleTests
Assert.Equal("WU_E_ALL_UPDATES_FAILED", AsString(symbol["value"]));
}
[Fact]
public async Task ANALYZE_DEV_004_fires_on_a_failed_download()
{
// EventID 31 from the Operational channel (D-310): the update never finished downloading.
var rule = BuiltInAnalyzeRules.GetAll().First(r => r.RuleId == "ANALYZE-DEV-004");
var events = new List<EnrollmentEvent>
{
WindowsUpdateFailed("2026-09 Cumulative Update for Windows 11 (KB5099999)", "0x80248007", "WU_E_DS_NODATA",
wuEventId: 31, wuPhase: "download"),
};
var outcome = await RunAsync(rule, events);
var result = Assert.Single(outcome.Results);
Assert.Equal("0x80248007", AsString(AsDict(result.MatchedConditions["wu_hresult"])["value"]));
}
[Fact]
public async Task ANALYZE_DEV_004_does_not_fire_on_an_update_scan_failure()
{
// EventID 25 names no update: a failed scan (often a transient network or proxy error) is
// not a failed update and must not raise a high-severity finding.
var rule = BuiltInAnalyzeRules.GetAll().First(r => r.RuleId == "ANALYZE-DEV-004");
var events = new List<EnrollmentEvent>
{
WindowsUpdateScanFailed("0x8024401C", "WU_E_PT_HTTP_STATUS_REQUEST_TIMEOUT"),
};
var outcome = await RunAsync(rule, events);
Assert.Empty(outcome.Results);
}
[Fact]
public async Task ANALYZE_DEV_004_takes_its_evidence_from_the_install_failure_not_the_scan_failure()
{
// The phase filter applies to every evidence condition: the explanation must name the
// failed update and its error, never the error of an earlier scan.
var rule = BuiltInAnalyzeRules.GetAll().First(r => r.RuleId == "ANALYZE-DEV-004");
var events = new List<EnrollmentEvent>
{
WindowsUpdateScanFailed("0x8024401C", "WU_E_PT_HTTP_STATUS_REQUEST_TIMEOUT"),
WindowsUpdateFailed("2026-09 Cumulative Update for Windows 11 (KB5099999)", "0x800F0922", "CBS_E_INSTALLERS_FAILED"),
};
var outcome = await RunAsync(rule, events);
var result = Assert.Single(outcome.Results);
Assert.Equal("0x800F0922", AsString(AsDict(result.MatchedConditions["wu_hresult"])["value"]));
Assert.Equal("CBS_E_INSTALLERS_FAILED", AsString(AsDict(result.MatchedConditions["wu_hresult_symbol"])["value"]));
Assert.Equal("2026-09 Cumulative Update for Windows 11 (KB5099999)",
AsString(AsDict(result.MatchedConditions["wu_update_title"])["value"]));
}
[Fact]
public async Task ANALYZE_DEV_004_does_not_fire_without_a_failed_update()
{
@@ -217,11 +274,12 @@ public class WindowsUpdateRuleTests
["wuClientCensus"] = "21=1,25=3",
["updateOrchestratorCensus"] = "200=2",
["lookbackMinutes"] = 60,
["targetedEventIds"] = "19,20,43,44",
["targetedEventIds"] = "19,20,25,26,31,41,43,44",
}
};
private static EnrollmentEvent WindowsUpdateFailed(string title, string hresult, string hresultSymbol) => new()
private static EnrollmentEvent WindowsUpdateFailed(
string title, string hresult, string hresultSymbol, int wuEventId = 20, string wuPhase = "install") => new()
{
EventId = Guid.NewGuid().ToString(),
TenantId = TenantId,
@@ -231,7 +289,10 @@ public class WindowsUpdateRuleTests
Sequence = 42,
Data = new Dictionary<string, object>
{
["wuEventId"] = 20,
["wuEventId"] = wuEventId,
["wuChannel"] = wuEventId == 20 ? "system" : "operational",
["wuPhase"] = wuPhase,
["updateClass"] = "os",
["updateTitle"] = title,
["updateGuid"] = "{8b1c8726-1111-2222-3333-444455556666}",
["hresult"] = hresult,
@@ -240,6 +301,25 @@ public class WindowsUpdateRuleTests
}
};
private static EnrollmentEvent WindowsUpdateScanFailed(string hresult, string hresultSymbol) => new()
{
EventId = Guid.NewGuid().ToString(),
TenantId = TenantId,
SessionId = SessionId,
EventType = "windows_update_failed",
Timestamp = DateTime.UtcNow,
Sequence = 41,
Data = new Dictionary<string, object>
{
["wuEventId"] = 25,
["wuChannel"] = "operational",
["wuPhase"] = "scan",
["hresult"] = hresult,
["hresultSymbol"] = hresultSymbol,
["backfilled"] = false,
}
};
private static EnrollmentEvent WindowsUpdateSucceeded(string title) => new()
{
EventId = Guid.NewGuid().ToString(),
@@ -251,6 +331,9 @@ public class WindowsUpdateRuleTests
Data = new Dictionary<string, object>
{
["wuEventId"] = 19,
["wuChannel"] = "system",
["wuPhase"] = "installed",
["updateClass"] = "os",
["updateTitle"] = title,
["updateGuid"] = "{8b1c8726-1111-2222-3333-444455556666}",
["backfilled"] = false,
@@ -101,7 +101,10 @@ namespace AutopilotMonitor.Functions.Services
bool RealmJoinDetected,
bool RealmJoinResolved,
bool HasAppInstallFailure,
bool HasAgentMaxLifetimeTimeout);
// The agent is gone for good: its max-lifetime watchdog fired, or its decision engine
// stopped waiting for the user (D-310, AgentStoppedWaiting).
bool AgentStoppedForGood,
bool AgentStoppedWaiting = false);
/// <summary>
/// Walk a session's events and distill the ESP rollup. Tolerant of missing/empty input
@@ -117,7 +120,7 @@ namespace AutopilotMonitor.Functions.Services
int acctBestN = 0, acctBestM = 0;
bool acctFallbackAll = false;
bool hasFailure = false, hasComplete = false, hasEnrollmentComplete = false, hasEmergencyBreak = false;
bool hasAppInstallFailure = false, hasMaxLifetimeTimeout = false;
bool hasAppInstallFailure = false, agentStoppedForGood = false, agentStoppedWaiting = false;
bool desktopArrived = false, helloResolved = false;
bool realmJoinDetected = false, realmJoinResolved = false;
bool sawHelloPolicyEnabled = false, sawHelloPolicyDisabled = false;
@@ -138,15 +141,19 @@ namespace AutopilotMonitor.Functions.Services
if (!Eq(TryGetDataString(evt, "failureType") ?? string.Empty, "agent_timeout"))
hasFailure = true;
else
hasMaxLifetimeTimeout = true;
agentStoppedForGood = true;
}
// The V2 watchdog shape of the same fact: the agent shut itself down at its
// max-lifetime cap. Either shape proves the agent is gone for good — rule 5 uses
// that to skip the AwaitingUser grace (a wait nothing can ever end).
// max-lifetime cap — or its decision engine stopped waiting for the user (D-310).
// Every shape proves the agent is gone for good — rule 5 uses that to skip the
// AwaitingUser grace (a wait nothing can ever end).
else if (Eq(type, "agent_shutting_down"))
{
if (Eq(TryGetDataString(evt, "reason") ?? string.Empty, "max_lifetime"))
hasMaxLifetimeTimeout = true;
var reason = TryGetDataString(evt, "reason") ?? string.Empty;
if (Eq(reason, "max_lifetime"))
agentStoppedForGood = true;
else if (Eq(reason, "stopped_waiting"))
agentStoppedForGood = agentStoppedWaiting = true;
}
else if (Eq(type, "app_install_failed"))
hasAppInstallFailure = true;
@@ -247,7 +254,8 @@ namespace AutopilotMonitor.Functions.Services
RealmJoinDetected: realmJoinDetected,
RealmJoinResolved: realmJoinResolved,
HasAppInstallFailure: hasAppInstallFailure,
HasAgentMaxLifetimeTimeout: hasMaxLifetimeTimeout);
AgentStoppedForGood: agentStoppedForGood,
AgentStoppedWaiting: agentStoppedWaiting);
}
/// <summary>
@@ -436,12 +444,13 @@ namespace AutopilotMonitor.Functions.Services
{
var elapsedHours = (nowUtc - startedAtUtc).TotalHours;
// The grace only buys time for a LIVE agent to come back with real evidence. Once
// the agent's own max-lifetime watchdog has fired it has cleaned up and exited —
// the agent stopped for good (its max-lifetime watchdog, or its engine stopped
// waiting for the user — D-310) it has cleaned up and exited —
// parking such a session AwaitingUser is a countdown, not a wait (all 5 observed
// maxlife:r5_awaiting episodes expired unhealed; calibration read 2026-08-27), so
// it skips straight to the terminal fork. Nothing is lost by deciding now: late
// straggler telemetry still heals a terminal verdict (TryLateTelemetryReconcileAsync).
if (elapsedHours < graceHours && !rollup.HasAgentMaxLifetimeTimeout)
if (elapsedHours < graceHours && !rollup.AgentStoppedForGood)
{
var acct = rollup.AccountSetupTotal > 0
? $" (Account Setup {rollup.AccountSetupSucceededCount}/{rollup.AccountSetupTotal})"
@@ -471,11 +480,14 @@ namespace AutopilotMonitor.Functions.Services
var acctDetail = rollup.AccountSetupTotal > 0
? $"last Account Setup {rollup.AccountSetupSucceededCount}/{rollup.AccountSetupTotal}"
: "Account Setup progress never observed";
// Inside the grace the only way here is the max-lifetime short-circuit — the
// "within {grace}h" wording would be false, so name the watchdog instead.
// Inside the grace the only way here is the agent-gone short-circuit — the
// "within {grace}h" wording would be false, so name what stopped the agent instead.
// The engine's own stop (D-310) states its reason in the verdict suffix.
if (elapsedHours < graceHours)
return (SessionStatus.Incomplete,
$"Agent max-lifetime watchdog fired — agent gone without completion after Device Setup completed ({acctDetail})",
rollup.AgentStoppedWaiting
? $"Device Setup completed ({acctDetail}); the agent stopped waiting for the user."
: $"Agent max-lifetime watchdog fired — agent gone without completion after Device Setup completed ({acctDetail})",
ClassifierRules.R5DeviceSetupIncomplete);
return (SessionStatus.Incomplete,
$"No completion signal within {graceHours}h grace after Device Setup completed ({acctDetail})",
@@ -1,3 +1,4 @@
using AutopilotMonitor.DecisionCore.State;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Logging;
@@ -92,11 +93,12 @@ namespace AutopilotMonitor.Functions.Services
classification.HasCompletionEvidenceCandidate = true;
break;
case "agent_shutting_down":
// Every shutdown ends this agent run's observation; the max-lifetime
// flavour additionally carries the verdict mapping below.
// Every shutdown ends this agent run's observation; the gave-up flavours
// (max-lifetime watchdog, engine stopped waiting) additionally carry the
// verdict mapping below.
classification.AgentShutdownEvent = evt;
if (IsMaxLifetimeAgentShutdown(evt))
classification.AgentMaxLifetimeShutdownEvent = evt;
if (IsAgentGaveUpShutdown(evt))
classification.AgentGaveUpShutdownEvent = evt;
break;
case "system_reboot_detected":
// Per-batch incremental reboot count (live value during enrollment).
@@ -177,14 +179,30 @@ namespace AutopilotMonitor.Functions.Services
return null;
}
internal static bool IsMaxLifetimeAgentShutdown(EnrollmentEvent? evt)
internal static bool IsMaxLifetimeAgentShutdown(EnrollmentEvent? evt) =>
string.Equals(AgentShutdownReason(evt), "max_lifetime", StringComparison.OrdinalIgnoreCase);
/// <summary>
/// True for an <c>agent_shutting_down</c> event whose <c>Data.reason</c> is
/// <c>stopped_waiting</c>: the agent's decision engine stopped waiting for the user — a
/// bounded wait expired (D-310, <c>Data.stopReason</c> names it, e.g. no sign-in after the
/// OOBE update). Same contract as <see cref="IsMaxLifetimeAgentShutdown"/>: the last event
/// of the session, no enrollment verdict — the backend classifies it honestly.
/// </summary>
internal static bool IsStoppedWaitingAgentShutdown(EnrollmentEvent? evt) =>
string.Equals(AgentShutdownReason(evt), "stopped_waiting", StringComparison.OrdinalIgnoreCase);
/// <summary>The agent gave up waiting for good — either shutdown shape above.</summary>
internal static bool IsAgentGaveUpShutdown(EnrollmentEvent? evt) =>
IsMaxLifetimeAgentShutdown(evt) || IsStoppedWaitingAgentShutdown(evt);
private static string? AgentShutdownReason(EnrollmentEvent? evt)
{
if (evt == null || !string.Equals(evt.EventType, "agent_shutting_down", StringComparison.Ordinal))
return false;
var reason = evt.Data != null && evt.Data.ContainsKey("reason")
return null;
return evt.Data != null && evt.Data.ContainsKey("reason")
? evt.Data["reason"]?.ToString()
: null;
return string.Equals(reason, "max_lifetime", StringComparison.OrdinalIgnoreCase);
}
/// <summary>
@@ -227,20 +245,24 @@ namespace AutopilotMonitor.Functions.Services
}
/// <summary>
/// Shared verdict path for both max-lifetime shapes (enrollment_failed/agent_timeout and
/// agent_shutting_down/max_lifetime): classify the session honestly via
/// <see cref="EnrollmentTimeoutClassifier"/> instead of hard-failing it. The watchdog
/// firing only proves the agent stopped waiting — the enrollment itself may be fully
/// Shared verdict path for every "the agent gave up waiting" shape — the max-lifetime
/// watchdog (enrollment_failed/agent_timeout, agent_shutting_down/max_lifetime) and the
/// engine's bounded wait (agent_shutting_down/stopped_waiting, D-310): classify the session
/// honestly via <see cref="EnrollmentTimeoutClassifier"/> instead of hard-failing it. The
/// agent stopping only proves it stopped waiting — the enrollment itself may be fully
/// provisioned with the user simply absent (AwaitingUser), silently dead (Incomplete),
/// or even provably finished (Succeeded reconcile). Mirrors the maintenance sweep's
/// stage-2 classification (misclassification audit 2026-07-16).
/// Returns (transitioned, failureReason) where failureReason is only set for the
/// terminal non-success outcomes so downstream failure notifications stay accurate.
/// </summary>
private async Task<(bool transitioned, string? failureReason)> ApplyMaxLifetimeVerdictAsync(
private async Task<(bool transitioned, string? failureReason)> ApplyAgentGaveUpVerdictAsync(
IngestEventsRequest request, string sessionPrefix, EventClassification c,
EnrollmentEvent triggerEvent, SessionStatus? preFetchedStatus)
{
var stoppedWaiting = IsStoppedWaitingAgentShutdown(triggerEvent);
var trigger = stoppedWaiting ? "stopped_waiting" : "max_lifetime";
var session = await _sessionRepo.GetSessionAsync(request.TenantId, request.SessionId);
var currentStatus = session?.Status ?? preFetchedStatus;
@@ -249,7 +271,7 @@ namespace AutopilotMonitor.Functions.Services
if (currentStatus == SessionStatus.Pending)
{
_logger.LogInformation(
"{SessionPrefix} max_lifetime verdict skipped — session is Pending (WhiteGlove)", sessionPrefix);
"{SessionPrefix} {Trigger} verdict skipped — session is Pending (WhiteGlove)", sessionPrefix, trigger);
return (false, null);
}
@@ -266,7 +288,7 @@ namespace AutopilotMonitor.Functions.Services
catch (Exception readEx)
{
_logger.LogWarning(readEx,
"{SessionPrefix} failed to read events for max_lifetime classification; proceeding with batch only", sessionPrefix);
"{SessionPrefix} failed to read events for {Trigger} classification; proceeding with batch only", sessionPrefix, trigger);
sessionEvents = new List<EnrollmentEvent>();
}
@@ -279,7 +301,7 @@ namespace AutopilotMonitor.Functions.Services
}
catch (Exception cfgEx)
{
_logger.LogWarning(cfgEx, "{SessionPrefix} failed to read tenant config for max_lifetime classification; using defaults", sessionPrefix);
_logger.LogWarning(cfgEx, "{SessionPrefix} failed to read tenant config for {Trigger} classification; using defaults", sessionPrefix, trigger);
}
var graceHours = EnrollmentTimeoutClassifier.ResolveGraceHours(tenantGraceHours, absoluteMaxHours);
@@ -289,7 +311,7 @@ namespace AutopilotMonitor.Functions.Services
var effectiveStart = session?.ResumedAt ?? session?.StartedAt ?? triggerEvent.Timestamp;
// The trigger event itself may not be persisted yet when this read runs — append it so
// the rollup always carries the max-lifetime fact (rule 5 skips the AwaitingUser grace
// the rollup always carries the agent-gone fact (rule 5 skips the AwaitingUser grace
// on it: the agent is provably gone). The rollup folds booleans, so a stored duplicate
// is harmless.
sessionEvents.Add(triggerEvent);
@@ -300,14 +322,20 @@ namespace AutopilotMonitor.Functions.Services
isPreProvisioned: session?.IsPreProvisioned == true, resumedAt: session?.ResumedAt,
isSelfDeployingProfile: session?.IsSelfDeployingProfile == true);
// Keep the max-lifetime trigger visible: the Succeeded reasons already carry their own
// silence-transparency clause, everything else gets the watchdog context appended.
// Keep the trigger visible: the Succeeded reasons already carry their own
// silence-transparency clause, everything else names what stopped the agent.
if (targetStatus != SessionStatus.Succeeded)
reason += " Verdict triggered by the agent's max-lifetime watchdog shutdown.";
{
reason += stoppedWaiting
? " " + StoppedWaitingReasons.Describe(triggerEvent.Data != null && triggerEvent.Data.ContainsKey("stopReason")
? triggerEvent.Data["stopReason"]?.ToString()
: null)
: " Verdict triggered by the agent's max-lifetime watchdog shutdown.";
}
if (targetStatus == currentStatus)
{
_logger.LogInformation("{SessionPrefix} max_lifetime verdict {Status} equals current status — no transition", sessionPrefix, targetStatus);
_logger.LogInformation("{SessionPrefix} {Trigger} verdict {Status} equals current status — no transition", sessionPrefix, trigger, targetStatus);
return (false, null);
}
@@ -319,20 +347,21 @@ namespace AutopilotMonitor.Functions.Services
try { snapshotJson = FailureSnapshotBuilder.Build(sessionEvents, now); }
catch (Exception snapEx)
{
_logger.LogWarning(snapEx, "{SessionPrefix} failed to build failure snapshot for max_lifetime verdict", sessionPrefix);
_logger.LogWarning(snapEx, "{SessionPrefix} failed to build failure snapshot for {Trigger} verdict", sessionPrefix, trigger);
}
}
var origin = stoppedWaiting ? VerdictPaths.OriginStoppedWaiting : VerdictPaths.OriginMaxLifetime;
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, targetStatus, VerdictPaths.Compose(VerdictPaths.OriginMaxLifetime, rule),
request.TenantId, request.SessionId, targetStatus, VerdictPaths.Compose(origin, rule),
triggerEvent.Phase, reason,
completedAt: isTerminalNonSuccess ? triggerEvent.Timestamp : (DateTime?)null,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
failureSource: isTerminalNonSuccess ? "max_lifetime_watchdog" : null,
failureSource: isTerminalNonSuccess ? (stoppedWaiting ? "stopped_waiting" : "max_lifetime_watchdog") : null,
failureSnapshotJson: snapshotJson);
_logger.LogWarning("{SessionPrefix} Status: {Status} via max_lifetime honest classification - {Reason} (transitioned={Transitioned})",
sessionPrefix, targetStatus, reason, transitioned);
_logger.LogWarning("{SessionPrefix} Status: {Status} via {Trigger} honest classification - {Reason} (transitioned={Transitioned})",
sessionPrefix, targetStatus, trigger, reason, transitioned);
return (transitioned, isTerminalNonSuccess ? reason : null);
}
@@ -495,7 +524,7 @@ namespace AutopilotMonitor.Functions.Services
// in (misclassification audit 2026-07-16, tenant a53e67ec: honest verdict
// AwaitingUser). Route through the same honest classification the maintenance
// sweep uses instead.
(statusTransitioned, failureReason) = await ApplyMaxLifetimeVerdictAsync(
(statusTransitioned, failureReason) = await ApplyAgentGaveUpVerdictAsync(
request, sessionPrefix, c, c.FailureEvent, preFetchedStatus);
}
else
@@ -608,10 +637,11 @@ namespace AutopilotMonitor.Functions.Services
_logger.LogInformation("{SessionPrefix} WhiteGlove resumed skipped, session already {Status}", sessionPrefix, currentSession?.Status);
}
}
else if (c.AgentMaxLifetimeShutdownEvent != null)
else if (c.AgentGaveUpShutdownEvent != null)
{
// Lowest-priority status writer (session 8bc1180f): the V2 max-lifetime
// watchdog stops the agent permanently without a session verdict — this
// watchdog — or the engine's bounded wait for the user (D-310, stopped_waiting) —
// stops the agent permanently without a session verdict — this
// shutdown event is the last one the session ever sends, so without this
// mapping the session stays InProgress forever. Any genuine terminal event
// in the same batch wins via the else-if chain above; an already-terminal
@@ -621,8 +651,8 @@ namespace AutopilotMonitor.Functions.Services
// verdict (misclassification audit 2026-07-16). Pending (WhiteGlove) sessions
// are skipped inside the helper — they are deliberately long-lived and resume
// via re-registration.
(statusTransitioned, failureReason) = await ApplyMaxLifetimeVerdictAsync(
request, sessionPrefix, c, c.AgentMaxLifetimeShutdownEvent, preFetchedStatus);
(statusTransitioned, failureReason) = await ApplyAgentGaveUpVerdictAsync(
request, sessionPrefix, c, c.AgentGaveUpShutdownEvent, preFetchedStatus);
}
if (c.WhiteGloveStartedEvent != null)
@@ -660,7 +690,7 @@ namespace AutopilotMonitor.Functions.Services
var batchWroteStatus = c.CompletionEvent != null || c.FailureEvent != null
|| c.EspFailureEvent != null || c.GatherCompletionEvent != null
|| c.WhiteGloveEvent != null || c.WhiteGloveResumedEvent != null
|| c.AgentMaxLifetimeShutdownEvent != null;
|| c.AgentGaveUpShutdownEvent != null;
var currentStatus = !batchWroteStatus && preFetchedStatus.HasValue
? preFetchedStatus
: (await _sessionRepo.GetSessionAsync(request.TenantId, request.SessionId))?.Status;
@@ -717,12 +747,12 @@ namespace AutopilotMonitor.Functions.Services
public EnrollmentEvent? AgentShutdownEvent { get; set; }
/// <summary>
/// <c>agent_shutting_down</c> with <c>Data.reason == "max_lifetime"</c> — the V2
/// watchdog shutdown that deliberately carries no enrollment verdict. Mapped to a
/// terminal Failed status as the lowest-priority status writer so the session does
/// not stay InProgress forever (session 8bc1180f).
/// <c>agent_shutting_down</c> with <c>Data.reason</c> <c>max_lifetime</c> (the V2
/// watchdog, session 8bc1180f) or <c>stopped_waiting</c> (the engine's bounded wait,
/// D-310) — shutdowns that deliberately carry no enrollment verdict. Classified honestly
/// as the lowest-priority status writer so the session does not stay InProgress forever.
/// </summary>
public EnrollmentEvent? AgentMaxLifetimeShutdownEvent { get; set; }
public EnrollmentEvent? AgentGaveUpShutdownEvent { get; set; }
public bool HasNonPeriodicRealEvent { get; set; }
public EnrollmentEvent? DeviceLocationEvent { get; set; }
public DateTime? EarliestEventTimestamp { get; set; }
@@ -41,7 +41,7 @@ namespace AutopilotMonitor.Functions.Services
// A failure alert requires an actual failure-ish verdict: an agent_timeout
// enrollment_failed can honestly classify to AwaitingUser or even Succeeded
// (ApplyMaxLifetimeVerdictAsync), in which case failureReason stays null and no
// (ApplyAgentGaveUpVerdictAsync), in which case failureReason stays null and no
// failure notification must go out for a session that did not fail.
var failureVerdictApplies = c.FailureEvent != null && failureReason != null;
if (statusTransitioned && (c.CompletionEvent != null || failureVerdictApplies))
@@ -1079,8 +1079,14 @@ export interface CollectorConfiguration {
modernDeploymentBackfillLookbackMinutes: number;
/** Master switch for the Windows Update watcher. When enabled the agent subscribes to Microsoft-Windows-WindowsUpdateClient/Operational and forwards update install/failure events (windows_update_succeeded / _failed / _started) so quality and cumulative updates that run DURING enrollment become visible with evidence. Default: true. */
windowsUpdateWatcherEnabled: boolean;
/** WindowsUpdateClient/Operational EventIDs to capture. Kept configurable so additional IDs can be enabled after real-device validation WITHOUT an agent redeploy. Verified set: 19=install success, 20=install failure (carries HRESULT), 43=install started, 44=download started. IDs whose semantics are still unverified (25/26/31, restart-required 21 vs 41) are intentionally excluded until confirmed on a device trace. Default: [19, 20, 43, 44]. */
/** WindowsUpdateClient EventIDs the install watcher captures: 19=install success, 20=install failure (carries HRESULT), 43=install started, 44=download started. The provider manifest writes these to the System log, so the agent routes every ID to the channel the manifest assigns it (System: 16–24, 27, 28, 32, 33, 43, 44, 212–218; Operational: all others). Default: [19, 20, 43, 44]. */
windowsUpdateTargetedEventIds: number[];
/** WindowsUpdateClient/Operational EventIDs the scan/download watcher captures: 25=scan failed, 26=scan found N updates, 31=download failed, 41=update downloaded. Separate from WindowsUpdateTargetedEventIds on purpose: agents that predate the update classification ignore this field, so they never receive the high-volume Store/Defender download events unthrottled. Default: [25, 26, 31, 41]. */
windowsUpdateOperationalEventIds: number[];
/** Master switch for the servicing watcher on the Setup log (Microsoft-Windows-Servicing EventIDs 1/2/3/4/6): the CBS package steps of an OS update — staged, installed, reboot required, failed. Default: true. */
servicingWatcherEnabled: boolean;
/** Mode of the OOBE update phase: "Off", "Shadow" or "Active". Off: Windows/.NET update activity is only shown on the timeline. Shadow: the update watchers also feed it into the decision engine, which records the phase and reports at the post-ESP-exit completion window what the phase would decide; today's rule still decides. Active: the phase holds that window while Windows installs an update in OOBE and waits a bounded time for the sign-in after the update restart instead of failing the session. Any other value counts as Shadow. Default: "Shadow". */
oobeUpdatePhaseMode: string;
/** Lookback window in minutes for the Windows Update backfill scan on startup. OOBE quality updates run at the early "Getting updates" screen — often BEFORE the agent starts — but the WindowsUpdateClient/Operational .evtx persists across the mid-OOBE reboot, so a generous backfill catches them. Set higher than the ModernDeployment backfill because the update scan/download can begin well before MDM enrollment. 0 = backfill disabled. Default: 60 minutes. */
windowsUpdateBackfillLookbackMinutes: number;
/** Self-evidencing blind-spot census (session 7443317c: an OOBE quality update installed — OS build changed across the reboot — yet the WindowsUpdateClient channel carried NONE of the targeted EventIDs). When enabled and the OS build changed across an agent restart while zero targeted WU events were captured, the tracker emits a one-shot windows_update_channel_census event with an unfiltered EventID histogram of the WindowsUpdateClient AND UpdateOrchestrator channels over the backfill lookback window — field evidence for which EventIDs/channel to add to WindowsUpdateTargetedEventIds via remote config, without diagnostics upload access. Default: true. */
@@ -41,6 +41,8 @@ export const EVENT_TYPES_CATALOG = {
'esp_config_detected',
'completion_check',
'completion_waiting',
'completion_wait_expired',
'oobe_update_phase_shadow',
'enrollment_complete',
'enrollment_failed',
'desktop_arrived',
@@ -162,6 +164,10 @@ export const EVENT_TYPES_CATALOG = {
'windows_update_succeeded',
'windows_update_failed',
'windows_update_started',
'windows_update_servicing',
'windows_update_activity_summary',
'oobe_update_page',
'oobe_update_state',
'windows_update_reboot_pending',
'windows_update_history',
'os_build_changed',
@@ -1033,7 +1033,7 @@ export function registerAdminTools(server: McpServer, ga: boolean, strictGa: boo
title: 'Verdict Calibration (classifier thermometer)',
description:
'Answer "which code path produced our session verdicts, and does any of them look wrong?". Per verdict ' +
'path (agent:complete, agent:failed, sweep:r5_incomplete, maxlife:r6, late:r4, retro:r6, rule:<id>, ' +
'path (agent:complete, agent:failed, sweep:r5_incomplete, maxlife:r6, stopwait:r5_incomplete, late:r4, retro:r6, rule:<id>, ' +
'manual:failed, register:superseded, legacy:* for pre-instrumentation rows derived read-side): session ' +
'count and share in the window, derivedCount (attributed by derivation — weaker evidence), the 7-day ' +
're-enrollment proxy (eligible7d = terminal sessions old enough to judge, reEnrolled7d = the same device ' +
@@ -64,7 +64,9 @@ namespace AutopilotMonitor.DecisionCore.Engine
/// 2026-07-10: a pre-sign-in IME AccountSetup line let a Device→Account handoff exit
/// arm the window, which then failed a live enrollment mid-install): a reboot while
/// armed cancels the window, and enforcement progress since arming re-arms it instead
/// of failing.
/// of failing. During the OOBE update phase (Windows/.NET update activity after the ESP
/// exit, no real user yet — D-310) the window re-arms within the phase's own bounds: the
/// update cap, the sign-in window after the update, one window after the sign-in.
/// </summary>
public const string AdvisoryCompletion = "advisory_completion";
@@ -201,6 +203,14 @@ namespace AutopilotMonitor.DecisionCore.Engine
/// </summary>
public const string HelloWaitTimeoutSeconds = "helloWaitTimeoutSeconds";
/// <summary>
/// On <c>EnrollmentFactsObserved</c>: the mode of the OOBE update phase (remote config
/// <c>OobeUpdatePhaseMode</c>, stamped by the agent at every start — <c>Off</c>,
/// <c>Shadow</c> or <c>Active</c>, see <see cref="State.OobeUpdatePhaseModes"/>). Missing →
/// the engine treats the phase as <c>Shadow</c>.
/// </summary>
public const string OobeUpdatePhaseMode = "oobeUpdatePhaseMode";
// --- InformationalEvent payload (plan §1.3, single-rail refactor) ------------
// Mirrors the EnrollmentEvent fields the reducer must reconstruct for the
// EmitEventTimelineEntry effect. EventType / Source are mandatory; the rest are
@@ -34,6 +34,7 @@ namespace AutopilotMonitor.DecisionCore.Engine
builder.SystemRebootUtc = new SignalFact<DateTime>(signal.OccurredAtUtc, signal.SessionSignalOrdinal);
}
RecordOobeUpdateRestart(state, builder, signal);
var rebaseEffects = RebaseEspExitVariantAdvisoryWindowOnReboot(state, builder, signal);
var newState = builder.Build();
@@ -961,6 +962,79 @@ namespace AutopilotMonitor.DecisionCore.Engine
return new DecisionStep(gateRearmedState, gateRearmedTransition, gateRearmedEffects);
}
// OOBE update phase (D-310, report 3a2207978e4c). The arming exit was the Device-ESP
// handoff and Windows has been installing its OOBE quality update since — 20–40 min
// plus a restart to the lock screen, where the device then waits for the user. No
// real user has signed in yet, so "the page closed and nothing happens" is not
// the shape this window resolves. The phase holds the window, each state bounded on
// its own (see DecisionEngine.OsUpdate.cs): while the update runs up to its cap, after
// it for the sign-in window, and for one window after the sign-in so the user's own
// setup (User ESP, Hello wizard) is not failed by a window re-armed against the
// update. Each re-arm is clamped to the state's bound, so its expiry is decided on
// time. Expired: a still-running update fails as oobe_update_not_finished below; an
// ended update without a sign-in stops waiting — no verdict, the backend classifies.
// The advisory variant is exempt for the same reason as above, unless the failure
// recovered or this is the Continue-Anyway observation (the user must still continue
// past the update page).
//
// Measurement before enforcement: only the Active mode lets the phase act. In Shadow
// mode the evaluation is reported next to today's verdict below; Off ignores it.
var oobePhaseMode = OobeUpdatePhaseModes.Normalize(state.ScenarioObservations.OobeUpdatePhaseMode?.Value);
var evaluatedOobePhase = (!hasAdvisoryAnchor || advisoryResolved || isObservationAdvisory)
&& oobePhaseMode != OobeUpdatePhaseModes.Off
? EvaluateOobeUpdatePhase(state, EffectiveDeadlineBase(state, signal))
: new OobeUpdatePhaseStatus(OobeUpdatePhaseKind.None);
var oobePhase = oobePhaseMode == OobeUpdatePhaseModes.Active
? evaluatedOobePhase
: new OobeUpdatePhaseStatus(OobeUpdatePhaseKind.None);
if (oobePhase.Holds)
{
var untilBound = oobePhase.BoundUtc!.Value - EffectiveDeadlineBase(state, signal);
var phaseRearmedDeadline = BuildAdvisoryCompletionDeadline(state, signal,
untilBound < s_advisoryCompletionWindow ? untilBound : s_advisoryCompletionWindow);
builder.AddDeadline(phaseRearmedDeadline);
var (triggerSuffix, message) = DescribeOobeUpdateHold(oobePhase.Kind);
var phaseTrigger = $"DeadlineFired:{DeadlineNames.AdvisoryCompletion}:{triggerSuffix}";
var phaseRearmedState = builder.Build();
var phaseRearmedTransition = BuildTakenTransition(
before: state,
signal: signal,
toStage: state.Stage,
nextStepIndex: nextStep,
trigger: phaseTrigger);
var phaseParameters = new Dictionary<string, string>
{
["eventType"] = SharedConstants.EventTypes.CompletionWaiting,
["source"] = "DecisionEngine",
["severity"] = "Info",
["immediateUpload"] = "false",
["message"] = message,
["missingPrerequisites"] = string.Join(",", BuildMissingCompletionPrerequisites(state)),
["trigger"] = phaseTrigger,
["stage"] = state.Stage.ToString(),
["resolutionDeadlineDueAtUtc"] = phaseRearmedDeadline.DueAtUtc.ToString("o"),
["oobeUpdateHoldsUntilUtc"] = oobePhase.BoundUtc.Value.ToString("o"),
};
AddOobeUpdateContext(phaseParameters, state, oobePhase.UpdateEndedUtc);
// Same deliberate dedupe bypass as the re-arm sites above — bounded to one event
// per window; the phase's own bounds cap the count.
var phaseRearmedEffects = new[]
{
new DecisionEffect(DecisionEffectKind.ScheduleDeadline, deadline: phaseRearmedDeadline),
new DecisionEffect(DecisionEffectKind.EmitEventTimelineEntry, parameters: phaseParameters),
};
return new DecisionStep(phaseRearmedState, phaseRearmedTransition, phaseRearmedEffects);
}
if (oobePhase.Kind == OobeUpdatePhaseKind.SignInWaitExpired)
{
return BuildOobeUpdateStopWaitingStep(state, builder, signal, nextStep, oobePhase);
}
// Conjunction not met — the session is failed. The two arming variants carry
// distinct context (see method doc): the advisory variant un-defangs the original
// ESP failure; the esp-exit variant never had an ESP failure, so it gets its own
@@ -968,7 +1042,18 @@ namespace AutopilotMonitor.DecisionCore.Engine
// stays the deadline, not EspTerminalFailure).
string reason;
Dictionary<string, string> parameters;
if (hasAdvisoryAnchor && advisoryResolved)
var failTrigger = $"DeadlineFired:{DeadlineNames.AdvisoryCompletion}";
if (oobePhase.Kind == OobeUpdatePhaseKind.UpdateNotFinished)
{
// D-310: the OOBE update still ran when its cap passed — a hanging update, not a
// device that went quiet after the ESP page. Its own reason, so the failure names
// the update instead of the generic window expiry.
reason = "oobe_update_not_finished";
parameters = BuildOobeUpdateNotFinishedParameters(state);
failTrigger += ":OobeUpdateNotFinished";
builder.WithLastFailureTrigger(nameof(DecisionSignalKind.DeadlineFired), signal.SessionSignalOrdinal);
}
else if (hasAdvisoryAnchor && advisoryResolved)
{
// Session 4910a5a5: the original ESP failure demonstrably recovered (failed
// category resolved to success after the advisory), so claiming
@@ -1038,22 +1123,25 @@ namespace AutopilotMonitor.DecisionCore.Engine
signal: signal,
toStage: SessionStage.Failed,
nextStepIndex: nextStep,
trigger: $"DeadlineFired:{DeadlineNames.AdvisoryCompletion}");
trigger: failTrigger);
var failedEffects = new[]
var failedEffects = new List<DecisionEffect>(2);
// Shadow mode: what the OOBE update phase would have decided, ahead of today's verdict.
if (evaluatedOobePhase.Kind != OobeUpdatePhaseKind.None && oobePhaseMode == OobeUpdatePhaseModes.Shadow)
{
new DecisionEffect(
DecisionEffectKind.EmitEventTimelineEntry,
parameters: parameters,
typedPayload: BuildEspFailureAuditTrail(
postState: failedState,
decidedStage: SessionStage.Failed,
trigger: $"DeadlineFired:{DeadlineNames.AdvisoryCompletion}",
failureReason: reason,
parameters: parameters)),
};
failedEffects.Add(BuildOobeUpdateShadowEffect(state, evaluatedOobePhase, reason));
}
failedEffects.Add(new DecisionEffect(
DecisionEffectKind.EmitEventTimelineEntry,
parameters: parameters,
typedPayload: BuildEspFailureAuditTrail(
postState: failedState,
decidedStage: SessionStage.Failed,
trigger: failTrigger,
failureReason: reason,
parameters: parameters)));
return new DecisionStep(failedState, failedTransition, failedEffects);
return new DecisionStep(failedState, failedTransition, failedEffects.ToArray());
}
private DecisionStep BuildFailedStep(
@@ -1226,6 +1314,7 @@ namespace AutopilotMonitor.DecisionCore.Engine
builder.SystemRebootUtc = new SignalFact<DateTime>(signal.OccurredAtUtc, signal.SessionSignalOrdinal);
}
RecordOobeUpdateRestart(state, builder, signal);
var rebaseEffects = RebaseEspExitVariantAdvisoryWindowOnReboot(state, builder, signal);
var newState = builder.Build();
@@ -0,0 +1,383 @@
using System;
using System.Collections.Generic;
using System.Globalization;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.DecisionCore.State;
using SharedConstants = AutopilotMonitor.Shared.Constants;
namespace AutopilotMonitor.DecisionCore.Engine
{
// OOBE update phase (D-310). With the ESP setting "Install Windows quality updates" Windows
// runs an update page at the end of OOBE — after the Device-ESP page exits and before the
// user signs in (CloudExperienceHost nodes OobeNDUP → RebootNDUP → OobeExit). Microsoft
// documents 20–40 minutes plus a restart, after which the user must sign in again at the lock
// screen. When the IME logged its AccountSetup phase line before that handoff exit (it does so
// as soon as the device phase completes), the exit armed the AdvisoryCompletion window and
// the session was failed mid-update (report 3a2207978e4c).
//
// The phase holds that window, but every hold is bounded on its own — the agent's max
// lifetime is no bound, it restarts with every agent start:
// - the update runs: at most OobeUpdateCap after the first update activity, then the
// session fails as oobe_update_not_finished;
// - the update ended: OobeSignInWindow for the sign-in, then the engine stops waiting
// (StoppedWaitingReason) and the agent ends the session like its
// max-lifetime watchdog — the backend classifies it honestly, never Failed;
// - the user arrived: one AdvisoryCompletion window of grace for the user's own setup.
public sealed partial class DecisionEngine
{
internal static readonly TimeSpan s_oobeUpdateCap = TimeSpan.FromMinutes(OobeUpdatePhaseBounds.UpdateCapMinutes);
internal static readonly TimeSpan s_oobeSignInWindow = TimeSpan.FromMinutes(OobeUpdatePhaseBounds.SignInWindowMinutes);
/// <summary>
/// Record one Windows/.NET update step. The earliest activity after the ESP exit while no
/// real user has signed in anchors the OOBE update phase
/// (<see cref="OsUpdateFacts.PostExitActivityUtc"/>). Bookkeeping only: the stage stays,
/// no deadline is armed — the phase only changes how
/// <see cref="HandleAdvisoryCompletionDeadlineFired"/> resolves.
/// </summary>
private DecisionStep HandleOsUpdateActivityV1(DecisionState state, DecisionSignal signal)
{
var nextStep = state.StepIndex + 1;
string? step = null;
string? update = null;
signal.Payload?.TryGetValue(OsUpdatePayloadKeys.Step, out step);
signal.Payload?.TryGetValue(OsUpdatePayloadKeys.Update, out update);
if (string.IsNullOrEmpty(step))
{
var bookkept = BumpStepBookkeeping(state, signal);
return new DecisionStep(
bookkept,
BuildDeadEndTransition(state, signal, bookkept.StepIndex,
trigger: nameof(DecisionSignalKind.OsUpdateActivity),
deadEndReason: "os_update_activity_without_step"),
Array.Empty<DecisionEffect>());
}
var phaseActivity = IsOobeUpdatePhaseActivity(state, signal);
var phaseStarts = phaseActivity && state.OsUpdateFacts.PostExitActivityUtc == null;
var facts = state.OsUpdateFacts.WithActivity(
signal.OccurredAtUtc, signal.SessionSignalOrdinal, step!, update, phaseActivity);
var newState = state.ToBuilder()
.WithStepIndex(nextStep)
.WithLastAppliedSignalOrdinal(signal.SessionSignalOrdinal)
.WithOsUpdateFacts(facts)
.Build();
var transition = BuildTakenTransition(
before: state,
signal: signal,
toStage: state.Stage,
nextStepIndex: nextStep,
trigger: nameof(DecisionSignalKind.OsUpdateActivity) + (phaseStarts ? ":OobeUpdatePhaseStarted" : string.Empty));
return new DecisionStep(newState, transition, Array.Empty<DecisionEffect>());
}
/// <summary>
/// Record a system restart that belongs to the OOBE update phase — the update page ends
/// with one. Called by the reboot handlers; a restart before the anchor or after a real
/// user arrived is not the update's restart.
/// </summary>
private static void RecordOobeUpdateRestart(DecisionState state, DecisionStateBuilder builder, DecisionSignal signal)
{
var anchor = state.OsUpdateFacts.PostExitActivityUtc;
if (anchor == null || signal.OccurredAtUtc < anchor.Value || HasRealUserEvidence(state)) return;
builder.WithOsUpdateFacts(state.OsUpdateFacts.WithRestart(signal.OccurredAtUtc, signal.SessionSignalOrdinal));
}
/// <summary>
/// True while the session is in the OOBE update phase: OS update activity was observed
/// after the ESP exit and no real user has signed in since. Says nothing about the bounds —
/// <see cref="EvaluateOobeUpdatePhase"/> decides what the phase holds.
/// </summary>
internal static bool IsInOobeUpdatePhase(DecisionState state) =>
state.OsUpdateFacts.PostExitActivityUtc != null && !HasRealUserEvidence(state);
internal enum OobeUpdatePhaseKind
{
/// <summary>No OOBE update phase in this session.</summary>
None,
/// <summary>The update runs; holds until the cap.</summary>
Updating,
/// <summary>The update ended; holds until the sign-in window closes.</summary>
AwaitingSignIn,
/// <summary>The user signed in less than one window ago; holds for the user's setup.</summary>
UserArrived,
/// <summary>The cap passed while the update still ran — the session fails as not finished.</summary>
UpdateNotFinished,
/// <summary>Nobody signed in within the sign-in window — the engine stops waiting.</summary>
SignInWaitExpired,
/// <summary>The user's grace passed; the window resolves the normal way.</summary>
Over,
}
internal readonly struct OobeUpdatePhaseStatus
{
public OobeUpdatePhaseStatus(OobeUpdatePhaseKind kind, DateTime? boundUtc = null, DateTime? updateEndedUtc = null)
{
Kind = kind;
BoundUtc = boundUtc;
UpdateEndedUtc = updateEndedUtc;
}
public OobeUpdatePhaseKind Kind { get; }
/// <summary>When the current state stops holding (cap, end of sign-in window or grace).</summary>
public DateTime? BoundUtc { get; }
/// <summary>When the update ended (its restart or terminal step); null while it runs.</summary>
public DateTime? UpdateEndedUtc { get; }
public bool Holds =>
Kind == OobeUpdatePhaseKind.Updating
|| Kind == OobeUpdatePhaseKind.AwaitingSignIn
|| Kind == OobeUpdatePhaseKind.UserArrived;
}
/// <summary>
/// Where the OOBE update phase stands at <paramref name="nowUtc"/>. The update counts as
/// ended when its latest event — by event time — is its restart or a terminal step
/// (installed, failed, reboot_required); any newer non-terminal step means it runs (again).
/// </summary>
internal static OobeUpdatePhaseStatus EvaluateOobeUpdatePhase(DecisionState state, DateTime nowUtc)
{
var facts = state.OsUpdateFacts;
var anchor = facts.PostExitActivityUtc;
if (anchor == null) return new OobeUpdatePhaseStatus(OobeUpdatePhaseKind.None);
var userArrivedUtc = FirstRealUserEvidenceUtc(state);
if (userArrivedUtc != null)
{
var graceEnd = userArrivedUtc.Value + s_advisoryCompletionWindow;
return nowUtc < graceEnd
? new OobeUpdatePhaseStatus(OobeUpdatePhaseKind.UserArrived, graceEnd)
: new OobeUpdatePhaseStatus(OobeUpdatePhaseKind.Over);
}
var endedUtc = UpdateEndedUtc(facts);
if (endedUtc == null)
{
var capEnd = anchor.Value + s_oobeUpdateCap;
return new OobeUpdatePhaseStatus(
nowUtc < capEnd ? OobeUpdatePhaseKind.Updating : OobeUpdatePhaseKind.UpdateNotFinished,
capEnd);
}
var signInEnd = endedUtc.Value + s_oobeSignInWindow;
return new OobeUpdatePhaseStatus(
nowUtc < signInEnd ? OobeUpdatePhaseKind.AwaitingSignIn : OobeUpdatePhaseKind.SignInWaitExpired,
signInEnd,
endedUtc);
}
private static DateTime? UpdateEndedUtc(OsUpdateFacts facts)
{
var last = facts.LastActivityUtc?.Value;
var restart = facts.RestartUtc?.Value;
if (restart != null && (last == null || restart.Value >= last.Value)) return restart;
if (last != null && OsUpdateSteps.IsTerminal(facts.LastStep?.Value)) return last;
return null;
}
/// <summary>
/// The <c>completion_waiting</c> trigger suffix and message for a holding phase state.
/// </summary>
private static (string TriggerSuffix, string Message) DescribeOobeUpdateHold(OobeUpdatePhaseKind kind)
{
switch (kind)
{
case OobeUpdatePhaseKind.Updating:
return ("OobeUpdatePhase",
"Completion resolution window re-armed: Windows is installing an update in OOBE");
case OobeUpdatePhaseKind.AwaitingSignIn:
return ("OobeUpdateAwaitingSignIn",
"Completion resolution window re-armed: the Windows update in OOBE has finished, waiting for the user to sign in");
default:
return ("OobeUpdatePhaseUserArrived",
"Completion resolution window re-armed: the user signed in after the OOBE update and gets one full window for the own setup");
}
}
/// <summary>
/// Nobody signed in within the sign-in window after the OOBE update: stop waiting. The
/// stage stays non-terminal and no verdict is declared — the device is fine, the user is
/// absent. <see cref="DecisionState.StoppedWaitingReason"/> makes the agent end the session
/// like its max-lifetime watchdog; the backend then classifies it (AwaitingUser for a
/// WhiteGlove Part 2, otherwise Incomplete), never Failed.
/// </summary>
private DecisionStep BuildOobeUpdateStopWaitingStep(
DecisionState state,
DecisionStateBuilder builder,
DecisionSignal signal,
int nextStep,
OobeUpdatePhaseStatus phase)
{
builder.StoppedWaitingReason = new SignalFact<string>(
StoppedWaitingReasons.OobeUpdateNoSignIn, signal.SessionSignalOrdinal);
builder.ClearDeadlines();
var trigger = $"DeadlineFired:{DeadlineNames.AdvisoryCompletion}:OobeUpdateSignInWaitExpired";
var parameters = new Dictionary<string, string>
{
["eventType"] = SharedConstants.EventTypes.CompletionWaitExpired,
["source"] = "DecisionEngine",
["severity"] = "Warning",
["immediateUpload"] = "true",
["message"] = "Nobody signed in within " +
s_oobeSignInWindow.TotalMinutes.ToString("0", CultureInfo.InvariantCulture) +
" minutes after the Windows update in OOBE — the agent stops waiting. Not a failure: " +
"the session is classified as awaiting the user or incomplete.",
["reason"] = StoppedWaitingReasons.OobeUpdateNoSignIn,
["signInWindowMinutes"] = s_oobeSignInWindow.TotalMinutes.ToString("0", CultureInfo.InvariantCulture),
["stage"] = state.Stage.ToString(),
};
AddOobeUpdateContext(parameters, state, phase.UpdateEndedUtc);
var stoppedState = builder.Build();
var transition = BuildTakenTransition(
before: state,
signal: signal,
toStage: state.Stage,
nextStepIndex: nextStep,
trigger: trigger);
return new DecisionStep(stoppedState, transition, new[]
{
new DecisionEffect(DecisionEffectKind.EmitEventTimelineEntry, parameters: parameters),
});
}
/// <summary>
/// Shadow mode: what the phase would decide at this advisory fire, emitted right before
/// today's verdict (<paramref name="decidedReason"/>). At most once per session — in Shadow
/// mode the fire that reaches the phase guard always ends in that verdict.
/// </summary>
private static DecisionEffect BuildOobeUpdateShadowEffect(
DecisionState state,
OobeUpdatePhaseStatus phase,
string decidedReason)
{
var (phaseState, wouldDecide, outlook) = DescribeOobeUpdateShadow(phase.Kind);
var parameters = new Dictionary<string, string>
{
["eventType"] = SharedConstants.EventTypes.OobeUpdatePhaseShadow,
["source"] = "DecisionEngine",
["severity"] = "Debug",
["immediateUpload"] = "true",
["message"] = $"OOBE update phase (shadow): {outlook} Today's rule decides ({decidedReason}).",
["phaseMode"] = OobeUpdatePhaseModes.Shadow,
["phaseState"] = phaseState,
["wouldDecide"] = wouldDecide,
["decidedReason"] = decidedReason,
["trigger"] = $"DeadlineFired:{DeadlineNames.AdvisoryCompletion}",
["stage"] = state.Stage.ToString(),
};
if (phase.BoundUtc != null)
parameters["phaseBoundUtc"] = phase.BoundUtc.Value.ToString("o");
var userArrivedUtc = FirstRealUserEvidenceUtc(state);
if (userArrivedUtc != null)
parameters["userArrivedUtc"] = userArrivedUtc.Value.ToString("o");
AddOobeUpdateContext(parameters, state, phase.UpdateEndedUtc);
return new DecisionEffect(DecisionEffectKind.EmitEventTimelineEntry, parameters: parameters);
}
/// <summary><c>phaseState</c>, <c>wouldDecide</c> and the message part of a shadow evaluation.</summary>
private static (string PhaseState, string WouldDecide, string Outlook) DescribeOobeUpdateShadow(OobeUpdatePhaseKind kind)
{
var capHours = s_oobeUpdateCap.TotalHours.ToString("0", CultureInfo.InvariantCulture);
var signInMinutes = s_oobeSignInWindow.TotalMinutes.ToString("0", CultureInfo.InvariantCulture);
switch (kind)
{
case OobeUpdatePhaseKind.Updating:
return ("update_running", "hold",
$"Windows is still installing an update in OOBE; the phase would keep waiting (cap {capHours} h).");
case OobeUpdatePhaseKind.AwaitingSignIn:
return ("awaiting_sign_in", "hold",
$"the Windows update in OOBE has finished; the phase would wait {signInMinutes} minutes for the sign-in.");
case OobeUpdatePhaseKind.UserArrived:
return ("user_arrived", "hold",
"the user signed in after the OOBE update; the phase would give the user's own setup one more window.");
case OobeUpdatePhaseKind.UpdateNotFinished:
return ("update_not_finished", "fail_oobe_update_not_finished",
$"the update still ran after {capHours} hours; the phase would fail the session as oobe_update_not_finished.");
case OobeUpdatePhaseKind.SignInWaitExpired:
return ("sign_in_wait_expired", "stop_waiting",
$"nobody signed in within {signInMinutes} minutes after the update; the phase would stop waiting without a verdict.");
default:
return ("over", "unchanged",
"the user's window after the OOBE update has passed; the phase would decide the same.");
}
}
/// <summary>
/// Failure parameters for an OOBE update that still ran when the cap passed — a hanging
/// update, reported with its own reason instead of <c>esp_exit_without_completion_evidence</c>.
/// </summary>
private static Dictionary<string, string> BuildOobeUpdateNotFinishedParameters(DecisionState state)
{
var parameters = new Dictionary<string, string>
{
["eventType"] = SharedConstants.EventTypes.EnrollmentFailed,
["reason"] = "oobe_update_not_finished",
["advisoryReason"] = "oobe_update_cap_expired_without_restart",
["oobeUpdateCapHours"] = s_oobeUpdateCap.TotalHours.ToString("0", CultureInfo.InvariantCulture),
};
AddOobeUpdateContext(parameters, state, updateEndedUtc: null);
return parameters;
}
private static void AddOobeUpdateContext(Dictionary<string, string> parameters, DecisionState state, DateTime? updateEndedUtc)
{
var facts = state.OsUpdateFacts;
if (facts.PostExitActivityUtc != null)
parameters["oobeUpdatePhaseStartedUtc"] = facts.PostExitActivityUtc.Value.ToString("o");
if (updateEndedUtc != null)
parameters["oobeUpdateEndedUtc"] = updateEndedUtc.Value.ToString("o");
if (facts.LastStep != null)
parameters["osUpdateLastStep"] = facts.LastStep.Value;
if (facts.LastUpdate != null)
parameters["osUpdate"] = facts.LastUpdate.Value;
}
/// <summary>
/// Real-user evidence: a DAD-validated desktop, a Hello wizard start or resolution, or a
/// confirmed Account Setup. Deliberately NOT evidence: the IME AccountSetup phase line and
/// the IME user-session completion — IME writes both under <c>defaultuser0</c> before the
/// sign-in (session 1924092e, report 3a2207978e4c).
/// </summary>
internal static bool HasRealUserEvidence(DecisionState state) => FirstRealUserEvidenceUtc(state) != null;
private static DateTime? FirstRealUserEvidenceUtc(DecisionState state)
{
DateTime? first = null;
foreach (var fact in new[]
{
state.DesktopArrivedUtc,
state.HelloWizardStartedUtc,
state.HelloResolvedUtc,
state.AccountSetupProvisioningSucceededUtc,
})
{
if (fact != null && (first == null || fact.Value < first.Value)) first = fact.Value;
}
return first;
}
private static bool IsOobeUpdatePhaseActivity(DecisionState state, DecisionSignal signal)
{
var exit = state.EspFinalExitUtc;
if (exit == null) return false;
if (signal.SessionSignalOrdinal <= exit.SourceSignalOrdinal) return false;
// The watchers backfill their logs after an agent restart: such a record gets a fresh
// ordinal but keeps its event-log time, so activity that predates the exit is not
// part of the phase.
if (signal.OccurredAtUtc < exit.Value) return false;
return !HasRealUserEvidence(state);
}
}
}
@@ -602,6 +602,15 @@ namespace AutopilotMonitor.DecisionCore.Engine
helloSafetyExtendEffect = ExtendArmedHelloSafetyToTenantWindow(state, builder, signal);
}
// OOBE update phase mode (D-310, remote config) — only Active lets the phase act; the
// observation can take Active away at a later agent start, never grant it.
if (signal.Payload != null
&& signal.Payload.TryGetValue(SignalPayloadKeys.OobeUpdatePhaseMode, out var rawOobeUpdatePhaseMode))
{
builder.ScenarioObservations = builder.ScenarioObservations
.WithOobeUpdatePhaseMode(rawOobeUpdatePhaseMode, signal.SessionSignalOrdinal);
}
var newState = builder.Build();
var transition = BuildTakenTransition(
before: state,
@@ -150,6 +150,9 @@ namespace AutopilotMonitor.DecisionCore.Engine
// Note: RealmJoinTimeout is a deadline-only kind (no direct adapter signal) —
// dispatched via DeadlineFiredV1 switch in DecisionEngine.Shared.cs.
// ----- OOBE update phase (DecisionEngine.OsUpdate.cs) — D-310 -----
(DecisionSignalKind.OsUpdateActivity, 1) => HandleOsUpdateActivityV1(state, signal),
// ----- Fall-through: unknown (kind, schemaVersion) pair → dead-end journal entry -----
_ => HandleUnhandledSignal(state, signal),
};
@@ -131,6 +131,25 @@ namespace AutopilotMonitor.DecisionCore.Engine
timestamps["helloWizardStarted"] = FormatUtc(state.HelloWizardStartedUtc.Value);
evidence["helloWizardStarted"] = TimestampedEvidence(state.HelloWizardStartedUtc);
}
// D-310 — OOBE update phase: Windows/.NET update activity after the ESP exit while no
// real user had signed in. Surfaced so terminal audit trails show that a long silence
// after the exit was Windows installing the OOBE quality update, not a hang — and in
// which mode the phase ran (only Active acts; Shadow only reports).
if (state.OsUpdateFacts.PostExitActivityUtc != null)
{
seen.Add("oobe_update_phase");
timestamps["oobeUpdatePhase"] = FormatUtc(state.OsUpdateFacts.PostExitActivityUtc.Value);
var phaseEvidence = TimestampedEvidence(state.OsUpdateFacts.PostExitActivityUtc);
phaseEvidence["mode"] = OobeUpdatePhaseModes.Normalize(state.ScenarioObservations.OobeUpdatePhaseMode?.Value);
evidence["oobeUpdatePhase"] = phaseEvidence;
}
// D-310 — the engine stopped waiting for the user without a verdict (bounded wait
// expired). Surfaced so final-status.json says why the agent ended the session.
if (state.StoppedWaitingReason != null)
{
seen.Add("stopped_waiting");
evidence["stoppedWaiting"] = TimestampedEvidence(state.StoppedWaitingReason);
}
// WhiteGlove Part-1 sealing observations — engine-internal, no UTC fact.
var obs = state.ScenarioObservations;
@@ -68,6 +68,7 @@ namespace AutopilotMonitor.DecisionCore.Engine
["imeUserSessionCompletedUtc"] = SerializeFact(state.ImeUserSessionCompletedUtc, FormatUtc),
["completionWaitingFingerprint"] = SerializeFact(state.CompletionWaitingFingerprint, v => v),
["helloWizardStartedUtc"] = SerializeFact(state.HelloWizardStartedUtc, FormatUtc),
["stoppedWaitingReason"] = SerializeFact(state.StoppedWaitingReason, v => v),
};
var scenario = new Dictionary<string, object?>(StringComparer.Ordinal)
@@ -131,5 +131,17 @@ namespace AutopilotMonitor.DecisionCore.Signals
// RealmJoinFacts.ProductVersion/ReleaseChannel so the persisted facts match what
// actually ran the deployment. Never touches the gate or the timeout deadline.
RealmJoinAutoUpdateDetected,
// --- OS update in OOBE (D-310) ---
// One step of a Windows/.NET update observed by the Windows Update client watchers
// (System + Operational channel) or the CBS servicing watcher (Setup log). Payload:
// { "step": download_started|downloaded|install_started|staging_started|staged|
// reboot_required|installed|failed, "source": wu|servicing, "update": title or KB }.
// Posted unless CollectorConfiguration.OobeUpdatePhaseMode is Off. The reducer records
// OsUpdateFacts; activity after the ESP exit without real-user evidence starts the OOBE
// update phase (Windows installs the OOBE quality update between the Device-ESP exit and
// the user's sign-in — report 3a2207978e4c), which holds the AdvisoryCompletion window in
// the Active mode and is only reported in the Shadow mode.
OsUpdateActivity,
}
}
@@ -66,7 +66,9 @@ namespace AutopilotMonitor.DecisionCore.State
SignalFact<string>? completionWaitingFingerprint = null,
SignalFact<DateTime>? helloWizardStartedUtc = null,
SignalFact<DateTime>? espAdvisoryFailureResolvedUtc = null,
SignalFact<string>? espAdvisoryFailureCategory = null)
SignalFact<string>? espAdvisoryFailureCategory = null,
OsUpdateFacts? osUpdateFacts = null,
SignalFact<string>? stoppedWaitingReason = null)
{
if (string.IsNullOrEmpty(sessionId))
{
@@ -112,6 +114,8 @@ namespace AutopilotMonitor.DecisionCore.State
HelloWizardStartedUtc = helloWizardStartedUtc;
EspAdvisoryFailureResolvedUtc = espAdvisoryFailureResolvedUtc;
EspAdvisoryFailureCategory = espAdvisoryFailureCategory;
OsUpdateFacts = osUpdateFacts ?? OsUpdateFacts.Empty;
StoppedWaitingReason = stoppedWaitingReason;
}
public string SessionId { get; }
@@ -354,6 +358,24 @@ namespace AutopilotMonitor.DecisionCore.State
/// </summary>
public SignalFact<string>? EspAdvisoryFailureCategory { get; }
/// <summary>
/// Windows/.NET update activity (D-310). Never null; defaults to
/// <see cref="State.OsUpdateFacts.Empty"/>. Its <see cref="State.OsUpdateFacts.PostExitActivityUtc"/>
/// anchors the bounded OOBE update phase that holds the <c>AdvisoryCompletion</c> window.
/// Additive, no snapshot-schema bump.
/// </summary>
public OsUpdateFacts OsUpdateFacts { get; }
/// <summary>
/// Set when the engine stopped waiting for the user without a verdict — a bounded wait
/// expired (D-310: nobody signed in within the sign-in window after the OOBE update).
/// The value is the reason code (<see cref="StoppedWaitingReasons"/>). The stage stays
/// non-terminal: the agent ends the session like its max-lifetime watchdog (outcome
/// TimedOut) and the backend classifies it honestly — never Failed. Set-once.
/// Additive-nullable, no snapshot-schema bump.
/// </summary>
public SignalFact<string>? StoppedWaitingReason { get; }
public string SchemaVersion { get; }
/// <summary>
@@ -53,6 +53,8 @@ namespace AutopilotMonitor.DecisionCore.State
HelloWizardStartedUtc = source.HelloWizardStartedUtc;
EspAdvisoryFailureResolvedUtc = source.EspAdvisoryFailureResolvedUtc;
EspAdvisoryFailureCategory = source.EspAdvisoryFailureCategory;
OsUpdateFacts = source.OsUpdateFacts;
StoppedWaitingReason = source.StoppedWaitingReason;
}
public string SessionId { get; set; }
@@ -88,6 +90,8 @@ namespace AutopilotMonitor.DecisionCore.State
public SignalFact<DateTime>? HelloWizardStartedUtc { get; set; }
public SignalFact<DateTime>? EspAdvisoryFailureResolvedUtc { get; set; }
public SignalFact<string>? EspAdvisoryFailureCategory { get; set; }
public OsUpdateFacts OsUpdateFacts { get; set; } = OsUpdateFacts.Empty;
public SignalFact<string>? StoppedWaitingReason { get; set; }
public string SchemaVersion { get; set; }
// ---------- fluent helpers for the most common reducer operations ----------
@@ -143,6 +147,12 @@ namespace AutopilotMonitor.DecisionCore.State
return this;
}
public DecisionStateBuilder WithOsUpdateFacts(OsUpdateFacts facts)
{
OsUpdateFacts = facts ?? throw new ArgumentNullException(nameof(facts));
return this;
}
public DecisionStateBuilder WithScenarioProfile(EnrollmentScenarioProfile profile)
{
ScenarioProfile = profile ?? throw new ArgumentNullException(nameof(profile));
@@ -258,6 +268,8 @@ namespace AutopilotMonitor.DecisionCore.State
completionWaitingFingerprint: CompletionWaitingFingerprint,
helloWizardStartedUtc: HelloWizardStartedUtc,
espAdvisoryFailureResolvedUtc: EspAdvisoryFailureResolvedUtc,
espAdvisoryFailureCategory: EspAdvisoryFailureCategory);
espAdvisoryFailureCategory: EspAdvisoryFailureCategory,
osUpdateFacts: OsUpdateFacts,
stoppedWaitingReason: StoppedWaitingReason);
}
}
@@ -123,6 +123,15 @@ namespace AutopilotMonitor.DecisionCore.State
/// </summary>
public SignalFact<int>? HelloWaitTimeoutSeconds { get; init; }
/// <summary>
/// Tenant-config fact from <see cref="Signals.DecisionSignalKind.EnrollmentFactsObserved"/>
/// (payload <c>oobeUpdatePhaseMode</c>, stamped by the agent at every start): the mode of
/// the OOBE update phase, normalized to <see cref="OobeUpdatePhaseModes"/> (D-310).
/// <c>null</c> = never stamped (an older agent), treated as <c>Shadow</c>. Not set-once:
/// see <see cref="WithOobeUpdatePhaseMode"/>.
/// </summary>
public SignalFact<string>? OobeUpdatePhaseMode { get; init; }
public EnrollmentScenarioObservations WithShellCoreWhiteGloveSuccessSeen(long sourceSignalOrdinal) =>
ShellCoreWhiteGloveSuccessSeen != null
? this
@@ -182,5 +191,21 @@ namespace AutopilotMonitor.DecisionCore.State
HelloWaitTimeoutSeconds != null
? this
: this with { HelloWaitTimeoutSeconds = new SignalFact<int>(value, sourceSignalOrdinal) };
/// <summary>
/// Records the stamped mode. The first stamp sets it; afterwards a stamp can only take
/// <c>Active</c> away, never grant it: switching the phase on mid-session applies from
/// the next session, switching it off applies at the next agent start (kill switch).
/// </summary>
public EnrollmentScenarioObservations WithOobeUpdatePhaseMode(string? value, long sourceSignalOrdinal)
{
var mode = OobeUpdatePhaseModes.Normalize(value);
if (OobeUpdatePhaseMode == null
|| (OobeUpdatePhaseMode.Value == OobeUpdatePhaseModes.Active && mode != OobeUpdatePhaseModes.Active))
{
return this with { OobeUpdatePhaseMode = new SignalFact<string>(mode, sourceSignalOrdinal) };
}
return this;
}
}
}
@@ -0,0 +1,223 @@
using System;
namespace AutopilotMonitor.DecisionCore.State
{
/// <summary>
/// Windows/.NET update activity observed during the session (D-310), fed by
/// <see cref="Signals.DecisionSignalKind.OsUpdateActivity"/> and the reboot signals. Drives the
/// bounded OOBE update phase: Windows installs the OOBE quality update after the Device-ESP exit
/// and before the user signs in, which can take longer than the <c>AdvisoryCompletion</c> window
/// and ends with a restart to the lock screen.
/// </summary>
/// <remarks>
/// Times follow the event time, not the arrival order: the watchers backfill their logs after an
/// agent restart, so a record written before the update restart can arrive after it. The
/// earliest-time facts (<see cref="FirstActivityUtc"/>, <see cref="PostExitActivityUtc"/>,
/// <see cref="RebootRequiredUtc"/>) keep the earliest time, the latest-time facts
/// (<see cref="LastActivityUtc"/>, <see cref="LastStep"/>, <see cref="LastUpdate"/>,
/// <see cref="FailedUtc"/>, <see cref="RestartUtc"/>) change only for a newer one — otherwise a
/// backfilled older step would read as the latest and the update would count as running again.
/// Immutable; additive on <see cref="DecisionState"/> (Empty default), so older snapshots
/// deserialize without a schema bump.
/// </remarks>
public sealed class OsUpdateFacts
{
public static readonly OsUpdateFacts Empty = new OsUpdateFacts(
firstActivityUtc: null,
postExitActivityUtc: null,
lastActivityUtc: null,
lastStep: null,
lastUpdate: null,
activityCount: 0,
rebootRequiredUtc: null,
failedUtc: null,
restartUtc: null);
public OsUpdateFacts(
SignalFact<DateTime>? firstActivityUtc,
SignalFact<DateTime>? postExitActivityUtc,
SignalFact<DateTime>? lastActivityUtc,
SignalFact<string>? lastStep,
SignalFact<string>? lastUpdate,
int activityCount,
SignalFact<DateTime>? rebootRequiredUtc,
SignalFact<DateTime>? failedUtc,
SignalFact<DateTime>? restartUtc = null)
{
FirstActivityUtc = firstActivityUtc;
PostExitActivityUtc = postExitActivityUtc;
LastActivityUtc = lastActivityUtc;
LastStep = lastStep;
LastUpdate = lastUpdate;
ActivityCount = activityCount;
RebootRequiredUtc = rebootRequiredUtc;
FailedUtc = failedUtc;
RestartUtc = restartUtc;
}
/// <summary>Earliest OS update activity of the session, wherever it happened.</summary>
public SignalFact<DateTime>? FirstActivityUtc { get; }
/// <summary>
/// Earliest OS update activity after the ESP exit while no real user had signed in — the
/// anchor of the OOBE update phase and the start of its cap.
/// </summary>
public SignalFact<DateTime>? PostExitActivityUtc { get; }
/// <summary>Latest OS update activity.</summary>
public SignalFact<DateTime>? LastActivityUtc { get; }
/// <summary>Step of the latest activity (e.g. <c>install_started</c>).</summary>
public SignalFact<string>? LastStep { get; }
/// <summary>Update title or KB of the latest activity that named one, bounded.</summary>
public SignalFact<string>? LastUpdate { get; }
public int ActivityCount { get; }
/// <summary>Earliest <c>reboot_required</c> step (CBS staged the update for the restart).</summary>
public SignalFact<DateTime>? RebootRequiredUtc { get; }
/// <summary>Latest <c>failed</c> step.</summary>
public SignalFact<DateTime>? FailedUtc { get; }
/// <summary>
/// Latest system restart observed after the phase anchor while no real user had signed in —
/// the update's own restart (the OOBE update page ends with one).
/// </summary>
public SignalFact<DateTime>? RestartUtc { get; }
/// <summary>
/// Record one activity. <paramref name="isPostExit"/> marks it as part of the OOBE update
/// phase (after the ESP exit, no real user yet).
/// </summary>
public OsUpdateFacts WithActivity(
DateTime utc, long sourceSignalOrdinal, string step, string? update, bool isPostExit)
{
var at = new SignalFact<DateTime>(utc, sourceSignalOrdinal);
var isNewest = LastActivityUtc == null || utc >= LastActivityUtc.Value;
var boundedUpdate = FactStringBounds.Bound(update);
return new OsUpdateFacts(
firstActivityUtc: Earliest(FirstActivityUtc, at),
postExitActivityUtc: isPostExit ? Earliest(PostExitActivityUtc, at) : PostExitActivityUtc,
lastActivityUtc: isNewest ? at : LastActivityUtc,
lastStep: isNewest
? new SignalFact<string>(FactStringBounds.Bound(step) ?? string.Empty, sourceSignalOrdinal)
: LastStep,
lastUpdate: isNewest && boundedUpdate != null
? new SignalFact<string>(boundedUpdate, sourceSignalOrdinal)
: LastUpdate,
activityCount: ActivityCount + 1,
rebootRequiredUtc: string.Equals(step, OsUpdateSteps.RebootRequired, StringComparison.Ordinal)
? Earliest(RebootRequiredUtc, at)
: RebootRequiredUtc,
failedUtc: string.Equals(step, OsUpdateSteps.Failed, StringComparison.Ordinal)
? Latest(FailedUtc, at)
: FailedUtc,
restartUtc: RestartUtc);
}
/// <summary>Record a system restart that happened during the OOBE update phase.</summary>
public OsUpdateFacts WithRestart(DateTime utc, long sourceSignalOrdinal) =>
new OsUpdateFacts(
firstActivityUtc: FirstActivityUtc,
postExitActivityUtc: PostExitActivityUtc,
lastActivityUtc: LastActivityUtc,
lastStep: LastStep,
lastUpdate: LastUpdate,
activityCount: ActivityCount,
rebootRequiredUtc: RebootRequiredUtc,
failedUtc: FailedUtc,
restartUtc: Latest(RestartUtc, new SignalFact<DateTime>(utc, sourceSignalOrdinal)));
private static SignalFact<DateTime> Earliest(SignalFact<DateTime>? current, SignalFact<DateTime> candidate) =>
current == null || candidate.Value < current.Value ? candidate : current;
private static SignalFact<DateTime> Latest(SignalFact<DateTime>? current, SignalFact<DateTime> candidate) =>
current == null || candidate.Value > current.Value ? candidate : current;
}
/// <summary>Step names carried in the <c>step</c> payload of <c>OsUpdateActivity</c>.</summary>
public static class OsUpdateSteps
{
public const string DownloadStarted = "download_started";
public const string Downloaded = "downloaded";
public const string InstallStarted = "install_started";
public const string StagingStarted = "staging_started";
public const string Staged = "staged";
public const string RebootRequired = "reboot_required";
public const string Installed = "installed";
public const string Failed = "failed";
/// <summary>
/// True for a step after which the update work is done: installed, failed, or installed
/// pending the restart. Any other step means the update is still running.
/// </summary>
public static bool IsTerminal(string? step) =>
string.Equals(step, Installed, StringComparison.Ordinal)
|| string.Equals(step, Failed, StringComparison.Ordinal)
|| string.Equals(step, RebootRequired, StringComparison.Ordinal);
}
/// <summary>Payload keys of <c>OsUpdateActivity</c>.</summary>
public static class OsUpdatePayloadKeys
{
public const string Step = "step";
/// <summary><c>wu</c> (Windows Update client) or <c>servicing</c> (CBS, Setup log).</summary>
public const string Source = "source";
/// <summary>Update title (WU) or package identifier (CBS).</summary>
public const string Update = "update";
}
/// <summary>
/// Modes of the OOBE update phase (D-310), from the remote config field
/// <c>OobeUpdatePhaseMode</c>. <see cref="Off"/>: the update watchers feed nothing into the
/// engine. <see cref="Shadow"/>: they do, the engine records the phase and reports at the
/// completion window what the phase would decide — today's rule still decides.
/// <see cref="Active"/>: the phase holds, stops or fails.
/// </summary>
public static class OobeUpdatePhaseModes
{
public const string Off = "Off";
public const string Shadow = "Shadow";
public const string Active = "Active";
/// <summary>
/// One of the three modes, case-insensitive. Anything else — empty, unknown, a value from
/// a newer backend — is <see cref="Shadow"/>: it changes no decision and still measures.
/// </summary>
public static string Normalize(string? mode)
{
if (string.Equals(mode, Off, StringComparison.OrdinalIgnoreCase)) return Off;
if (string.Equals(mode, Active, StringComparison.OrdinalIgnoreCase)) return Active;
return Shadow;
}
/// <summary>True when the update watchers post <c>OsUpdateActivity</c> to the engine.</summary>
public static bool PostsSignals(string? mode) => Normalize(mode) != Off;
}
/// <summary>Bounds of the OOBE update phase (D-310, user decision 2026-10-02).</summary>
public static class OobeUpdatePhaseBounds
{
/// <summary>Longest the update may run (from its first activity) before the session fails as not finished.</summary>
public const int UpdateCapMinutes = 180;
/// <summary>Sign-in window after the update ended (its restart or terminal step).</summary>
public const int SignInWindowMinutes = 60;
}
/// <summary>Reason codes of <see cref="DecisionState.StoppedWaitingReason"/>.</summary>
public static class StoppedWaitingReasons
{
/// <summary>Nobody signed in within the sign-in window after the OOBE update (D-310).</summary>
public const string OobeUpdateNoSignIn = "oobe_update_no_sign_in";
/// <summary>One sentence for the session's end reason (final status, shutdown event).</summary>
public static string Describe(string? reason) =>
string.Equals(reason, OobeUpdateNoSignIn, StringComparison.Ordinal)
? $"Nobody signed in within {OobeUpdatePhaseBounds.SignInWindowMinutes} minutes after the Windows update during setup; the agent stopped waiting."
: "The agent stopped waiting for the user without a verdict.";
}
}
@@ -520,6 +520,14 @@ namespace AutopilotMonitor.Shared
// the engine is still waiting on (account_setup_provisioning_complete,
// hello_resolution, desktop_arrival, realmjoin_resolution).
public const string CompletionWaiting = "completion_waiting";
// D-310 — the DecisionEngine stopped waiting for the user without a verdict: a bounded
// wait expired (Data.reason, e.g. oobe_update_no_sign_in). The agent then ends the
// session like its max-lifetime watchdog; the backend classifies it, never Failed.
public const string CompletionWaitExpired = "completion_wait_expired";
// D-310 shadow mode — at the post-ESP-exit completion window the DecisionEngine reports
// what the OOBE update phase would decide (Data.phaseState, wouldDecide, phaseBoundUtc)
// right before today's verdict (Data.decidedReason). Debug; at most once per session.
public const string OobeUpdatePhaseShadow = "oobe_update_phase_shadow";
public const string ScriptStarted = "script_started";
public const string ScriptCompleted = "script_completed";
public const string ScriptFailed = "script_failed";
@@ -626,15 +634,25 @@ namespace AutopilotMonitor.Shared
public const string ModernDeploymentWarning = "modern_deployment_warning"; // Level 3 (Warning)
public const string ModernDeploymentError = "modern_deployment_error"; // Level 1-2 (Critical/Error)
// Windows Update during OOBE (WindowsUpdateTracker — Microsoft-Windows-WindowsUpdateClient/
// Operational live watcher + startup backfill). Surfaces quality/cumulative updates that
// install DURING enrollment — a blind spot no other tool (Intune console included) covers.
// A cumulative update installing mid-OOBE can break the enrollment (r/Intune KB5095189) and
// is becoming more common with the ESP "Install Windows quality updates during OOBE" feature.
// WU Client EventIDs: 19=success, 20=failure (carries HRESULT), 43=install started, 44=download started.
// Windows Update during OOBE (WindowsUpdateTracker — Microsoft-Windows-WindowsUpdateClient
// live watchers + startup backfill). Surfaces quality/cumulative updates that install DURING
// enrollment — a blind spot no other tool (Intune console included) covers. The provider
// manifest splits the IDs across two channels: 19=success, 20=failure (HRESULT), 43=install
// started, 44=download started live in the System log; 25=scan failed, 26=scan found N,
// 31=download failed, 41=downloaded live in the Operational channel. Only Windows/.NET
// updates (Data.updateClass: os) are emitted one by one; Store/Defender/other activity is
// counted into windows_update_activity_summary (D-310).
public const string WindowsUpdateSucceeded = "windows_update_succeeded"; // WU Client EventID 19
public const string WindowsUpdateFailed = "windows_update_failed"; // WU Client EventID 20 — Data.hresult(hex)+hresultSymbol(decoded)+updateTitle+updateGuid
public const string WindowsUpdateStarted = "windows_update_started"; // WU Client EventID 43 (install) / 44 (download) — Debug context
public const string WindowsUpdateFailed = "windows_update_failed"; // WU Client EventID 20 (install) / 31 (download) / 25 (scan) — Data.hresult(hex)+hresultSymbol(decoded)+wuPhase+updateTitle+updateGuid
public const string WindowsUpdateStarted = "windows_update_started"; // WU Client EventID 44 (download started) / 41 (downloaded) / 43 (install started) / 26 (scan found N) — Data.wuPhase
public const string WindowsUpdateServicing = "windows_update_servicing"; // CBS servicing step of an OS update (Setup log, Microsoft-Windows-Servicing 1/2/3/4/6) — Data.step+package+targetState+client
public const string WindowsUpdateActivitySummary = "windows_update_activity_summary"; // Counted non-OS update activity (Store/Defender/other) + suppressed repeats, emitted when the watchers stop
// OOBE update page measurement (D-310, ShellCoreTracker, Debug). The CloudExperienceHost
// update page (CXID OobeNDUP / RebootNDUP) and its ExpeditedUpdate_* event names, plus the
// registry state behind it (NDUP keys and both ESP "Install Windows quality updates"
// policy paths). Bounded per agent run.
public const string OobeUpdatePage = "oobe_update_page"; // Shell-Core 62404/62405 of an NDUP page, 62406/62407 update-page names — Data.cxhEvent+page+result+name(+value: boolean/integer/HRESULT only)
public const string OobeUpdateState = "oobe_update_state"; // NDUP + ESP-policy registry values at agent start/stop and at the update page — Data.moment + one entry per key alias; only when values exist and changed
// Corroboration snapshots (gather rules, config-delivered). Secondary evidence that an
// update landed during enrollment even when the watcher's backfill window missed it.
public const string WindowsUpdateRebootPending = "windows_update_reboot_pending"; // CBS / WU Auto-Update pending-reboot regkey snapshot
@@ -485,15 +485,44 @@ namespace AutopilotMonitor.Shared.Models
public bool WindowsUpdateWatcherEnabled { get; set; } = true;
/// <summary>
/// WindowsUpdateClient/Operational EventIDs to capture. Kept configurable so additional
/// IDs can be enabled after real-device validation WITHOUT an agent redeploy. Verified set:
/// 19=install success, 20=install failure (carries HRESULT), 43=install started, 44=download
/// started. IDs whose semantics are still unverified (25/26/31, restart-required 21 vs 41)
/// are intentionally excluded until confirmed on a device trace.
/// WindowsUpdateClient EventIDs the install watcher captures: 19=install success,
/// 20=install failure (carries HRESULT), 43=install started, 44=download started. The
/// provider manifest writes these to the System log, so the agent routes every ID to the
/// channel the manifest assigns it (System: 16–24, 27, 28, 32, 33, 43, 44, 212–218;
/// Operational: all others).
/// Default: [19, 20, 43, 44].
/// </summary>
public int[] WindowsUpdateTargetedEventIds { get; set; } = new[] { 19, 20, 43, 44 };
/// <summary>
/// WindowsUpdateClient/Operational EventIDs the scan/download watcher captures: 25=scan
/// failed, 26=scan found N updates, 31=download failed, 41=update downloaded. Separate from
/// <see cref="WindowsUpdateTargetedEventIds"/> on purpose: agents that predate the update
/// classification ignore this field, so they never receive the high-volume Store/Defender
/// download events unthrottled.
/// Default: [25, 26, 31, 41].
/// </summary>
public int[] WindowsUpdateOperationalEventIds { get; set; } = new[] { 25, 26, 31, 41 };
/// <summary>
/// Master switch for the servicing watcher on the Setup log (Microsoft-Windows-Servicing
/// EventIDs 1/2/3/4/6): the CBS package steps of an OS update — staged, installed,
/// reboot required, failed. Default: true.
/// </summary>
public bool ServicingWatcherEnabled { get; set; } = true;
/// <summary>
/// Mode of the OOBE update phase: "Off", "Shadow" or "Active".
/// Off: Windows/.NET update activity is only shown on the timeline.
/// Shadow: the update watchers also feed it into the decision engine, which records the
/// phase and reports at the post-ESP-exit completion window what the phase would decide;
/// today's rule still decides.
/// Active: the phase holds that window while Windows installs an update in OOBE and waits
/// a bounded time for the sign-in after the update restart instead of failing the session.
/// Any other value counts as Shadow. Default: "Shadow".
/// </summary>
public string OobeUpdatePhaseMode { get; set; } = "Shadow";
/// <summary>
/// Lookback window in minutes for the Windows Update backfill scan on startup. OOBE quality
/// updates run at the early "Getting updates" screen — often BEFORE the agent starts — but
@@ -44,6 +44,13 @@ namespace AutopilotMonitor.Shared.Models
// ── Origins whose detail is a classifier rule id (see <see cref="ClassifierRules"/>) ──
public const string OriginMaxLifetime = "maxlife";
/// <summary>
/// The agent's decision engine stopped waiting for the user — a bounded wait expired
/// (D-310: no sign-in within the window after the OOBE update) — and the session was
/// classified like a max-lifetime shutdown. A deliberate decision, not a silent agent:
/// counted as a classifier verdict, never in the silence share.
/// </summary>
public const string OriginStoppedWaiting = "stopwait";
public const string OriginLateReconcile = "late";
public const string OriginSweep = "sweep";
public const string OriginRetro = "retro";
@@ -75,7 +82,8 @@ namespace AutopilotMonitor.Shared.Models
{
if (string.IsNullOrEmpty(path)) return false;
var origin = Origin(path);
if (origin != OriginSweep && origin != OriginMaxLifetime && origin != OriginLateReconcile && origin != OriginRetro)
if (origin != OriginSweep && origin != OriginMaxLifetime && origin != OriginStoppedWaiting
&& origin != OriginLateReconcile && origin != OriginRetro)
return false;
var detail = path.Length > origin.Length + 1 ? path.Substring(origin.Length + 1) : string.Empty;
return detail.Length >= 2 && detail[0] == 'r' && char.IsDigit(detail[1]);
@@ -50,7 +50,7 @@ describe("groupPathsByOrigin", () => {
});
it("has a label for every origin in the order list", () => {
for (const origin of ["agent", "ingest", "sweep", "maxlife", "late", "retro", "register", "rule", "manual", "legacy"]) {
for (const origin of ["agent", "ingest", "sweep", "maxlife", "stopwait", "late", "retro", "register", "rule", "manual", "legacy"]) {
expect(VERDICT_PATH_ORIGIN_LABELS[origin]).toBeTruthy();
}
});
@@ -108,6 +108,7 @@ export const VERDICT_PATH_ORIGIN_LABELS: Record<string, string> = {
ingest: "Ingest mapping",
sweep: "Maintenance sweep (silence classifier)",
maxlife: "Agent max-lifetime shutdown (silence classifier)",
stopwait: "Agent stopped waiting for the user (silence classifier)",
late: "Late telemetry reconcile (silence classifier)",
retro: "Retro reclassification",
register: "Session registration",
@@ -325,6 +325,9 @@ export const COLLECTOR_FIELDS: Record<keyof CollectorConfiguration, RuntimeRow |
windowsUpdateTargetedEventIds: internal("Windows Update Targeted Event IDs"),
windowsUpdateBackfillLookbackMinutes: internal("Windows Update Backfill Lookback (min)"),
windowsUpdateChannelCensusEnabled: internal("Windows Update Channel Census"),
windowsUpdateOperationalEventIds: internal("Windows Update Operational Event IDs"),
servicingWatcherEnabled: internal("Servicing Watcher"),
oobeUpdatePhaseMode: internal("OOBE Update Phase Mode"),
mdmRebootPolicyWatcherEnabled: internal("MDM Reboot Policy Watcher"),
mdmRebootPolicyBackfillLookbackMinutes: internal("MDM Reboot Policy Backfill Lookback (min)"),
enableSystemTimelineWatcher: internal("System Timeline Watcher"),
@@ -38,6 +38,10 @@ export const KNOWN_EVENT_TYPES: EventTypeEntry[] = [
description: "Throttled state-machine snapshot during completion evaluation." },
{ value: "completion_waiting", label: "completion_waiting", category: "enrollment",
description: "Engine blocked or deferred a completion attempt; missingPrerequisites lists what it is still waiting on (deduped, emitted only when the set changes)." },
{ value: "completion_wait_expired", label: "completion_wait_expired", category: "enrollment",
description: "Engine stopped waiting for the user without a verdict, e.g. nobody signed in within 60 minutes after the Windows update in OOBE (reason). The agent then ends the session; it is classified as awaiting user or incomplete, never failed." },
{ value: "oobe_update_phase_shadow", label: "oobe_update_phase_shadow", category: "enrollment",
description: "Measurement of the OOBE update phase: at the completion window after the ESP page exit, what the phase would decide while Windows installs its OOBE quality update (phaseState, wouldDecide, phaseBoundUtc), reported right before today's verdict (decidedReason). Debug, at most once per session." },
{ value: "agent_started", label: "agent_started", category: "enrollment",
description: "Agent process started." },
{ value: "agent_shutdown", label: "agent_shutdown (V1)", category: "enrollment",
@@ -73,11 +77,19 @@ export const KNOWN_EVENT_TYPES: EventTypeEntry[] = [
// -------- Windows Update during OOBE --------
{ value: "windows_update_failed", label: "windows_update_failed", category: "esp",
description: "A Windows quality/cumulative update FAILED to install during enrollment (WindowsUpdateClient EventID 20). Carries the decoded HRESULT (hresult + hresultSymbol), updateTitle and updateGuid. A cumulative update failing mid-OOBE is a known enrollment-breaker otherwise invisible in the Intune console." },
description: "A Windows/.NET update step FAILED during enrollment: install (WindowsUpdateClient EventID 20), download (31) or the update scan (25) — see wuPhase. Carries the decoded HRESULT (hresult + hresultSymbol), updateTitle and updateGuid. A cumulative update failing mid-OOBE is a known enrollment-breaker otherwise invisible in the Intune console." },
{ value: "windows_update_succeeded", label: "windows_update_succeeded", category: "esp",
description: "A Windows quality/cumulative update installed during enrollment (WindowsUpdateClient EventID 19). Carries updateTitle and updateGuid." },
description: "A Windows Update installed during enrollment (WindowsUpdateClient EventID 19, System log). Carries updateTitle, updateGuid and updateClass." },
{ value: "windows_update_started", label: "windows_update_started", category: "esp",
description: "Windows Update download/install started during enrollment (WindowsUpdateClient EventID 43/44). Debug context for the timeline." },
description: "Windows Update activity during enrollment: download started (EventID 44), downloaded (41), install started (43) or scan found N updates (26) — see wuPhase. Only Windows/.NET updates (updateClass os) are listed; Store/Defender/other updates are counted in windows_update_activity_summary." },
{ value: "windows_update_servicing", label: "windows_update_servicing", category: "esp",
description: "A servicing step of a Windows update from the Setup log (Microsoft-Windows-Servicing): initiating, state_reached, reboot_required or failed — see step. Carries package (KB), initialState, targetState and client (UpdateAgentLCU for the cumulative update). Shows the long installation part of an OOBE quality update up to the restart." },
{ value: "windows_update_activity_summary", label: "windows_update_activity_summary", category: "esp",
description: "Counts of update activity that was counted instead of listed, emitted when the update watchers stop: Store, Defender and other non-OS update events per class and EventID, repeated scan results, servicing steps of non-OS packages." },
{ value: "oobe_update_page", label: "oobe_update_page", category: "esp",
description: "The Windows update page at the end of OOBE (CloudExperienceHost OobeNDUP / RebootNDUP): page started and stopped with its result, and the page's ExpeditedUpdate_* event names — a value only when it is a boolean, an integer or an HRESULT. Debug, bounded per agent run." },
{ value: "oobe_update_state", label: "oobe_update_state", category: "esp",
description: "Registry state behind the OOBE quality update at agent start and stop and at the update page: the NDUP keys and both policy paths of the ESP setting \"Install Windows quality updates\", one entry per key with values. Only when a key holds values and the state changed; Debug." },
{ value: "windows_update_reboot_pending", label: "windows_update_reboot_pending", category: "esp",
description: "Startup snapshot of the CBS RebootPending registry key. Present at agent start = a high-confidence corroborator that a (often pre-agent, OOBE-time) update landed and needs a reboot, which can delay/block completion. Gather rule GATHER-DEVICE-005 (on by default)." },
{ value: "windows_update_history", label: "windows_update_history", category: "esp",
@@ -2,7 +2,7 @@ import { LandingNavbar } from "../../components/landing/LandingNavbar";
import { SiteFooter } from "../../components/SiteFooter";
import { DOCS_URL } from "@/utils/config";
const LAST_UPDATED = "25 September 2026";
const LAST_UPDATED = "1 October 2026";
const DOCS_SECURITY_FAQ = `${DOCS_URL}/trust/security-faq`;
const DOCS_DATA_FLOWS = `${DOCS_URL}/trust/data-flows`;
const DOCS_DPA = `${DOCS_URL}/legal/data-privacy-agreement-dpa`;
@@ -70,7 +70,7 @@ export default function PrivacyPage() {
<h3 className="text-lg font-medium text-gray-800 mt-4">From enrolling devices</h3>
<ul className="list-disc list-inside space-y-2 text-gray-700 ml-4">
<li><strong>Device identity</strong> — serial number, device name, manufacturer, model, and the Entra ID tenant the device enrolls into</li>
<li><strong>Enrollment progress</strong> — phases, ESP stages, application and script results, policy activity, reboots, timings, and failure codes</li>
<li><strong>Enrollment progress</strong> — phases, ESP stages, application and script results, policy activity, Windows updates installed during the enrollment, reboots, timings, and failure codes</li>
<li><strong>Device context</strong> — OS build, hardware characteristics, disk and network state, plus whatever your own gather rules request</li>
<li><strong>Approximate location</strong> — country, region, city, and approximate coordinates, if geolocation is enabled for your tenant</li>
</ul>
@@ -424,6 +424,14 @@ export const SHARED_MANIFEST = {
43,
44
],
"windowsUpdateOperationalEventIds": [
25,
26,
31,
41
],
"servicingWatcherEnabled": true,
"oobeUpdatePhaseMode": "Shadow",
"windowsUpdateBackfillLookbackMinutes": 60,
"windowsUpdateChannelCensusEnabled": true,
"mdmRebootPolicyWatcherEnabled": true,
@@ -761,6 +769,7 @@ export const SHARED_MANIFEST = {
"cert_validation",
"collector_degraded",
"completion_check",
"completion_wait_expired",
"completion_waiting",
"configmgr_client_detected",
"console_prefetch_detected",
@@ -856,6 +865,9 @@ export const SHARED_MANIFEST = {
"office_preinstalled_detected",
"oobe_console_spawned",
"oobe_state_completed",
"oobe_update_page",
"oobe_update_phase_shadow",
"oobe_update_state",
"os_build_changed",
"os_info",
"outbound_ip",
@@ -916,10 +928,12 @@ export const SHARED_MANIFEST = {
"whiteglove_resumed",
"whiteglove_started",
"wifi_signal_info",
"windows_update_activity_summary",
"windows_update_channel_census",
"windows_update_failed",
"windows_update_history",
"windows_update_reboot_pending",
"windows_update_servicing",
"windows_update_started",
"windows_update_succeeded"
],
@@ -420,6 +420,14 @@
43,
44
],
"windowsUpdateOperationalEventIds": [
25,
26,
31,
41
],
"servicingWatcherEnabled": true,
"oobeUpdatePhaseMode": "Shadow",
"windowsUpdateBackfillLookbackMinutes": 60,
"windowsUpdateChannelCensusEnabled": true,
"mdmRebootPolicyWatcherEnabled": true,
@@ -757,6 +765,7 @@
"cert_validation",
"collector_degraded",
"completion_check",
"completion_wait_expired",
"completion_waiting",
"configmgr_client_detected",
"console_prefetch_detected",
@@ -852,6 +861,9 @@
"office_preinstalled_detected",
"oobe_console_spawned",
"oobe_state_completed",
"oobe_update_page",
"oobe_update_phase_shadow",
"oobe_update_state",
"os_build_changed",
"os_info",
"outbound_ip",
@@ -912,10 +924,12 @@
"whiteglove_resumed",
"whiteglove_started",
"wifi_signal_info",
"windows_update_activity_summary",
"windows_update_channel_census",
"windows_update_failed",
"windows_update_history",
"windows_update_reboot_pending",
"windows_update_servicing",
"windows_update_started",
"windows_update_succeeded"
],
@@ -5644,7 +5658,34 @@
"name": "number"
}
},
"doc": "WindowsUpdateClient/Operational EventIDs to capture. Kept configurable so additional IDs can be enabled after real-device validation WITHOUT an agent redeploy. Verified set: 19=install success, 20=install failure (carries HRESULT), 43=install started, 44=download started. IDs whose semantics are still unverified (25/26/31, restart-required 21 vs 41) are intentionally excluded until confirmed on a device trace. Default: [19, 20, 43, 44]."
"doc": "WindowsUpdateClient EventIDs the install watcher captures: 19=install success, 20=install failure (carries HRESULT), 43=install started, 44=download started. The provider manifest writes these to the System log, so the agent routes every ID to the channel the manifest assigns it (System: 16–24, 27, 28, 32, 33, 43, 44, 212–218; Operational: all others). Default: [19, 20, 43, 44]."
},
{
"name": "windowsUpdateOperationalEventIds",
"type": {
"kind": "array",
"element": {
"kind": "primitive",
"name": "number"
}
},
"doc": "WindowsUpdateClient/Operational EventIDs the scan/download watcher captures: 25=scan failed, 26=scan found N updates, 31=download failed, 41=update downloaded. Separate from WindowsUpdateTargetedEventIds on purpose: agents that predate the update classification ignore this field, so they never receive the high-volume Store/Defender download events unthrottled. Default: [25, 26, 31, 41]."
},
{
"name": "servicingWatcherEnabled",
"type": {
"kind": "primitive",
"name": "boolean"
},
"doc": "Master switch for the servicing watcher on the Setup log (Microsoft-Windows-Servicing EventIDs 1/2/3/4/6): the CBS package steps of an OS update — staged, installed, reboot required, failed. Default: true."
},
{
"name": "oobeUpdatePhaseMode",
"type": {
"kind": "primitive",
"name": "string"
},
"doc": "Mode of the OOBE update phase: \"Off\", \"Shadow\" or \"Active\". Off: Windows/.NET update activity is only shown on the timeline. Shadow: the update watchers also feed it into the decision engine, which records the phase and reports at the post-ESP-exit completion window what the phase would decide; today's rule still decides. Active: the phase holds that window while Windows installs an update in OOBE and waits a bounded time for the sign-in after the update restart instead of failing the session. Any other value counts as Shadow. Default: \"Shadow\"."
},
{
"name": "windowsUpdateBackfillLookbackMinutes",
@@ -1079,8 +1079,14 @@ export interface CollectorConfiguration {
modernDeploymentBackfillLookbackMinutes: number;
/** Master switch for the Windows Update watcher. When enabled the agent subscribes to Microsoft-Windows-WindowsUpdateClient/Operational and forwards update install/failure events (windows_update_succeeded / _failed / _started) so quality and cumulative updates that run DURING enrollment become visible with evidence. Default: true. */
windowsUpdateWatcherEnabled: boolean;
/** WindowsUpdateClient/Operational EventIDs to capture. Kept configurable so additional IDs can be enabled after real-device validation WITHOUT an agent redeploy. Verified set: 19=install success, 20=install failure (carries HRESULT), 43=install started, 44=download started. IDs whose semantics are still unverified (25/26/31, restart-required 21 vs 41) are intentionally excluded until confirmed on a device trace. Default: [19, 20, 43, 44]. */
/** WindowsUpdateClient EventIDs the install watcher captures: 19=install success, 20=install failure (carries HRESULT), 43=install started, 44=download started. The provider manifest writes these to the System log, so the agent routes every ID to the channel the manifest assigns it (System: 16–24, 27, 28, 32, 33, 43, 44, 212–218; Operational: all others). Default: [19, 20, 43, 44]. */
windowsUpdateTargetedEventIds: number[];
/** WindowsUpdateClient/Operational EventIDs the scan/download watcher captures: 25=scan failed, 26=scan found N updates, 31=download failed, 41=update downloaded. Separate from WindowsUpdateTargetedEventIds on purpose: agents that predate the update classification ignore this field, so they never receive the high-volume Store/Defender download events unthrottled. Default: [25, 26, 31, 41]. */
windowsUpdateOperationalEventIds: number[];
/** Master switch for the servicing watcher on the Setup log (Microsoft-Windows-Servicing EventIDs 1/2/3/4/6): the CBS package steps of an OS update — staged, installed, reboot required, failed. Default: true. */
servicingWatcherEnabled: boolean;
/** Mode of the OOBE update phase: "Off", "Shadow" or "Active". Off: Windows/.NET update activity is only shown on the timeline. Shadow: the update watchers also feed it into the decision engine, which records the phase and reports at the post-ESP-exit completion window what the phase would decide; today's rule still decides. Active: the phase holds that window while Windows installs an update in OOBE and waits a bounded time for the sign-in after the update restart instead of failing the session. Any other value counts as Shadow. Default: "Shadow". */
oobeUpdatePhaseMode: string;
/** Lookback window in minutes for the Windows Update backfill scan on startup. OOBE quality updates run at the early "Getting updates" screen — often BEFORE the agent starts — but the WindowsUpdateClient/Operational .evtx persists across the mid-OOBE reboot, so a generous backfill catches them. Set higher than the ModernDeployment backfill because the update scan/download can begin well before MDM enrollment. 0 = backfill disabled. Default: 60 minutes. */
windowsUpdateBackfillLookbackMinutes: number;
/** Self-evidencing blind-spot census (session 7443317c: an OOBE quality update installed — OS build changed across the reboot — yet the WindowsUpdateClient channel carried NONE of the targeted EventIDs). When enabled and the OS build changed across an agent restart while zero targeted WU events were captured, the tracker emits a one-shot windows_update_channel_census event with an unfiltered EventID histogram of the WindowsUpdateClient AND UpdateOrchestrator channels over the backfill lookback window — field evidence for which EventIDs/channel to add to WindowsUpdateTargetedEventIds via remote config, without diagnostics upload access. Default: true. */
+15
View File
@@ -0,0 +1,15 @@
# OsUpdateActivity v1 — OOBE update phase (D-310, report 3a2207978e4c, 2026-10-01).
# Posted by WindowsUpdateWatcherHost (CollectorConfiguration.OobeUpdatePhaseMode Shadow or Active)
# for every step of a Windows/.NET update: the Windows Update client on the System/Operational
# logs (source "wu": download_started, downloaded, install_started, installed, failed) and the
# CBS servicing steps on the Setup log (source "servicing": staging_started, staged,
# install_started, reboot_required, installed, failed). Store, Defender and other updates never
# post it. Payload: step (mandatory), source, update (title or KB).
# Reducer: HandleOsUpdateActivityV1 → records DecisionState.OsUpdateFacts (event-time ordered);
# the earliest activity after the ESP exit while no real user signed in anchors the OOBE update
# phase (PostExitActivityUtc). The phase holds the AdvisoryCompletion deadline within its own
# bounds: 3 h while the update runs (then Failed oobe_update_not_finished), 60 min for the
# sign-in after it ended (then the engine stops waiting — StoppedWaitingReason, no verdict).
# No stage change, no deadline. A signal without a step is a dead end.
{"SessionSignalOrdinal":1,"SessionTraceOrdinal":1,"Kind":"OsUpdateActivity","KindSchemaVersion":1,"OccurredAtUtc":"2026-10-01T11:41:12.0000000Z","SourceOrigin":"WindowsUpdateWatcher","Evidence":{"Kind":"Raw","Identifier":"os-update-activity:wu","Summary":"install_started: 2026-09 Cumulative Update for Windows 11, version 25H2 for x64-based Systems (KB5099999)"},"Payload":{"step":"install_started","source":"wu","update":"2026-09 Cumulative Update for Windows 11, version 25H2 for x64-based Systems (KB5099999)"}}
{"SessionSignalOrdinal":2,"SessionTraceOrdinal":2,"Kind":"OsUpdateActivity","KindSchemaVersion":1,"OccurredAtUtc":"2026-10-01T12:06:40.0000000Z","SourceOrigin":"ServicingWatcher","Evidence":{"Kind":"Raw","Identifier":"os-update-activity:servicing","Summary":"reboot_required: KB5099999"},"Payload":{"step":"reboot_required","source":"servicing","update":"KB5099999"}}