Compare commits

...

4 Commits

Author SHA1 Message Date
Oliver Kieselbach 3224774752 fix(test): EDR-safe straggler replay — no more renamed system executable
The straggler integration test copied powershell.exe under a synthetic
agent name and launched it — textbook MITRE T1036.003 ("Rename Legitimate
Utilities"). Defender for Endpoint flagged the run ("System executable
renamed and launched", initiating process testhost.net48.exe) and ISOLATED
the dev machine.

Replaced with function shadowing: a harness script defines Get-Process /
Stop-Process stub functions (PowerShell resolves functions before cmdlets)
and dot-sources the cleanup script. The stubs report a live agent-named
straggler on every probe and record the kill, pinning the identical
contract — poll by NAME until the deadline, force-kill by NAME, then
delete — with zero process spawns, zero PE copies, nothing EDR-visible.

The production routine never renames or copies executables; the flagged
technique existed only in this test. Lesson recorded in tasks/lessons.md.

22/22 cleanup tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 17:24:07 +02:00
Oliver Kieselbach e7ea431ccf feat(esp): recovery story for terminally reported ESP failures (session 4910a5a5)
DeviceSetup/Apps failed, the user pressed "Try again" ~23 min later, apps
recovered to 10/10 and DeviceSetup resolved - yet the advisory_completion
window un-defanged the stale failure and failed the live enrollment,
because the advisory variant was exempt from all re-arm/rebase guards.

Engine fix (DecisionCore):
- New facts EspAdvisoryFailureResolvedUtc + EspAdvisoryFailureCategory
  (state-borne; deadline payload would be lost on re-arm/rebase).
- TryResolveAdvisoryOnCategoryRecovery in both ProvisioningComplete
  handlers: category match sets the fact + emits
  esp_failure_advisory_resolved.
- Guards now (!hasAdvisoryAnchor || advisoryResolved): enforcement-progress
  re-arm, hello-never-observed promote and the reboot rebase apply again
  once the failure demonstrably un-happened; unresolved advisories keep
  the legacy un-defang exactly.
- Truthful terminal reason esp_recovered_without_completion_evidence
  (+ failureRecovered=true, LastFailureTrigger=DeadlineFired so the
  likely-stuck app promotion stays off).

Story events (agent tracker):
- esp_failure_retry_detected: failed subcategory leaves "failed" after the
  terminal fire - consistent with the user pressing "Try again".
- esp_failure_recovered: previously failed category reaches success.
- EspFailureDetectedEventArgs.LikelyCulpritApps (never-started first,
  cap 3) rides the EspTerminalFailure payload so esp_failure_advisory /
  enrollment_failed name the culprit app directly.

Surfacing: MCP resource-catalog, web timeline badges (Recovered /
User retry pills, culprit chip on advisory), gather-rule autocomplete,
ANALYZE-ESP-006 "ESP Failure Recovered After User Retry".

Tests: AdvisoryRecoveryTests (13, incl. the 527/611/635/671 field-ordinal
replay that no longer fails the session), EspFailureRecoveryStoryTests (5),
snapshot-builder guards 18 -> 20. All suites green (DecisionCore 493,
V2.Core 2018, Backend 3201, web vitest 589, MCP 449 incl. drift).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 17:03:44 +02:00
Oliver Kieselbach 9e24b92fbf feat(agent): ESP policy-provider stall detector + Autopilot registry guardrail
Customer field case: pre-provisioned devices with the ConfigMgr client as a
blocking app occasionally park the user ESP at "Apps (Identifying)" for days.
Root cause (verified against the EnrollmentStatusTracking CSP doc): only a
"ConfigMgr" provider is registered under Device\Setup\Apps\PolicyProviders
(no "Sidecar" = IME) and TrackingPoliciesCreated never becomes 1 - a wait the
CSP defines WITHOUT any timeout. Renaming the key to Sidecar unblocks
immediately. Microsoft documents that ESP providers are not aware of each
other and that pre-provisioning is unsupported with co-management.

New always-on esp_policy_provider_stalled tripwire, generalized to any
provider in both kinds (Setup/Apps TrackingPoliciesCreated != 1,
DevicePreparation InstallationState not in {2,3}):

- EspPolicyProviderProbe: fail-soft registry read of the
  EnrollmentStatusTracking mirror (device + user-SID scopes), defensive
  value parsing (missing/unparseable = incomplete), ScopedOverride test seam.
  Research and CSP contract quotes captured in the XML doc header.
- EspPolicyProviderStallDetector: pure EvaluateStallTransition on a 15-min
  continuous-incompleteness dwell (resets on completion/disappearance;
  absent key = normal early enrollment), one-shot per provider,
  StartupEventGate fingerprint over the sorted stalled key set for
  cross-restart dedup. Warning + ImmediateUpload via InformationalEventPost;
  purely observational, no decision-engine coupling, no config toggle,
  no ConfigVersion bump (disk_space_low precedent).
- EspPolicyProviderStallHost: 60-s wall-clock tick, deliberately outside the
  StallProbe idle gate (idle clock resets on session activity) and the
  idle-stopped periodic collectors.
- Registered in Constants.EventTypes, MCP resource-catalog (health_events),
  web gather-rule autocomplete (stall category).

Guardrail companion: SOFTWARE\Microsoft\Windows\Autopilot joins the
gather-rule registry allowlist ("Autopilot / OOBE / Setup") so tenants can
build custom rules on the subtree (e.g. not_exists on the Sidecar provider
key); generated web mirror regenerated via combine.js; positive + sibling-
spoof guard tests added.

Docs: OKF concept doc agent/esp-policy-provider-stall.md (+ index/log).

Tests: 22 new detector tests (pure transition matrix + emission surface incl.
the ConfigMgr-only customer scenario and gate-across-restart dedup); agent
suite 2013/2013, web vitest 589/589 + tsc clean, MCP 449 passed incl.
event-types-drift.

Both the detector and the embedded allowlist take effect with the next agent
release.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 15:20:47 +02:00
Oliver Kieselbach 45d7c03ccf fix(web): WhiteGlove per-block durations use sequence-canonical edges, not timestamp min/max
Customer report (session fcb44595): dashboard said 115 min, session detail
said 5h 2m for the same pre-provisioned enrollment. The detail's per-block
duration calc took min/max over ALL event timestamps inside each
sequence-block, which two real-world timestamp shapes break:

- an app_install_started re-emitted by the ImeLogTracker after the Part-2
  resume legitimately carries its historical Part-1 event time -> the User
  Enrollment block stretched back across the sealed pause (56m -> 3h18m)
- a do_telemetry with a +1h source timestamp (IME log line with a wrong
  timezone bias, just under the agent's 1h future-skew clamp) stretched
  Pre-Provisioning (58m -> 1h44m)

Sum: exactly the "5h 2m" the customer saw.

Block spans are now anchored on the first/last event BY SEQUENCE (sequence
is the canonical order), with Part 1 starting at session.startedAt for
backend DurationSeconds parity. Extracted as pure helper
computeWhiteGloveDurations with tests modeled on the real session; replay
against the customer's 880 events yields 1h 54m == backend 6872s.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 14:26:33 +02:00
34 changed files with 2720 additions and 111 deletions
+125
View File
@@ -0,0 +1,125 @@
---
type: Concept
title: ESP Policy-Provider Stall Detection — the no-timeout Setup/Apps wait and the co-management Sidecar gap
description: Why a registered EnrollmentStatusTracking CSP policy provider that never sets TrackingPoliciesCreated parks the user ESP at "Apps (Identifying)" indefinitely, the pre-provisioning + ConfigMgr field case, and how the always-on esp_policy_provider_stalled tripwire (15-min wall-clock dwell, StartupEventGate cross-restart dedup) surfaces it.
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/EspPolicyProviderProbe.cs
tags:
- agent
- esp
- enrollment-status-tracking
- co-management
- stall-detection
timestamp: 2026-07-23T00:00:00+02:00
---
# ESP Policy-Provider Stall Detection
## Schema
The Enrollment Status Page decides when its "Apps" category may progress from
"Identifying" using the **EnrollmentStatusTracking CSP**
(<https://learn.microsoft.com/en-us/windows/client-management/mdm/enrollmentstatustracking-csp>),
mirrored in the registry under
`HKLM\SOFTWARE\Microsoft\Windows\Autopilot\EnrollmentStatusTracking\{Device|<UserSID>}\…`.
Two contracts matter:
1. **`Setup/Apps/PolicyProviders/<Name>`** — per the CSP doc: *"Existence of this node
indicates to the ESP that it shouldn't show the tracking status message until the
TrackingPoliciesCreated node has been set to true."* This wait has **no timeout**. A
registered provider that never sets `TrackingPoliciesCreated=1` parks the ESP at
"Apps (Identifying)" indefinitely — observed for days in the field. Exists in device
scope (`Device`) and user scope (`S-<SID>`, appears at sign-in).
2. **`DevicePreparation/PolicyProviders/<Name>/InstallationState`** — must reach 2
(NotRequired) or 3 (Completed); 1=NotInstalled, 4=Error. Device scope only. The ESP
applies a default 15-minute timeout here, so these stalls self-surface as ESP errors —
the detector still records them for the diagnostic trail.
Known provider names: **`Sidecar` is the Intune Management Extension** (the CSP doc names
"Intune's agents, such as SideCar"); **`ConfigMgr`** is the Configuration Manager client,
registered during Autopilot-into-co-management — the troubleshooting section of
<https://learn.microsoft.com/en-us/intune/configmgr/comanage/autopilot-enrollment> queries
exactly these registry paths. The same document states that ESP policy providers **are not
aware of each other** and that **pre-provisioning is unsupported** with Autopilot into
co-management.
## The field case
A tenant pre-provisions devices and installs the ConfigMgr client as a blocking app.
Occasionally the user-phase ESP reaches "Apps (Identifying)" and never leaves it — for
hours to days. On those devices `Device\Setup\Apps\PolicyProviders` contains **only a
`ConfigMgr` key (no `Sidecar`)** and `TrackingPoliciesCreated` is never set. Manually
renaming the key to `Sidecar` unblocks the ESP immediately (the IME then finds its
provider node and reports tracking-policies-created). The mechanism is exactly the
documented no-timeout wait; the unsupported pre-provisioning + co-management combination
explains why only some devices are affected. Why the Sidecar registration is missing or
displaced on those devices is not publicly documented — irrelevant for detection, since
the signature is deterministic.
## Detection
`EspPolicyProviderProbe` (fail-soft, static, `ScopedOverride` test seam — structural clone
of `EspTrackingInfoProbe`) enumerates all providers in both kinds and scopes.
`EspPolicyProviderStallDetector` evaluates a pure transition function on a 60-second tick
from the always-on `EspPolicyProviderStallHost`:
* A provider **continuously incomplete for ≥ 15 minutes** (wall-clock) is stalled. The
dwell clock starts at first incomplete observation and resets on completion or
disappearance. Missing values parse as incomplete (fail-toward-observation).
* Missing root key / empty provider list = normal early enrollment: clears all dwell
state, never fires.
* One `esp_policy_provider_stalled` **Warning** per newly stalled provider set
(`ImmediateUpload`), one-shot per provider in-process, cross-restart deduped via a
`StartupEventGate` fingerprint over the sorted currently-stalled key set — a restart
into the same stall stays silent, a new provider joining the stall re-reports.
* Payload: `providers[]` (full table: name, scope, kind, raw values, complete flag),
`stalledProviders[]` with `stalledForMinutes`, `sidecarRegistered`, `dwellMinutes`.
* **Purely observational**: emitted via `InformationalEventPost` (dispatch-guard exempt
pass-through) — never touches decision state. No config toggle, no ConfigVersion bump
(precedent: `disk_space_low`).
Deliberately **not** built on `StallProbeCollector` (its idle clock resets on any session
activity — a provider stall coexists with a chatty session) and not on the periodic
collectors (idle-stopped after 15 minutes — exactly the dormant window the dwell must
observe).
## Guardrail companion
`rules/guardrails.json` allow-lists `SOFTWARE\Microsoft\Windows\Autopilot` (category
"Autopilot / OOBE / Setup") so tenants can build their own gather rules on the subtree —
e.g. a `not_exists` registry rule on the `Sidecar` provider key. The subtree carries
deployment state only (provider registrations, tracking lists, app IDs, per-user SIDs);
sensitivity is at or below existing entries. Enforcement is agent-side (embedded
resource), so both the detector and the guardrail change ship with the agent release.
## Examples
Customer-case event (abbreviated):
```json
{
"eventType": "esp_policy_provider_stalled",
"severity": "Warning",
"data": {
"dwellMinutes": 15,
"sidecarRegistered": false,
"providers": [
{ "name": "ConfigMgr", "scope": "device", "kind": "setupApps",
"trackingPoliciesCreated": null, "installationState": null, "complete": false }
],
"stalledProviders": [
{ "name": "ConfigMgr", "scope": "device", "kind": "setupApps", "stalledForMinutes": 15.0 }
]
}
}
```
`sidecarRegistered=false` while `setupApps` providers exist is the co-management
signature; `sidecarRegistered=true` with a stalled third-party provider generalizes the
same contract violation.
# Citations
* EnrollmentStatusTracking CSP — <https://learn.microsoft.com/en-us/windows/client-management/mdm/enrollmentstatustracking-csp>
* Windows Autopilot into co-management (requirements, provider-conflict warning, troubleshooting registry paths) — <https://learn.microsoft.com/en-us/intune/configmgr/comanage/autopilot-enrollment>
* Agent code: `EspPolicyProviderProbe.cs`, `EspPolicyProviderStallDetector.cs`, `EspPolicyProviderStallHost.cs` (SystemSignals / Orchestration.Hosts)
* Related: [MDM Reboot Coalescing](mdm-reboot-coalescing.md), [Gather-Rule Guardrails](../rules/gather-rule-guardrails.md)
+1
View File
@@ -22,6 +22,7 @@ https://docs.autopilotmonitor.com (separate repository).
* [Agent Endpoint Migration](agent/endpoint-migration.md) - How the backend re-homes agents to a new API base URL via the config channel (backend move or per-tenant region move): live-fetch-only, allowlist-validated on both sides, one hop, kill wins.
* [MDM Reboot Coalescing](agent/mdm-reboot-coalescing.md) - Attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: aggregated DM-Enterprise 2800 observations (neutral, flush-time watermark) + the ANALYZE-ESP-005 advisory gated on an actually-observed reboot.
* [Secure Boot CA-2023 Detection](agent/secureboot-ca2023-detection.md) - Why ANALYZE-SEC-001 v3 reads the UEFI db/KEK variables directly instead of trusting the SecureBoot\Servicing registry, and how the one-sided uefiCa2023FirmwareConfirmed marker plus a not_exists precondition give absence-tolerant suppression across old and new agents.
* [ESP Policy-Provider Stall Detection](agent/esp-policy-provider-stall.md) - The EnrollmentStatusTracking CSP's no-timeout Setup/Apps wait: a registered provider that never sets TrackingPoliciesCreated (co-management "ConfigMgr" without the IME's "Sidecar") parks the user ESP at "Apps (Identifying)" for days; the always-on esp_policy_provider_stalled tripwire and the SOFTWARE\Microsoft\Windows\Autopilot gather-rule allowlist entry that surface it.
# Rules
+2
View File
@@ -2,6 +2,8 @@
## 2026-07-23
* **Creation**: Added `agent/esp-policy-provider-stall.md` — a customer diagnosed that pre-provisioned devices with the ConfigMgr client as a blocking app occasionally park the user ESP at "Apps (Identifying)" for days: only a `ConfigMgr` provider is registered under `EnrollmentStatusTracking\Device\Setup\Apps\PolicyProviders` (no `Sidecar` = IME) and `TrackingPoliciesCreated` never becomes 1 — a wait the CSP defines without any timeout; renaming the key to `Sidecar` unblocks immediately. The agent now ships the always-on `esp_policy_provider_stalled` tripwire (60-s tick, 15-min continuous-incompleteness dwell, one-shot per provider, `StartupEventGate` fingerprint across restarts, purely observational via the InformationalEvent rail — no toggle, no ConfigVersion bump), generalized to any provider in both `Setup/Apps` and `DevicePreparation` kinds rather than hardcoding the ConfigMgr case. Companion change: `SOFTWARE\Microsoft\Windows\Autopilot` joins the gather-rule registry allowlist so tenants can target the subtree with custom rules (e.g. `not_exists` on the Sidecar key).
* **Creation**: Added `agent/secureboot-ca2023-detection.md` — ANALYZE-SEC-001 warned on devices whose firmware already carried the 2023 certificate, because the agent only read the `SecureBoot\Servicing\UEFICA2023Status` registry value, which mirrors the Windows Update rollout rather than the firmware (customer report). The agent now reads the UEFI `db`/`KEK` variables natively (`GetFirmwareEnvironmentVariableExW` with `SeSystemEnvironmentPrivilege` enabled on the process token — first privilege-adjustment code in the repo) and emits per-certificate booleans on `secureboot_status`. Suppression rides on a one-sided marker, `uefiCa2023FirmwareConfirmed`, emitted only on positive confirmation: the engine's `not_exists` precondition fails on any present value, so a two-sided boolean could not distinguish "confirmed" from "checked and missing", while the marker's absence keeps old agents and firmware-missing devices on the v2 path. Rule v3.0.0 is therefore safe to seed before the agent release.
* **Update**: `url-registry.md` — the first release after the alias migration surfaced Front Door caching as a correctness hazard: the alias served the previous `version.json` (`X-Cache: TCP_REMOTE_HIT`) after the origin had rotated, and since manifest and ZIP are independent cache entries, a stale manifest with a fresh ZIP fails the bootstrap SHA-256 check. Route caching on `/agent/*` is now disabled and every blob upload stamps `Cache-Control: no-cache` (CI `Send-Blob`, local `build.ps1`, `versions/*.json` manifests) — all agent-container blobs rotate in place, so nothing there is safely cacheable.
+90
View File
@@ -0,0 +1,90 @@
{
"$schema": "../schema/analyze-rule.schema.json",
"ruleId": "ANALYZE-ESP-006",
"title": "ESP Failure Recovered After User Retry",
"description": "Detects the recovery story: the ESP reported a terminal failure (typically DeviceSetup/Apps), the user most likely pressed 'Try again' on the failure page, the failed step re-ran and completed successfully. The earlier failure did not stick — the enrollment continued past it.",
"severity": "info",
"category": "esp",
"version": "1.0.0",
"enabled": true,
"baseConfidence": 85,
"conditions": [
{
"signal": "advisory",
"source": "event_type",
"eventType": "esp_failure_advisory",
"operator": "exists",
"value": "",
"required": true
},
{
"signal": "recovered",
"source": "event_type",
"eventType": "esp_failure_recovered",
"operator": "exists",
"value": "",
"required": true
},
{
"signal": "retry",
"source": "event_type",
"eventType": "esp_failure_retry_detected",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "category",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "category",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "failedSubcategory",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "failedSubcategory",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "likelyCulpritApps",
"source": "event_type",
"eventType": "esp_failure_advisory",
"dataField": "likelyCulpritApps",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "minutesSinceFailure",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "minutesSinceFailure",
"operator": "exists",
"value": "",
"required": false
}
],
"confidenceFactors": [
{ "signal": "retry", "condition": "exists", "weight": 10 },
{ "signal": "likelyCulpritApps", "condition": "exists", "weight": 5 }
],
"explanation": "The Enrollment Status Page reported a **failure** in the `{{category}}` phase (`{{failedSubcategory}}` subcategory, most likely app(s): `{{likelyCulpritApps}}`) — and then **recovered**: the failed step left its failed state and completed successfully about `{{minutesSinceFailure}}` min later.\n\nThat pattern is consistent with the user pressing **'Try again'** on the ESP failure page (the profile allows failure-page actions; the agent cannot observe the click itself). The failure the user saw was real, but it did not stick — the ESP re-ran the step, it succeeded, and the enrollment continued.\n\nRead the session timeline as one arc: `esp_failure_advisory` (the failure, with the likely culprit app) → `esp_failure_retry_detected` (the retry) → `esp_failure_recovered` (the recovery). If this pattern repeats across devices for the same app, that app's install is flaky on first attempt — fix the app rather than the devices. Any literal `{{…}}` placeholder means that detail was not recorded for this session.",
"remediation": [
{
"title": "Make the first install attempt stick",
"steps": [
"Check the likely culprit app(s) (`{{likelyCulpritApps}}`) in Intune: an app that fails on the first ESP pass but succeeds on retry usually has a dependency/ordering or detection-timing problem.",
"If the app shows 'never started installing' in esp_failure_settle_started, verify its assignment and applicability rules — ESP counted it as required, but IME never processed it.",
"If the pattern repeats across the tenant, consider removing the app from the ESP blocking list until the first-attempt install is reliable — users are seeing an avoidable failure screen.",
"No device-side action is needed for this session: the failure recovered and the enrollment continued."
]
}
],
"tags": ["esp", "recovery", "try-again", "retry", "continue-anyway", "enrollment"]
}
+104
View File
@@ -1884,6 +1884,110 @@
"user-experience"
]
},
{
"ruleId": "ANALYZE-ESP-006",
"title": "ESP Failure Recovered After User Retry",
"description": "Detects the recovery story: the ESP reported a terminal failure (typically DeviceSetup/Apps), the user most likely pressed 'Try again' on the failure page, the failed step re-ran and completed successfully. The earlier failure did not stick — the enrollment continued past it.",
"severity": "info",
"category": "esp",
"version": "1.0.0",
"enabled": true,
"baseConfidence": 85,
"conditions": [
{
"signal": "advisory",
"source": "event_type",
"eventType": "esp_failure_advisory",
"operator": "exists",
"value": "",
"required": true
},
{
"signal": "recovered",
"source": "event_type",
"eventType": "esp_failure_recovered",
"operator": "exists",
"value": "",
"required": true
},
{
"signal": "retry",
"source": "event_type",
"eventType": "esp_failure_retry_detected",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "category",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "category",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "failedSubcategory",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "failedSubcategory",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "likelyCulpritApps",
"source": "event_type",
"eventType": "esp_failure_advisory",
"dataField": "likelyCulpritApps",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "minutesSinceFailure",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "minutesSinceFailure",
"operator": "exists",
"value": "",
"required": false
}
],
"confidenceFactors": [
{
"signal": "retry",
"condition": "exists",
"weight": 10
},
{
"signal": "likelyCulpritApps",
"condition": "exists",
"weight": 5
}
],
"explanation": "The Enrollment Status Page reported a **failure** in the `{{category}}` phase (`{{failedSubcategory}}` subcategory, most likely app(s): `{{likelyCulpritApps}}`) — and then **recovered**: the failed step left its failed state and completed successfully about `{{minutesSinceFailure}}` min later.\n\nThat pattern is consistent with the user pressing **'Try again'** on the ESP failure page (the profile allows failure-page actions; the agent cannot observe the click itself). The failure the user saw was real, but it did not stick — the ESP re-ran the step, it succeeded, and the enrollment continued.\n\nRead the session timeline as one arc: `esp_failure_advisory` (the failure, with the likely culprit app) → `esp_failure_retry_detected` (the retry) → `esp_failure_recovered` (the recovery). If this pattern repeats across devices for the same app, that app's install is flaky on first attempt — fix the app rather than the devices. Any literal `{{…}}` placeholder means that detail was not recorded for this session.",
"remediation": [
{
"title": "Make the first install attempt stick",
"steps": [
"Check the likely culprit app(s) (`{{likelyCulpritApps}}`) in Intune: an app that fails on the first ESP pass but succeeds on retry usually has a dependency/ordering or detection-timing problem.",
"If the app shows 'never started installing' in esp_failure_settle_started, verify its assignment and applicability rules — ESP counted it as required, but IME never processed it.",
"If the pattern repeats across the tenant, consider removing the app from the ESP blocking list until the first-attempt install is reliable — users are seeing an avoidable failure screen.",
"No device-side action is needed for this session: the failure recovered and the enrollment continued."
]
}
],
"tags": [
"esp",
"recovery",
"try-again",
"retry",
"continue-anyway",
"enrollment"
]
},
{
"ruleId": "ANALYZE-ID-001",
"title": "Expected Machine Certificate Not Found",
+2 -1
View File
@@ -43,7 +43,8 @@
"prefixes": [
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Setup",
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\OOBE",
"SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Winlogon"
"SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Winlogon",
"SOFTWARE\\Microsoft\\Windows\\Autopilot"
]
},
{
@@ -290,9 +290,20 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
[InlineData(@"SAM\SAM\Domains\Account\Users")]
[InlineData(@"SECURITY\Policy\Secrets")]
[InlineData(@"HKLM\SOFTWARE\Microsoft\EnrollmentsSomethingElse")] // prefix spoofing
[InlineData(@"HKLM\SOFTWARE\Microsoft\Windows\AutopilotSomethingElse")] // sibling spoof of the Autopilot prefix
public void RegistryCollector_OffAllowlistPath_IsBlockedBeforeTheHiveIsOpened(string target)
=> AssertBlocked(new RegistryCollector().Execute(Rule("registry", target), Context()), target);
// Session 5d735290: the EnrollmentStatusTracking subtree (ESP policy providers / tracking
// state) is allow-listed so tenants can build gather rules on the co-management stall
// signature the esp_policy_provider_stalled detector reports (e.g. not_exists on the
// Sidecar provider key).
[Theory]
[InlineData(@"SOFTWARE\Microsoft\Windows\Autopilot")]
[InlineData(@"SOFTWARE\Microsoft\Windows\Autopilot\EnrollmentStatusTracking\Device\Setup\Apps\PolicyProviders\Sidecar")]
public void RegistryGuard_AutopilotEnrollmentTrackingSubtree_IsAllowed(string subPath)
=> Assert.True(GatherRuleGuards.IsRegistryPathAllowed(subPath));
[Theory]
[InlineData("SELECT * FROM Win32_UserAccount")]
[InlineData("SELECT * FROM Win32_ShadowCopy")]
@@ -15,7 +15,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
/// <item>a locked tree must abort WITHOUT deleting anything (old script force-deleted
/// marker + state around the locks and orphaned the binaries),</item>
/// <item>a straggler agent process (the incident's second instance) must be found by
/// NAME, killed, and the cleanup must then complete,</item>
/// NAME, killed, and the cleanup must then complete — simulated via function
/// shadowing, NEVER by launching a renamed executable (EDR isolation, see the test),</item>
/// <item>concurrent cleanup scripts must be serialized by the lock file.</item>
/// </list>
/// Timings are dialed down via the test-only BuildCleanupScript parameters so each case
@@ -111,60 +112,55 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
}
[Fact]
public void Straggler_process_holding_the_tree_is_killed_and_cleanup_completes()
public void Straggler_agent_process_is_waited_on_and_force_killed_via_name_backstop()
{
// Full incident replay: a SECOND agent-named process (here: a renamed
// powershell.exe, matching how only the process NAME identifies agents) holds a
// file in the tree open. The old script waited on the launcher PID only and ran
// its force-delete against the live locks. The hardened script must find the
// straggler by name, kill it after the wait deadline, and then delete everything.
var fakeExe = Path.Combine(_root, _fakeProcessName + ".exe");
File.Copy(SystemPaths.PowerShell, fakeExe);
// Incident replay of the second-instance mechanics WITHOUT spawning a process:
// an earlier version of this test copied powershell.exe under the fake agent name
// and launched it — textbook MITRE T1036.003 ("Rename Legitimate Utilities"),
// which Defender for Endpoint answered with a machine ISOLATION on the dev box
// (see tasks/lessons.md). Instead, a harness script shadows Get-Process /
// Stop-Process with functions (PowerShell resolves functions before cmdlets, so
// the dot-sourced cleanup script binds to the stubs): Get-Process reports a live
// agent-named straggler on every probe; Stop-Process records the kill. This pins
// the same contract — the script polls by NAME until the deadline, then
// force-kills by NAME before deleting — with zero EDR-visible behavior.
var probeLog = Path.Combine(_root, "probe.log");
var cleanupScriptPath = Path.Combine(_root, $"cleanup-{Guid.NewGuid():N}.ps1");
File.WriteAllText(cleanupScriptPath, BuildScript(processWaitSeconds: 2));
var lockTarget = Path.Combine(_basePath, "Agent", "fake.exe").Replace("'", "''");
Process straggler = null;
try
{
straggler = Process.Start(new ProcessStartInfo
{
FileName = fakeExe,
Arguments = "-NoProfile -Command \"$f = [IO.File]::Open('" + lockTarget +
"', 'Open', 'Read', 'None'); Start-Sleep -Seconds 120\"",
UseShellExecute = false,
CreateNoWindow = true,
});
var harness = @"
$probeLog = 'LOG_PATH'
$global:AmProbeCount = 0
function Get-Process {
[CmdletBinding()]
param([string[]]$Name)
if ($Name -contains 'FAKE_NAME') {
$global:AmProbeCount++
Add-Content -Path $probeLog -Value ('probe ' + $global:AmProbeCount)
return [pscustomobject]@{ Name = 'FAKE_NAME'; Id = 424242 }
}
Microsoft.PowerShell.Management\Get-Process @PSBoundParameters
}
function Stop-Process {
[CmdletBinding()]
param([Parameter(ValueFromPipeline = $true)]$InputObject, [switch]$Force)
process { Add-Content -Path $probeLog -Value ('kill ' + $InputObject.Id + ' force=' + $Force) }
}
. 'SCRIPT_PATH'
"
.Replace("LOG_PATH", probeLog)
.Replace("FAKE_NAME", _fakeProcessName)
.Replace("SCRIPT_PATH", cleanupScriptPath);
// Give the straggler time to acquire its file handle before cleanup starts.
var acquireDeadline = DateTime.UtcNow.AddSeconds(20);
var acquired = false;
while (!acquired && DateTime.UtcNow < acquireDeadline)
{
try
{
using (File.Open(Path.Combine(_basePath, "Agent", "fake.exe"),
FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
}
}
catch (IOException)
{
acquired = true; // share violation ⇒ straggler holds the file
}
if (!acquired) System.Threading.Thread.Sleep(200);
}
Assert.True(acquired, "straggler never acquired its file lock");
RunScript(harness);
RunScript(BuildScript(processWaitSeconds: 2));
Assert.True(straggler.HasExited, "straggler must be force-killed by the name-based backstop");
Assert.False(Directory.Exists(_basePath), "after the kill the tree must be fully deleted");
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
}
finally
{
try { if (straggler != null && !straggler.HasExited) straggler.Kill(); } catch { }
try { straggler?.Dispose(); } catch { }
}
var log = File.ReadAllLines(probeLog);
Assert.Contains("kill 424242 force=True", log);
Assert.True(Array.FindAll(log, l => l.StartsWith("probe")).Length >= 2,
"script must poll by name in the wait loop AND re-probe for the kill backstop");
Assert.False(Directory.Exists(_basePath), "after the kill the tree must be fully deleted");
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
Assert.False(File.Exists(LockPath));
}
[Fact]
@@ -0,0 +1,256 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
{
/// <summary>
/// Session 4910a5a5 (2026-07-23) — post-terminal recovery story. Field sequence:
/// DeviceSetup/Apps failed (culprit app never started installing) → settle window expired
/// → terminal EspFailureDetected → the user pressed "Try again" ~23 min later → Apps left
/// the failed state → 10/10 installed → DeviceSetup resolved. Pre-fix, that recovery only
/// appeared as a generic <c>subcategory_state_change</c>. Coverage:
/// <list type="bullet">
/// <item><c>esp_failure_retry_detected</c> fires once when the failed subcategory leaves
/// the failed state AFTER the terminal fire.</item>
/// <item><c>esp_failure_recovered</c> fires once when the category then resolves to
/// success.</item>
/// <item>An in-settle-window recovery keeps emitting only
/// <c>esp_failure_settle_recovered</c> — the post-terminal pair stays silent.</item>
/// <item>Apps failures carry <c>LikelyCulpritApps</c> on the EspFailureDetected args,
/// never-started apps ranked first.</item>
/// </list>
/// </summary>
public sealed class EspFailureRecoveryStoryTests
{
private static readonly DateTime Fixed = new DateTime(2026, 7, 23, 4, 30, 0, DateTimeKind.Utc);
private sealed class Fixture : IDisposable
{
public TempDirectory Tmp { get; } = new TempDirectory();
public FakeSignalIngressSink Sink { get; } = new FakeSignalIngressSink();
public List<EspFailureDetectedEventArgs> EspFailures { get; } = new List<EspFailureDetectedEventArgs>();
public ProvisioningStatusTracker Tracker { get; }
public Fixture(Func<IReadOnlyList<AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppPackageState>>? packageStatesProbe = null)
{
var clock = new VirtualClock(Fixed);
var post = new InformationalEventPost(Sink, clock);
var logger = new AgentLogger(Tmp.Path, AgentLogLevel.Info);
Tracker = new ProvisioningStatusTracker(
sessionId: "S1",
tenantId: "T1",
post: post,
logger: logger,
appxScanner: new FakeAppxDeploymentFailureScanner(),
backgroundDispatcher: action => action(),
packageStatesProbe: packageStatesProbe);
Tracker.EspFailureDetected += (_, args) => EspFailures.Add(args);
}
public IReadOnlyList<(string EventType, IReadOnlyDictionary<string, object> Data)> CapturedEvents()
{
return Sink.Posted
.Where(p => p.Kind == DecisionSignalKind.InformationalEvent && p.Payload != null)
.Select(p =>
{
var eventType = p.Payload!.TryGetValue(SignalPayloadKeys.EventType, out var et) ? et : "";
var data = p.TypedPayload as IReadOnlyDictionary<string, object>
?? (IReadOnlyDictionary<string, object>)new Dictionary<string, object>();
return (eventType, data);
})
.ToList();
}
/// <summary>Drive the session-4910a5a5 shape up to the terminal fire.</summary>
public void DriveToTerminalDeviceSetupAppsFailure()
{
Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Apps (9 of 10 installed)""}
}");
Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""failed"",""subcategoryStatusText"":""Apps (Error)""}
}");
Tracker.TriggerSettleTimerForTest("DeviceSetupCategory.Status");
}
public void Dispose()
{
Tracker.Dispose();
Tmp.Dispose();
}
}
[Fact]
public void PostTerminalRetry_EmitsRetryDetected_Once()
{
using var f = new Fixture();
f.DriveToTerminalDeviceSetupAppsFailure();
Assert.Single(f.EspFailures);
// "Try again": Apps leaves the failed state well after the settle window.
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Apps (Working on it...)""}
}");
var retry = Assert.Single(f.CapturedEvents(), e => e.EventType == "esp_failure_retry_detected");
Assert.Equal("DeviceSetup", (string)retry.Data["category"]);
Assert.Equal("Apps", (string)retry.Data["subcategory"]);
Assert.Equal("failed", (string)retry.Data["previousState"]);
Assert.Equal("inProgress", (string)retry.Data["newState"]);
Assert.Equal("Provisioning_DeviceSetup_Apps_Failed", (string)retry.Data["failureType"]);
Assert.True(retry.Data.ContainsKey("minutesSinceFailure"));
// A later unrelated transition does not re-emit.
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Apps (10 of 10 installed)""}
}");
Assert.Single(f.CapturedEvents(), e => e.EventType == "esp_failure_retry_detected");
}
[Fact]
public void PostTerminalRecovery_EmitsRecovered_OnCategorySuccess()
{
using var f = new Fixture();
f.DriveToTerminalDeviceSetupAppsFailure();
// Retry → all apps installed → Windows resolves the category boolean.
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Apps (Working on it...)""}
}");
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": true,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Apps (10 of 10 installed)""}
}");
var recovered = Assert.Single(f.CapturedEvents(), e => e.EventType == "esp_failure_recovered");
Assert.Equal("DeviceSetup", (string)recovered.Data["category"]);
Assert.Equal("Apps", (string)recovered.Data["failedSubcategory"]);
Assert.Equal("Provisioning_DeviceSetup_Apps_Failed", (string)recovered.Data["failureType"]);
Assert.True(recovered.Data.ContainsKey("minutesSinceFailure"));
// Duplicate category-success writes do not re-emit (fire-once).
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": true,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Apps (10 of 10 installed, take 2)""}
}");
Assert.Single(f.CapturedEvents(), e => e.EventType == "esp_failure_recovered");
}
[Fact]
public void InSettleWindowRecovery_KeepsPostTerminalPairSilent()
{
using var f = new Fixture();
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""AppsSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Apps (0 of 1 installed)""}
}");
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""AppsSubcategory"": {""subcategoryState"":""failed"",""subcategoryStatusText"":""Apps (Error)""}
}");
// Recovery INSIDE the settle window — terminal fire is suppressed.
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""AppsSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Apps (Working on it...)""}
}");
f.Tracker.TriggerSettleTimerForTest("DeviceSetupCategory.Status");
Assert.Empty(f.EspFailures);
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": true,
""AppsSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Apps (1 of 1 installed)""}
}");
var events = f.CapturedEvents();
Assert.Contains(events, e => e.EventType == "esp_failure_settle_recovered");
Assert.DoesNotContain(events, e => e.EventType == "esp_failure_retry_detected");
Assert.DoesNotContain(events, e => e.EventType == "esp_failure_recovered");
}
[Fact]
public void AppsFailure_TerminalArgs_CarryLikelyCulprits_NeverStartedFirst()
{
var neverStarted = BuildApp("app-bg", "Teams Backgrounds v2",
AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Unknown,
downloadingSeen: false);
var startedButPending = BuildApp("app-slow", "Slow Installer",
AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Unknown,
downloadingSeen: true);
var installed = BuildApp("app-ok", "Falcon Sensor",
AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Installed,
downloadingSeen: true);
using var f = new Fixture(() => new[] { startedButPending, neverStarted, installed });
f.DriveToTerminalDeviceSetupAppsFailure();
var fired = Assert.Single(f.EspFailures);
Assert.Equal(2, fired.LikelyCulpritApps.Count);
// Never-started app ranked first — it starved the ESP gate without any IME event.
Assert.Equal("Teams Backgrounds v2", fired.LikelyCulpritApps[0]);
Assert.Equal("Slow Installer", fired.LikelyCulpritApps[1]);
}
[Fact]
public void NonAppsFailure_TerminalArgs_CarryNoCulprits()
{
var neverStarted = BuildApp("app-bg", "Teams Backgrounds v2",
AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Unknown,
downloadingSeen: false);
using var f = new Fixture(() => new[] { neverStarted });
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Working""}
}");
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""failed"",""subcategoryStatusText"":""Failed""}
}");
f.Tracker.TriggerSettleTimerForTest("DeviceSetupCategory.Status");
var fired = Assert.Single(f.EspFailures);
Assert.Empty(fired.LikelyCulpritApps);
}
private static AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppPackageState BuildApp(
string id, string name,
AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState state,
bool downloadingSeen)
{
var pkg = new AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppPackageState(id, 0);
pkg.UpdateName(name);
pkg.UpdateIntent(AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppIntent.Install);
pkg.UpdateTargeted(AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppTargeted.Device);
// DownloadingOrInstallingSeen is derived inside UpdateState (Downloading..Installing
// range) — drive it through a Downloading transition instead of setting it directly.
if (downloadingSeen)
pkg.UpdateState(AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Downloading);
if (state != AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Unknown
&& state != AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Downloading)
pkg.UpdateState(state);
return pkg;
}
}
}
@@ -0,0 +1,359 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Persistence;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
{
/// <summary>
/// Session 5d735290 — pure detection logic of <see cref="EspPolicyProviderStallDetector"/>:
/// the CSP completion rule and the dwell/one-shot transition. No registry, no timers.
/// </summary>
public sealed class EspPolicyProviderStallLogicTests
{
private static readonly DateTime T0 = new DateTime(2026, 7, 23, 10, 0, 0, DateTimeKind.Utc);
private static readonly TimeSpan Dwell = TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes);
private static PolicyProviderState SetupApps(string name, int? tracking, string scope = "device") =>
new PolicyProviderState(name, scope, EspPolicyProviderProbe.KindSetupApps, tracking, null);
private static PolicyProviderState DevicePrep(string name, int? installState) =>
new PolicyProviderState(name, "device", EspPolicyProviderProbe.KindDevicePreparation, null, installState);
private static EspPolicyProviderSnapshot Snapshot(params PolicyProviderState[] providers) =>
new EspPolicyProviderSnapshot(providers);
// ------------------------------------------------- completion rule (CSP contract)
[Theory]
[InlineData(1, true)] // TrackingPoliciesCreated=true
[InlineData(0, false)] // explicit false (the CSP default)
[InlineData(null, false)] // value missing — the field case
[InlineData(2, false)] // anything else is not "true"
public void SetupApps_provider_complete_only_at_tracking_1(int? tracking, bool expected)
=> Assert.Equal(expected, EspPolicyProviderProbe.IsProviderComplete(
EspPolicyProviderProbe.KindSetupApps, tracking, null));
[Theory]
[InlineData(2, true)] // NotRequired
[InlineData(3, true)] // Completed
[InlineData(1, false)] // NotInstalled
[InlineData(4, false)] // Error
[InlineData(null, false)]
public void DevicePreparation_provider_complete_at_state_2_or_3(int? state, bool expected)
=> Assert.Equal(expected, EspPolicyProviderProbe.IsProviderComplete(
EspPolicyProviderProbe.KindDevicePreparation, null, state));
[Fact]
public void TryReadInt_tolerates_dword_string_and_bool_forms()
{
Assert.Equal(1, EspPolicyProviderProbe.TryReadInt(1));
Assert.Equal(3, EspPolicyProviderProbe.TryReadInt(3L));
Assert.Equal(1, EspPolicyProviderProbe.TryReadInt(" 1 "));
Assert.Equal(1, EspPolicyProviderProbe.TryReadInt(true));
Assert.Equal(0, EspPolicyProviderProbe.TryReadInt(false));
Assert.Null(EspPolicyProviderProbe.TryReadInt(null));
Assert.Null(EspPolicyProviderProbe.TryReadInt("not-a-number"));
Assert.Null(EspPolicyProviderProbe.TryReadInt(new byte[] { 1 }));
}
// ------------------------------------------------- dwell transition
[Fact]
public void Empty_or_missing_snapshot_clears_dwell_state_and_never_stalls()
{
var previous = new Dictionary<string, DateTime> { ["setupApps|device|ConfigMgr"] = T0 };
var missing = EspPolicyProviderStallDetector.EvaluateStallTransition(
EspPolicyProviderSnapshot.Empty, previous, new HashSet<string>(), T0.AddHours(2), Dwell);
Assert.Empty(missing.UpdatedFirstSeenUtc);
Assert.Empty(missing.NewlyStalled);
var registeredButEmpty = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(), previous, new HashSet<string>(), T0.AddHours(2), Dwell);
Assert.Empty(registeredButEmpty.UpdatedFirstSeenUtc);
Assert.Empty(registeredButEmpty.NewlyStalled);
}
[Fact]
public void Incomplete_provider_accrues_dwell_but_does_not_stall_before_threshold()
{
var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: null));
var round1 = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, new Dictionary<string, DateTime>(), new HashSet<string>(), T0, Dwell);
Assert.Equal(T0, round1.UpdatedFirstSeenUtc["setupApps|device|ConfigMgr"]);
Assert.Empty(round1.NewlyStalled);
var round2 = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, round1.UpdatedFirstSeenUtc, new HashSet<string>(), T0.AddMinutes(14), Dwell);
Assert.Equal(T0, round2.UpdatedFirstSeenUtc["setupApps|device|ConfigMgr"]); // first-seen preserved
Assert.Empty(round2.NewlyStalled);
}
[Fact]
public void Provider_stalls_at_dwell_threshold_with_correct_minutes()
{
var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: 0));
var seen = new Dictionary<string, DateTime> { ["setupApps|device|ConfigMgr"] = T0 };
var eval = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, seen, new HashSet<string>(), T0.AddMinutes(15), Dwell);
var stalled = Assert.Single(eval.NewlyStalled);
Assert.Equal("ConfigMgr", stalled.Provider.Name);
Assert.Equal(15.0, stalled.StalledForMinutes, precision: 3);
}
[Fact]
public void Completion_or_disappearance_resets_the_dwell_clock()
{
var key = "setupApps|device|ConfigMgr";
var seen = new Dictionary<string, DateTime> { [key] = T0 };
// Completes mid-dwell → dropped from the map.
var completed = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("ConfigMgr", tracking: 1)), seen, new HashSet<string>(), T0.AddMinutes(10), Dwell);
Assert.Empty(completed.UpdatedFirstSeenUtc);
// Re-appears incomplete much later → dwell restarts from the new now, no stall yet.
var reappeared = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("ConfigMgr", tracking: 0)), completed.UpdatedFirstSeenUtc,
new HashSet<string>(), T0.AddHours(3), Dwell);
Assert.Equal(T0.AddHours(3), reappeared.UpdatedFirstSeenUtc[key]);
Assert.Empty(reappeared.NewlyStalled);
// Disappearance behaves the same as completion.
var disappeared = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("Sidecar", tracking: 1)), new Dictionary<string, DateTime> { [key] = T0 },
new HashSet<string>(), T0.AddMinutes(20), Dwell);
Assert.False(disappeared.UpdatedFirstSeenUtc.ContainsKey(key));
}
[Fact]
public void Already_fired_provider_is_never_reported_again()
{
var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: null));
var seen = new Dictionary<string, DateTime> { ["setupApps|device|ConfigMgr"] = T0 };
var fired = new HashSet<string>(StringComparer.OrdinalIgnoreCase) { "setupApps|device|ConfigMgr" };
var eval = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, seen, fired, T0.AddHours(5), Dwell);
Assert.Empty(eval.NewlyStalled);
Assert.True(eval.UpdatedFirstSeenUtc.ContainsKey("setupApps|device|ConfigMgr")); // still tracked
}
[Fact]
public void Mixed_kinds_and_scopes_are_tracked_under_distinct_keys()
{
var snapshot = Snapshot(
SetupApps("ConfigMgr", tracking: null),
SetupApps("ConfigMgr", tracking: null, scope: "user:S-1-5-21-1111"),
DevicePrep("ConfigMgr", installState: 1));
var eval = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, new Dictionary<string, DateTime>(), new HashSet<string>(), T0, Dwell);
Assert.Equal(3, eval.UpdatedFirstSeenUtc.Count);
Assert.Contains("setupApps|device|ConfigMgr", eval.UpdatedFirstSeenUtc.Keys);
Assert.Contains("setupApps|user:S-1-5-21-1111|ConfigMgr", eval.UpdatedFirstSeenUtc.Keys);
Assert.Contains("devicePreparation|device|ConfigMgr", eval.UpdatedFirstSeenUtc.Keys);
}
[Fact]
public void SidecarRegistered_matches_setupApps_sidecar_only()
{
Assert.True(Snapshot(SetupApps("Sidecar", tracking: 0)).SidecarRegistered);
Assert.True(Snapshot(SetupApps("sidecar", tracking: 1, scope: "user:S-1-5-21-1")).SidecarRegistered);
Assert.False(Snapshot(SetupApps("ConfigMgr", tracking: 1)).SidecarRegistered);
Assert.False(Snapshot(DevicePrep("Sidecar", installState: 3)).SidecarRegistered); // wrong kind
Assert.False(EspPolicyProviderSnapshot.Empty.SidecarRegistered);
}
}
/// <summary>
/// Emission surface of the detector via the real <see cref="InformationalEventPost"/> rail.
/// Static probe override → serial collection (same contract as
/// <c>EspConfigDetectedTrackingFieldsTests</c>).
/// </summary>
[Collection("SerialThreading")]
public sealed class EspPolicyProviderStallDetectorEmissionTests : IDisposable
{
private static readonly DateTime At = new DateTime(2026, 7, 23, 12, 0, 0, DateTimeKind.Utc);
private readonly TempDirectory _tmp = new TempDirectory();
public void Dispose() => _tmp.Dispose();
private (EspPolicyProviderStallDetector sut, FakeSignalIngressSink sink, VirtualClock clock)
BuildDetector(StartupEventGate? gate = null)
{
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
var sink = new FakeSignalIngressSink();
var clock = new VirtualClock(At);
var post = new InformationalEventPost(sink, clock);
var sut = new EspPolicyProviderStallDetector("S1", "T1", post, logger, clock, gate);
return (sut, sink, clock);
}
private static EspPolicyProviderSnapshot ConfigMgrOnly() => new EspPolicyProviderSnapshot(new[]
{
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindSetupApps,
trackingPoliciesCreated: null, installationState: null),
});
private static EspPolicyProviderSnapshot SidecarCompleteConfigMgrIncomplete() => new EspPolicyProviderSnapshot(new[]
{
new PolicyProviderState("Sidecar", "device", EspPolicyProviderProbe.KindSetupApps, 1, null),
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindSetupApps, 0, null),
});
private static IReadOnlyList<FakeSignalIngressSink.PostedSignal> StalledEvents(FakeSignalIngressSink sink) =>
sink.Posted.Where(p =>
p.Payload != null
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
&& et == "esp_policy_provider_stalled").ToList();
[Fact]
public void CustomerScenario_configmgr_without_sidecar_emits_one_warning_after_dwell()
{
using var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => ConfigMgrOnly());
var (sut, sink, clock) = BuildDetector();
sut.Tick(); // first sighting — dwell starts
Assert.Empty(StalledEvents(sink));
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick(); // threshold crossed → emit
var evt = Assert.Single(StalledEvents(sink));
Assert.Equal(DecisionSignalKind.InformationalEvent, evt.Kind);
Assert.Equal("Warning", evt.Payload![SignalPayloadKeys.Severity]);
Assert.Equal("true", evt.Payload[SignalPayloadKeys.ImmediateUpload]);
Assert.Equal(EspPolicyProviderStallDetector.SourceLabel, evt.Payload[SignalPayloadKeys.Source]);
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(evt.TypedPayload);
Assert.Equal(false, data["sidecarRegistered"]);
Assert.Equal(EspPolicyProviderStallDetector.DwellMinutes, data["dwellMinutes"]);
var stalled = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object>>>(data["stalledProviders"]);
var entry = Assert.Single(stalled);
Assert.Equal("ConfigMgr", entry["name"]);
Assert.Equal(EspPolicyProviderProbe.KindSetupApps, entry["kind"]);
Assert.Equal(15.0, entry["stalledForMinutes"]);
}
[Fact]
public void Sidecar_complete_configmgr_incomplete_reports_only_configmgr_with_sidecar_flag()
{
using var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => SidecarCompleteConfigMgrIncomplete());
var (sut, sink, clock) = BuildDetector();
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
var evt = Assert.Single(StalledEvents(sink));
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(evt.TypedPayload);
Assert.Equal(true, data["sidecarRegistered"]);
var stalled = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object>>>(data["stalledProviders"]);
Assert.Equal("ConfigMgr", Assert.Single(stalled)["name"]);
var providers = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object?>>>(data["providers"]);
Assert.Equal(2, providers.Count); // full table always attached
}
[Fact]
public void Latch_prevents_second_emission_for_the_same_provider()
{
using var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => ConfigMgrOnly());
var (sut, sink, clock) = BuildDetector();
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(30));
sut.Tick();
sut.Tick();
Assert.Single(StalledEvents(sink));
}
[Fact]
public void Startup_gate_suppresses_same_stall_across_restart_but_reports_new_provider_set()
{
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
// Run 1: fires and persists the fingerprint.
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => ConfigMgrOnly()))
{
var (sut, sink, clock) = BuildDetector(new StartupEventGate(_tmp.Path, logger));
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
Assert.Single(StalledEvents(sink));
}
// Run 2 (simulated restart — fresh detector, gate reloaded from the same state dir):
// identical stall re-accrues dwell but lands on the same fingerprint → suppressed.
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => ConfigMgrOnly()))
{
var (sut, sink, clock) = BuildDetector(new StartupEventGate(_tmp.Path, logger));
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
Assert.Empty(StalledEvents(sink));
}
// Run 3: a SECOND provider joins the stall → fingerprint changes → reported.
var widened = new EspPolicyProviderSnapshot(new[]
{
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindSetupApps, null, null),
new PolicyProviderState("Contoso", "device", EspPolicyProviderProbe.KindSetupApps, 0, null),
});
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => widened))
{
var (sut, sink, clock) = BuildDetector(new StartupEventGate(_tmp.Path, logger));
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
Assert.Single(StalledEvents(sink));
}
}
[Fact]
public void Healthy_or_absent_tracking_never_emits()
{
var healthy = new EspPolicyProviderSnapshot(new[]
{
new PolicyProviderState("Sidecar", "device", EspPolicyProviderProbe.KindSetupApps, 1, null),
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindDevicePreparation, null, 3),
});
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => healthy))
{
var (sut, sink, clock) = BuildDetector();
sut.Tick();
clock.Advance(TimeSpan.FromHours(4));
sut.Tick();
Assert.Empty(StalledEvents(sink));
}
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => EspPolicyProviderSnapshot.Empty))
{
var (sut, sink, clock) = BuildDetector();
sut.Tick();
clock.Advance(TimeSpan.FromHours(4));
sut.Tick();
Assert.Empty(StalledEvents(sink));
}
}
}
}
@@ -1,4 +1,5 @@
using System;
using System.Collections.Generic;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
@@ -38,16 +39,30 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
/// </summary>
public string Category { get; }
/// <summary>
/// Session 4910a5a5: names of the tracked apps most likely behind an Apps-subcategory
/// failure (capped, never-started apps ranked first — those starve ESP's app gate
/// without ever producing an IME install event). Snapshot from
/// <c>ProvisioningStatusTracker.SnapshotTrackedAppsNotCompleted</c> at failure-arming
/// time; empty for non-Apps failures and event-log-derived failures. Propagated onto
/// the EspTerminalFailure signal payload so the DecisionEngine's
/// <c>esp_failure_advisory</c> / <c>enrollment_failed</c> events can name the culprit
/// directly.
/// </summary>
public IReadOnlyList<string> LikelyCulpritApps { get; }
public EspFailureDetectedEventArgs(
string failureType,
string errorCode = null,
string failedSubcategory = null,
string category = null)
string category = null,
IReadOnlyList<string> likelyCulpritApps = null)
{
FailureType = string.IsNullOrEmpty(failureType) ? "unknown" : failureType;
ErrorCode = string.IsNullOrEmpty(errorCode) ? null : errorCode;
FailedSubcategory = string.IsNullOrEmpty(failedSubcategory) ? null : failedSubcategory;
Category = string.IsNullOrEmpty(category) ? null : category;
LikelyCulpritApps = likelyCulpritApps ?? Array.Empty<string>();
}
}
}
@@ -0,0 +1,301 @@
using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using Microsoft.Win32;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
/// <summary>
/// Session 5d735290 (2026-07-23) — reads the ESP's registered policy providers from
/// <c>HKLM\SOFTWARE\Microsoft\Windows\Autopilot\EnrollmentStatusTracking</c>, the registry
/// mirror of the EnrollmentStatusTracking CSP
/// (https://learn.microsoft.com/en-us/windows/client-management/mdm/enrollmentstatustracking-csp).
/// <para>
/// CSP contract the ESP enforces (quotes from the doc above):
/// <list type="bullet">
/// <item><c>Setup/Apps/PolicyProviders/&lt;Name&gt;</c> — "Existence of this node indicates
/// to the ESP that it shouldn't show the tracking status message until the
/// TrackingPoliciesCreated node has been set to true." This wait has NO timeout: a
/// registered provider that never sets <c>TrackingPoliciesCreated=1</c> parks the ESP at
/// "Apps (Identifying)" indefinitely (observed for days in the field).</item>
/// <item><c>DevicePreparation/PolicyProviders/&lt;Name&gt;/InstallationState</c> — must reach
/// 2 (NotRequired) or 3 (Completed); 1=NotInstalled, 4=Error. ESP applies a default
/// 15-minute timeout here, so stalls in this stage are self-surfacing — still reported
/// for the diagnostic trail.</item>
/// </list>
/// Registry layout: <c>Device</c> subkey for device scope, <c>S-&lt;SID&gt;</c> subkeys for user
/// scope (appear at sign-in), each containing <c>Setup\Apps\PolicyProviders\&lt;Name&gt;</c>;
/// <c>DevicePreparation\PolicyProviders</c> exists in device scope only.
/// </para>
/// <para>
/// Known provider names: <c>Sidecar</c> is the Intune Management Extension ("Intune's agents,
/// such as SideCar" per the CSP doc); <c>ConfigMgr</c> is the Configuration Manager client
/// (registered during Autopilot-into-co-management, see the troubleshooting registry paths in
/// https://learn.microsoft.com/en-us/intune/configmgr/comanage/autopilot-enrollment). Microsoft
/// warns that ESP policy providers are not aware of each other and that pre-provisioning is
/// unsupported with co-management. Field case that motivated this probe: a pre-provisioned
/// device with the ConfigMgr client installed as a blocking app ended up with ONLY
/// <c>ConfigMgr</c> registered under <c>Device\Setup\Apps\PolicyProviders</c> (no
/// <c>Sidecar</c>) and <c>TrackingPoliciesCreated</c> never set — the user ESP hung at
/// "Apps (Identifying)" for hours to days; manually renaming the key to <c>Sidecar</c>
/// unblocked it immediately. <see cref="EspPolicyProviderStallDetector"/> turns this contract
/// into the <c>esp_policy_provider_stalled</c> tripwire.
/// </para>
/// <para>
/// Fail-soft like <see cref="EspTrackingInfoProbe"/>: any failure is Debug-logged and yields
/// <see cref="EspPolicyProviderSnapshot.Empty"/>; a missing root key (early enrollment,
/// non-Autopilot device) is not an error. Values are parsed defensively — a missing or
/// unparseable value counts as incomplete (fail-toward-observation).
/// </para>
/// </summary>
internal static class EspPolicyProviderProbe
{
internal const string TrackingRootKeyPath =
@"SOFTWARE\Microsoft\Windows\Autopilot\EnrollmentStatusTracking";
/// <summary>Provider kind: <c>Setup\Apps\PolicyProviders</c> (TrackingPoliciesCreated wait, no timeout).</summary>
internal const string KindSetupApps = "setupApps";
/// <summary>Provider kind: <c>DevicePreparation\PolicyProviders</c> (InstallationState wait, 15-min OS timeout).</summary>
internal const string KindDevicePreparation = "devicePreparation";
/// <summary>The IME's provider name — its absence while other providers are registered is the field-case signature.</summary>
internal const string SidecarProviderName = "Sidecar";
internal const string DeviceScope = "device";
private const string TrackingPoliciesCreatedValueName = "TrackingPoliciesCreated";
private const string InstallationStateValueName = "InstallationState";
/// <summary>
/// Test seam — when non-null, <see cref="Read"/> delegates to this func instead of
/// touching the live registry. Use <see cref="ScopedOverride"/> to guarantee cleanup.
/// </summary>
internal static Func<AgentLogger, EspPolicyProviderSnapshot> TestOverride;
/// <summary>
/// Reads all registered ESP policy providers. <see cref="EspPolicyProviderSnapshot.HasData"/>
/// is <c>true</c> when the EnrollmentStatusTracking root key existed and was readable
/// (even with no providers registered); <see cref="EspPolicyProviderSnapshot.Empty"/> otherwise.
/// </summary>
/// <param name="logger">Optional logger — Debug-level trace only; no warn/error.</param>
public static EspPolicyProviderSnapshot Read(AgentLogger logger = null)
{
var probe = TestOverride;
if (probe != null) return probe(logger);
try
{
using (var rootKey = Registry.LocalMachine.OpenSubKey(TrackingRootKeyPath))
{
if (rootKey == null)
{
logger?.Debug("EspPolicyProviderProbe: EnrollmentStatusTracking registry key not found");
return EspPolicyProviderSnapshot.Empty;
}
var providers = new List<PolicyProviderState>();
foreach (var scopeName in rootKey.GetSubKeyNames())
{
var isDevice = string.Equals(scopeName, "Device", StringComparison.OrdinalIgnoreCase);
// User scope repeats the Setup structure under the signed-in user's SID.
var isUserSid = scopeName.StartsWith("S-", StringComparison.OrdinalIgnoreCase);
if (!isDevice && !isUserSid) continue;
var scopeLabel = isDevice ? DeviceScope : "user:" + scopeName;
using (var scopeKey = rootKey.OpenSubKey(scopeName))
{
if (scopeKey == null) continue;
CollectProviders(
scopeKey, @"Setup\Apps\PolicyProviders", scopeLabel, KindSetupApps,
TrackingPoliciesCreatedValueName, providers, logger);
if (isDevice)
{
CollectProviders(
scopeKey, @"DevicePreparation\PolicyProviders", scopeLabel, KindDevicePreparation,
InstallationStateValueName, providers, logger);
}
}
}
return new EspPolicyProviderSnapshot(providers);
}
}
catch (Exception ex)
{
logger?.Debug($"EspPolicyProviderProbe: read threw: {ex.Message}");
return EspPolicyProviderSnapshot.Empty;
}
}
private static void CollectProviders(
RegistryKey scopeKey,
string providersSubPath,
string scopeLabel,
string kind,
string stateValueName,
List<PolicyProviderState> providers,
AgentLogger logger)
{
try
{
using (var providersKey = scopeKey.OpenSubKey(providersSubPath))
{
if (providersKey == null) return;
foreach (var providerName in providersKey.GetSubKeyNames())
{
using (var providerKey = providersKey.OpenSubKey(providerName))
{
if (providerKey == null) continue;
var value = TryReadInt(providerKey.GetValue(stateValueName));
providers.Add(new PolicyProviderState(
name: providerName,
scope: scopeLabel,
kind: kind,
trackingPoliciesCreated: kind == KindSetupApps ? value : null,
installationState: kind == KindDevicePreparation ? value : null));
}
}
}
}
catch (Exception ex)
{
logger?.Debug($"EspPolicyProviderProbe: '{providersSubPath}' ({scopeLabel}) read threw: {ex.Message}");
}
}
/// <summary>
/// Defensive value parse: REG_DWORD expected per CSP (booleans stored as 0/1), but string
/// forms are tolerated. Anything missing or unparseable is <c>null</c> — which the
/// completion rule treats as incomplete.
/// </summary>
internal static int? TryReadInt(object rawValue)
{
switch (rawValue)
{
case null: return null;
case int i: return i;
case long l when l >= int.MinValue && l <= int.MaxValue: return (int)l;
case string s when int.TryParse(s.Trim(), NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsed):
return parsed;
case bool b: return b ? 1 : 0;
default: return null;
}
}
/// <summary>
/// Pure completion rule per the CSP contract: <c>setupApps</c> providers are complete when
/// <c>TrackingPoliciesCreated == 1</c>; <c>devicePreparation</c> providers when
/// <c>InstallationState</c> is 2 (NotRequired) or 3 (Completed).
/// </summary>
internal static bool IsProviderComplete(string kind, int? trackingPoliciesCreated, int? installationState)
=> kind == KindSetupApps
? trackingPoliciesCreated == 1
: installationState == 2 || installationState == 3;
/// <summary>
/// Disposable scope that sets <see cref="TestOverride"/> for the lifetime of the scope
/// and restores the previous value on Dispose. Nestable. Internal test-only helper.
/// </summary>
internal sealed class ScopedOverride : IDisposable
{
private readonly Func<AgentLogger, EspPolicyProviderSnapshot> _previous;
private int _disposed;
public ScopedOverride(Func<AgentLogger, EspPolicyProviderSnapshot> probe)
{
_previous = TestOverride;
TestOverride = probe;
}
public void Dispose()
{
if (System.Threading.Interlocked.Exchange(ref _disposed, 1) == 1) return;
TestOverride = _previous;
}
}
}
/// <summary>
/// One registered ESP policy provider with its completion-relevant state.
/// <see cref="Key"/> is the stable identity used for dwell tracking and one-shot latching.
/// </summary>
internal readonly struct PolicyProviderState
{
public PolicyProviderState(string name, string scope, string kind, int? trackingPoliciesCreated, int? installationState)
{
Name = name ?? string.Empty;
Scope = scope ?? string.Empty;
Kind = kind ?? string.Empty;
TrackingPoliciesCreated = trackingPoliciesCreated;
InstallationState = installationState;
IsComplete = EspPolicyProviderProbe.IsProviderComplete(kind, trackingPoliciesCreated, installationState);
}
/// <summary>Provider key name, e.g. "Sidecar" or "ConfigMgr".</summary>
public string Name { get; }
/// <summary><c>"device"</c> or <c>"user:S-1-5-21-…"</c>.</summary>
public string Scope { get; }
/// <summary><see cref="EspPolicyProviderProbe.KindSetupApps"/> or <see cref="EspPolicyProviderProbe.KindDevicePreparation"/>.</summary>
public string Kind { get; }
/// <summary>Raw <c>TrackingPoliciesCreated</c> value (setupApps only); <c>null</c> when missing/unparseable.</summary>
public int? TrackingPoliciesCreated { get; }
/// <summary>Raw <c>InstallationState</c> value (devicePreparation only); <c>null</c> when missing/unparseable.</summary>
public int? InstallationState { get; }
/// <summary>Completion per <see cref="EspPolicyProviderProbe.IsProviderComplete"/>.</summary>
public bool IsComplete { get; }
/// <summary>Stable dwell/latch identity: <c>kind|scope|name</c>.</summary>
public string Key => Kind + "|" + Scope + "|" + Name;
}
/// <summary>
/// Snapshot of all registered ESP policy providers. <see cref="HasData"/> is <c>false</c> only
/// when the EnrollmentStatusTracking root key was missing or unreadable — an empty provider
/// list with <see cref="HasData"/> = <c>true</c> is valid data (nothing registered yet).
/// </summary>
internal readonly struct EspPolicyProviderSnapshot
{
public static readonly EspPolicyProviderSnapshot Empty = default;
public EspPolicyProviderSnapshot(IReadOnlyList<PolicyProviderState> providers)
{
HasData = true;
Providers = providers ?? Array.Empty<PolicyProviderState>();
}
/// <summary><c>true</c> when the EnrollmentStatusTracking root key existed and was readable.</summary>
public bool HasData { get; }
/// <summary>All registered providers across scopes and kinds; empty when none registered.</summary>
public IReadOnlyList<PolicyProviderState> Providers { get; }
/// <summary>
/// <c>true</c> when a <c>Setup\Apps</c> provider named "Sidecar" (the IME) is registered in
/// any scope — its absence while other providers are registered is the co-management
/// field-case signature.
/// </summary>
public bool SidecarRegistered
{
get
{
var providers = Providers;
if (providers == null) return false;
return providers.Any(p =>
p.Kind == EspPolicyProviderProbe.KindSetupApps &&
string.Equals(p.Name, EspPolicyProviderProbe.SidecarProviderName, StringComparison.OrdinalIgnoreCase));
}
}
}
}
@@ -0,0 +1,226 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Persistence;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
/// <summary>
/// Turns the ESP policy-provider contract read by <see cref="EspPolicyProviderProbe"/> into
/// the one-shot <c>esp_policy_provider_stalled</c> Warning: a registered provider that stays
/// continuously incomplete for <see cref="DwellMinutes"/> minutes is reported once, with the
/// full provider table in the payload. See the probe's header for the CSP contract, the
/// official sources, and the co-management field case (ConfigMgr registered without Sidecar
/// → user ESP parked at "Apps (Identifying)" with no OS timeout) this tripwire exists for.
/// <para>
/// Dwell semantics: the dwell clock per provider starts when it is first observed incomplete
/// and RESETS when the provider completes or disappears — only <i>continuously</i> incomplete
/// providers fire. A missing root key / empty provider list is normal early enrollment and
/// clears all dwell state (probe D4). Purely observational: emission goes through
/// <see cref="InformationalEventPost"/> (InformationalEvent pass-through, dispatch-guard
/// exempt), never into decision state.
/// </para>
/// <para>
/// Duplicate suppression: in-process one-shot per provider key, plus a
/// <see cref="StartupEventGate"/> fingerprint over the sorted set of currently-stalled keys so
/// an agent restart into the same stall does not re-report it — while a NEW provider joining
/// the stall changes the fingerprint and is reported (the event carries the full list).
/// </para>
/// </summary>
internal sealed class EspPolicyProviderStallDetector
{
/// <summary>
/// Wall-clock minutes a provider must stay continuously incomplete before it is reported.
/// Mirrors the CSP's own DevicePreparation default timeout; the Setup/Apps wait this
/// tripwire mainly targets has no OS timeout at all.
/// </summary>
internal const int DwellMinutes = 15;
internal const string SourceLabel = "EspPolicyProviderStallDetector";
private readonly string _sessionId;
private readonly string _tenantId;
private readonly InformationalEventPost _post;
private readonly AgentLogger _logger;
private readonly IClock _clock;
private readonly StartupEventGate? _startupGate;
private readonly TimeSpan _dwell;
private readonly Dictionary<string, DateTime> _firstSeenIncompleteUtc =
new Dictionary<string, DateTime>(StringComparer.OrdinalIgnoreCase);
private readonly HashSet<string> _firedKeys =
new HashSet<string>(StringComparer.OrdinalIgnoreCase);
public EspPolicyProviderStallDetector(
string sessionId,
string tenantId,
InformationalEventPost post,
AgentLogger logger,
IClock clock,
StartupEventGate? startupGate = null,
TimeSpan? dwell = null)
{
_sessionId = sessionId;
_tenantId = tenantId;
_post = post ?? throw new ArgumentNullException(nameof(post));
_logger = logger;
_clock = clock ?? throw new ArgumentNullException(nameof(clock));
_startupGate = startupGate;
_dwell = dwell ?? TimeSpan.FromMinutes(DwellMinutes);
}
/// <summary>Host timer callback and test seam — one probe + evaluation round.</summary>
internal void Tick()
{
var snapshot = EspPolicyProviderProbe.Read(_logger);
var eval = EvaluateStallTransition(snapshot, _firstSeenIncompleteUtc, _firedKeys, _clock.UtcNow, _dwell);
_firstSeenIncompleteUtc.Clear();
foreach (var kv in eval.UpdatedFirstSeenUtc)
_firstSeenIncompleteUtc[kv.Key] = kv.Value;
if (eval.NewlyStalled.Count > 0)
EmitStalled(snapshot, eval);
}
/// <summary>
/// Pure evaluation of one probe round — the entire detection decision, testable without
/// registry, timer, or ingress (pattern: <c>PerformanceCollector.EvaluateDiskLowTransition</c>).
/// </summary>
/// <param name="snapshot">Current provider snapshot.</param>
/// <param name="firstSeenIncompleteUtc">Dwell state from the previous round (provider key → first-seen-incomplete).</param>
/// <param name="alreadyFired">Provider keys already reported this run (never re-report).</param>
/// <param name="nowUtc">Evaluation timestamp.</param>
/// <param name="dwell">Continuous-incompleteness threshold.</param>
internal static StallEvaluation EvaluateStallTransition(
EspPolicyProviderSnapshot snapshot,
IReadOnlyDictionary<string, DateTime> firstSeenIncompleteUtc,
ISet<string> alreadyFired,
DateTime nowUtc,
TimeSpan dwell)
{
var result = new StallEvaluation();
// Missing root key or no registered providers = normal early enrollment: clear all
// dwell state (a provider must be continuously OBSERVED incomplete to accrue dwell).
if (!snapshot.HasData || snapshot.Providers.Count == 0)
return result;
foreach (var provider in snapshot.Providers)
{
if (provider.IsComplete)
continue; // completed → falls out of the dwell map (reset)
var firstSeen = firstSeenIncompleteUtc.TryGetValue(provider.Key, out var seen)
? seen
: nowUtc;
result.UpdatedFirstSeenUtc[provider.Key] = firstSeen;
var stalledFor = nowUtc - firstSeen;
if (stalledFor >= dwell && !alreadyFired.Contains(provider.Key))
result.NewlyStalled.Add(new StalledProvider(provider, stalledFor.TotalMinutes));
}
return result;
}
private void EmitStalled(EspPolicyProviderSnapshot snapshot, StallEvaluation eval)
{
foreach (var stalled in eval.NewlyStalled)
_firedKeys.Add(stalled.Provider.Key);
// Cross-restart dedup: fingerprint = sorted set of all fired keys that are still
// incomplete. A restart into the same stall re-accrues dwell and lands on the same
// fingerprint → suppressed; a new provider joining the stall changes it → re-emitted.
var currentlyStalledKeys = snapshot.Providers
.Where(p => !p.IsComplete && _firedKeys.Contains(p.Key))
.Select(p => p.Key)
.OrderBy(k => k, StringComparer.OrdinalIgnoreCase)
.ToList();
var fingerprint = StartupEventGate.HashString(string.Join(";", currentlyStalledKeys));
if (_startupGate != null &&
!_startupGate.ShouldEmit(Constants.EventTypes.EspPolicyProviderStalled, fingerprint))
{
_logger?.Debug(
"EspPolicyProviderStallDetector: stalled provider set already reported by a " +
"previous agent run (fingerprint match) — suppressed.");
return;
}
var stalledNames = string.Join(", ",
eval.NewlyStalled.Select(s => $"{s.Provider.Name} ({s.Provider.Scope}/{s.Provider.Kind})"));
var data = new Dictionary<string, object>(StringComparer.Ordinal)
{
["dwellMinutes"] = DwellMinutes,
["sidecarRegistered"] = snapshot.SidecarRegistered,
["providers"] = snapshot.Providers.Select(p => new Dictionary<string, object?>(StringComparer.Ordinal)
{
["name"] = p.Name,
["scope"] = p.Scope,
["kind"] = p.Kind,
["trackingPoliciesCreated"] = p.TrackingPoliciesCreated,
["installationState"] = p.InstallationState,
["complete"] = p.IsComplete,
}).ToList(),
["stalledProviders"] = eval.NewlyStalled.Select(s => new Dictionary<string, object>(StringComparer.Ordinal)
{
["name"] = s.Provider.Name,
["scope"] = s.Provider.Scope,
["kind"] = s.Provider.Kind,
["stalledForMinutes"] = Math.Round(s.StalledForMinutes, 1),
}).ToList(),
};
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
Timestamp = _clock.UtcNow,
EventType = Constants.EventTypes.EspPolicyProviderStalled,
Severity = EventSeverity.Warning,
Source = SourceLabel,
Phase = EnrollmentPhase.Unknown,
ImmediateUpload = true,
Message = $"ESP policy provider(s) incomplete for >= {DwellMinutes} min: {stalledNames} " +
$"(sidecarRegistered={snapshot.SidecarRegistered}) — the ESP waits on " +
"Setup/Apps providers without any timeout",
Data = data
});
_startupGate?.MarkEmitted(Constants.EventTypes.EspPolicyProviderStalled);
_logger?.Warning(
$"EspPolicyProviderStallDetector: {stalledNames} incomplete for >= {DwellMinutes} min " +
$"(sidecarRegistered={snapshot.SidecarRegistered}).");
}
/// <summary>Result of one <see cref="EvaluateStallTransition"/> round.</summary>
internal sealed class StallEvaluation
{
/// <summary>Next round's dwell state — only currently-incomplete providers survive.</summary>
public Dictionary<string, DateTime> UpdatedFirstSeenUtc { get; } =
new Dictionary<string, DateTime>(StringComparer.OrdinalIgnoreCase);
/// <summary>Providers that crossed the dwell threshold this round and were not yet reported.</summary>
public List<StalledProvider> NewlyStalled { get; } = new List<StalledProvider>();
}
internal readonly struct StalledProvider
{
public StalledProvider(PolicyProviderState provider, double stalledForMinutes)
{
Provider = provider;
StalledForMinutes = stalledForMinutes;
}
public PolicyProviderState Provider { get; }
public double StalledForMinutes { get; }
}
}
}
@@ -116,6 +116,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
// failure details that fire after the settle window expires.
private Dictionary<string, Timer> _provisioningFailureSettleTimers;
private Dictionary<string, EspFailureDetectedEventArgs> _provisioningFailureSettleArgs;
// Session 4910a5a5 — post-terminal recovery story. Populated when the settle window
// expires and EspFailureDetected actually fires (terminal path); consumed by the
// esp_failure_retry_detected / esp_failure_recovered emitters so a late "Try again"
// recovery (23 min after the failure in the field session) is visible on the timeline
// instead of hiding inside a generic subcategory_state_change. Keyed by category-name.
private Dictionary<string, EspFailureDetectedEventArgs> _provisioningFailureTerminalArgs;
private Dictionary<string, DateTime> _provisioningFailureTerminalUtc;
private HashSet<string> _postFailureRetryEmitted;
private HashSet<string> _postFailureRecoveredEmitted;
// Session 2bc884b6: AppX enrichment scan seams + fire-once guard. The scanner and the
// dispatcher are injectable so tests never touch the real event log or the threadpool.
private readonly IAppxDeploymentFailureScanner _appxScanner;
@@ -285,6 +294,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
_lastSubcategoryStates = new Dictionary<string, Dictionary<string, string>>(StringComparer.OrdinalIgnoreCase);
_provisioningFailureSettleTimers = new Dictionary<string, Timer>(StringComparer.OrdinalIgnoreCase);
_provisioningFailureSettleArgs = new Dictionary<string, EspFailureDetectedEventArgs>(StringComparer.OrdinalIgnoreCase);
_provisioningFailureTerminalArgs = new Dictionary<string, EspFailureDetectedEventArgs>(StringComparer.OrdinalIgnoreCase);
_provisioningFailureTerminalUtc = new Dictionary<string, DateTime>(StringComparer.OrdinalIgnoreCase);
_postFailureRetryEmitted = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
_postFailureRecoveredEmitted = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
_deviceSetupProvisioningCompleteFired = false;
_deviceSetupFallbackTimer = null;
_accountSetupProvisioningCompleteFired = false;
@@ -665,6 +678,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
// null-ing here is correctness-neutral on the start-condition side.
ResetAccountSetupFallbackTimer();
_accountSetupProvisioningCompleteFired = true;
MaybeEmitPostFailureRecovered("AccountSetupCategory.Status", "AccountSetup");
try { AccountSetupProvisioningComplete?.Invoke(this, EventArgs.Empty); }
catch (Exception ex) { _logger.Error("AccountSetupProvisioningComplete handler failed (fallback path)", ex); }
}
@@ -833,6 +847,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
// null-ing here is correctness-neutral on the start-condition side.
ResetDeviceSetupFallbackTimer();
_deviceSetupProvisioningCompleteFired = true;
MaybeEmitPostFailureRecovered("DeviceSetupCategory.Status", "DeviceSetup");
try { DeviceSetupProvisioningComplete?.Invoke(this, EventArgs.Empty); }
catch (Exception ex) { _logger.Error("DeviceSetupProvisioningComplete handler failed (fallback path)", ex); }
}
@@ -946,12 +961,25 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
var severity = categorySucceeded == false ? EventSeverity.Warning : EventSeverity.Info;
EmitProvisioningEvent(categoryLabel, categorySucceeded, statusText, subcategories, severity);
EmitRawRegistryDump(categoryLabel, jsonValue, $"category_resolved_{(categorySucceeded == true ? "success" : "failed")}");
// Session 4910a5a5: category resolving to success after a terminally
// fired failure = the failure recovered (fallback-path successes call
// this from their own timer handlers).
if (categorySucceeded == true)
{
MaybeEmitPostFailureRecovered(categoryName, categoryLabel);
}
}
else
{
var transitions = DetectSubcategoryTransitions(categoryName, subcategories);
StoreSubcategoryStates(categoryName, subcategories);
// Session 4910a5a5: after a terminally fired failure, the failed
// subcategory leaving "failed" is the ESP re-running the step —
// consistent with the user pressing "Try again" on the failure page.
MaybeEmitPostFailureRetry(categoryName, categoryLabel, transitions);
var failureTransitions = transitions.Where(t => t.IsFailure).ToList();
if (transitions.Count > 0)
{
@@ -1025,6 +1053,21 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
$"escalating as {failureTypeName} (errorCode={errorCode ?? "n/a"})");
_logger.Warning($"Provisioning failure detected: {failureTypeName}");
// Session 4910a5a5: name the likely culprit app(s) directly on the failure args so
// the DecisionEngine's advisory/terminal events carry them — never-started apps
// first (they starve ESP's app gate without ever producing an IME install event),
// capped at 3 names.
IReadOnlyList<string> likelyCulpritApps = null;
if (string.Equals(failedSubcategoryName, "Apps", StringComparison.OrdinalIgnoreCase))
{
likelyCulpritApps = SnapshotTrackedAppsNotCompleted()
.OrderBy(a => a["everStartedInstalling"] is bool started && started ? 1 : 0)
.Select(a => a["appName"]?.ToString())
.Where(n => !string.IsNullOrEmpty(n))
.Take(3)
.ToList();
}
// Caller already holds _stateLock.
ArmFailureSettleTimer(
categoryName: categoryName,
@@ -1032,7 +1075,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
failureType: failureTypeName,
errorCode: errorCode,
failedSubcategory: failedSubcategoryName,
category: categoryLabel));
category: categoryLabel,
likelyCulpritApps: likelyCulpritApps));
// Settle-window owns the actual EspFailureDetected fire. The legacy "fire-after-emit"
// path in CheckProvisioningStatus is now a no-op for this code-path; it stays in place
@@ -1292,6 +1336,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
_provisioningCategoriesResolved.Remove(categoryName);
}
}
else
{
// Session 4910a5a5: remember the terminally fired failure so a LATE
// recovery (outside the settle window — e.g. user pressing "Try again"
// minutes later) can be told as esp_failure_retry_detected /
// esp_failure_recovered instead of hiding in a generic state-change.
_provisioningFailureTerminalArgs[categoryName] = args;
_provisioningFailureTerminalUtc[categoryName] = DateTime.UtcNow;
}
}
if (recovered)
@@ -1382,6 +1435,133 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
});
}
/// <summary>
/// Session 4910a5a5: after a terminally fired failure (settle window expired,
/// EspFailureDetected raised), the failed subcategory leaving the <c>failed</c> state
/// means the ESP re-ran the step — consistent with the user pressing "Try again" on
/// the ESP failure page (the agent cannot observe the click itself). Emits
/// <c>esp_failure_retry_detected</c> once per terminally fired category. Caller holds
/// <see cref="_stateLock"/>.
/// </summary>
private void MaybeEmitPostFailureRetry(string categoryName, string categoryLabel, List<SubcategoryTransition> transitions)
{
if (transitions == null || transitions.Count == 0) return;
if (_postFailureRetryEmitted.Contains(categoryName)) return;
if (_provisioningFailureTerminalArgs == null
|| !_provisioningFailureTerminalArgs.TryGetValue(categoryName, out var terminalArgs))
{
return; // No terminal fire on record — in-window recovery is settle_recovered's job.
}
SubcategoryTransition? retryTransition = null;
foreach (var t in transitions)
{
if (string.Equals(t.OldState, "failed", StringComparison.OrdinalIgnoreCase)
&& !string.Equals(t.NewState, "failed", StringComparison.OrdinalIgnoreCase))
{
retryTransition = t;
break;
}
}
if (retryTransition == null) return;
var retry = retryTransition.Value;
_postFailureRetryEmitted.Add(categoryName);
var minutesSinceFailure = MinutesSinceTerminalFailure(categoryName);
var data = new Dictionary<string, object>
{
{ "category", categoryLabel },
{ "subcategory", retry.SubcategoryName },
{ "previousState", retry.OldState },
{ "newState", retry.NewState },
{ "failureType", terminalArgs.FailureType },
{ "minutesSinceFailure", minutesSinceFailure },
};
if (!string.IsNullOrEmpty(terminalArgs.ErrorCode))
data["errorCode"] = terminalArgs.ErrorCode;
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
EventType = Constants.EventTypes.EspFailureRetryDetected,
Severity = EventSeverity.Info,
Source = "EspAndHelloTracker",
Phase = EnrollmentPhase.Unknown,
Message = $"ESP retry detected — {categoryLabel}/{retry.SubcategoryName} left the failed state " +
$"({retry.OldState} → {retry.NewState}) {minutesSinceFailure} min after the failure — " +
"consistent with the user pressing 'Try again' on the ESP failure page.",
Data = data,
ImmediateUpload = true
});
}
/// <summary>
/// Session 4910a5a5: the category behind a terminally fired failure reached success
/// (categorySucceeded=true or the all-subcategories-succeeded fallback) — the failure
/// demonstrably recovered. Emits <c>esp_failure_recovered</c> once per terminally
/// fired category so the timeline tells "failed → retried → recovered" instead of
/// leaving the stale failure as the last word. Caller holds <see cref="_stateLock"/>.
/// </summary>
private void MaybeEmitPostFailureRecovered(string categoryName, string categoryLabel)
{
if (_postFailureRecoveredEmitted == null || _postFailureRecoveredEmitted.Contains(categoryName)) return;
if (_provisioningFailureTerminalArgs == null
|| !_provisioningFailureTerminalArgs.TryGetValue(categoryName, out var terminalArgs))
{
return;
}
_postFailureRecoveredEmitted.Add(categoryName);
var minutesSinceFailure = MinutesSinceTerminalFailure(categoryName);
string subcategoryStatus = null;
if (terminalArgs.FailedSubcategory != null
&& _lastSubcategoryStates != null
&& _lastSubcategoryStates.TryGetValue(categoryName, out var subs)
&& subs.TryGetValue(terminalArgs.FailedSubcategory, out var currentState))
{
subcategoryStatus = currentState;
}
var data = new Dictionary<string, object>
{
{ "category", categoryLabel },
{ "failedSubcategory", terminalArgs.FailedSubcategory ?? "category-level" },
{ "failureType", terminalArgs.FailureType },
{ "minutesSinceFailure", minutesSinceFailure },
};
if (!string.IsNullOrEmpty(terminalArgs.ErrorCode))
data["errorCode"] = terminalArgs.ErrorCode;
if (subcategoryStatus != null)
data["subcategoryState"] = subcategoryStatus;
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
EventType = Constants.EventTypes.EspFailureRecovered,
Severity = EventSeverity.Info,
Source = "EspAndHelloTracker",
Phase = EnrollmentPhase.Unknown,
Message = $"ESP failure recovered — {categoryLabel}/{terminalArgs.FailedSubcategory ?? "category-level"} " +
$"completed successfully {minutesSinceFailure} min after the failure; the earlier ESP failure no longer applies.",
Data = data,
ImmediateUpload = true
});
}
private int MinutesSinceTerminalFailure(string categoryName)
{
if (_provisioningFailureTerminalUtc != null
&& _provisioningFailureTerminalUtc.TryGetValue(categoryName, out var firedAt))
{
return Math.Max(0, (int)Math.Round((DateTime.UtcNow - firedAt).TotalMinutes));
}
return 0;
}
private void EmitFailureSettleStarted(EspFailureDetectedEventArgs args)
{
var data = new Dictionary<string, object>
@@ -1863,6 +2043,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
_provisioningFailureSettleTimers = new Dictionary<string, Timer>(StringComparer.OrdinalIgnoreCase);
if (_provisioningFailureSettleArgs == null)
_provisioningFailureSettleArgs = new Dictionary<string, EspFailureDetectedEventArgs>(StringComparer.OrdinalIgnoreCase);
if (_provisioningFailureTerminalArgs == null)
_provisioningFailureTerminalArgs = new Dictionary<string, EspFailureDetectedEventArgs>(StringComparer.OrdinalIgnoreCase);
if (_provisioningFailureTerminalUtc == null)
_provisioningFailureTerminalUtc = new Dictionary<string, DateTime>(StringComparer.OrdinalIgnoreCase);
if (_postFailureRetryEmitted == null)
_postFailureRetryEmitted = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
if (_postFailureRecoveredEmitted == null)
_postFailureRecoveredEmitted = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
}
}
@@ -36,6 +36,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
/// <item><b>StallProbeHost</b> — <see cref="StallProbeCollector"/> +
/// <see cref="StallProbeCollectorAdapter"/>. Owns its 60-s idle-check timer (the
/// collector itself has no timer — it's a pure probe invoked from outside).</item>
/// <item><b>EspPolicyProviderStallHost</b> — always-on 60-s wall-clock tick for the
/// <c>esp_policy_provider_stalled</c> tripwire (registered EnrollmentStatusTracking CSP
/// provider continuously incomplete ≥ 15 min — the no-timeout ESP wait).</item>
/// </list>
/// Optional peripheral hosts (driven by <see cref="CollectorConfiguration"/> toggles):
/// <list type="bullet">
@@ -303,6 +306,20 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
harmlessModernDeploymentEventIds: collectors.ModernDeploymentHarmlessEventIds));
}
// ESP policy-provider stall tripwire — always-on kernel host (no config gate, like
// disk_space_low): a registered EnrollmentStatusTracking CSP provider that never
// completes (e.g. co-management "ConfigMgr" without "Sidecar" on a pre-provisioned
// device) parks the user ESP at "Apps (Identifying)" WITHOUT any OS timeout. Wall-clock
// dwell, deliberately outside the StallProbe gate: the idle clock there resets on any
// session activity, and the stall must accrue regardless of it.
hosts.Add(new EspPolicyProviderStallHost(
sessionId: sessionId,
tenantId: tenantId,
logger: logger,
ingress: ingress,
clock: clock,
startupGate: _startupEventGate));
// Deterministic update corroboration — compares the persisted OS build (CurrentBuild.UBR)
// across agent restarts and emits os_build_changed when it differs (session 7443317c:
// build jump proved an OOBE quality update the WU channel never showed). Must be added
@@ -0,0 +1,94 @@
#nullable enable
using System;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Persistence;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
namespace AutopilotMonitor.Agent.V2.Core.Orchestration
{
/// <summary>
/// Drives the <see cref="EspPolicyProviderStallDetector"/> from a 60-s wall-clock tick.
/// Deliberately NOT idle-coupled (a policy-provider stall coexists with a chatty session —
/// <see cref="StallProbeHost"/>'s idle clock would keep resetting) and NOT idle-stopped
/// (<see cref="PeriodicCollectorLifecycleHost"/> would sleep through exactly the dormant
/// window the dwell must observe). Always-on kernel host, no config gate — precedent:
/// the <c>disk_space_low</c> tripwire.
/// <para>
/// First tick fires after 60 s, not at Start: the EnrollmentStatusTracking key legitimately
/// doesn't exist at agent start, and nothing can be 15 min stale at t=0.
/// </para>
/// </summary>
internal sealed class EspPolicyProviderStallHost : ICollectorHost
{
public string Name => "EspPolicyProviderStallDetector";
private static readonly TimeSpan PollInterval = TimeSpan.FromSeconds(60);
private readonly EspPolicyProviderStallDetector _detector;
private readonly AgentLogger _logger;
private Timer? _tickTimer;
private int _disposed;
public EspPolicyProviderStallHost(
string sessionId,
string tenantId,
AgentLogger logger,
ISignalIngressSink ingress,
IClock clock,
StartupEventGate? startupGate)
{
if (ingress == null) throw new ArgumentNullException(nameof(ingress));
if (clock == null) throw new ArgumentNullException(nameof(clock));
_logger = logger;
var post = new InformationalEventPost(ingress, clock);
_detector = new EspPolicyProviderStallDetector(
sessionId: sessionId,
tenantId: tenantId,
post: post,
logger: logger,
clock: clock,
startupGate: startupGate);
}
public void Start()
{
_tickTimer = new Timer(
_ => SafeTick(),
state: null,
dueTime: PollInterval,
period: PollInterval);
_logger.Info($"EspPolicyProviderStallHost: started (tick every {PollInterval.TotalSeconds}s).");
}
public void Stop()
{
try
{
_tickTimer?.Dispose();
_tickTimer = null;
}
catch (Exception ex) { _logger.Warning($"EspPolicyProviderStallHost: timer dispose failed: {ex.Message}"); }
}
public void Dispose()
{
if (Interlocked.Exchange(ref _disposed, 1) == 1) return;
Stop();
}
private void SafeTick()
{
try
{
_detector.Tick();
}
catch (Exception ex)
{
_logger.Error("EspPolicyProviderStallHost: tick failed.", ex);
}
}
}
}
@@ -1,6 +1,7 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Globalization;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
@@ -267,6 +268,14 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
payload["failedSubcategory"] = failedSubcategory!;
if (!string.IsNullOrEmpty(category))
payload["category"] = category!;
// Session 4910a5a5: culprit-app names ride the signal so the DecisionEngine's
// esp_failure_advisory / enrollment_failed events can name them directly.
var culprits = args?.LikelyCulpritApps;
if (culprits != null && culprits.Count > 0)
{
payload["likelyCulpritApps"] = string.Join(", ", culprits);
payload["likelyCulpritAppCount"] = culprits.Count.ToString(CultureInfo.InvariantCulture);
}
_ingress.Post(
kind: DecisionSignalKind.EspTerminalFailure,
@@ -0,0 +1,382 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Serialization;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.DecisionCore.State;
using Xunit;
namespace AutopilotMonitor.DecisionCore.Tests
{
/// <summary>
/// Session 4910a5a5 (2026-07-23) — advisory recovery. Field failure: DeviceSetup/Apps
/// failed (culprit app never started installing), the advisory defang armed the
/// <c>advisory_completion</c> window, the user pressed "Try again" ~23 min later, the apps
/// re-ran to 10/10 and <c>DeviceSetupProvisioningComplete</c> arrived — yet the window
/// fired 30 min after the advisory and un-defanged the stale failure
/// (<c>enrollment_failed: esp_terminal_failure</c>) on a live enrollment mid-AccountSetup,
/// because the advisory variant was exempt from ALL re-arm/rebase guards. Coverage:
/// <list type="bullet">
/// <item>The advisory records the failed category + culprit apps.</item>
/// <item>A ProvisioningComplete of the SAME category sets the resolved fact + emits
/// <c>esp_failure_advisory_resolved</c>; a different category does not.</item>
/// <item>Deadline fire after recovery + enforcement progress → re-arm, not Failed.</item>
/// <item>Deadline fire after recovery without progress → Failed with the truthful
/// <c>esp_recovered_without_completion_evidence</c> reason and
/// LastFailureTrigger=DeadlineFired (likely-stuck promotion off).</item>
/// <item>Unrecovered advisory keeps the legacy un-defang exactly (regression guard).</item>
/// <item>Reboot re-bases a recovered-advisory window (and still does NOT touch an
/// unrecovered one).</item>
/// <item>Snapshot round-trip + census surfacing of the new facts.</item>
/// </list>
/// </summary>
public sealed class AdvisoryRecoveryTests
{
private static readonly DateTime T0 = new DateTime(2026, 7, 23, 3, 30, 0, DateTimeKind.Utc);
private static DecisionSignal MakeSignal(
long ordinal,
DecisionSignalKind kind,
DateTime occurredAtUtc,
IReadOnlyDictionary<string, string>? payload = null)
{
return new DecisionSignal(
sessionSignalOrdinal: ordinal,
sessionTraceOrdinal: ordinal,
kind: kind,
kindSchemaVersion: 1,
occurredAtUtc: occurredAtUtc,
sourceOrigin: "test",
evidence: new Evidence(EvidenceKind.Synthetic, $"{kind}-{ordinal}", "test"),
payload: payload);
}
private static DecisionSignal DeadlineFired(long ordinal, DateTime occurredAtUtc) =>
MakeSignal(ordinal, DecisionSignalKind.DeadlineFired, occurredAtUtc,
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.AdvisoryCompletion });
private static DecisionEffect? FindTimelineEffect(DecisionStep step, string eventType) =>
step.Effects.FirstOrDefault(e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry
&& e.Parameters != null
&& e.Parameters.TryGetValue("eventType", out var et)
&& et == eventType);
private static ActiveDeadline? FindDeadline(DecisionState state, string name) =>
state.Deadlines.FirstOrDefault(d => d.Name == name);
/// <summary>
/// Replays the session-4910a5a5 shape up to (but excluding) the ESP terminal failure:
/// Classic UserDriven, ContinueAnyway profile, Hello disabled, DeviceSetup entered,
/// AccountSetup entered (04:18, BEFORE the 04:34 DeviceSetup/Apps failure — the IME
/// phase line landed while device apps were still at 9/10).
/// </summary>
private static DecisionState SetupSession(DecisionEngine engine)
{
var state = DecisionState.CreateInitial("sess-4910a5a5", "tenant-e46bc88e", T0);
state = engine.Reduce(state, MakeSignal(0, DecisionSignalKind.SessionStarted, T0)).NewState;
state = engine.Reduce(state, MakeSignal(
5, DecisionSignalKind.EspConfigDetected, T0.AddMinutes(1),
new Dictionary<string, string>
{
[SignalPayloadKeys.SkipUserEsp] = "false",
[SignalPayloadKeys.SkipDeviceEsp] = "false",
[SignalPayloadKeys.EspSyncFailureTimeoutMinutes] = "60",
[SignalPayloadKeys.EspAllowContinueAnyway] = "true",
})).NewState;
state = engine.Reduce(state, MakeSignal(
8, DecisionSignalKind.HelloPolicyDetected, T0.AddMinutes(2),
new Dictionary<string, string> { [SignalPayloadKeys.HelloEnabled] = "false" })).NewState;
state = engine.Reduce(state, MakeSignal(
10, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(6),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "DeviceSetup" })).NewState;
state = engine.Reduce(state, MakeSignal(
20, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(48),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState;
return state;
}
private static DecisionStep ApplyDeviceSetupAppsFailure(DecisionEngine engine, DecisionState state, long ordinal = 50) =>
engine.Reduce(state, MakeSignal(
ordinal, DecisionSignalKind.EspTerminalFailure, T0.AddMinutes(64),
new Dictionary<string, string>
{
["failureType"] = "Provisioning_DeviceSetup_Apps_Failed",
["failedSubcategory"] = "Apps",
["category"] = "DeviceSetup",
["likelyCulpritApps"] = "Oriflame Cosmectics - Teams Backgroundsv v2",
["likelyCulpritAppCount"] = "1",
}));
private static DecisionStep ApplyDeviceSetupResolved(DecisionEngine engine, DecisionState state, long ordinal = 60) =>
engine.Reduce(state, MakeSignal(
ordinal, DecisionSignalKind.DeviceSetupProvisioningComplete, T0.AddMinutes(87)));
// ================================================== advisory arming context ====
[Fact]
public void Advisory_RecordsFailedCategory_AndCulpritApps()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
var step = ApplyDeviceSetupAppsFailure(engine, state);
Assert.NotNull(step.NewState.EspAdvisoryFailureRecordedUtc);
Assert.Equal("DeviceSetup", step.NewState.EspAdvisoryFailureCategory?.Value);
Assert.Null(step.NewState.EspAdvisoryFailureResolvedUtc);
var advisory = FindTimelineEffect(step, "esp_failure_advisory");
Assert.NotNull(advisory);
Assert.Equal("Oriflame Cosmectics - Teams Backgroundsv v2", advisory!.Parameters!["likelyCulpritApps"]);
}
// ===================================================== recovery hook ====
[Fact]
public void DeviceSetupComplete_SameCategory_ResolvesAdvisory_AndEmitsStoryEvent()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
var step = ApplyDeviceSetupResolved(engine, state);
Assert.NotNull(step.NewState.EspAdvisoryFailureResolvedUtc);
Assert.Equal(60, step.NewState.EspAdvisoryFailureResolvedUtc!.SourceSignalOrdinal);
// The anchor itself stays — the advisory happened; it just no longer blocks.
Assert.NotNull(step.NewState.EspAdvisoryFailureRecordedUtc);
var resolved = FindTimelineEffect(step, "esp_failure_advisory_resolved");
Assert.NotNull(resolved);
Assert.Equal("DeviceSetup", resolved!.Parameters!["category"]);
Assert.Equal("23", resolved.Parameters["minutesSinceAdvisory"]);
}
[Fact]
public void DeviceSetupComplete_DifferentCategory_DoesNotResolve()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
// Advisory for AccountSetup — DeviceSetup resolving must not defuse it.
state = engine.Reduce(state, MakeSignal(
50, DecisionSignalKind.EspTerminalFailure, T0.AddMinutes(64),
new Dictionary<string, string>
{
["failureType"] = "Provisioning_AccountSetup_Apps_Failed",
["failedSubcategory"] = "Apps",
["category"] = "AccountSetup",
})).NewState;
var step = ApplyDeviceSetupResolved(engine, state);
Assert.Null(step.NewState.EspAdvisoryFailureResolvedUtc);
Assert.Null(FindTimelineEffect(step, "esp_failure_advisory_resolved"));
}
[Fact]
public void DeviceSetupComplete_Duplicate_ResolvesOnlyOnce()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
state = ApplyDeviceSetupResolved(engine, state).NewState;
// Post-reboot duplicate (AnchorAlreadySet passthrough branch).
var duplicate = engine.Reduce(state, MakeSignal(
65, DecisionSignalKind.DeviceSetupProvisioningComplete, T0.AddMinutes(92)));
Assert.Equal(60, duplicate.NewState.EspAdvisoryFailureResolvedUtc!.SourceSignalOrdinal);
Assert.Null(FindTimelineEffect(duplicate, "esp_failure_advisory_resolved"));
}
// ================================================ deadline fire semantics ====
[Fact]
public void DeadlineFire_AfterRecovery_WithEnforcementProgress_ReArmsInsteadOfFailing()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
state = ApplyDeviceSetupResolved(engine, state).NewState;
// Post-reboot IME re-assert of AccountSetup (session shape: 05:02:59, ordinal 654)
// — user-phase enforcement demonstrably progressed since the arming at ordinal 50.
state = engine.Reduce(state, MakeSignal(
70, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(92),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState;
var step = engine.Reduce(state, DeadlineFired(80, T0.AddMinutes(94)));
// Not failed — the window re-arms with fresh baselines instead.
Assert.Null(step.NewState.Outcome);
Assert.NotEqual(SessionStage.Failed, step.NewState.Stage);
var rearmed = FindDeadline(step.NewState, DeadlineNames.AdvisoryCompletion);
Assert.NotNull(rearmed);
Assert.Equal(T0.AddMinutes(94).AddMinutes(30), rearmed!.DueAtUtc);
Assert.Contains(step.Effects, e => e.Kind == DecisionEffectKind.ScheduleDeadline);
}
[Fact]
public void DeadlineFire_AfterRecovery_NoProgress_FailsWithRecoveredReason()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
state = ApplyDeviceSetupResolved(engine, state).NewState;
var step = engine.Reduce(state, DeadlineFired(80, T0.AddMinutes(94)));
Assert.Equal(SessionStage.Failed, step.NewState.Stage);
Assert.Equal(SessionOutcome.EnrollmentFailed, step.NewState.Outcome);
var failed = FindTimelineEffect(step, "enrollment_failed");
Assert.NotNull(failed);
// Truthful reason: the original failure recovered — do not blame it.
Assert.Equal("esp_recovered_without_completion_evidence", failed!.Parameters!["reason"]);
Assert.Equal("true", failed.Parameters["failureRecovered"]);
Assert.Equal("advisory_recovered_window_expired_without_completion_evidence", failed.Parameters["advisoryReason"]);
// Likely-stuck app promotion stays off: apps are not stuck, they succeeded.
Assert.Equal(nameof(DecisionSignalKind.DeadlineFired), step.NewState.LastFailureTrigger?.Value);
}
[Fact]
public void DeadlineFire_WithoutRecovery_KeepsLegacyUndefang()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
// No recovery — even with enforcement progress the advisory variant must
// un-defang exactly as before (its anchor failure never un-happened).
state = engine.Reduce(state, MakeSignal(
70, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(92),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState;
var step = engine.Reduce(state, DeadlineFired(80, T0.AddMinutes(94)));
Assert.Equal(SessionStage.Failed, step.NewState.Stage);
var failed = FindTimelineEffect(step, "enrollment_failed");
Assert.NotNull(failed);
Assert.Equal("esp_terminal_failure", failed!.Parameters!["reason"]);
Assert.Equal(nameof(DecisionSignalKind.EspTerminalFailure), step.NewState.LastFailureTrigger?.Value);
}
// ======================================================= reboot rebase ====
[Fact]
public void Reboot_AfterRecovery_RebasesAdvisoryWindow()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
state = ApplyDeviceSetupResolved(engine, state).NewState;
var step = engine.Reduce(state, MakeSignal(
70, DecisionSignalKind.SystemRebootObserved, T0.AddMinutes(92)));
Assert.Contains(step.Effects, e =>
e.Kind == DecisionEffectKind.CancelDeadline
&& e.CancelDeadlineName == DeadlineNames.AdvisoryCompletion);
var rebased = FindDeadline(step.NewState, DeadlineNames.AdvisoryCompletion);
Assert.NotNull(rebased);
Assert.Equal(T0.AddMinutes(92).AddMinutes(30), rebased!.DueAtUtc);
}
[Fact]
public void Reboot_WithoutRecovery_DoesNotTouchAdvisoryWindow()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
var originalDue = FindDeadline(state, DeadlineNames.AdvisoryCompletion)!.DueAtUtc;
var step = engine.Reduce(state, MakeSignal(
70, DecisionSignalKind.SystemRebootObserved, T0.AddMinutes(92)));
Assert.DoesNotContain(step.Effects, e => e.Kind == DecisionEffectKind.CancelDeadline);
Assert.Equal(originalDue, FindDeadline(step.NewState, DeadlineNames.AdvisoryCompletion)!.DueAtUtc);
}
// ============================================== full session-4910a5a5 replay ====
[Fact]
public void Session4910a5a5_FailureRecoveryRebootFire_DoesNotFailTheLiveEnrollment()
{
// Field ordinals: 527 EspTerminalFailure → 611 DeviceSetupProvisioningComplete
// (post "Try again", apps 10/10) → 635 SystemRebootObserved → 654 AccountSetup
// re-assert → 671 DeadlineFired. Pre-fix outcome: enrollment_failed at the fire.
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state, ordinal: 527).NewState;
state = ApplyDeviceSetupResolved(engine, state, ordinal: 611).NewState;
state = engine.Reduce(state, MakeSignal(
635, DecisionSignalKind.SystemRebootObserved, T0.AddMinutes(92))).NewState;
state = engine.Reduce(state, MakeSignal(
654, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(93),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState;
var step = engine.Reduce(state, DeadlineFired(671, T0.AddMinutes(94)));
Assert.Null(step.NewState.Outcome);
Assert.NotEqual(SessionStage.Failed, step.NewState.Stage);
Assert.NotNull(FindDeadline(step.NewState, DeadlineNames.AdvisoryCompletion));
}
// =============================================== serialization + census ====
[Fact]
public void SnapshotRoundTrip_PreservesRecoveryFacts()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
state = ApplyDeviceSetupResolved(engine, state).NewState;
var roundtripped = StateSerializer.Deserialize(StateSerializer.Serialize(state));
Assert.Equal("DeviceSetup", roundtripped.EspAdvisoryFailureCategory?.Value);
Assert.Equal(state.EspAdvisoryFailureResolvedUtc!.Value, roundtripped.EspAdvisoryFailureResolvedUtc?.Value);
Assert.Equal(state.EspAdvisoryFailureResolvedUtc.SourceSignalOrdinal, roundtripped.EspAdvisoryFailureResolvedUtc?.SourceSignalOrdinal);
}
[Fact]
public void LegacySnapshot_WithoutRecoveryFacts_DeserializesToNull()
{
// Simulate a snapshot written by a pre-fix agent: strip the new properties from
// the JSON entirely — deserialization must default them to null (additive compat).
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
state = ApplyDeviceSetupResolved(engine, state).NewState;
var legacy = Newtonsoft.Json.Linq.JObject.Parse(StateSerializer.Serialize(state));
foreach (var prop in legacy.Properties()
.Where(p => p.Name.IndexOf("EspAdvisoryFailureResolved", StringComparison.OrdinalIgnoreCase) >= 0
|| p.Name.IndexOf("EspAdvisoryFailureCategory", StringComparison.OrdinalIgnoreCase) >= 0)
.ToList())
{
prop.Remove();
}
var roundtripped = StateSerializer.Deserialize(legacy.ToString());
Assert.Null(roundtripped.EspAdvisoryFailureResolvedUtc);
Assert.Null(roundtripped.EspAdvisoryFailureCategory);
// The pre-existing advisory anchor still round-trips.
Assert.NotNull(roundtripped.EspAdvisoryFailureRecordedUtc);
}
[Fact]
public void Census_SurfacesAdvisoryResolved()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
state = ApplyDeviceSetupResolved(engine, state).NewState;
var census = DecisionStateSignalCensus.Build(state);
Assert.Contains("esp_advisory_failure_resolved", census.SignalsSeen);
Assert.True(census.SignalTimestamps.ContainsKey("espAdvisoryFailureResolved"));
}
}
}
@@ -114,7 +114,10 @@ namespace AutopilotMonitor.DecisionCore.Tests
// state-change-only dedupe anchor for the completion_waiting event → 17.
// Session 772fe502 (2026-07-13) added `helloWizardStartedUtc` — genuine
// Shell-Core 62404 wizard launch; vetoes the policy-disabled Hello skip → 18.
Assert.Equal(18, facts.Count);
// Session 4910a5a5 (2026-07-23) added `espAdvisoryFailureResolvedUtc` (the failed
// category recovered — lifts the advisory's re-arm/rebase guard exemption) and
// `espAdvisoryFailureCategory` (which category the advisory was for) → 20.
Assert.Equal(20, facts.Count);
Assert.All(facts.Values, v => Assert.Null(v));
}
@@ -236,10 +239,12 @@ namespace AutopilotMonitor.DecisionCore.Tests
// `completionWaitingFingerprint` as the state-change-only dedupe anchor for
// the completion_waiting event → 17. Session 772fe502 (2026-07-13) added
// `helloWizardStartedUtc` — the genuine Shell-Core 62404 wizard-launch fact
// that vetoes/retracts the policy-disabled Hello skip → 18. If this number
// ever changes, both the count expectation AND the actual snapshot output
// need to evolve in lockstep.
Assert.Equal(18, expectedFactKeys.Count);
// that vetoes/retracts the policy-disabled Hello skip → 18. Session 4910a5a5
// (2026-07-23) added `espAdvisoryFailureResolvedUtc` + `espAdvisoryFailureCategory`
// — advisory-recovery facts that lift the AdvisoryCompletion guard exemption → 20.
// If this number ever changes, both the count expectation AND the actual snapshot
// output need to evolve in lockstep.
Assert.Equal(20, expectedFactKeys.Count);
var snapshot = DecisionStateSnapshotBuilder.Build(DecisionState.CreateInitial("s", "t"));
var facts = (Dictionary<string, object?>)snapshot["facts"]!;
@@ -23,6 +23,9 @@ export const EVENT_TYPES_CATALOG = {
'esp_failure_advisory',
'esp_failure_settle_started',
'esp_failure_settle_recovered',
'esp_failure_retry_detected',
'esp_failure_recovered',
'esp_failure_advisory_resolved',
'esp_exiting',
'mdm_policy_reboot_required',
'esp_config_detected',
@@ -167,6 +170,7 @@ export const EVENT_TYPES_CATALOG = {
'telemetry_upload_blocked',
'session_parked_without_deadline',
'disk_space_low',
'esp_policy_provider_stalled',
'agent_late_start',
],
metrics_events: ['performance_snapshot', 'agent_metrics_snapshot', 'ingress_backpressure'],
@@ -73,6 +73,11 @@ namespace AutopilotMonitor.DecisionCore.Engine
/// The advisory variant (<see cref="DecisionState.EspAdvisoryFailureRecordedUtc"/>
/// set) is deliberately NOT touched: its anchor is a real ESP terminal failure and
/// a reboot does not un-happen that failure — the window must still un-defang it.
/// EXCEPTION (session 4910a5a5, 2026-07-23): when
/// <see cref="DecisionState.EspAdvisoryFailureResolvedUtc"/> is set the failed
/// category demonstrably recovered to success — the failure DID un-happen, and a
/// reboot mid-recovery (ESP continuing into AccountSetup) must re-base the window
/// exactly like the esp-exit variant instead of letting it fail a live enrollment.
/// </para>
/// Returns the effects (cancel + schedule + a fingerprint-bypassing
/// <c>completion_waiting</c> carrying the re-based due-time, mirroring the
@@ -84,7 +89,9 @@ namespace AutopilotMonitor.DecisionCore.Engine
DecisionStateBuilder builder,
DecisionSignal signal)
{
if (state.EspAdvisoryFailureRecordedUtc != null || !HasAdvisoryCompletionDeadline(state))
var unresolvedAdvisoryAnchored = state.EspAdvisoryFailureRecordedUtc != null
&& state.EspAdvisoryFailureResolvedUtc == null;
if (unresolvedAdvisoryAnchored || !HasAdvisoryCompletionDeadline(state))
{
return Array.Empty<DecisionEffect>();
}
@@ -139,6 +146,10 @@ namespace AutopilotMonitor.DecisionCore.Engine
"mayHaveContinuedAnyway",
"continueAnywayHint",
"advisoryReason",
// Session 4910a5a5 — recovery story context.
"failureRecovered",
"likelyCulpritApps",
"likelyCulpritAppCount",
};
/// <summary>
@@ -408,10 +419,17 @@ namespace AutopilotMonitor.DecisionCore.Engine
string reason,
Dictionary<string, string> parameters)
{
// Session 4910a5a5: remember WHICH category failed so the recovery hook
// (TryResolveAdvisoryOnCategoryRecovery) can later match a ProvisioningComplete
// signal of the same category against this advisory. Absent on event-log-derived
// failures — the hook then never matches.
string? failedCategory = null;
signal.Payload?.TryGetValue("category", out failedCategory);
var builder = state.ToBuilder()
.WithStepIndex(nextStep)
.WithLastAppliedSignalOrdinal(signal.SessionSignalOrdinal)
.WithEspAdvisoryFailureRecorded(signal.OccurredAtUtc, signal.SessionSignalOrdinal);
.WithEspAdvisoryFailureRecorded(signal.OccurredAtUtc, signal.SessionSignalOrdinal, failedCategory);
// Stage / Outcome deliberately untouched — monitoring continues. The session's
// normal completion paths (Hello/Desktop, IME pattern, AccountSetup provisioning
@@ -454,6 +472,63 @@ namespace AutopilotMonitor.DecisionCore.Engine
return new DecisionStep(newState, transition, effects);
}
/// <summary>
/// Session 4910a5a5 (2026-07-23) — recovery hook for a defanged ESP failure. When a
/// <c>DeviceSetupProvisioningComplete</c> / <c>AccountSetupProvisioningComplete</c>
/// signal reports the SAME category the advisory failure was recorded for, the failure
/// demonstrably un-happened (DeviceSetup/Apps failed → user pressed "Try again" ~23 min
/// later → apps re-ran to 10/10 → category resolved). Sets the set-once
/// <see cref="DecisionState.EspAdvisoryFailureResolvedUtc"/> fact on
/// <paramref name="builder"/> and returns an <c>esp_failure_advisory_resolved</c>
/// timeline effect telling the recovery story; returns null (builder untouched) when
/// there is no advisory, it is already resolved, or the category does not match.
/// <para>
/// The fact lifts the advisory variant's exemption from the AdvisoryCompletion
/// re-arm/rebase guards (see <see cref="HandleAdvisoryCompletionDeadlineFired"/> and
/// <see cref="RebaseEspExitVariantAdvisoryWindowOnReboot"/>) — without it the window
/// un-defangs the stale failure and fails a live enrollment mid-AccountSetup.
/// </para>
/// </summary>
private static DecisionEffect? TryResolveAdvisoryOnCategoryRecovery(
DecisionState state,
DecisionStateBuilder builder,
DecisionSignal signal,
string resolvedCategory)
{
if (state.EspAdvisoryFailureRecordedUtc == null) return null;
if (state.EspAdvisoryFailureResolvedUtc != null) return null;
var advisoryCategory = state.EspAdvisoryFailureCategory?.Value;
if (!string.Equals(advisoryCategory, resolvedCategory, StringComparison.OrdinalIgnoreCase))
{
return null;
}
builder.WithEspAdvisoryFailureResolved(signal.OccurredAtUtc, signal.SessionSignalOrdinal);
var minutesSinceAdvisory = Math.Max(0,
(int)Math.Round((signal.OccurredAtUtc - state.EspAdvisoryFailureRecordedUtc.Value).TotalMinutes));
var continueAnywayAllowed = state.ScenarioObservations?.EspAllowContinueAnyway?.Value == true;
var parameters = new Dictionary<string, string>
{
["eventType"] = SharedConstants.EventTypes.EspFailureAdvisoryResolved,
["source"] = "DecisionEngine",
["severity"] = "Info",
["immediateUpload"] = "false",
["message"] = $"ESP failure advisory resolved: {resolvedCategory} completed successfully "
+ $"{minutesSinceAdvisory} min after the advisory — the ESP re-ran the failed step "
+ "(consistent with a user-initiated retry on the ESP failure page"
+ (continueAnywayAllowed ? "; the profile allows Try again / Continue anyway" : "")
+ "). The earlier failure no longer blocks completion.",
["category"] = resolvedCategory,
["minutesSinceAdvisory"] = minutesSinceAdvisory.ToString(CultureInfo.InvariantCulture),
["espAllowContinueAnyway"] = continueAnywayAllowed ? "true" : "false",
};
return new DecisionEffect(DecisionEffectKind.EmitEventTimelineEntry, parameters: parameters);
}
/// <summary>
/// <see cref="DeadlineNames.AdvisoryCompletion"/> fired — the resolution window after a
/// completion dead-end expired without one of the normal completion paths terminating
@@ -511,6 +586,12 @@ namespace AutopilotMonitor.DecisionCore.Engine
var deadlineStillArmed = armedDeadline != null;
var hasAdvisoryAnchor = state.EspAdvisoryFailureRecordedUtc != null;
// Session 4910a5a5: once the failed category demonstrably recovered
// (TryResolveAdvisoryOnCategoryRecovery set the fact), the advisory variant's
// "a real ESP terminal failure does not un-happen" premise is disproven — the
// guards below that protect live enrollments (enforcement-progress re-arm,
// hello-never-observed promote) apply to it again.
var advisoryResolved = state.EspAdvisoryFailureResolvedUtc != null;
// Presence-only anchor (L5, delta review 2026-07-02). The esp-exit arming site only
// runs when AccountSetupEnteredUtc is already set, so both facts being present is
// the anchor; `deadlineStillArmed` below is what rejects stray timer fires. The old
@@ -635,8 +716,13 @@ namespace AutopilotMonitor.DecisionCore.Engine
// When user-phase enforcement demonstrably progressed since arming, the dead-end
// shape is disproven: re-arm a fresh window with updated baselines instead of
// failing a live enrollment mid-install. The advisory variant is exempt — its
// anchor is a real ESP terminal failure that a busy IME does not un-happen.
if (!hasAdvisoryAnchor && HasEnforcementProgressSinceArming(state, armedDeadline!))
// anchor is a real ESP terminal failure that a busy IME does not un-happen —
// UNLESS the failed category later resolved to success (session 4910a5a5:
// DeviceSetup/Apps failed, user retried, apps reached 10/10, DeviceSetup resolved
// — the failure DID un-happen, so a live AccountSetup must not be failed on the
// stale advisory). Convergent either way: each re-arm renews the baselines, so a
// session that stalls after the recovery still fails one window later.
if ((!hasAdvisoryAnchor || advisoryResolved) && HasEnforcementProgressSinceArming(state, armedDeadline!))
{
var rearmedDeadline = BuildAdvisoryCompletionDeadline(state, signal);
builder.AddDeadline(rearmedDeadline);
@@ -692,7 +778,7 @@ namespace AutopilotMonitor.DecisionCore.Engine
// through Finalizing. Deliberately does NOT require the IME user-session gate —
// the FP sessions never saw an IME user-session completion (that gate guards
// completion evidence, not user presence; presence is the DAD-validated desktop).
if (!hasAdvisoryAnchor
if ((!hasAdvisoryAnchor || advisoryResolved)
&& desktopArrived
&& state.HelloResolvedUtc == null
&& state.HelloPolicyEnabled == null
@@ -741,7 +827,30 @@ namespace AutopilotMonitor.DecisionCore.Engine
// stays the deadline, not EspTerminalFailure).
string reason;
Dictionary<string, string> parameters;
if (hasAdvisoryAnchor)
if (hasAdvisoryAnchor && advisoryResolved)
{
// Session 4910a5a5: the original ESP failure demonstrably recovered (failed
// category resolved to success after the advisory), so claiming
// `esp_terminal_failure` would blame a failure that un-happened. The device
// recovered its device phase but then produced no completion evidence for a
// whole (re-armed, progress-gated) window. LastFailureTrigger stays the
// deadline so the EnrollmentTerminationHandler's likely-stuck app promotion
// stays OFF — the apps are not stuck, they succeeded.
reason = "esp_recovered_without_completion_evidence";
parameters = BuildEspFailureParameters(
signal: signal,
reason: reason,
advisoryPath: false,
observations: state.ScenarioObservations);
parameters["advisoryReason"] = "advisory_recovered_window_expired_without_completion_evidence";
parameters["failureRecovered"] = "true";
if (state.EspAdvisoryFailureCategory != null)
{
parameters["category"] = state.EspAdvisoryFailureCategory.Value;
}
builder.WithLastFailureTrigger(nameof(DecisionSignalKind.DeadlineFired), signal.SessionSignalOrdinal);
}
else if (hasAdvisoryAnchor)
{
// Registry-derived failure context (failureType/errorCode/...) lived on the
// original signal and is already on the wire via the esp_failure_advisory
@@ -885,6 +994,12 @@ namespace AutopilotMonitor.DecisionCore.Engine
parameters["failedSubcategory"] = failedSubcategory;
if (signal.Payload.TryGetValue("category", out var category) && !string.IsNullOrEmpty(category))
parameters["category"] = category;
// Session 4910a5a5 — tracker-snapshotted culprit-app names for Apps failures,
// so the advisory/terminal event tells WHICH app ESP most likely failed on.
if (signal.Payload.TryGetValue("likelyCulpritApps", out var culpritApps) && !string.IsNullOrEmpty(culpritApps))
parameters["likelyCulpritApps"] = culpritApps;
if (signal.Payload.TryGetValue("likelyCulpritAppCount", out var culpritCount) && !string.IsNullOrEmpty(culpritCount))
parameters["likelyCulpritAppCount"] = culpritCount;
}
// FirstSync-derived ESP failure-handling settings — observable hint that
@@ -98,17 +98,28 @@ namespace AutopilotMonitor.DecisionCore.Engine
// was already armed (or already cancelled by AccountSetup). Pass-through.
if (state.DeviceSetupResolvedUtc != null)
{
var passthroughState = state.ToBuilder()
var passthroughBuilder = state.ToBuilder()
.WithStepIndex(nextStep)
.WithLastAppliedSignalOrdinal(signal.SessionSignalOrdinal)
.Build();
.WithLastAppliedSignalOrdinal(signal.SessionSignalOrdinal);
// Session 4910a5a5 recovery hook: a duplicate arrival (e.g. the post-reboot
// registry re-read) can still be the first evidence that the advisory's failed
// category recovered — set-once + category-match inside the helper keep this
// a no-op everywhere else.
var passthroughResolveEffect = TryResolveAdvisoryOnCategoryRecovery(
state, passthroughBuilder, signal, resolvedCategory: "DeviceSetup");
var passthroughState = passthroughBuilder.Build();
var passthroughTransition = BuildTakenTransition(
before: state,
signal: signal,
toStage: state.Stage,
nextStepIndex: nextStep,
trigger: nameof(DecisionSignalKind.DeviceSetupProvisioningComplete) + ":AnchorAlreadySet");
return new DecisionStep(passthroughState, passthroughTransition, Array.Empty<DecisionEffect>());
return new DecisionStep(
passthroughState,
passthroughTransition,
passthroughResolveEffect != null
? new[] { passthroughResolveEffect }
: Array.Empty<DecisionEffect>());
}
// Step 2 — Set the DeviceSetupResolvedUtc anchor UNCONDITIONALLY. Even when Classic
@@ -119,6 +130,13 @@ namespace AutopilotMonitor.DecisionCore.Engine
.WithLastAppliedSignalOrdinal(signal.SessionSignalOrdinal);
builder.DeviceSetupResolvedUtc = new SignalFact<DateTime>(signal.OccurredAtUtc, signal.SessionSignalOrdinal);
// Session 4910a5a5 recovery hook: DeviceSetup resolving to success while a
// DeviceSetup advisory failure is on record means the failure un-happened (user
// "Try again" retry). Sets the set-once resolved fact + emits the
// esp_failure_advisory_resolved story event; no-op otherwise.
var advisoryResolveEffect = TryResolveAdvisoryOnCategoryRecovery(
state, builder, signal, resolvedCategory: "DeviceSetup");
// Step 3 — Classic-path short-circuit: AccountSetup already started. The DeviceOnly
// deadline is moot (already cancelled by HandleEspPhaseChangedV1's AccountSetup branch,
// or never armed under the new code). No deadline arm here.
@@ -136,7 +154,12 @@ namespace AutopilotMonitor.DecisionCore.Engine
toStage: state.Stage,
nextStepIndex: nextStep,
trigger: nameof(DecisionSignalKind.DeviceSetupProvisioningComplete) + ":AccountSetupAlreadyEntered");
return new DecisionStep(classicState, classicTransition, Array.Empty<DecisionEffect>());
return new DecisionStep(
classicState,
classicTransition,
advisoryResolveEffect != null
? new[] { advisoryResolveEffect }
: Array.Empty<DecisionEffect>());
}
// Step 4 — Arm the deadline. Cancel any existing DeviceOnlyEspDetection first
@@ -156,13 +179,14 @@ namespace AutopilotMonitor.DecisionCore.Engine
nextStepIndex: nextStep,
trigger: nameof(DecisionSignalKind.DeviceSetupProvisioningComplete) + ":DeadlineArmed");
var effects = new[]
var effects = new List<DecisionEffect>(3)
{
new DecisionEffect(DecisionEffectKind.CancelDeadline, cancelDeadlineName: DeadlineNames.DeviceOnlyEspDetection),
new DecisionEffect(DecisionEffectKind.ScheduleDeadline, deadline: deadline),
};
if (advisoryResolveEffect != null) effects.Add(advisoryResolveEffect);
return new DecisionStep(newState, transition, effects);
return new DecisionStep(newState, transition, effects.ToArray());
}
/// <summary>
@@ -836,6 +836,13 @@ namespace AutopilotMonitor.DecisionCore.Engine
.WithStepIndex(nextStep)
.WithLastAppliedSignalOrdinal(signal.SessionSignalOrdinal);
// Session 4910a5a5 recovery hook (AccountSetup counterpart): the user-phase
// category resolving to success while an AccountSetup advisory failure is on
// record means that failure un-happened. Sets the set-once resolved fact + emits
// the esp_failure_advisory_resolved story event; no-op otherwise.
var advisoryResolveEffect = TryResolveAdvisoryOnCategoryRecovery(
state, builder, signal, resolvedCategory: "AccountSetup");
var alreadyRecorded = state.AccountSetupProvisioningSucceededUtc != null;
if (!alreadyRecorded)
{
@@ -876,10 +883,10 @@ namespace AutopilotMonitor.DecisionCore.Engine
toStage: state.Stage,
nextStepIndex: nextStep,
trigger: nameof(DecisionSignalKind.AccountSetupProvisioningComplete));
return new DecisionStep(
newState,
transition,
waitingEffect != null ? new[] { waitingEffect } : Array.Empty<DecisionEffect>());
var noPromoteEffects = new List<DecisionEffect>(2);
if (advisoryResolveEffect != null) noPromoteEffects.Add(advisoryResolveEffect);
if (waitingEffect != null) noPromoteEffects.Add(waitingEffect);
return new DecisionStep(newState, transition, noPromoteEffects.ToArray());
}
// Deferred-completion parity (session caa6cf50 fix, 2026-06-11): when the strong gate
@@ -910,14 +917,18 @@ namespace AutopilotMonitor.DecisionCore.Engine
builder.CancelDeadline(DeadlineNames.HelloSafety);
}
var deferredLeadingEffects = new List<DecisionEffect>(2);
if (advisoryResolveEffect != null) deferredLeadingEffects.Add(advisoryResolveEffect);
if (helloSafetyCancelEffect != null) deferredLeadingEffects.Add(helloSafetyCancelEffect);
return CompleteThroughFinalizingOrDefer(
state: state,
signal: signal,
preparedBuilder: builder,
nextStepIndex: nextStep,
trigger: nameof(DecisionSignalKind.AccountSetupProvisioningComplete) + ":DeferredCompletion",
leadingEffects: helloSafetyCancelEffect != null
? new[] { helloSafetyCancelEffect }
leadingEffects: deferredLeadingEffects.Count > 0
? deferredLeadingEffects.ToArray()
: null);
}
@@ -946,12 +957,13 @@ namespace AutopilotMonitor.DecisionCore.Engine
nextStepIndex: nextStep,
trigger: nameof(DecisionSignalKind.AccountSetupProvisioningComplete) + ":DeferredPromote");
var promotedEffects = new[]
var promotedEffects = new List<DecisionEffect>(2)
{
new DecisionEffect(DecisionEffectKind.ScheduleDeadline, deadline: helloSafety),
};
if (advisoryResolveEffect != null) promotedEffects.Add(advisoryResolveEffect);
return new DecisionStep(promotedState, promotedTransition, promotedEffects);
return new DecisionStep(promotedState, promotedTransition, promotedEffects.ToArray());
}
/// <summary>
@@ -110,6 +110,17 @@ namespace AutopilotMonitor.DecisionCore.Engine
timestamps["imeUserSessionCompleted"] = FormatUtc(state.ImeUserSessionCompletedUtc.Value);
evidence["imeUserSessionCompleted"] = TimestampedEvidence(state.ImeUserSessionCompletedUtc);
}
// Session 4910a5a5 — the ESP category behind a defanged advisory failure later
// resolved to success (user "Try again" recovery). Surfaced so terminal audit
// trails show that the original failure demonstrably un-happened — the
// discriminant between "failed and stayed failed" and "failed, recovered, but
// produced no completion evidence".
if (state.EspAdvisoryFailureResolvedUtc != null)
{
seen.Add("esp_advisory_failure_resolved");
timestamps["espAdvisoryFailureResolved"] = FormatUtc(state.EspAdvisoryFailureResolvedUtc.Value);
evidence["espAdvisoryFailureResolved"] = TimestampedEvidence(state.EspAdvisoryFailureResolvedUtc);
}
// Session 772fe502 — genuine Hello-wizard launch (Shell-Core 62404, CXID AADHello/
// NGC). Surfaced in the census so terminal audit trails show whether a wizard was
// observed — the discriminant between "Hello skipped by policy" and "Hello skipped
@@ -63,6 +63,8 @@ namespace AutopilotMonitor.DecisionCore.Engine
["helloPolicyEnabled"] = SerializeFact(state.HelloPolicyEnabled, v => (object)v),
["lastFailureTrigger"] = SerializeFact(state.LastFailureTrigger, v => v),
["espAdvisoryFailureRecordedUtc"] = SerializeFact(state.EspAdvisoryFailureRecordedUtc, FormatUtc),
["espAdvisoryFailureResolvedUtc"] = SerializeFact(state.EspAdvisoryFailureResolvedUtc, FormatUtc),
["espAdvisoryFailureCategory"] = SerializeFact(state.EspAdvisoryFailureCategory, v => v),
["imeUserSessionCompletedUtc"] = SerializeFact(state.ImeUserSessionCompletedUtc, FormatUtc),
["completionWaitingFingerprint"] = SerializeFact(state.CompletionWaitingFingerprint, v => v),
["helloWizardStartedUtc"] = SerializeFact(state.HelloWizardStartedUtc, FormatUtc),
@@ -64,7 +64,9 @@ namespace AutopilotMonitor.DecisionCore.State
SignalFact<DateTime>? espAdvisoryFailureRecordedUtc = null,
SignalFact<DateTime>? imeUserSessionCompletedUtc = null,
SignalFact<string>? completionWaitingFingerprint = null,
SignalFact<DateTime>? helloWizardStartedUtc = null)
SignalFact<DateTime>? helloWizardStartedUtc = null,
SignalFact<DateTime>? espAdvisoryFailureResolvedUtc = null,
SignalFact<string>? espAdvisoryFailureCategory = null)
{
if (string.IsNullOrEmpty(sessionId))
{
@@ -108,6 +110,8 @@ namespace AutopilotMonitor.DecisionCore.State
ImeUserSessionCompletedUtc = imeUserSessionCompletedUtc;
CompletionWaitingFingerprint = completionWaitingFingerprint;
HelloWizardStartedUtc = helloWizardStartedUtc;
EspAdvisoryFailureResolvedUtc = espAdvisoryFailureResolvedUtc;
EspAdvisoryFailureCategory = espAdvisoryFailureCategory;
}
public string SessionId { get; }
@@ -320,6 +324,36 @@ namespace AutopilotMonitor.DecisionCore.State
/// </summary>
public SignalFact<DateTime>? HelloWizardStartedUtc { get; }
/// <summary>
/// Set when the ESP category behind a recorded advisory failure
/// (<see cref="EspAdvisoryFailureRecordedUtc"/>) later resolves to success — i.e. the
/// failure demonstrably un-happened. Session 4910a5a5 (2026-07-23): DeviceSetup/Apps
/// failed, the user pressed "Try again" ~23 min later, the apps re-ran to 10/10 and
/// <c>DeviceSetupProvisioningComplete</c> arrived — yet the <c>AdvisoryCompletion</c>
/// window still un-defanged the stale failure and failed the live enrollment.
/// <para>
/// Set-once by the shared recovery hook in the ProvisioningComplete handlers when the
/// resolved category matches the <c>armFailedCategory</c> carried on the armed
/// <c>AdvisoryCompletion</c> deadline. Once set, the advisory variant loses its
/// exemption from the enforcement-progress re-arm, the reboot rebase and the Hello
/// promotes — those guards exist to protect live enrollments, and with this fact the
/// "a real ESP terminal failure does not un-happen" premise is disproven for this
/// session. Additive-nullable, no snapshot-schema bump.
/// </para>
/// </summary>
public SignalFact<DateTime>? EspAdvisoryFailureResolvedUtc { get; }
/// <summary>
/// The ESP category (<c>DeviceSetup</c> / <c>AccountSetup</c>) named by the advisory
/// failure's signal payload, recorded alongside <see cref="EspAdvisoryFailureRecordedUtc"/>.
/// Lives on state — not on the <c>AdvisoryCompletion</c> deadline payload — because
/// re-arm and reboot-rebase sites rebuild that deadline and would silently drop a
/// payload-carried category. Null when the arming signal carried no category (e.g.
/// event-log-derived failures); the recovery hook then never matches and behavior is
/// unchanged. Additive-nullable, no snapshot-schema bump.
/// </summary>
public SignalFact<string>? EspAdvisoryFailureCategory { get; }
public string SchemaVersion { get; }
/// <summary>
@@ -51,6 +51,8 @@ namespace AutopilotMonitor.DecisionCore.State
ImeUserSessionCompletedUtc = source.ImeUserSessionCompletedUtc;
CompletionWaitingFingerprint = source.CompletionWaitingFingerprint;
HelloWizardStartedUtc = source.HelloWizardStartedUtc;
EspAdvisoryFailureResolvedUtc = source.EspAdvisoryFailureResolvedUtc;
EspAdvisoryFailureCategory = source.EspAdvisoryFailureCategory;
}
public string SessionId { get; set; }
@@ -84,6 +86,8 @@ namespace AutopilotMonitor.DecisionCore.State
public SignalFact<DateTime>? ImeUserSessionCompletedUtc { get; set; }
public SignalFact<string>? CompletionWaitingFingerprint { get; set; }
public SignalFact<DateTime>? HelloWizardStartedUtc { get; set; }
public SignalFact<DateTime>? EspAdvisoryFailureResolvedUtc { get; set; }
public SignalFact<string>? EspAdvisoryFailureCategory { get; set; }
public string SchemaVersion { get; set; }
// ---------- fluent helpers for the most common reducer operations ----------
@@ -197,9 +201,25 @@ namespace AutopilotMonitor.DecisionCore.State
/// Fire-once gate: subsequent <c>EspTerminalFailure</c> signals see this fact set and
/// dead-end without further effect.
/// </summary>
public DecisionStateBuilder WithEspAdvisoryFailureRecorded(DateTime utc, long sourceSignalOrdinal)
public DecisionStateBuilder WithEspAdvisoryFailureRecorded(
DateTime utc, long sourceSignalOrdinal, string? failedCategory = null)
{
EspAdvisoryFailureRecordedUtc = new SignalFact<DateTime>(utc, sourceSignalOrdinal);
if (!string.IsNullOrEmpty(failedCategory))
{
EspAdvisoryFailureCategory = new SignalFact<string>(failedCategory!, sourceSignalOrdinal);
}
return this;
}
/// <summary>
/// Record that the ESP category behind the recorded advisory failure later resolved to
/// success (session 4910a5a5 — "Try again" recovery). Set-once; lifts the advisory
/// variant's exemption from the AdvisoryCompletion re-arm/rebase guards.
/// </summary>
public DecisionStateBuilder WithEspAdvisoryFailureResolved(DateTime utc, long sourceSignalOrdinal)
{
EspAdvisoryFailureResolvedUtc = new SignalFact<DateTime>(utc, sourceSignalOrdinal);
return this;
}
@@ -236,6 +256,8 @@ namespace AutopilotMonitor.DecisionCore.State
espAdvisoryFailureRecordedUtc: EspAdvisoryFailureRecordedUtc,
imeUserSessionCompletedUtc: ImeUserSessionCompletedUtc,
completionWaitingFingerprint: CompletionWaitingFingerprint,
helloWizardStartedUtc: HelloWizardStartedUtc);
helloWizardStartedUtc: HelloWizardStartedUtc,
espAdvisoryFailureResolvedUtc: EspAdvisoryFailureResolvedUtc,
espAdvisoryFailureCategory: EspAdvisoryFailureCategory);
}
}
@@ -368,6 +368,7 @@ namespace AutopilotMonitor.Shared
public const string TelemetryUploadBlocked = "telemetry_upload_blocked"; // Warning (one-shot) when the drain is blocked by a retained unknown permanent 4xx (400 contract bug / 404 route mismatch / tenant-validation). Data is NOT discarded. LOCAL diagnostic (agent log + diag ZIP): this marker is queued behind the blocking batch and only reaches the backend after the block clears — no out-of-band bypass by design.
public const string SessionParkedWithoutDeadline = "session_parked_without_deadline"; // One-shot Warning tripwire (liveness plan PR1): a non-terminal session entered the post-AccountSetup dead-end zone (ESP final exit or advisory failure) with NO resolution-capable deadline armed. After the 2026-06-12 dead-end fixes this must never fire in production — every occurrence is a bug report for an unknown parking variant.
public const string DiskSpaceLow = "disk_space_low"; // One-shot Warning (state-change-only, 1 GB hysteresis — NOT periodic) when the system drive drops below 2 GB free. Review MON-B10: disk-full was previously only visible inside Debug performance_snapshot payloads (filtered out of timelines), so a disk-starved enrollment carried no actionable threshold signal.
public const string EspPolicyProviderStalled = "esp_policy_provider_stalled"; // One-shot-per-provider Warning: a registered EnrollmentStatusTracking CSP policy provider stayed incomplete >= 15 min (Setup/Apps TrackingPoliciesCreated != 1, or DevicePreparation InstallationState not in {2,3}). The ESP waits on Setup/Apps providers WITHOUT any OS timeout — e.g. pre-provisioning + co-management leaves only "ConfigMgr" registered (no "Sidecar" = IME) and the user ESP sits at "Apps (Identifying)" for days. Observational only; payload carries the full provider list. See EspPolicyProviderProbe.
public const string KeepAwakeEngaged = "keep_awake_engaged"; // Info (state-change, NOT periodic) emitted by the opt-in UserEspKeepAwake host when it holds the device awake (system + display) on entering the User-ESP/AccountSetup phase, so idle standby cannot stall app installs / account provisioning. Reboots are unaffected. Per-tenant toggle (AnalyzerConfiguration.KeepAwakeDuringUserEsp, default off).
public const string KeepAwakeReleased = "keep_awake_released"; // Info (Warning when reason=safety_cap) emitted when the keep-awake hold is released — reason = account_setup_complete | host_stop | safety_cap. Brackets KeepAwakeEngaged; the OS also auto-clears the hold on process exit/reboot.
@@ -538,6 +539,17 @@ namespace AutopilotMonitor.Shared
public const string EspFailureSettleRecovered = "esp_failure_settle_recovered"; // The failed subcategory left the "failed" state during the settle window (e.g. ESP "Try again" retry, session c071e92b) — the terminal EspFailureDetected fire is suppressed and monitoring continues; a subsequent failure re-arms a fresh settle window.
public const string EspAppxFailureAnalysis = "esp_appx_failure_analysis"; // One-shot AppXDeploymentServer/Operational scan during the ESP failure settle window: candidate MSIX/Store packages behind an "Apps (0x…)" subcategory failure invisible to ImeLogTracker (session 2bc884b6). Assessment, not a confirmed root cause.
public const string EspProvisioningRaw = "esp_provisioning_raw";
// Recovery story AFTER a terminally fired ESP failure (session 4910a5a5, 2026-07-23 —
// DeviceSetup/Apps failed, user pressed "Try again" ~23 min later, apps re-ran to
// 10/10). Distinct from EspFailureSettleRecovered, which covers recovery INSIDE the
// 30s settle window (terminal fire suppressed); these fire when the failure already
// went terminal (advisory/enrollment_failed) and the registry later disproves it.
public const string EspFailureRetryDetected = "esp_failure_retry_detected"; // failed subcategory left "failed" post-terminal — consistent with the user pressing "Try again"
public const string EspFailureRecovered = "esp_failure_recovered"; // previously failed category reached all-subcategories-succeeded post-terminal
// DecisionEngine counterpart (survives reboots via snapshot, unlike the in-run tracker
// pair above): the category behind a defanged esp_failure_advisory later resolved to
// success — the advisory no longer blocks; re-arm/rebase guards apply again.
public const string EspFailureAdvisoryResolved = "esp_failure_advisory_resolved";
public const string EspFailure = "esp_failure";
public const string EspExiting = "esp_exiting";
@@ -58,6 +58,8 @@ export const KNOWN_EVENT_TYPES: EventTypeEntry[] = [
description: "Stall probe found an anomaly (ModernDeployment error, DeploymentErrorCode≠0, IME EnforcementState Failed)." },
{ value: "stall_probe_check", label: "stall_probe_check", category: "stall",
description: "Trace-level heartbeat from a stall probe — proves the logic ran (default: only the 15 min probe)." },
{ value: "esp_policy_provider_stalled", label: "esp_policy_provider_stalled", category: "stall",
description: "A registered ESP policy provider (EnrollmentStatusTracking CSP) stayed incomplete for 15+ minutes — e.g. co-management leaves only ConfigMgr registered without the Intune Sidecar provider, which parks the user ESP at 'Apps (Identifying)' with no OS timeout. One-shot Warning; payload carries the full provider table, per-provider state, and sidecarRegistered." },
{ value: "agent_late_start", label: "agent_late_start", category: "stall",
description: "Low observation coverage: the agent started long after device boot (≥10 min) AND lived only briefly (≤5 min) before a terminal outcome — it arrived after the enrollment had already decided, so its diagnosis is a post-mortem of the end-state, not a live observation. Carries bootToAgentStartSeconds, agentUptimeSeconds and outcome. Often paired with script_timeout_suspected (a script that hung ahead of the agent bootstrap)." },
@@ -96,6 +98,12 @@ export const KNOWN_EVENT_TYPES: EventTypeEntry[] = [
description: "ESP reported a subcategory failure but the device had already progressed to AccountSetup with ContinueAnyway enabled — non-terminal advisory, the agent continues monitoring." },
{ value: "esp_appx_failure_analysis", label: "esp_appx_failure_analysis", category: "esp",
description: "AppX deployment log scan during the ESP failure settle window: suspected MSIX/Store package candidates behind an Apps-subcategory failure (assessment, not a confirmed root cause)." },
{ value: "esp_failure_retry_detected", label: "esp_failure_retry_detected", category: "esp",
description: "After a terminally reported ESP failure, the failed subcategory left the failed state — consistent with the user pressing 'Try again' on the ESP failure page." },
{ value: "esp_failure_recovered", label: "esp_failure_recovered", category: "esp",
description: "The ESP category behind a terminally reported failure later completed successfully — the earlier failure recovered (e.g. after a user retry)." },
{ value: "esp_failure_advisory_resolved", label: "esp_failure_advisory_resolved", category: "esp",
description: "DecisionEngine: the category behind a defanged esp_failure_advisory resolved to success — the advisory no longer blocks completion." },
// -------- App installs --------
{ value: "app_install_started", label: "app_install_started", category: "app",
@@ -521,6 +521,46 @@ function EventRow({ event, showScriptOutput }: { event: EnrollmentEvent; showScr
subcategory: <span className="font-mono">{String(sub)}</span>
</span>
)}
{event.data?.likelyCulpritApps && (
<span
className="px-1.5 py-0.5 rounded bg-orange-100 text-orange-800 font-medium"
title="Tracked app(s) ESP most likely failed on — never-started apps ranked first (snapshot at failure time)."
>
Likely app: {String(event.data.likelyCulpritApps)}
</span>
)}
</div>
);
})()}
{/* Recovery-story badges (session 4910a5a5): a terminally reported ESP failure that
later un-happened — the user retried ("Try again"), the failed step re-ran and
recovered. Green/teal palette so the timeline visually closes the earlier red/amber
failure arc. */}
{(event.eventType === "esp_failure_retry_detected" || event.eventType === "esp_failure_recovered" || event.eventType === "esp_failure_advisory_resolved") && (() => {
const isRetry = event.eventType === "esp_failure_retry_detected";
const label = isRetry ? "User retry" : "Recovered";
const badgeCls = isRetry ? "bg-sky-100 text-sky-800" : "bg-emerald-100 text-emerald-800";
const cat = event.data?.category;
const sub = event.data?.subcategory ?? event.data?.failedSubcategory;
const mins = event.data?.minutesSinceFailure ?? event.data?.minutesSinceAdvisory;
return (
<div className="mt-1 flex flex-wrap items-center gap-2 text-xs">
<span
className={`px-1.5 py-0.5 rounded ${badgeCls} font-medium`}
title={isRetry
? "The failed ESP step left the failed state — consistent with the user pressing 'Try again' on the ESP failure page."
: "The previously failed ESP category completed successfully — the earlier failure no longer applies."}
>
{label}
</span>
{cat && (
<span className="text-gray-500">
{String(cat)}{sub ? <> / <span className="font-mono">{String(sub)}</span></> : null}
</span>
)}
{mins !== undefined && mins !== null && (
<span className="text-gray-500">{String(mins)} min after the failure</span>
)}
</div>
);
})()}
@@ -3,7 +3,7 @@
import { useMemo } from "react";
import { EnrollmentEvent, Session } from "@/types";
import { V1_PHASE_NAMES, V2_PHASE_NAMES, V1_PHASE_ORDER, V2_PHASE_ORDER } from "../utils/phaseConstants";
import { computeWhiteGloveSplitSequence, groupEventsByPhase } from "../utils/eventHelpers";
import { computeWhiteGloveDurations, computeWhiteGloveSplitSequence, groupEventsByPhase } from "../utils/eventHelpers";
interface PhaseGrouping {
eventsByPhase: Record<string, EnrollmentEvent[]>;
@@ -176,33 +176,8 @@ export function useSessionDerivedData(
if (!isWhiteGloveSession) {
return { preProvDuration: null as string | null, userEnrollDuration: null as string | null, combinedDuration: null as string | null };
}
const preProvEvts = whiteGloveSplitSequence < 0 ? events : events.filter(e => e.sequence <= whiteGloveSplitSequence);
const userEnrollEvts = whiteGloveSplitSequence < 0 ? [] : events.filter(e => e.sequence > whiteGloveSplitSequence);
const calcMs = (evts: EnrollmentEvent[]): number => {
if (evts.length === 0) return 0;
const ts = evts.map(e => new Date(e.timestamp).getTime());
return Math.max(...ts) - Math.min(...ts);
};
const fmt = (ms: number): string | null => {
const sec = Math.round(ms / 1000);
if (sec < 1) return null;
if (sec < 60) return `${sec}s`;
if (sec < 3600) return `${Math.floor(sec / 60)}m ${sec % 60}s`;
return `${Math.floor(sec / 3600)}h ${Math.floor((sec % 3600) / 60)}m`;
};
const preProvMs = calcMs(preProvEvts);
const userEnrollMs = calcMs(userEnrollEvts);
return {
preProvDuration: fmt(preProvMs),
userEnrollDuration: fmt(userEnrollMs),
combinedDuration: fmt(preProvMs + userEnrollMs),
};
}, [events, isWhiteGloveSession, whiteGloveSplitSequence]);
return computeWhiteGloveDurations(events, whiteGloveSplitSequence, session?.startedAt);
}, [events, isWhiteGloveSession, whiteGloveSplitSequence, session?.startedAt]);
// Group events by phase — single timeline for normal sessions, two groups for WhiteGlove
const { eventsByPhase, orderedPhases } = useMemo(() => {
@@ -66,6 +66,69 @@ export function computeWhiteGloveSplitSequence(events: EnrollmentEvent[]): numbe
return -1;
}
// Pure helper — per-block durations for WhiteGlove sessions.
// Duration 1 = pre-provisioning, Duration 2 = user enrollment, combined = D1 + D2
// (the sealed pause between reseal and resume is excluded).
//
// Block spans are anchored on the SEQUENCE-canonical edge events (first/last event by
// sequence), NOT on min/max over all timestamps in the block. Two real-world cases break
// the min/max approach:
// - Events re-emitted after the Part-2 resume can legitimately carry historical event
// times (e.g. an app_install_started the ImeLogTracker re-parses from the IME log,
// stamped with the Part-1 install start). min() would stretch the User Enrollment
// block back across the sealed pause.
// - A single clock-skewed source timestamp (e.g. an IME log line with a wrong timezone
// bias that slips under the agent's 1h future-skew clamp) would stretch the block
// the other way.
// Sequence is the canonical event order, so the block edges by sequence are the robust
// span anchors. Part 1 additionally starts at session.startedAt (registration) when
// provided — the same anchor the backend uses for DurationSeconds.
export interface WhiteGloveDurations {
preProvDuration: string | null;
userEnrollDuration: string | null;
combinedDuration: string | null;
}
export function computeWhiteGloveDurations(
events: EnrollmentEvent[],
splitSequence: number,
startedAt?: string,
): WhiteGloveDurations {
const preProvEvts = splitSequence < 0 ? events : events.filter(e => e.sequence <= splitSequence);
const userEnrollEvts = splitSequence < 0 ? [] : events.filter(e => e.sequence > splitSequence);
const spanMs = (evts: EnrollmentEvent[], startOverrideMs?: number): number => {
if (evts.length === 0) return 0;
let first = evts[0];
let last = evts[0];
for (const e of evts) {
if (e.sequence < first.sequence) first = e;
if (e.sequence > last.sequence) last = e;
}
const start = startOverrideMs ?? new Date(first.timestamp).getTime();
const end = new Date(last.timestamp).getTime();
return Math.max(0, end - start);
};
const fmt = (ms: number): string | null => {
const sec = Math.round(ms / 1000);
if (sec < 1) return null;
if (sec < 60) return `${sec}s`;
if (sec < 3600) return `${Math.floor(sec / 60)}m ${sec % 60}s`;
return `${Math.floor(sec / 3600)}h ${Math.floor((sec % 3600) / 60)}m`;
};
const startedAtMs = startedAt ? new Date(startedAt).getTime() : NaN;
const preProvMs = spanMs(preProvEvts, Number.isFinite(startedAtMs) ? startedAtMs : undefined);
const userEnrollMs = spanMs(userEnrollEvts);
return {
preProvDuration: fmt(preProvMs),
userEnrollDuration: fmt(userEnrollMs),
combinedDuration: fmt(preProvMs + userEnrollMs),
};
}
// Pure helper — groups a flat event list into phase buckets.
// Extracted from the useMemo so it can be called multiple times for WhiteGlove split timelines.
//
@@ -0,0 +1,103 @@
import { describe, it, expect } from "vitest";
import { computeWhiteGloveDurations } from "../../app/sessions/[sessionId]/utils/eventHelpers";
import type { EnrollmentEvent } from "@/types";
// Fixture builder — sequence and timestamp are the only fields that matter here.
function ev(seq: number, eventType: string, isoTs: string): EnrollmentEvent {
return {
eventId: `e${seq}`,
sessionId: "s1",
timestamp: isoTs,
eventType,
severity: "Info",
source: "Agent",
phase: 0,
message: eventType,
sequence: seq,
};
}
describe("computeWhiteGloveDurations", () => {
it("computes both blocks from the sequence-canonical edge events", () => {
const events = [
ev(1, "agent_started", "2026-07-23T08:31:00Z"),
ev(2, "esp_phase_changed", "2026-07-23T08:35:00Z"),
ev(3, "agent_shutting_down", "2026-07-23T09:28:00Z"), // Part 1 ends (reseal)
ev(4, "agent_started", "2026-07-23T11:00:00Z"), // Part 2 boot after sealed pause
ev(5, "enrollment_complete", "2026-07-23T11:56:00Z"),
];
const d = computeWhiteGloveDurations(events, 3);
expect(d.preProvDuration).toBe("57m 0s");
expect(d.userEnrollDuration).toBe("56m 0s");
expect(d.combinedDuration).toBe("1h 53m"); // pause 09:28→11:00 excluded
});
it("ignores a historical event time re-emitted into the user block (post-resume IME re-parse)", () => {
// Real case (session fcb44595): after the Part-2 resume the ImeLogTracker re-emits an
// app_install_started carrying the ORIGINAL Part-1 install time. min/max over timestamps
// stretched the user block back across the sealed pause (56m became 3h18m).
const events = [
ev(1, "agent_started", "2026-07-23T08:31:00Z"),
ev(2, "agent_shutting_down", "2026-07-23T09:28:00Z"),
ev(3, "agent_started", "2026-07-23T11:00:00Z"),
ev(4, "app_install_started", "2026-07-23T08:37:57Z"), // historical event time, Part-2 sequence
ev(5, "enrollment_complete", "2026-07-23T11:56:00Z"),
];
const d = computeWhiteGloveDurations(events, 2);
expect(d.userEnrollDuration).toBe("56m 0s");
expect(d.combinedDuration).toBe("1h 53m");
});
it("ignores a clock-skewed outlier timestamp inside the pre-provisioning block", () => {
// Real case (session fcb44595): a do_telemetry event carried a +1h source timestamp
// (IME log line with a wrong timezone bias, just under the agent's future-skew clamp).
// min/max stretched Part 1 from 57m to 1h44m.
const events = [
ev(1, "agent_started", "2026-07-23T08:31:00Z"),
ev(2, "do_telemetry", "2026-07-23T10:13:42Z"), // +1h skewed, mid-block by sequence
ev(3, "agent_shutting_down", "2026-07-23T09:28:00Z"),
ev(4, "agent_started", "2026-07-23T11:00:00Z"),
ev(5, "enrollment_complete", "2026-07-23T11:56:00Z"),
];
const d = computeWhiteGloveDurations(events, 3);
expect(d.preProvDuration).toBe("57m 0s");
});
it("anchors Part 1 at session.startedAt when provided (backend DurationSeconds parity)", () => {
const events = [
ev(1, "agent_started", "2026-07-23T08:31:00Z"),
ev(2, "agent_shutting_down", "2026-07-23T09:28:00Z"),
ev(3, "agent_started", "2026-07-23T11:00:00Z"),
ev(4, "enrollment_complete", "2026-07-23T11:56:00Z"),
];
const d = computeWhiteGloveDurations(events, 2, "2026-07-23T08:29:41Z");
expect(d.preProvDuration).toBe("58m 19s"); // 08:29:41 → 09:28:00
});
it("treats everything as Part 1 while no split point exists yet", () => {
const events = [
ev(1, "agent_started", "2026-07-23T08:31:00Z"),
ev(2, "esp_phase_changed", "2026-07-23T08:45:00Z"),
];
const d = computeWhiteGloveDurations(events, -1);
expect(d.preProvDuration).toBe("14m 0s");
expect(d.userEnrollDuration).toBeNull();
expect(d.combinedDuration).toBe("14m 0s");
});
it("returns nulls for an empty event list", () => {
const d = computeWhiteGloveDurations([], -1);
expect(d.preProvDuration).toBeNull();
expect(d.userEnrollDuration).toBeNull();
expect(d.combinedDuration).toBeNull();
});
it("never returns a negative span when edge timestamps are disordered", () => {
const events = [
ev(1, "agent_started", "2026-07-23T09:00:00Z"),
ev(2, "agent_shutting_down", "2026-07-23T08:59:00Z"), // straggler stamped before the boot
];
const d = computeWhiteGloveDurations(events, 2, "2026-07-23T09:30:00Z");
expect(d.preProvDuration).toBeNull(); // clamped to 0 → formatted as null
});
});
@@ -57,6 +57,7 @@ export const REGISTRY_PREFIX_CATEGORIES: readonly GuardrailCategory[] = [
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Setup",
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\OOBE",
"SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Winlogon",
"SOFTWARE\\Microsoft\\Windows\\Autopilot",
],
},
{
@@ -242,6 +243,7 @@ export const ALLOWED_REGISTRY_PREFIXES: readonly string[] = [
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Setup",
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\OOBE",
"SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Winlogon",
"SOFTWARE\\Microsoft\\Windows\\Autopilot",
"SYSTEM\\CurrentControlSet\\Services\\TPM",
"SOFTWARE\\Microsoft\\Tpm",
"SYSTEM\\CurrentControlSet\\Control\\SecureBoot",