diff --git a/docs/agent/esp-policy-provider-stall.md b/docs/agent/esp-policy-provider-stall.md index b1cd78d5..9aa87de7 100644 --- a/docs/agent/esp-policy-provider-stall.md +++ b/docs/agent/esp-policy-provider-stall.md @@ -1,7 +1,7 @@ --- type: Concept title: ESP Policy-Provider Stall Detection — the no-timeout Setup/Apps wait and the co-management Sidecar gap -description: Why a registered EnrollmentStatusTracking CSP policy provider that never sets TrackingPoliciesCreated parks the user ESP at "Apps (Identifying)" indefinitely, the pre-provisioning + ConfigMgr field case, and how the always-on esp_policy_provider_stalled tripwire (15-min wall-clock dwell, StartupEventGate cross-restart dedup) surfaces it. +description: Why the ESP's Apps wait is keyed to the Intune-registered "Sidecar" provider by name — a foreign provider (ConfigMgr) with TrackingPoliciesCreated=1 does not satisfy it — plus the classic never-completes stall, and how the always-on two-arm esp_policy_provider_stalled tripwire (reason=provider_incomplete | sidecar_provider_missing, 15-min wall-clock dwell, StartupEventGate cross-restart dedup) surfaces both. resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/EspPolicyProviderProbe.cs tags: - agent @@ -9,7 +9,7 @@ tags: - enrollment-status-tracking - co-management - stall-detection -timestamp: 2026-07-23T00:00:00+02:00 +timestamp: 2026-07-24T00:00:00+02:00 --- # ESP Policy-Provider Stall Detection @@ -42,37 +42,59 @@ exactly these registry paths. The same document states that ESP policy providers aware of each other** and that **pre-provisioning is unsupported** with Autopilot into co-management. -## The field case +## The field case (issue #106) A tenant pre-provisions devices and installs the ConfigMgr client as a blocking app. Occasionally the user-phase ESP reaches "Apps (Identifying)" and never leaves it — for hours to days. On those devices `Device\Setup\Apps\PolicyProviders` contains **only a -`ConfigMgr` key (no `Sidecar`)** and `TrackingPoliciesCreated` is never set. Manually -renaming the key to `Sidecar` unblocks the ESP immediately (the IME then finds its -provider node and reports tracking-policies-created). The mechanism is exactly the -documented no-timeout wait; the unsupported pre-provisioning + co-management combination -explains why only some devices are affected. Why the Sidecar registration is missing or -displaced on those devices is not publicly documented — irrelevant for detection, since -the signature is deterministic. +`ConfigMgr` key (no `Sidecar`)** — and, per the customer's follow-up, **`TrackingPoliciesCreated=1` +IS set under it**. The provider is "complete" by the CSP value contract, yet the ESP +still waits. Manually renaming the key to `Sidecar` — which creates no tracking entries, +only satisfies the name — unblocks the ESP instantly. + +The naive CSP-doc reading ("any provider with TrackingPoliciesCreated=1 satisfies the +wait") is therefore wrong for this case. What the evidence supports instead: provider +nodes are registered **by the Intune service at enrollment**, and its ESP-policy WAP +payload always registers `Sidecar` (ConfigMgr joins only via a co-management settings +policy — see the payload documented by Niehaus). The ESP waits on the provider it +expects **by name**; a foreign provider occupying the slot does not satisfy it, +regardless of its completion values. On the affected devices the CM client (installed +as a plain Win32 app, no co-management settings policy) evidently displaced the +Sidecar node — consistent with Microsoft's warning that providers "aren't currently +aware of others" and that pre-provisioning is unsupported with co-management. The exact +CloudExperienceHost mechanism is not publicly documented; the registry signature — +setupApps providers present, `Sidecar` absent — is deterministic either way. ## Detection `EspPolicyProviderProbe` (fail-soft, static, `ScopedOverride` test seam — structural clone of `EspTrackingInfoProbe`) enumerates all providers in both kinds and scopes. `EspPolicyProviderStallDetector` evaluates a pure transition function on a 60-second tick -from the always-on `EspPolicyProviderStallHost`: +from the always-on `EspPolicyProviderStallHost` — **two arms**, same 15-minute wall-clock +dwell, one shared event type discriminated by `reason`: -* A provider **continuously incomplete for ≥ 15 minutes** (wall-clock) is stalled. The - dwell clock starts at first incomplete observation and resets on completion or - disappearance. Missing values parse as incomplete (fail-toward-observation). +* **Arm 1, `reason=provider_incomplete`** — a provider **continuously incomplete for + ≥ 15 minutes**. The dwell clock starts at first incomplete observation and resets on + completion or disappearance. Missing values parse as incomplete + (fail-toward-observation). Catches e.g. Sidecar itself never reporting. +* **Arm 2, `reason=sidecar_provider_missing`** — **≥ 1 `Setup\Apps` provider registered + but none named `Sidecar`**, continuously for ≥ 15 minutes. Deliberately ignores + provider completeness (the field case had `TrackingPoliciesCreated=1`). Sidecar + registering later — the legitimate startup ordering — clears the condition and resets + the clock. DevicePreparation-only snapshots don't activate it. +* When both arms cross in the same round, ONE event is emitted and + `sidecar_provider_missing` wins as `reason` (it explains the hang regardless of the + foreign provider's state). * Missing root key / empty provider list = normal early enrollment: clears all dwell - state, never fires. -* One `esp_policy_provider_stalled` **Warning** per newly stalled provider set - (`ImmediateUpload`), one-shot per provider in-process, cross-restart deduped via a - `StartupEventGate` fingerprint over the sorted currently-stalled key set — a restart - into the same stall stays silent, a new provider joining the stall re-reports. -* Payload: `providers[]` (full table: name, scope, kind, raw values, complete flag), - `stalledProviders[]` with `stalledForMinutes`, `sidecarRegistered`, `dwellMinutes`. + state (both arms), never fires. +* One `esp_policy_provider_stalled` **Warning** per newly stalled condition set + (`ImmediateUpload`), one-shot per condition in-process, cross-restart deduped via a + `StartupEventGate` fingerprint over the sorted currently-active fired conditions + (provider keys + a `sidecarMissing` sentinel) — a restart into the same stall stays + silent, a new condition joining re-reports. +* Payload: `reason`, `providers[]` (full table: name, scope, kind, raw values, complete + flag), `stalledProviders[]` with `stalledForMinutes`, `sidecarMissingForMinutes` + (arm 2 only), `sidecarRegistered`, `dwellMinutes`. * **Purely observational**: emitted via `InformationalEventPost` (dispatch-guard exempt pass-through) — never touches decision state. No config toggle, no ConfigVersion bump (precedent: `disk_space_low`). @@ -93,33 +115,37 @@ resource), so both the detector and the guardrail change ship with the agent rel ## Examples -Customer-case event (abbreviated): +Customer-case event (abbreviated) — note the provider is "complete" by CSP values, the +hang is carried entirely by `reason=sidecar_provider_missing`: ```json { "eventType": "esp_policy_provider_stalled", "severity": "Warning", "data": { + "reason": "sidecar_provider_missing", "dwellMinutes": 15, "sidecarRegistered": false, + "sidecarMissingForMinutes": 15.0, "providers": [ { "name": "ConfigMgr", "scope": "device", "kind": "setupApps", - "trackingPoliciesCreated": null, "installationState": null, "complete": false } + "trackingPoliciesCreated": 1, "installationState": null, "complete": true } ], - "stalledProviders": [ - { "name": "ConfigMgr", "scope": "device", "kind": "setupApps", "stalledForMinutes": 15.0 } - ] + "stalledProviders": [] } } ``` -`sidecarRegistered=false` while `setupApps` providers exist is the co-management -signature; `sidecarRegistered=true` with a stalled third-party provider generalizes the -same contract violation. +`reason=sidecar_provider_missing` is the co-management signature; +`reason=provider_incomplete` with `sidecarRegistered=true` covers a stalled Sidecar or +third-party provider — the generalized contract violation. # Citations * EnrollmentStatusTracking CSP — * Windows Autopilot into co-management (requirements, provider-conflict warning, troubleshooting registry paths) — +* Troubleshoot the ESP (models app tracking exclusively via the `Sidecar` subkey) — +* Niehaus: the Intune ESP-policy WAP payload registering the provider nodes (`Sidecar` always, `ConfigMgr` only with a co-management settings policy) — +* Niehaus: ESP tracks arbitrarily named providers when they are properly registered — * Agent code: `EspPolicyProviderProbe.cs`, `EspPolicyProviderStallDetector.cs`, `EspPolicyProviderStallHost.cs` (SystemSignals / Orchestration.Hosts) * Related: [MDM Reboot Coalescing](mdm-reboot-coalescing.md), [Gather-Rule Guardrails](../rules/gather-rule-guardrails.md) diff --git a/docs/index.md b/docs/index.md index 9f2841a5..6326daf8 100644 --- a/docs/index.md +++ b/docs/index.md @@ -22,7 +22,7 @@ https://docs.autopilotmonitor.com (separate repository). * [Agent Endpoint Migration](agent/endpoint-migration.md) - How the backend re-homes agents to a new API base URL via the config channel (backend move or per-tenant region move): live-fetch-only, allowlist-validated on both sides, one hop, kill wins. * [MDM Reboot Coalescing](agent/mdm-reboot-coalescing.md) - Attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: aggregated DM-Enterprise 2800 observations (neutral, flush-time watermark) + the ANALYZE-ESP-005 advisory gated on an actually-observed reboot. * [Secure Boot CA-2023 Detection](agent/secureboot-ca2023-detection.md) - Why ANALYZE-SEC-001 v3 reads the UEFI db/KEK variables directly instead of trusting the SecureBoot\Servicing registry, and how the one-sided uefiCa2023FirmwareConfirmed marker plus a not_exists precondition give absence-tolerant suppression across old and new agents. -* [ESP Policy-Provider Stall Detection](agent/esp-policy-provider-stall.md) - The EnrollmentStatusTracking CSP's no-timeout Setup/Apps wait: a registered provider that never sets TrackingPoliciesCreated (co-management "ConfigMgr" without the IME's "Sidecar") parks the user ESP at "Apps (Identifying)" for days; the always-on esp_policy_provider_stalled tripwire and the SOFTWARE\Microsoft\Windows\Autopilot gather-rule allowlist entry that surface it. +* [ESP Policy-Provider Stall Detection](agent/esp-policy-provider-stall.md) - The EnrollmentStatusTracking CSP's no-timeout Setup/Apps wait is keyed to the Intune-registered "Sidecar" provider by NAME: co-management's "ConfigMgr" parks the user ESP at "Apps (Identifying)" for days even with TrackingPoliciesCreated=1 (issue #106); the always-on two-arm esp_policy_provider_stalled tripwire (provider_incomplete | sidecar_provider_missing) and the SOFTWARE\Microsoft\Windows\Autopilot gather-rule allowlist entry that surface it. # Rules diff --git a/docs/log.md b/docs/log.md index 83ceaaeb..bce208f3 100644 --- a/docs/log.md +++ b/docs/log.md @@ -1,6 +1,8 @@ # Log -## 2026-07-23 +## 2026-07-24 + +* **Update**: `agent/esp-policy-provider-stall.md` — the customer's follow-up on issue #106 corrected the field-case mechanics: `TrackingPoliciesCreated=1` WAS set under the foreign `ConfigMgr` provider, so the ESP ignores a foreign provider's completion values and waits on the Intune-registered `Sidecar` provider by name (the rename to `Sidecar` creates no tracking entries yet unblocks instantly). The completeness-keyed detector would have missed exactly this — the incident it was built for. The tripwire is now two-armed under one event type discriminated by `reason`: `provider_incomplete` (unchanged dwell on continuously-incomplete providers) and `sidecar_provider_missing` (≥ 1 Setup/Apps provider registered without `Sidecar`, completeness deliberately ignored, clock reset when Sidecar registers later). Sidecar-missing wins the `reason` when both arms cross in one round; the cross-restart fingerprint gains a `sidecarMissing` sentinel; payload adds `reason` and `sidecarMissingForMinutes`. Evidence for the name-keyed wait: Niehaus' documented Intune WAP payload always registers `Sidecar` (ConfigMgr only via a co-management settings policy) and Microsoft's ESP troubleshooting models app tracking exclusively through the Sidecar subkey. * **Creation**: Added `agent/esp-policy-provider-stall.md` — a customer diagnosed that pre-provisioned devices with the ConfigMgr client as a blocking app occasionally park the user ESP at "Apps (Identifying)" for days: only a `ConfigMgr` provider is registered under `EnrollmentStatusTracking\Device\Setup\Apps\PolicyProviders` (no `Sidecar` = IME) and `TrackingPoliciesCreated` never becomes 1 — a wait the CSP defines without any timeout; renaming the key to `Sidecar` unblocks immediately. The agent now ships the always-on `esp_policy_provider_stalled` tripwire (60-s tick, 15-min continuous-incompleteness dwell, one-shot per provider, `StartupEventGate` fingerprint across restarts, purely observational via the InformationalEvent rail — no toggle, no ConfigVersion bump), generalized to any provider in both `Setup/Apps` and `DevicePreparation` kinds rather than hardcoding the ConfigMgr case. Companion change: `SOFTWARE\Microsoft\Windows\Autopilot` joins the gather-rule registry allowlist so tenants can target the subtree with custom rules (e.g. `not_exists` on the Sidecar key). diff --git a/src/Agent/AutopilotMonitor.Agent.V2.Core.Tests/Monitoring/SystemSignals/EspPolicyProviderStallDetectorTests.cs b/src/Agent/AutopilotMonitor.Agent.V2.Core.Tests/Monitoring/SystemSignals/EspPolicyProviderStallDetectorTests.cs index f59a3e7b..d6b2ebb8 100644 --- a/src/Agent/AutopilotMonitor.Agent.V2.Core.Tests/Monitoring/SystemSignals/EspPolicyProviderStallDetectorTests.cs +++ b/src/Agent/AutopilotMonitor.Agent.V2.Core.Tests/Monitoring/SystemSignals/EspPolicyProviderStallDetectorTests.cs @@ -74,14 +74,17 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals var previous = new Dictionary { ["setupApps|device|ConfigMgr"] = T0 }; var missing = EspPolicyProviderStallDetector.EvaluateStallTransition( - EspPolicyProviderSnapshot.Empty, previous, new HashSet(), T0.AddHours(2), Dwell); + EspPolicyProviderSnapshot.Empty, previous, new HashSet(), T0, false, T0.AddHours(2), Dwell); Assert.Empty(missing.UpdatedFirstSeenUtc); Assert.Empty(missing.NewlyStalled); + Assert.Null(missing.UpdatedSidecarMissingSinceUtc); // arm 2 resets too + Assert.Null(missing.SidecarMissingStalledForMinutes); var registeredButEmpty = EspPolicyProviderStallDetector.EvaluateStallTransition( - Snapshot(), previous, new HashSet(), T0.AddHours(2), Dwell); + Snapshot(), previous, new HashSet(), T0, false, T0.AddHours(2), Dwell); Assert.Empty(registeredButEmpty.UpdatedFirstSeenUtc); Assert.Empty(registeredButEmpty.NewlyStalled); + Assert.Null(registeredButEmpty.UpdatedSidecarMissingSinceUtc); } [Fact] @@ -90,14 +93,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: null)); var round1 = EspPolicyProviderStallDetector.EvaluateStallTransition( - snapshot, new Dictionary(), new HashSet(), T0, Dwell); + snapshot, new Dictionary(), new HashSet(), null, false, T0, Dwell); Assert.Equal(T0, round1.UpdatedFirstSeenUtc["setupApps|device|ConfigMgr"]); Assert.Empty(round1.NewlyStalled); var round2 = EspPolicyProviderStallDetector.EvaluateStallTransition( - snapshot, round1.UpdatedFirstSeenUtc, new HashSet(), T0.AddMinutes(14), Dwell); + snapshot, round1.UpdatedFirstSeenUtc, new HashSet(), + round1.UpdatedSidecarMissingSinceUtc, false, T0.AddMinutes(14), Dwell); Assert.Equal(T0, round2.UpdatedFirstSeenUtc["setupApps|device|ConfigMgr"]); // first-seen preserved Assert.Empty(round2.NewlyStalled); + Assert.Null(round2.SidecarMissingStalledForMinutes); // 14 min < dwell on arm 2 as well } [Fact] @@ -107,7 +112,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals var seen = new Dictionary { ["setupApps|device|ConfigMgr"] = T0 }; var eval = EspPolicyProviderStallDetector.EvaluateStallTransition( - snapshot, seen, new HashSet(), T0.AddMinutes(15), Dwell); + snapshot, seen, new HashSet(), null, false, T0.AddMinutes(15), Dwell); var stalled = Assert.Single(eval.NewlyStalled); Assert.Equal("ConfigMgr", stalled.Provider.Name); @@ -122,20 +127,20 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals // Completes mid-dwell → dropped from the map. var completed = EspPolicyProviderStallDetector.EvaluateStallTransition( - Snapshot(SetupApps("ConfigMgr", tracking: 1)), seen, new HashSet(), T0.AddMinutes(10), Dwell); + Snapshot(SetupApps("ConfigMgr", tracking: 1)), seen, new HashSet(), null, false, T0.AddMinutes(10), Dwell); Assert.Empty(completed.UpdatedFirstSeenUtc); // Re-appears incomplete much later → dwell restarts from the new now, no stall yet. var reappeared = EspPolicyProviderStallDetector.EvaluateStallTransition( Snapshot(SetupApps("ConfigMgr", tracking: 0)), completed.UpdatedFirstSeenUtc, - new HashSet(), T0.AddHours(3), Dwell); + new HashSet(), null, false, T0.AddHours(3), Dwell); Assert.Equal(T0.AddHours(3), reappeared.UpdatedFirstSeenUtc[key]); Assert.Empty(reappeared.NewlyStalled); // Disappearance behaves the same as completion. var disappeared = EspPolicyProviderStallDetector.EvaluateStallTransition( Snapshot(SetupApps("Sidecar", tracking: 1)), new Dictionary { [key] = T0 }, - new HashSet(), T0.AddMinutes(20), Dwell); + new HashSet(), null, false, T0.AddMinutes(20), Dwell); Assert.False(disappeared.UpdatedFirstSeenUtc.ContainsKey(key)); } @@ -147,7 +152,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals var fired = new HashSet(StringComparer.OrdinalIgnoreCase) { "setupApps|device|ConfigMgr" }; var eval = EspPolicyProviderStallDetector.EvaluateStallTransition( - snapshot, seen, fired, T0.AddHours(5), Dwell); + snapshot, seen, fired, null, false, T0.AddHours(5), Dwell); Assert.Empty(eval.NewlyStalled); Assert.True(eval.UpdatedFirstSeenUtc.ContainsKey("setupApps|device|ConfigMgr")); // still tracked @@ -162,7 +167,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals DevicePrep("ConfigMgr", installState: 1)); var eval = EspPolicyProviderStallDetector.EvaluateStallTransition( - snapshot, new Dictionary(), new HashSet(), T0, Dwell); + snapshot, new Dictionary(), new HashSet(), null, false, T0, Dwell); Assert.Equal(3, eval.UpdatedFirstSeenUtc.Count); Assert.Contains("setupApps|device|ConfigMgr", eval.UpdatedFirstSeenUtc.Keys); @@ -179,6 +184,71 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals Assert.False(Snapshot(DevicePrep("Sidecar", installState: 3)).SidecarRegistered); // wrong kind Assert.False(EspPolicyProviderSnapshot.Empty.SidecarRegistered); } + + // ------------------------------------------------- arm 2: sidecar-missing transition + + [Fact] + public void Issue106_foreign_provider_complete_but_sidecar_missing_stalls_after_dwell() + { + // The actual field case: ConfigMgr with TrackingPoliciesCreated=1 — "complete" by the + // CSP value contract, arm 1 never fires — yet the ESP waits on the absent Sidecar. + var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: 1)); + + var round1 = EspPolicyProviderStallDetector.EvaluateStallTransition( + snapshot, new Dictionary(), new HashSet(), null, false, T0, Dwell); + Assert.Empty(round1.NewlyStalled); // arm 1 stays silent... + Assert.Equal(T0, round1.UpdatedSidecarMissingSinceUtc); // ...but arm 2 starts its clock + Assert.Null(round1.SidecarMissingStalledForMinutes); + + var round2 = EspPolicyProviderStallDetector.EvaluateStallTransition( + snapshot, round1.UpdatedFirstSeenUtc, new HashSet(), + round1.UpdatedSidecarMissingSinceUtc, false, T0.AddMinutes(15), Dwell); + Assert.Empty(round2.NewlyStalled); + Assert.Equal(15.0, round2.SidecarMissingStalledForMinutes!.Value, precision: 3); + } + + [Fact] + public void Sidecar_registering_resets_the_sidecar_missing_clock() + { + // ConfigMgr first (legitimate startup ordering) → clock runs... + var round1 = EspPolicyProviderStallDetector.EvaluateStallTransition( + Snapshot(SetupApps("ConfigMgr", tracking: 1)), + new Dictionary(), new HashSet(), null, false, T0, Dwell); + Assert.Equal(T0, round1.UpdatedSidecarMissingSinceUtc); + + // ...Sidecar joins → condition clears, state resets. + var round2 = EspPolicyProviderStallDetector.EvaluateStallTransition( + Snapshot(SetupApps("ConfigMgr", tracking: 1), SetupApps("Sidecar", tracking: 0)), + round1.UpdatedFirstSeenUtc, new HashSet(), + round1.UpdatedSidecarMissingSinceUtc, false, T0.AddMinutes(10), Dwell); + Assert.Null(round2.UpdatedSidecarMissingSinceUtc); + Assert.Null(round2.SidecarMissingStalledForMinutes); + + // Sidecar gone again much later → clock restarts from the new now, no instant stall. + var round3 = EspPolicyProviderStallDetector.EvaluateStallTransition( + Snapshot(SetupApps("ConfigMgr", tracking: 1)), + round2.UpdatedFirstSeenUtc, new HashSet(), + round2.UpdatedSidecarMissingSinceUtc, false, T0.AddHours(3), Dwell); + Assert.Equal(T0.AddHours(3), round3.UpdatedSidecarMissingSinceUtc); + Assert.Null(round3.SidecarMissingStalledForMinutes); + } + + [Fact] + public void Sidecar_missing_one_shot_and_deviceprep_only_scoping() + { + // Already fired → never re-reported, but the clock keeps running. + var latched = EspPolicyProviderStallDetector.EvaluateStallTransition( + Snapshot(SetupApps("ConfigMgr", tracking: 1)), + new Dictionary(), new HashSet(), T0, true, T0.AddHours(5), Dwell); + Assert.Null(latched.SidecarMissingStalledForMinutes); + Assert.Equal(T0, latched.UpdatedSidecarMissingSinceUtc); + + // DevicePreparation providers alone don't constitute the Setup/Apps wait — arm 2 off. + var devicePrepOnly = EspPolicyProviderStallDetector.EvaluateStallTransition( + Snapshot(DevicePrep("ConfigMgr", installState: 1)), + new Dictionary(), new HashSet(), null, false, T0, Dwell); + Assert.Null(devicePrepOnly.UpdatedSidecarMissingSinceUtc); + } } /// @@ -243,7 +313,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals Assert.Equal(EspPolicyProviderStallDetector.SourceLabel, evt.Payload[SignalPayloadKeys.Source]); var data = Assert.IsAssignableFrom>(evt.TypedPayload); + // Both arms cross in the same round (ConfigMgr incomplete AND Sidecar absent) → + // ONE event; sidecar-missing is the dominant reason. + Assert.Equal(EspPolicyProviderStallDetector.ReasonSidecarMissing, data["reason"]); Assert.Equal(false, data["sidecarRegistered"]); + Assert.Equal(15.0, data["sidecarMissingForMinutes"]); Assert.Equal(EspPolicyProviderStallDetector.DwellMinutes, data["dwellMinutes"]); var stalled = Assert.IsAssignableFrom>>(data["stalledProviders"]); var entry = Assert.Single(stalled); @@ -252,6 +326,36 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals Assert.Equal(15.0, entry["stalledForMinutes"]); } + [Fact] + public void Issue106_configmgr_complete_without_sidecar_still_emits_sidecar_missing() + { + // Regression test for the customer's follow-up on issue #106: TrackingPoliciesCreated=1 + // WAS present under ConfigMgr — the provider looks "complete", arm 1 never fires, yet + // the ESP hangs because Sidecar is absent. Arm 2 must catch exactly this. + var configMgrComplete = new EspPolicyProviderSnapshot(new[] + { + new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindSetupApps, + trackingPoliciesCreated: 1, installationState: null), + }); + using var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => configMgrComplete); + var (sut, sink, clock) = BuildDetector(); + + sut.Tick(); + Assert.Empty(StalledEvents(sink)); + + clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes)); + sut.Tick(); + + var evt = Assert.Single(StalledEvents(sink)); + var data = Assert.IsAssignableFrom>(evt.TypedPayload); + Assert.Equal(EspPolicyProviderStallDetector.ReasonSidecarMissing, data["reason"]); + Assert.Equal(false, data["sidecarRegistered"]); + Assert.Equal(15.0, data["sidecarMissingForMinutes"]); + Assert.Empty(Assert.IsAssignableFrom>>(data["stalledProviders"])); + var providers = Assert.IsAssignableFrom>>(data["providers"]); + Assert.Equal(true, Assert.Single(providers)["complete"]); // the misleading "complete" is visible + } + [Fact] public void Sidecar_complete_configmgr_incomplete_reports_only_configmgr_with_sidecar_flag() { @@ -264,7 +368,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals var evt = Assert.Single(StalledEvents(sink)); var data = Assert.IsAssignableFrom>(evt.TypedPayload); + Assert.Equal(EspPolicyProviderStallDetector.ReasonProviderIncomplete, data["reason"]); // arm 2 off — Sidecar is there Assert.Equal(true, data["sidecarRegistered"]); + Assert.False(data.ContainsKey("sidecarMissingForMinutes")); var stalled = Assert.IsAssignableFrom>>(data["stalledProviders"]); Assert.Equal("ConfigMgr", Assert.Single(stalled)["name"]); var providers = Assert.IsAssignableFrom>>(data["providers"]); diff --git a/src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/EspPolicyProviderProbe.cs b/src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/EspPolicyProviderProbe.cs index c0809a05..809bf0e2 100644 --- a/src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/EspPolicyProviderProbe.cs +++ b/src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/EspPolicyProviderProbe.cs @@ -35,13 +35,19 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals /// (registered during Autopilot-into-co-management, see the troubleshooting registry paths in /// https://learn.microsoft.com/en-us/intune/configmgr/comanage/autopilot-enrollment). Microsoft /// warns that ESP policy providers are not aware of each other and that pre-provisioning is - /// unsupported with co-management. Field case that motivated this probe: a pre-provisioned - /// device with the ConfigMgr client installed as a blocking app ended up with ONLY - /// ConfigMgr registered under Device\Setup\Apps\PolicyProviders (no - /// Sidecar) and TrackingPoliciesCreated never set — the user ESP hung at - /// "Apps (Identifying)" for hours to days; manually renaming the key to Sidecar - /// unblocked it immediately. turns this contract - /// into the esp_policy_provider_stalled tripwire. + /// unsupported with co-management. Field case that motivated this probe (issue #106): a + /// pre-provisioned device with the ConfigMgr client installed as a blocking app ended up with + /// ONLY ConfigMgr registered under Device\Setup\Apps\PolicyProviders (no + /// Sidecar) — and although TrackingPoliciesCreated=1 WAS set under it, the user + /// ESP hung at "Apps (Identifying)" for hours to days; manually renaming the key to + /// Sidecar unblocked it immediately. So the ESP's Apps wait is keyed to the provider + /// the Intune service registered at enrollment (always Sidecar; ConfigMgr only joins + /// via a co-management settings policy — see the WAP payload documented in + /// https://oofhours.com/2023/08/25/run-an-sccm-task-sequence-during-autopilot/), and a + /// foreign provider's completion values do not satisfy it. Detection therefore keys on the + /// provider NAME topology, not only on completeness: + /// turns this contract into the two-arm + /// esp_policy_provider_stalled tripwire. /// /// /// Fail-soft like : any failure is Debug-logged and yields diff --git a/src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/EspPolicyProviderStallDetector.cs b/src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/EspPolicyProviderStallDetector.cs index 84bf58d3..5fb9f30d 100644 --- a/src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/EspPolicyProviderStallDetector.cs +++ b/src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/EspPolicyProviderStallDetector.cs @@ -13,18 +13,30 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals { /// /// Turns the ESP policy-provider contract read by into - /// the one-shot esp_policy_provider_stalled Warning: a registered provider that stays - /// continuously incomplete for minutes is reported once, with the - /// full provider table in the payload. See the probe's header for the CSP contract, the - /// official sources, and the co-management field case (ConfigMgr registered without Sidecar - /// → user ESP parked at "Apps (Identifying)" with no OS timeout) this tripwire exists for. + /// the one-shot esp_policy_provider_stalled Warning via two independent arms, each + /// with the same -minute continuous-dwell rule and the full + /// provider table in the payload: + /// + /// Arm 1 — reason=provider_incomplete: a registered provider stays + /// continuously incomplete (e.g. Sidecar itself never sets TrackingPoliciesCreated). + /// Arm 2 — reason=sidecar_provider_missing: at least one + /// Setup\Apps provider is registered but none of them is Sidecar (the IME). + /// This is the actual co-management field-case signature (issue #106): the foreign + /// ConfigMgr provider had TrackingPoliciesCreated=1 — i.e. was "complete" + /// by the CSP value contract — yet the user ESP stayed parked at "Apps (Identifying)" + /// until the key was renamed to Sidecar. The ESP's Apps wait is keyed to the + /// Intune-registered provider by NAME, so provider completeness alone must not gate this + /// detection. See the probe's header for sources. + /// /// - /// Dwell semantics: the dwell clock per provider starts when it is first observed incomplete - /// and RESETS when the provider completes or disappears — only continuously incomplete - /// providers fire. A missing root key / empty provider list is normal early enrollment and - /// clears all dwell state (probe D4). Purely observational: emission goes through - /// (InformationalEvent pass-through, dispatch-guard - /// exempt), never into decision state. + /// Dwell semantics: arm 1's clock per provider starts when it is first observed incomplete + /// and RESETS when the provider completes or disappears; arm 2's clock starts when the + /// sidecar-missing condition is first observed and RESETS when Sidecar registers (covers the + /// legitimate "ConfigMgr registered before Sidecar" startup ordering) — only + /// continuously bad states fire. A missing root key / empty provider list is normal + /// early enrollment and clears all dwell state (probe D4). Purely observational: emission + /// goes through (InformationalEvent pass-through, + /// dispatch-guard exempt), never into decision state. /// /// /// Duplicate suppression: in-process one-shot per provider key, plus a @@ -44,6 +56,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals internal const string SourceLabel = "EspPolicyProviderStallDetector"; + /// Payload reason: a registered provider stayed continuously incomplete (arm 1). + internal const string ReasonProviderIncomplete = "provider_incomplete"; + + /// Payload reason: Setup\Apps providers registered but Sidecar absent (arm 2 — dominant when both arms fire in one round). + internal const string ReasonSidecarMissing = "sidecar_provider_missing"; + + /// + /// Sentinel token representing the arm-2 condition in the cross-restart fingerprint. + /// Cannot collide with provider keys — those always contain | separators. + /// + private const string SidecarMissingFingerprintToken = "sidecarMissing"; + private readonly string _sessionId; private readonly string _tenantId; private readonly InformationalEventPost _post; @@ -57,6 +81,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals private readonly HashSet _firedKeys = new HashSet(StringComparer.OrdinalIgnoreCase); + /// Arm-2 dwell state: first observation of "setupApps providers without Sidecar"; null = condition not active. + private DateTime? _sidecarMissingSinceUtc; + + /// Arm-2 in-process one-shot latch (mirror of ). + private bool _sidecarMissingFired; + public EspPolicyProviderStallDetector( string sessionId, string tenantId, @@ -79,13 +109,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals internal void Tick() { var snapshot = EspPolicyProviderProbe.Read(_logger); - var eval = EvaluateStallTransition(snapshot, _firstSeenIncompleteUtc, _firedKeys, _clock.UtcNow, _dwell); + var eval = EvaluateStallTransition( + snapshot, _firstSeenIncompleteUtc, _firedKeys, + _sidecarMissingSinceUtc, _sidecarMissingFired, _clock.UtcNow, _dwell); _firstSeenIncompleteUtc.Clear(); foreach (var kv in eval.UpdatedFirstSeenUtc) _firstSeenIncompleteUtc[kv.Key] = kv.Value; + _sidecarMissingSinceUtc = eval.UpdatedSidecarMissingSinceUtc; - if (eval.NewlyStalled.Count > 0) + if (eval.NewlyStalled.Count > 0 || eval.SidecarMissingStalledForMinutes != null) EmitStalled(snapshot, eval); } @@ -94,24 +127,29 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals /// registry, timer, or ingress (pattern: PerformanceCollector.EvaluateDiskLowTransition). /// /// Current provider snapshot. - /// Dwell state from the previous round (provider key → first-seen-incomplete). + /// Arm-1 dwell state from the previous round (provider key → first-seen-incomplete). /// Provider keys already reported this run (never re-report). + /// Arm-2 dwell state from the previous round; null = condition was not active. + /// Arm-2 one-shot latch (never re-report). /// Evaluation timestamp. - /// Continuous-incompleteness threshold. + /// Continuous-condition threshold (shared by both arms). internal static StallEvaluation EvaluateStallTransition( EspPolicyProviderSnapshot snapshot, IReadOnlyDictionary firstSeenIncompleteUtc, ISet alreadyFired, + DateTime? sidecarMissingSinceUtc, + bool sidecarMissingAlreadyFired, DateTime nowUtc, TimeSpan dwell) { var result = new StallEvaluation(); // Missing root key or no registered providers = normal early enrollment: clear all - // dwell state (a provider must be continuously OBSERVED incomplete to accrue dwell). + // dwell state (a condition must be continuously OBSERVED to accrue dwell). if (!snapshot.HasData || snapshot.Providers.Count == 0) return result; + // Arm 1 — a registered provider continuously incomplete. foreach (var provider in snapshot.Providers) { if (provider.IsComplete) @@ -127,22 +165,44 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals result.NewlyStalled.Add(new StalledProvider(provider, stalledFor.TotalMinutes)); } + // Arm 2 — Setup\Apps providers registered, but none of them is Sidecar. Deliberately + // ignores provider completeness: the field case (issue #106) had ConfigMgr with + // TrackingPoliciesCreated=1 and the ESP still waited. Sidecar registering later + // (legitimate startup ordering) clears the condition and resets the clock. + if (HasSetupAppsProvider(snapshot) && !snapshot.SidecarRegistered) + { + var since = sidecarMissingSinceUtc ?? nowUtc; + result.UpdatedSidecarMissingSinceUtc = since; + + var missingFor = nowUtc - since; + if (missingFor >= dwell && !sidecarMissingAlreadyFired) + result.SidecarMissingStalledForMinutes = missingFor.TotalMinutes; + } + return result; } + private static bool HasSetupAppsProvider(EspPolicyProviderSnapshot snapshot) + => snapshot.Providers.Any(p => p.Kind == EspPolicyProviderProbe.KindSetupApps); + private void EmitStalled(EspPolicyProviderSnapshot snapshot, StallEvaluation eval) { foreach (var stalled in eval.NewlyStalled) _firedKeys.Add(stalled.Provider.Key); + if (eval.SidecarMissingStalledForMinutes != null) + _sidecarMissingFired = true; - // Cross-restart dedup: fingerprint = sorted set of all fired keys that are still - // incomplete. A restart into the same stall re-accrues dwell and lands on the same - // fingerprint → suppressed; a new provider joining the stall changes it → re-emitted. + // Cross-restart dedup: fingerprint = sorted set of all fired conditions that are + // still active (incomplete providers + the sidecar-missing sentinel). A restart into + // the same stall re-accrues dwell and lands on the same fingerprint → suppressed; a + // new condition joining the stall changes it → re-emitted. var currentlyStalledKeys = snapshot.Providers .Where(p => !p.IsComplete && _firedKeys.Contains(p.Key)) .Select(p => p.Key) - .OrderBy(k => k, StringComparer.OrdinalIgnoreCase) .ToList(); + if (_sidecarMissingFired && HasSetupAppsProvider(snapshot) && !snapshot.SidecarRegistered) + currentlyStalledKeys.Add(SidecarMissingFingerprintToken); + currentlyStalledKeys.Sort(StringComparer.OrdinalIgnoreCase); var fingerprint = StartupEventGate.HashString(string.Join(";", currentlyStalledKeys)); if (_startupGate != null && !_startupGate.ShouldEmit(Constants.EventTypes.EspPolicyProviderStalled, fingerprint)) @@ -153,11 +213,30 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals return; } + // Sidecar-missing is the dominant reason when both arms fire in one round: it + // explains the ESP hang regardless of the foreign providers' completeness. + var reason = eval.SidecarMissingStalledForMinutes != null + ? ReasonSidecarMissing + : ReasonProviderIncomplete; + + var messageParts = new List(); + if (eval.SidecarMissingStalledForMinutes != null) + { + var registeredNames = string.Join(", ", snapshot.Providers + .Where(p => p.Kind == EspPolicyProviderProbe.KindSetupApps) + .Select(p => $"{p.Name} ({p.Scope})")); + messageParts.Add( + $"expected IME provider '{EspPolicyProviderProbe.SidecarProviderName}' not registered for " + + $">= {DwellMinutes} min while Setup/Apps providers exist: {registeredNames}"); + } var stalledNames = string.Join(", ", eval.NewlyStalled.Select(s => $"{s.Provider.Name} ({s.Provider.Scope}/{s.Provider.Kind})")); + if (eval.NewlyStalled.Count > 0) + messageParts.Add($"provider(s) incomplete for >= {DwellMinutes} min: {stalledNames}"); var data = new Dictionary(StringComparer.Ordinal) { + ["reason"] = reason, ["dwellMinutes"] = DwellMinutes, ["sidecarRegistered"] = snapshot.SidecarRegistered, ["providers"] = snapshot.Providers.Select(p => new Dictionary(StringComparer.Ordinal) @@ -177,6 +256,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals ["stalledForMinutes"] = Math.Round(s.StalledForMinutes, 1), }).ToList(), }; + if (eval.SidecarMissingStalledForMinutes != null) + data["sidecarMissingForMinutes"] = Math.Round(eval.SidecarMissingStalledForMinutes.Value, 1); + + var message = + "ESP Setup/Apps tracking stalled: " + string.Join("; ", messageParts) + + " — the ESP waits on its app-tracking providers without any timeout"; _post.Emit(new EnrollmentEvent { @@ -188,27 +273,31 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals Source = SourceLabel, Phase = EnrollmentPhase.Unknown, ImmediateUpload = true, - Message = $"ESP policy provider(s) incomplete for >= {DwellMinutes} min: {stalledNames} " + - $"(sidecarRegistered={snapshot.SidecarRegistered}) — the ESP waits on " + - "Setup/Apps providers without any timeout", + Message = message, Data = data }); _startupGate?.MarkEmitted(Constants.EventTypes.EspPolicyProviderStalled); _logger?.Warning( - $"EspPolicyProviderStallDetector: {stalledNames} incomplete for >= {DwellMinutes} min " + - $"(sidecarRegistered={snapshot.SidecarRegistered})."); + $"EspPolicyProviderStallDetector: {string.Join("; ", messageParts)} " + + $"(reason={reason}, sidecarRegistered={snapshot.SidecarRegistered})."); } /// Result of one round. internal sealed class StallEvaluation { - /// Next round's dwell state — only currently-incomplete providers survive. + /// Next round's arm-1 dwell state — only currently-incomplete providers survive. public Dictionary UpdatedFirstSeenUtc { get; } = new Dictionary(StringComparer.OrdinalIgnoreCase); - /// Providers that crossed the dwell threshold this round and were not yet reported. + /// Providers that crossed the dwell threshold this round and were not yet reported (arm 1). public List NewlyStalled { get; } = new List(); + + /// Next round's arm-2 dwell state; null = sidecar-missing condition not active (reset). + public DateTime? UpdatedSidecarMissingSinceUtc { get; set; } + + /// Non-null when the sidecar-missing condition crossed the dwell threshold this round and was not yet reported (arm 2). + public double? SidecarMissingStalledForMinutes { get; set; } } internal readonly struct StalledProvider diff --git a/src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/DefaultComponentFactory.cs b/src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/DefaultComponentFactory.cs index ef4bd862..9b744253 100644 --- a/src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/DefaultComponentFactory.cs +++ b/src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/DefaultComponentFactory.cs @@ -37,8 +37,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration /// . Owns its 60-s idle-check timer (the /// collector itself has no timer — it's a pure probe invoked from outside). /// EspPolicyProviderStallHost — always-on 60-s wall-clock tick for the - /// esp_policy_provider_stalled tripwire (registered EnrollmentStatusTracking CSP - /// provider continuously incomplete ≥ 15 min — the no-timeout ESP wait). + /// two-arm esp_policy_provider_stalled tripwire (EnrollmentStatusTracking CSP + /// provider continuously incomplete ≥ 15 min, OR Setup\Apps providers registered + /// without the expected Sidecar/IME provider — the no-timeout ESP wait). /// /// Optional peripheral hosts (driven by toggles): /// @@ -307,11 +308,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration } // ESP policy-provider stall tripwire — always-on kernel host (no config gate, like - // disk_space_low): a registered EnrollmentStatusTracking CSP provider that never - // completes (e.g. co-management "ConfigMgr" without "Sidecar" on a pre-provisioned - // device) parks the user ESP at "Apps (Identifying)" WITHOUT any OS timeout. Wall-clock - // dwell, deliberately outside the StallProbe gate: the idle clock there resets on any - // session activity, and the stall must accrue regardless of it. + // disk_space_low): a provider that never completes, or co-management "ConfigMgr" + // registered without the expected "Sidecar"/IME provider (issue #106 — even with + // TrackingPoliciesCreated=1 the ESP ignores foreign provider names), parks the user + // ESP at "Apps (Identifying)" WITHOUT any OS timeout. Wall-clock dwell, deliberately + // outside the StallProbe gate: the idle clock there resets on any session activity, + // and the stall must accrue regardless of it. hosts.Add(new EspPolicyProviderStallHost( sessionId: sessionId, tenantId: tenantId, diff --git a/src/Web/autopilot-monitor-web/app/gather-rules/eventTypes.ts b/src/Web/autopilot-monitor-web/app/gather-rules/eventTypes.ts index fcddce0b..1053738a 100644 --- a/src/Web/autopilot-monitor-web/app/gather-rules/eventTypes.ts +++ b/src/Web/autopilot-monitor-web/app/gather-rules/eventTypes.ts @@ -59,7 +59,7 @@ export const KNOWN_EVENT_TYPES: EventTypeEntry[] = [ { value: "stall_probe_check", label: "stall_probe_check", category: "stall", description: "Trace-level heartbeat from a stall probe — proves the logic ran (default: only the 15 min probe)." }, { value: "esp_policy_provider_stalled", label: "esp_policy_provider_stalled", category: "stall", - description: "A registered ESP policy provider (EnrollmentStatusTracking CSP) stayed incomplete for 15+ minutes — e.g. co-management leaves only ConfigMgr registered without the Intune Sidecar provider, which parks the user ESP at 'Apps (Identifying)' with no OS timeout. One-shot Warning; payload carries the full provider table, per-provider state, and sidecarRegistered." }, + description: "The ESP's app-tracking wait (EnrollmentStatusTracking CSP) made no progress for 15+ minutes: a registered policy provider stayed incomplete (reason=provider_incomplete), or Setup/Apps providers exist without the expected Intune 'Sidecar' provider (reason=sidecar_provider_missing) — the co-management signature where only ConfigMgr is registered (even with TrackingPoliciesCreated=1) and the user ESP parks at 'Apps (Identifying)' with no OS timeout. One-shot Warning; payload carries reason, the full provider table, per-provider state, and sidecarRegistered." }, { value: "agent_late_start", label: "agent_late_start", category: "stall", description: "Low observation coverage: the agent started long after device boot (≥10 min) AND lived only briefly (≤5 min) before a terminal outcome — it arrived after the enrollment had already decided, so its diagnosis is a post-mortem of the end-state, not a live observation. Carries bootToAgentStartSeconds, agentUptimeSeconds and outcome. Often paired with script_timeout_suspected (a script that hung ahead of the agent bootstrap)." },