mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-24 12:42:38 +00:00
fix(agent): esp_policy_provider_stalled gains sidecar-missing arm (issue #106)
Customer follow-up corrected the field-case mechanics: TrackingPoliciesCreated=1 WAS set under the foreign ConfigMgr provider - the ESP ignores a foreign provider's completion values and waits on the Intune-registered "Sidecar" provider by name (renaming the key unblocks instantly without creating any tracking entries). The completeness-keyed detector would have missed exactly the incident it was built for. - Arm 2: >=1 Setup\Apps provider registered without Sidecar, continuously for the 15-min dwell -> reason=sidecar_provider_missing; clock resets when Sidecar registers later (legitimate startup ordering); DevicePreparation-only snapshots don't activate it - Arm 1 (continuously-incomplete providers) unchanged -> reason=provider_incomplete; sidecar-missing wins when both arms cross in one round (single event) - Cross-restart fingerprint gains a "sidecarMissing" sentinel; payload adds reason + sidecarMissingForMinutes - Regression test: ConfigMgr complete + Sidecar absent -> fires (previously classified healthy); probe/doc headers and OKF doc corrected, web event-type description updated Evidence (research session c7e3db08): Intune's ESP-policy WAP payload always registers Sidecar (ConfigMgr only via co-management settings policy, per Niehaus' documented payload); Microsoft's ESP troubleshooting models app tracking exclusively through the Sidecar subkey. Agent tests 2043/2043, web tsc clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: Concept
|
||||
title: ESP Policy-Provider Stall Detection — the no-timeout Setup/Apps wait and the co-management Sidecar gap
|
||||
description: Why a registered EnrollmentStatusTracking CSP policy provider that never sets TrackingPoliciesCreated parks the user ESP at "Apps (Identifying)" indefinitely, the pre-provisioning + ConfigMgr field case, and how the always-on esp_policy_provider_stalled tripwire (15-min wall-clock dwell, StartupEventGate cross-restart dedup) surfaces it.
|
||||
description: Why the ESP's Apps wait is keyed to the Intune-registered "Sidecar" provider by name — a foreign provider (ConfigMgr) with TrackingPoliciesCreated=1 does not satisfy it — plus the classic never-completes stall, and how the always-on two-arm esp_policy_provider_stalled tripwire (reason=provider_incomplete | sidecar_provider_missing, 15-min wall-clock dwell, StartupEventGate cross-restart dedup) surfaces both.
|
||||
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/EspPolicyProviderProbe.cs
|
||||
tags:
|
||||
- agent
|
||||
@@ -9,7 +9,7 @@ tags:
|
||||
- enrollment-status-tracking
|
||||
- co-management
|
||||
- stall-detection
|
||||
timestamp: 2026-07-23T00:00:00+02:00
|
||||
timestamp: 2026-07-24T00:00:00+02:00
|
||||
---
|
||||
|
||||
# ESP Policy-Provider Stall Detection
|
||||
@@ -42,37 +42,59 @@ exactly these registry paths. The same document states that ESP policy providers
|
||||
aware of each other** and that **pre-provisioning is unsupported** with Autopilot into
|
||||
co-management.
|
||||
|
||||
## The field case
|
||||
## The field case (issue #106)
|
||||
|
||||
A tenant pre-provisions devices and installs the ConfigMgr client as a blocking app.
|
||||
Occasionally the user-phase ESP reaches "Apps (Identifying)" and never leaves it — for
|
||||
hours to days. On those devices `Device\Setup\Apps\PolicyProviders` contains **only a
|
||||
`ConfigMgr` key (no `Sidecar`)** and `TrackingPoliciesCreated` is never set. Manually
|
||||
renaming the key to `Sidecar` unblocks the ESP immediately (the IME then finds its
|
||||
provider node and reports tracking-policies-created). The mechanism is exactly the
|
||||
documented no-timeout wait; the unsupported pre-provisioning + co-management combination
|
||||
explains why only some devices are affected. Why the Sidecar registration is missing or
|
||||
displaced on those devices is not publicly documented — irrelevant for detection, since
|
||||
the signature is deterministic.
|
||||
`ConfigMgr` key (no `Sidecar`)** — and, per the customer's follow-up, **`TrackingPoliciesCreated=1`
|
||||
IS set under it**. The provider is "complete" by the CSP value contract, yet the ESP
|
||||
still waits. Manually renaming the key to `Sidecar` — which creates no tracking entries,
|
||||
only satisfies the name — unblocks the ESP instantly.
|
||||
|
||||
The naive CSP-doc reading ("any provider with TrackingPoliciesCreated=1 satisfies the
|
||||
wait") is therefore wrong for this case. What the evidence supports instead: provider
|
||||
nodes are registered **by the Intune service at enrollment**, and its ESP-policy WAP
|
||||
payload always registers `Sidecar` (ConfigMgr joins only via a co-management settings
|
||||
policy — see the payload documented by Niehaus). The ESP waits on the provider it
|
||||
expects **by name**; a foreign provider occupying the slot does not satisfy it,
|
||||
regardless of its completion values. On the affected devices the CM client (installed
|
||||
as a plain Win32 app, no co-management settings policy) evidently displaced the
|
||||
Sidecar node — consistent with Microsoft's warning that providers "aren't currently
|
||||
aware of others" and that pre-provisioning is unsupported with co-management. The exact
|
||||
CloudExperienceHost mechanism is not publicly documented; the registry signature —
|
||||
setupApps providers present, `Sidecar` absent — is deterministic either way.
|
||||
|
||||
## Detection
|
||||
|
||||
`EspPolicyProviderProbe` (fail-soft, static, `ScopedOverride` test seam — structural clone
|
||||
of `EspTrackingInfoProbe`) enumerates all providers in both kinds and scopes.
|
||||
`EspPolicyProviderStallDetector` evaluates a pure transition function on a 60-second tick
|
||||
from the always-on `EspPolicyProviderStallHost`:
|
||||
from the always-on `EspPolicyProviderStallHost` — **two arms**, same 15-minute wall-clock
|
||||
dwell, one shared event type discriminated by `reason`:
|
||||
|
||||
* A provider **continuously incomplete for ≥ 15 minutes** (wall-clock) is stalled. The
|
||||
dwell clock starts at first incomplete observation and resets on completion or
|
||||
disappearance. Missing values parse as incomplete (fail-toward-observation).
|
||||
* **Arm 1, `reason=provider_incomplete`** — a provider **continuously incomplete for
|
||||
≥ 15 minutes**. The dwell clock starts at first incomplete observation and resets on
|
||||
completion or disappearance. Missing values parse as incomplete
|
||||
(fail-toward-observation). Catches e.g. Sidecar itself never reporting.
|
||||
* **Arm 2, `reason=sidecar_provider_missing`** — **≥ 1 `Setup\Apps` provider registered
|
||||
but none named `Sidecar`**, continuously for ≥ 15 minutes. Deliberately ignores
|
||||
provider completeness (the field case had `TrackingPoliciesCreated=1`). Sidecar
|
||||
registering later — the legitimate startup ordering — clears the condition and resets
|
||||
the clock. DevicePreparation-only snapshots don't activate it.
|
||||
* When both arms cross in the same round, ONE event is emitted and
|
||||
`sidecar_provider_missing` wins as `reason` (it explains the hang regardless of the
|
||||
foreign provider's state).
|
||||
* Missing root key / empty provider list = normal early enrollment: clears all dwell
|
||||
state, never fires.
|
||||
* One `esp_policy_provider_stalled` **Warning** per newly stalled provider set
|
||||
(`ImmediateUpload`), one-shot per provider in-process, cross-restart deduped via a
|
||||
`StartupEventGate` fingerprint over the sorted currently-stalled key set — a restart
|
||||
into the same stall stays silent, a new provider joining the stall re-reports.
|
||||
* Payload: `providers[]` (full table: name, scope, kind, raw values, complete flag),
|
||||
`stalledProviders[]` with `stalledForMinutes`, `sidecarRegistered`, `dwellMinutes`.
|
||||
state (both arms), never fires.
|
||||
* One `esp_policy_provider_stalled` **Warning** per newly stalled condition set
|
||||
(`ImmediateUpload`), one-shot per condition in-process, cross-restart deduped via a
|
||||
`StartupEventGate` fingerprint over the sorted currently-active fired conditions
|
||||
(provider keys + a `sidecarMissing` sentinel) — a restart into the same stall stays
|
||||
silent, a new condition joining re-reports.
|
||||
* Payload: `reason`, `providers[]` (full table: name, scope, kind, raw values, complete
|
||||
flag), `stalledProviders[]` with `stalledForMinutes`, `sidecarMissingForMinutes`
|
||||
(arm 2 only), `sidecarRegistered`, `dwellMinutes`.
|
||||
* **Purely observational**: emitted via `InformationalEventPost` (dispatch-guard exempt
|
||||
pass-through) — never touches decision state. No config toggle, no ConfigVersion bump
|
||||
(precedent: `disk_space_low`).
|
||||
@@ -93,33 +115,37 @@ resource), so both the detector and the guardrail change ship with the agent rel
|
||||
|
||||
## Examples
|
||||
|
||||
Customer-case event (abbreviated):
|
||||
Customer-case event (abbreviated) — note the provider is "complete" by CSP values, the
|
||||
hang is carried entirely by `reason=sidecar_provider_missing`:
|
||||
|
||||
```json
|
||||
{
|
||||
"eventType": "esp_policy_provider_stalled",
|
||||
"severity": "Warning",
|
||||
"data": {
|
||||
"reason": "sidecar_provider_missing",
|
||||
"dwellMinutes": 15,
|
||||
"sidecarRegistered": false,
|
||||
"sidecarMissingForMinutes": 15.0,
|
||||
"providers": [
|
||||
{ "name": "ConfigMgr", "scope": "device", "kind": "setupApps",
|
||||
"trackingPoliciesCreated": null, "installationState": null, "complete": false }
|
||||
"trackingPoliciesCreated": 1, "installationState": null, "complete": true }
|
||||
],
|
||||
"stalledProviders": [
|
||||
{ "name": "ConfigMgr", "scope": "device", "kind": "setupApps", "stalledForMinutes": 15.0 }
|
||||
]
|
||||
"stalledProviders": []
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`sidecarRegistered=false` while `setupApps` providers exist is the co-management
|
||||
signature; `sidecarRegistered=true` with a stalled third-party provider generalizes the
|
||||
same contract violation.
|
||||
`reason=sidecar_provider_missing` is the co-management signature;
|
||||
`reason=provider_incomplete` with `sidecarRegistered=true` covers a stalled Sidecar or
|
||||
third-party provider — the generalized contract violation.
|
||||
|
||||
# Citations
|
||||
|
||||
* EnrollmentStatusTracking CSP — <https://learn.microsoft.com/en-us/windows/client-management/mdm/enrollmentstatustracking-csp>
|
||||
* Windows Autopilot into co-management (requirements, provider-conflict warning, troubleshooting registry paths) — <https://learn.microsoft.com/en-us/intune/configmgr/comanage/autopilot-enrollment>
|
||||
* Troubleshoot the ESP (models app tracking exclusively via the `Sidecar` subkey) — <https://learn.microsoft.com/en-us/troubleshoot/mem/intune/device-enrollment/understand-troubleshoot-esp>
|
||||
* Niehaus: the Intune ESP-policy WAP payload registering the provider nodes (`Sidecar` always, `ConfigMgr` only with a co-management settings policy) — <https://oofhours.com/2023/08/25/run-an-sccm-task-sequence-during-autopilot/>
|
||||
* Niehaus: ESP tracks arbitrarily named providers when they are properly registered — <https://oofhours.com/2023/09/18/track-anything-using-esp/>
|
||||
* Agent code: `EspPolicyProviderProbe.cs`, `EspPolicyProviderStallDetector.cs`, `EspPolicyProviderStallHost.cs` (SystemSignals / Orchestration.Hosts)
|
||||
* Related: [MDM Reboot Coalescing](mdm-reboot-coalescing.md), [Gather-Rule Guardrails](../rules/gather-rule-guardrails.md)
|
||||
|
||||
+1
-1
@@ -22,7 +22,7 @@ https://docs.autopilotmonitor.com (separate repository).
|
||||
* [Agent Endpoint Migration](agent/endpoint-migration.md) - How the backend re-homes agents to a new API base URL via the config channel (backend move or per-tenant region move): live-fetch-only, allowlist-validated on both sides, one hop, kill wins.
|
||||
* [MDM Reboot Coalescing](agent/mdm-reboot-coalescing.md) - Attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: aggregated DM-Enterprise 2800 observations (neutral, flush-time watermark) + the ANALYZE-ESP-005 advisory gated on an actually-observed reboot.
|
||||
* [Secure Boot CA-2023 Detection](agent/secureboot-ca2023-detection.md) - Why ANALYZE-SEC-001 v3 reads the UEFI db/KEK variables directly instead of trusting the SecureBoot\Servicing registry, and how the one-sided uefiCa2023FirmwareConfirmed marker plus a not_exists precondition give absence-tolerant suppression across old and new agents.
|
||||
* [ESP Policy-Provider Stall Detection](agent/esp-policy-provider-stall.md) - The EnrollmentStatusTracking CSP's no-timeout Setup/Apps wait: a registered provider that never sets TrackingPoliciesCreated (co-management "ConfigMgr" without the IME's "Sidecar") parks the user ESP at "Apps (Identifying)" for days; the always-on esp_policy_provider_stalled tripwire and the SOFTWARE\Microsoft\Windows\Autopilot gather-rule allowlist entry that surface it.
|
||||
* [ESP Policy-Provider Stall Detection](agent/esp-policy-provider-stall.md) - The EnrollmentStatusTracking CSP's no-timeout Setup/Apps wait is keyed to the Intune-registered "Sidecar" provider by NAME: co-management's "ConfigMgr" parks the user ESP at "Apps (Identifying)" for days even with TrackingPoliciesCreated=1 (issue #106); the always-on two-arm esp_policy_provider_stalled tripwire (provider_incomplete | sidecar_provider_missing) and the SOFTWARE\Microsoft\Windows\Autopilot gather-rule allowlist entry that surface it.
|
||||
|
||||
# Rules
|
||||
|
||||
|
||||
+3
-1
@@ -1,6 +1,8 @@
|
||||
# Log
|
||||
|
||||
## 2026-07-23
|
||||
## 2026-07-24
|
||||
|
||||
* **Update**: `agent/esp-policy-provider-stall.md` — the customer's follow-up on issue #106 corrected the field-case mechanics: `TrackingPoliciesCreated=1` WAS set under the foreign `ConfigMgr` provider, so the ESP ignores a foreign provider's completion values and waits on the Intune-registered `Sidecar` provider by name (the rename to `Sidecar` creates no tracking entries yet unblocks instantly). The completeness-keyed detector would have missed exactly this — the incident it was built for. The tripwire is now two-armed under one event type discriminated by `reason`: `provider_incomplete` (unchanged dwell on continuously-incomplete providers) and `sidecar_provider_missing` (≥ 1 Setup/Apps provider registered without `Sidecar`, completeness deliberately ignored, clock reset when Sidecar registers later). Sidecar-missing wins the `reason` when both arms cross in one round; the cross-restart fingerprint gains a `sidecarMissing` sentinel; payload adds `reason` and `sidecarMissingForMinutes`. Evidence for the name-keyed wait: Niehaus' documented Intune WAP payload always registers `Sidecar` (ConfigMgr only via a co-management settings policy) and Microsoft's ESP troubleshooting models app tracking exclusively through the Sidecar subkey.
|
||||
|
||||
* **Creation**: Added `agent/esp-policy-provider-stall.md` — a customer diagnosed that pre-provisioned devices with the ConfigMgr client as a blocking app occasionally park the user ESP at "Apps (Identifying)" for days: only a `ConfigMgr` provider is registered under `EnrollmentStatusTracking\Device\Setup\Apps\PolicyProviders` (no `Sidecar` = IME) and `TrackingPoliciesCreated` never becomes 1 — a wait the CSP defines without any timeout; renaming the key to `Sidecar` unblocks immediately. The agent now ships the always-on `esp_policy_provider_stalled` tripwire (60-s tick, 15-min continuous-incompleteness dwell, one-shot per provider, `StartupEventGate` fingerprint across restarts, purely observational via the InformationalEvent rail — no toggle, no ConfigVersion bump), generalized to any provider in both `Setup/Apps` and `DevicePreparation` kinds rather than hardcoding the ConfigMgr case. Companion change: `SOFTWARE\Microsoft\Windows\Autopilot` joins the gather-rule registry allowlist so tenants can target the subtree with custom rules (e.g. `not_exists` on the Sidecar key).
|
||||
|
||||
|
||||
+116
-10
@@ -74,14 +74,17 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
|
||||
var previous = new Dictionary<string, DateTime> { ["setupApps|device|ConfigMgr"] = T0 };
|
||||
|
||||
var missing = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
EspPolicyProviderSnapshot.Empty, previous, new HashSet<string>(), T0.AddHours(2), Dwell);
|
||||
EspPolicyProviderSnapshot.Empty, previous, new HashSet<string>(), T0, false, T0.AddHours(2), Dwell);
|
||||
Assert.Empty(missing.UpdatedFirstSeenUtc);
|
||||
Assert.Empty(missing.NewlyStalled);
|
||||
Assert.Null(missing.UpdatedSidecarMissingSinceUtc); // arm 2 resets too
|
||||
Assert.Null(missing.SidecarMissingStalledForMinutes);
|
||||
|
||||
var registeredButEmpty = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
Snapshot(), previous, new HashSet<string>(), T0.AddHours(2), Dwell);
|
||||
Snapshot(), previous, new HashSet<string>(), T0, false, T0.AddHours(2), Dwell);
|
||||
Assert.Empty(registeredButEmpty.UpdatedFirstSeenUtc);
|
||||
Assert.Empty(registeredButEmpty.NewlyStalled);
|
||||
Assert.Null(registeredButEmpty.UpdatedSidecarMissingSinceUtc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -90,14 +93,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
|
||||
var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: null));
|
||||
|
||||
var round1 = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
snapshot, new Dictionary<string, DateTime>(), new HashSet<string>(), T0, Dwell);
|
||||
snapshot, new Dictionary<string, DateTime>(), new HashSet<string>(), null, false, T0, Dwell);
|
||||
Assert.Equal(T0, round1.UpdatedFirstSeenUtc["setupApps|device|ConfigMgr"]);
|
||||
Assert.Empty(round1.NewlyStalled);
|
||||
|
||||
var round2 = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
snapshot, round1.UpdatedFirstSeenUtc, new HashSet<string>(), T0.AddMinutes(14), Dwell);
|
||||
snapshot, round1.UpdatedFirstSeenUtc, new HashSet<string>(),
|
||||
round1.UpdatedSidecarMissingSinceUtc, false, T0.AddMinutes(14), Dwell);
|
||||
Assert.Equal(T0, round2.UpdatedFirstSeenUtc["setupApps|device|ConfigMgr"]); // first-seen preserved
|
||||
Assert.Empty(round2.NewlyStalled);
|
||||
Assert.Null(round2.SidecarMissingStalledForMinutes); // 14 min < dwell on arm 2 as well
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -107,7 +112,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
|
||||
var seen = new Dictionary<string, DateTime> { ["setupApps|device|ConfigMgr"] = T0 };
|
||||
|
||||
var eval = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
snapshot, seen, new HashSet<string>(), T0.AddMinutes(15), Dwell);
|
||||
snapshot, seen, new HashSet<string>(), null, false, T0.AddMinutes(15), Dwell);
|
||||
|
||||
var stalled = Assert.Single(eval.NewlyStalled);
|
||||
Assert.Equal("ConfigMgr", stalled.Provider.Name);
|
||||
@@ -122,20 +127,20 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
|
||||
|
||||
// Completes mid-dwell → dropped from the map.
|
||||
var completed = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
Snapshot(SetupApps("ConfigMgr", tracking: 1)), seen, new HashSet<string>(), T0.AddMinutes(10), Dwell);
|
||||
Snapshot(SetupApps("ConfigMgr", tracking: 1)), seen, new HashSet<string>(), null, false, T0.AddMinutes(10), Dwell);
|
||||
Assert.Empty(completed.UpdatedFirstSeenUtc);
|
||||
|
||||
// Re-appears incomplete much later → dwell restarts from the new now, no stall yet.
|
||||
var reappeared = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
Snapshot(SetupApps("ConfigMgr", tracking: 0)), completed.UpdatedFirstSeenUtc,
|
||||
new HashSet<string>(), T0.AddHours(3), Dwell);
|
||||
new HashSet<string>(), null, false, T0.AddHours(3), Dwell);
|
||||
Assert.Equal(T0.AddHours(3), reappeared.UpdatedFirstSeenUtc[key]);
|
||||
Assert.Empty(reappeared.NewlyStalled);
|
||||
|
||||
// Disappearance behaves the same as completion.
|
||||
var disappeared = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
Snapshot(SetupApps("Sidecar", tracking: 1)), new Dictionary<string, DateTime> { [key] = T0 },
|
||||
new HashSet<string>(), T0.AddMinutes(20), Dwell);
|
||||
new HashSet<string>(), null, false, T0.AddMinutes(20), Dwell);
|
||||
Assert.False(disappeared.UpdatedFirstSeenUtc.ContainsKey(key));
|
||||
}
|
||||
|
||||
@@ -147,7 +152,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
|
||||
var fired = new HashSet<string>(StringComparer.OrdinalIgnoreCase) { "setupApps|device|ConfigMgr" };
|
||||
|
||||
var eval = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
snapshot, seen, fired, T0.AddHours(5), Dwell);
|
||||
snapshot, seen, fired, null, false, T0.AddHours(5), Dwell);
|
||||
|
||||
Assert.Empty(eval.NewlyStalled);
|
||||
Assert.True(eval.UpdatedFirstSeenUtc.ContainsKey("setupApps|device|ConfigMgr")); // still tracked
|
||||
@@ -162,7 +167,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
|
||||
DevicePrep("ConfigMgr", installState: 1));
|
||||
|
||||
var eval = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
snapshot, new Dictionary<string, DateTime>(), new HashSet<string>(), T0, Dwell);
|
||||
snapshot, new Dictionary<string, DateTime>(), new HashSet<string>(), null, false, T0, Dwell);
|
||||
|
||||
Assert.Equal(3, eval.UpdatedFirstSeenUtc.Count);
|
||||
Assert.Contains("setupApps|device|ConfigMgr", eval.UpdatedFirstSeenUtc.Keys);
|
||||
@@ -179,6 +184,71 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
|
||||
Assert.False(Snapshot(DevicePrep("Sidecar", installState: 3)).SidecarRegistered); // wrong kind
|
||||
Assert.False(EspPolicyProviderSnapshot.Empty.SidecarRegistered);
|
||||
}
|
||||
|
||||
// ------------------------------------------------- arm 2: sidecar-missing transition
|
||||
|
||||
[Fact]
|
||||
public void Issue106_foreign_provider_complete_but_sidecar_missing_stalls_after_dwell()
|
||||
{
|
||||
// The actual field case: ConfigMgr with TrackingPoliciesCreated=1 — "complete" by the
|
||||
// CSP value contract, arm 1 never fires — yet the ESP waits on the absent Sidecar.
|
||||
var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: 1));
|
||||
|
||||
var round1 = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
snapshot, new Dictionary<string, DateTime>(), new HashSet<string>(), null, false, T0, Dwell);
|
||||
Assert.Empty(round1.NewlyStalled); // arm 1 stays silent...
|
||||
Assert.Equal(T0, round1.UpdatedSidecarMissingSinceUtc); // ...but arm 2 starts its clock
|
||||
Assert.Null(round1.SidecarMissingStalledForMinutes);
|
||||
|
||||
var round2 = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
snapshot, round1.UpdatedFirstSeenUtc, new HashSet<string>(),
|
||||
round1.UpdatedSidecarMissingSinceUtc, false, T0.AddMinutes(15), Dwell);
|
||||
Assert.Empty(round2.NewlyStalled);
|
||||
Assert.Equal(15.0, round2.SidecarMissingStalledForMinutes!.Value, precision: 3);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Sidecar_registering_resets_the_sidecar_missing_clock()
|
||||
{
|
||||
// ConfigMgr first (legitimate startup ordering) → clock runs...
|
||||
var round1 = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
Snapshot(SetupApps("ConfigMgr", tracking: 1)),
|
||||
new Dictionary<string, DateTime>(), new HashSet<string>(), null, false, T0, Dwell);
|
||||
Assert.Equal(T0, round1.UpdatedSidecarMissingSinceUtc);
|
||||
|
||||
// ...Sidecar joins → condition clears, state resets.
|
||||
var round2 = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
Snapshot(SetupApps("ConfigMgr", tracking: 1), SetupApps("Sidecar", tracking: 0)),
|
||||
round1.UpdatedFirstSeenUtc, new HashSet<string>(),
|
||||
round1.UpdatedSidecarMissingSinceUtc, false, T0.AddMinutes(10), Dwell);
|
||||
Assert.Null(round2.UpdatedSidecarMissingSinceUtc);
|
||||
Assert.Null(round2.SidecarMissingStalledForMinutes);
|
||||
|
||||
// Sidecar gone again much later → clock restarts from the new now, no instant stall.
|
||||
var round3 = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
Snapshot(SetupApps("ConfigMgr", tracking: 1)),
|
||||
round2.UpdatedFirstSeenUtc, new HashSet<string>(),
|
||||
round2.UpdatedSidecarMissingSinceUtc, false, T0.AddHours(3), Dwell);
|
||||
Assert.Equal(T0.AddHours(3), round3.UpdatedSidecarMissingSinceUtc);
|
||||
Assert.Null(round3.SidecarMissingStalledForMinutes);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Sidecar_missing_one_shot_and_deviceprep_only_scoping()
|
||||
{
|
||||
// Already fired → never re-reported, but the clock keeps running.
|
||||
var latched = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
Snapshot(SetupApps("ConfigMgr", tracking: 1)),
|
||||
new Dictionary<string, DateTime>(), new HashSet<string>(), T0, true, T0.AddHours(5), Dwell);
|
||||
Assert.Null(latched.SidecarMissingStalledForMinutes);
|
||||
Assert.Equal(T0, latched.UpdatedSidecarMissingSinceUtc);
|
||||
|
||||
// DevicePreparation providers alone don't constitute the Setup/Apps wait — arm 2 off.
|
||||
var devicePrepOnly = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
Snapshot(DevicePrep("ConfigMgr", installState: 1)),
|
||||
new Dictionary<string, DateTime>(), new HashSet<string>(), null, false, T0, Dwell);
|
||||
Assert.Null(devicePrepOnly.UpdatedSidecarMissingSinceUtc);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -243,7 +313,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
|
||||
Assert.Equal(EspPolicyProviderStallDetector.SourceLabel, evt.Payload[SignalPayloadKeys.Source]);
|
||||
|
||||
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(evt.TypedPayload);
|
||||
// Both arms cross in the same round (ConfigMgr incomplete AND Sidecar absent) →
|
||||
// ONE event; sidecar-missing is the dominant reason.
|
||||
Assert.Equal(EspPolicyProviderStallDetector.ReasonSidecarMissing, data["reason"]);
|
||||
Assert.Equal(false, data["sidecarRegistered"]);
|
||||
Assert.Equal(15.0, data["sidecarMissingForMinutes"]);
|
||||
Assert.Equal(EspPolicyProviderStallDetector.DwellMinutes, data["dwellMinutes"]);
|
||||
var stalled = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object>>>(data["stalledProviders"]);
|
||||
var entry = Assert.Single(stalled);
|
||||
@@ -252,6 +326,36 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
|
||||
Assert.Equal(15.0, entry["stalledForMinutes"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Issue106_configmgr_complete_without_sidecar_still_emits_sidecar_missing()
|
||||
{
|
||||
// Regression test for the customer's follow-up on issue #106: TrackingPoliciesCreated=1
|
||||
// WAS present under ConfigMgr — the provider looks "complete", arm 1 never fires, yet
|
||||
// the ESP hangs because Sidecar is absent. Arm 2 must catch exactly this.
|
||||
var configMgrComplete = new EspPolicyProviderSnapshot(new[]
|
||||
{
|
||||
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindSetupApps,
|
||||
trackingPoliciesCreated: 1, installationState: null),
|
||||
});
|
||||
using var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => configMgrComplete);
|
||||
var (sut, sink, clock) = BuildDetector();
|
||||
|
||||
sut.Tick();
|
||||
Assert.Empty(StalledEvents(sink));
|
||||
|
||||
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
|
||||
sut.Tick();
|
||||
|
||||
var evt = Assert.Single(StalledEvents(sink));
|
||||
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(evt.TypedPayload);
|
||||
Assert.Equal(EspPolicyProviderStallDetector.ReasonSidecarMissing, data["reason"]);
|
||||
Assert.Equal(false, data["sidecarRegistered"]);
|
||||
Assert.Equal(15.0, data["sidecarMissingForMinutes"]);
|
||||
Assert.Empty(Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object>>>(data["stalledProviders"]));
|
||||
var providers = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object?>>>(data["providers"]);
|
||||
Assert.Equal(true, Assert.Single(providers)["complete"]); // the misleading "complete" is visible
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Sidecar_complete_configmgr_incomplete_reports_only_configmgr_with_sidecar_flag()
|
||||
{
|
||||
@@ -264,7 +368,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
|
||||
|
||||
var evt = Assert.Single(StalledEvents(sink));
|
||||
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(evt.TypedPayload);
|
||||
Assert.Equal(EspPolicyProviderStallDetector.ReasonProviderIncomplete, data["reason"]); // arm 2 off — Sidecar is there
|
||||
Assert.Equal(true, data["sidecarRegistered"]);
|
||||
Assert.False(data.ContainsKey("sidecarMissingForMinutes"));
|
||||
var stalled = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object>>>(data["stalledProviders"]);
|
||||
Assert.Equal("ConfigMgr", Assert.Single(stalled)["name"]);
|
||||
var providers = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object?>>>(data["providers"]);
|
||||
|
||||
+13
-7
@@ -35,13 +35,19 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
/// (registered during Autopilot-into-co-management, see the troubleshooting registry paths in
|
||||
/// https://learn.microsoft.com/en-us/intune/configmgr/comanage/autopilot-enrollment). Microsoft
|
||||
/// warns that ESP policy providers are not aware of each other and that pre-provisioning is
|
||||
/// unsupported with co-management. Field case that motivated this probe: a pre-provisioned
|
||||
/// device with the ConfigMgr client installed as a blocking app ended up with ONLY
|
||||
/// <c>ConfigMgr</c> registered under <c>Device\Setup\Apps\PolicyProviders</c> (no
|
||||
/// <c>Sidecar</c>) and <c>TrackingPoliciesCreated</c> never set — the user ESP hung at
|
||||
/// "Apps (Identifying)" for hours to days; manually renaming the key to <c>Sidecar</c>
|
||||
/// unblocked it immediately. <see cref="EspPolicyProviderStallDetector"/> turns this contract
|
||||
/// into the <c>esp_policy_provider_stalled</c> tripwire.
|
||||
/// unsupported with co-management. Field case that motivated this probe (issue #106): a
|
||||
/// pre-provisioned device with the ConfigMgr client installed as a blocking app ended up with
|
||||
/// ONLY <c>ConfigMgr</c> registered under <c>Device\Setup\Apps\PolicyProviders</c> (no
|
||||
/// <c>Sidecar</c>) — and although <c>TrackingPoliciesCreated=1</c> WAS set under it, the user
|
||||
/// ESP hung at "Apps (Identifying)" for hours to days; manually renaming the key to
|
||||
/// <c>Sidecar</c> unblocked it immediately. So the ESP's Apps wait is keyed to the provider
|
||||
/// the Intune service registered at enrollment (always <c>Sidecar</c>; ConfigMgr only joins
|
||||
/// via a co-management settings policy — see the WAP payload documented in
|
||||
/// https://oofhours.com/2023/08/25/run-an-sccm-task-sequence-during-autopilot/), and a
|
||||
/// foreign provider's completion values do not satisfy it. Detection therefore keys on the
|
||||
/// provider NAME topology, not only on completeness:
|
||||
/// <see cref="EspPolicyProviderStallDetector"/> turns this contract into the two-arm
|
||||
/// <c>esp_policy_provider_stalled</c> tripwire.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Fail-soft like <see cref="EspTrackingInfoProbe"/>: any failure is Debug-logged and yields
|
||||
|
||||
+116
-27
@@ -13,18 +13,30 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
{
|
||||
/// <summary>
|
||||
/// Turns the ESP policy-provider contract read by <see cref="EspPolicyProviderProbe"/> into
|
||||
/// the one-shot <c>esp_policy_provider_stalled</c> Warning: a registered provider that stays
|
||||
/// continuously incomplete for <see cref="DwellMinutes"/> minutes is reported once, with the
|
||||
/// full provider table in the payload. See the probe's header for the CSP contract, the
|
||||
/// official sources, and the co-management field case (ConfigMgr registered without Sidecar
|
||||
/// → user ESP parked at "Apps (Identifying)" with no OS timeout) this tripwire exists for.
|
||||
/// the one-shot <c>esp_policy_provider_stalled</c> Warning via two independent arms, each
|
||||
/// with the same <see cref="DwellMinutes"/>-minute continuous-dwell rule and the full
|
||||
/// provider table in the payload:
|
||||
/// <list type="bullet">
|
||||
/// <item><b>Arm 1 — <c>reason=provider_incomplete</c></b>: a registered provider stays
|
||||
/// continuously incomplete (e.g. Sidecar itself never sets TrackingPoliciesCreated).</item>
|
||||
/// <item><b>Arm 2 — <c>reason=sidecar_provider_missing</c></b>: at least one
|
||||
/// <c>Setup\Apps</c> provider is registered but none of them is <c>Sidecar</c> (the IME).
|
||||
/// This is the actual co-management field-case signature (issue #106): the foreign
|
||||
/// <c>ConfigMgr</c> provider had <c>TrackingPoliciesCreated=1</c> — i.e. was "complete"
|
||||
/// by the CSP value contract — yet the user ESP stayed parked at "Apps (Identifying)"
|
||||
/// until the key was renamed to <c>Sidecar</c>. The ESP's Apps wait is keyed to the
|
||||
/// Intune-registered provider by NAME, so provider completeness alone must not gate this
|
||||
/// detection. See the probe's header for sources.</item>
|
||||
/// </list>
|
||||
/// <para>
|
||||
/// Dwell semantics: the dwell clock per provider starts when it is first observed incomplete
|
||||
/// and RESETS when the provider completes or disappears — only <i>continuously</i> incomplete
|
||||
/// providers fire. A missing root key / empty provider list is normal early enrollment and
|
||||
/// clears all dwell state (probe D4). Purely observational: emission goes through
|
||||
/// <see cref="InformationalEventPost"/> (InformationalEvent pass-through, dispatch-guard
|
||||
/// exempt), never into decision state.
|
||||
/// Dwell semantics: arm 1's clock per provider starts when it is first observed incomplete
|
||||
/// and RESETS when the provider completes or disappears; arm 2's clock starts when the
|
||||
/// sidecar-missing condition is first observed and RESETS when Sidecar registers (covers the
|
||||
/// legitimate "ConfigMgr registered before Sidecar" startup ordering) — only
|
||||
/// <i>continuously</i> bad states fire. A missing root key / empty provider list is normal
|
||||
/// early enrollment and clears all dwell state (probe D4). Purely observational: emission
|
||||
/// goes through <see cref="InformationalEventPost"/> (InformationalEvent pass-through,
|
||||
/// dispatch-guard exempt), never into decision state.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Duplicate suppression: in-process one-shot per provider key, plus a
|
||||
@@ -44,6 +56,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
|
||||
internal const string SourceLabel = "EspPolicyProviderStallDetector";
|
||||
|
||||
/// <summary>Payload <c>reason</c>: a registered provider stayed continuously incomplete (arm 1).</summary>
|
||||
internal const string ReasonProviderIncomplete = "provider_incomplete";
|
||||
|
||||
/// <summary>Payload <c>reason</c>: Setup\Apps providers registered but Sidecar absent (arm 2 — dominant when both arms fire in one round).</summary>
|
||||
internal const string ReasonSidecarMissing = "sidecar_provider_missing";
|
||||
|
||||
/// <summary>
|
||||
/// Sentinel token representing the arm-2 condition in the cross-restart fingerprint.
|
||||
/// Cannot collide with provider keys — those always contain <c>|</c> separators.
|
||||
/// </summary>
|
||||
private const string SidecarMissingFingerprintToken = "sidecarMissing";
|
||||
|
||||
private readonly string _sessionId;
|
||||
private readonly string _tenantId;
|
||||
private readonly InformationalEventPost _post;
|
||||
@@ -57,6 +81,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
private readonly HashSet<string> _firedKeys =
|
||||
new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
/// <summary>Arm-2 dwell state: first observation of "setupApps providers without Sidecar"; null = condition not active.</summary>
|
||||
private DateTime? _sidecarMissingSinceUtc;
|
||||
|
||||
/// <summary>Arm-2 in-process one-shot latch (mirror of <see cref="_firedKeys"/>).</summary>
|
||||
private bool _sidecarMissingFired;
|
||||
|
||||
public EspPolicyProviderStallDetector(
|
||||
string sessionId,
|
||||
string tenantId,
|
||||
@@ -79,13 +109,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
internal void Tick()
|
||||
{
|
||||
var snapshot = EspPolicyProviderProbe.Read(_logger);
|
||||
var eval = EvaluateStallTransition(snapshot, _firstSeenIncompleteUtc, _firedKeys, _clock.UtcNow, _dwell);
|
||||
var eval = EvaluateStallTransition(
|
||||
snapshot, _firstSeenIncompleteUtc, _firedKeys,
|
||||
_sidecarMissingSinceUtc, _sidecarMissingFired, _clock.UtcNow, _dwell);
|
||||
|
||||
_firstSeenIncompleteUtc.Clear();
|
||||
foreach (var kv in eval.UpdatedFirstSeenUtc)
|
||||
_firstSeenIncompleteUtc[kv.Key] = kv.Value;
|
||||
_sidecarMissingSinceUtc = eval.UpdatedSidecarMissingSinceUtc;
|
||||
|
||||
if (eval.NewlyStalled.Count > 0)
|
||||
if (eval.NewlyStalled.Count > 0 || eval.SidecarMissingStalledForMinutes != null)
|
||||
EmitStalled(snapshot, eval);
|
||||
}
|
||||
|
||||
@@ -94,24 +127,29 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
/// registry, timer, or ingress (pattern: <c>PerformanceCollector.EvaluateDiskLowTransition</c>).
|
||||
/// </summary>
|
||||
/// <param name="snapshot">Current provider snapshot.</param>
|
||||
/// <param name="firstSeenIncompleteUtc">Dwell state from the previous round (provider key → first-seen-incomplete).</param>
|
||||
/// <param name="firstSeenIncompleteUtc">Arm-1 dwell state from the previous round (provider key → first-seen-incomplete).</param>
|
||||
/// <param name="alreadyFired">Provider keys already reported this run (never re-report).</param>
|
||||
/// <param name="sidecarMissingSinceUtc">Arm-2 dwell state from the previous round; null = condition was not active.</param>
|
||||
/// <param name="sidecarMissingAlreadyFired">Arm-2 one-shot latch (never re-report).</param>
|
||||
/// <param name="nowUtc">Evaluation timestamp.</param>
|
||||
/// <param name="dwell">Continuous-incompleteness threshold.</param>
|
||||
/// <param name="dwell">Continuous-condition threshold (shared by both arms).</param>
|
||||
internal static StallEvaluation EvaluateStallTransition(
|
||||
EspPolicyProviderSnapshot snapshot,
|
||||
IReadOnlyDictionary<string, DateTime> firstSeenIncompleteUtc,
|
||||
ISet<string> alreadyFired,
|
||||
DateTime? sidecarMissingSinceUtc,
|
||||
bool sidecarMissingAlreadyFired,
|
||||
DateTime nowUtc,
|
||||
TimeSpan dwell)
|
||||
{
|
||||
var result = new StallEvaluation();
|
||||
|
||||
// Missing root key or no registered providers = normal early enrollment: clear all
|
||||
// dwell state (a provider must be continuously OBSERVED incomplete to accrue dwell).
|
||||
// dwell state (a condition must be continuously OBSERVED to accrue dwell).
|
||||
if (!snapshot.HasData || snapshot.Providers.Count == 0)
|
||||
return result;
|
||||
|
||||
// Arm 1 — a registered provider continuously incomplete.
|
||||
foreach (var provider in snapshot.Providers)
|
||||
{
|
||||
if (provider.IsComplete)
|
||||
@@ -127,22 +165,44 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
result.NewlyStalled.Add(new StalledProvider(provider, stalledFor.TotalMinutes));
|
||||
}
|
||||
|
||||
// Arm 2 — Setup\Apps providers registered, but none of them is Sidecar. Deliberately
|
||||
// ignores provider completeness: the field case (issue #106) had ConfigMgr with
|
||||
// TrackingPoliciesCreated=1 and the ESP still waited. Sidecar registering later
|
||||
// (legitimate startup ordering) clears the condition and resets the clock.
|
||||
if (HasSetupAppsProvider(snapshot) && !snapshot.SidecarRegistered)
|
||||
{
|
||||
var since = sidecarMissingSinceUtc ?? nowUtc;
|
||||
result.UpdatedSidecarMissingSinceUtc = since;
|
||||
|
||||
var missingFor = nowUtc - since;
|
||||
if (missingFor >= dwell && !sidecarMissingAlreadyFired)
|
||||
result.SidecarMissingStalledForMinutes = missingFor.TotalMinutes;
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
private static bool HasSetupAppsProvider(EspPolicyProviderSnapshot snapshot)
|
||||
=> snapshot.Providers.Any(p => p.Kind == EspPolicyProviderProbe.KindSetupApps);
|
||||
|
||||
private void EmitStalled(EspPolicyProviderSnapshot snapshot, StallEvaluation eval)
|
||||
{
|
||||
foreach (var stalled in eval.NewlyStalled)
|
||||
_firedKeys.Add(stalled.Provider.Key);
|
||||
if (eval.SidecarMissingStalledForMinutes != null)
|
||||
_sidecarMissingFired = true;
|
||||
|
||||
// Cross-restart dedup: fingerprint = sorted set of all fired keys that are still
|
||||
// incomplete. A restart into the same stall re-accrues dwell and lands on the same
|
||||
// fingerprint → suppressed; a new provider joining the stall changes it → re-emitted.
|
||||
// Cross-restart dedup: fingerprint = sorted set of all fired conditions that are
|
||||
// still active (incomplete providers + the sidecar-missing sentinel). A restart into
|
||||
// the same stall re-accrues dwell and lands on the same fingerprint → suppressed; a
|
||||
// new condition joining the stall changes it → re-emitted.
|
||||
var currentlyStalledKeys = snapshot.Providers
|
||||
.Where(p => !p.IsComplete && _firedKeys.Contains(p.Key))
|
||||
.Select(p => p.Key)
|
||||
.OrderBy(k => k, StringComparer.OrdinalIgnoreCase)
|
||||
.ToList();
|
||||
if (_sidecarMissingFired && HasSetupAppsProvider(snapshot) && !snapshot.SidecarRegistered)
|
||||
currentlyStalledKeys.Add(SidecarMissingFingerprintToken);
|
||||
currentlyStalledKeys.Sort(StringComparer.OrdinalIgnoreCase);
|
||||
var fingerprint = StartupEventGate.HashString(string.Join(";", currentlyStalledKeys));
|
||||
if (_startupGate != null &&
|
||||
!_startupGate.ShouldEmit(Constants.EventTypes.EspPolicyProviderStalled, fingerprint))
|
||||
@@ -153,11 +213,30 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
return;
|
||||
}
|
||||
|
||||
// Sidecar-missing is the dominant reason when both arms fire in one round: it
|
||||
// explains the ESP hang regardless of the foreign providers' completeness.
|
||||
var reason = eval.SidecarMissingStalledForMinutes != null
|
||||
? ReasonSidecarMissing
|
||||
: ReasonProviderIncomplete;
|
||||
|
||||
var messageParts = new List<string>();
|
||||
if (eval.SidecarMissingStalledForMinutes != null)
|
||||
{
|
||||
var registeredNames = string.Join(", ", snapshot.Providers
|
||||
.Where(p => p.Kind == EspPolicyProviderProbe.KindSetupApps)
|
||||
.Select(p => $"{p.Name} ({p.Scope})"));
|
||||
messageParts.Add(
|
||||
$"expected IME provider '{EspPolicyProviderProbe.SidecarProviderName}' not registered for " +
|
||||
$">= {DwellMinutes} min while Setup/Apps providers exist: {registeredNames}");
|
||||
}
|
||||
var stalledNames = string.Join(", ",
|
||||
eval.NewlyStalled.Select(s => $"{s.Provider.Name} ({s.Provider.Scope}/{s.Provider.Kind})"));
|
||||
if (eval.NewlyStalled.Count > 0)
|
||||
messageParts.Add($"provider(s) incomplete for >= {DwellMinutes} min: {stalledNames}");
|
||||
|
||||
var data = new Dictionary<string, object>(StringComparer.Ordinal)
|
||||
{
|
||||
["reason"] = reason,
|
||||
["dwellMinutes"] = DwellMinutes,
|
||||
["sidecarRegistered"] = snapshot.SidecarRegistered,
|
||||
["providers"] = snapshot.Providers.Select(p => new Dictionary<string, object?>(StringComparer.Ordinal)
|
||||
@@ -177,6 +256,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
["stalledForMinutes"] = Math.Round(s.StalledForMinutes, 1),
|
||||
}).ToList(),
|
||||
};
|
||||
if (eval.SidecarMissingStalledForMinutes != null)
|
||||
data["sidecarMissingForMinutes"] = Math.Round(eval.SidecarMissingStalledForMinutes.Value, 1);
|
||||
|
||||
var message =
|
||||
"ESP Setup/Apps tracking stalled: " + string.Join("; ", messageParts) +
|
||||
" — the ESP waits on its app-tracking providers without any timeout";
|
||||
|
||||
_post.Emit(new EnrollmentEvent
|
||||
{
|
||||
@@ -188,27 +273,31 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
Source = SourceLabel,
|
||||
Phase = EnrollmentPhase.Unknown,
|
||||
ImmediateUpload = true,
|
||||
Message = $"ESP policy provider(s) incomplete for >= {DwellMinutes} min: {stalledNames} " +
|
||||
$"(sidecarRegistered={snapshot.SidecarRegistered}) — the ESP waits on " +
|
||||
"Setup/Apps providers without any timeout",
|
||||
Message = message,
|
||||
Data = data
|
||||
});
|
||||
_startupGate?.MarkEmitted(Constants.EventTypes.EspPolicyProviderStalled);
|
||||
|
||||
_logger?.Warning(
|
||||
$"EspPolicyProviderStallDetector: {stalledNames} incomplete for >= {DwellMinutes} min " +
|
||||
$"(sidecarRegistered={snapshot.SidecarRegistered}).");
|
||||
$"EspPolicyProviderStallDetector: {string.Join("; ", messageParts)} " +
|
||||
$"(reason={reason}, sidecarRegistered={snapshot.SidecarRegistered}).");
|
||||
}
|
||||
|
||||
/// <summary>Result of one <see cref="EvaluateStallTransition"/> round.</summary>
|
||||
internal sealed class StallEvaluation
|
||||
{
|
||||
/// <summary>Next round's dwell state — only currently-incomplete providers survive.</summary>
|
||||
/// <summary>Next round's arm-1 dwell state — only currently-incomplete providers survive.</summary>
|
||||
public Dictionary<string, DateTime> UpdatedFirstSeenUtc { get; } =
|
||||
new Dictionary<string, DateTime>(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
/// <summary>Providers that crossed the dwell threshold this round and were not yet reported.</summary>
|
||||
/// <summary>Providers that crossed the dwell threshold this round and were not yet reported (arm 1).</summary>
|
||||
public List<StalledProvider> NewlyStalled { get; } = new List<StalledProvider>();
|
||||
|
||||
/// <summary>Next round's arm-2 dwell state; null = sidecar-missing condition not active (reset).</summary>
|
||||
public DateTime? UpdatedSidecarMissingSinceUtc { get; set; }
|
||||
|
||||
/// <summary>Non-null when the sidecar-missing condition crossed the dwell threshold this round and was not yet reported (arm 2).</summary>
|
||||
public double? SidecarMissingStalledForMinutes { get; set; }
|
||||
}
|
||||
|
||||
internal readonly struct StalledProvider
|
||||
|
||||
@@ -37,8 +37,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
|
||||
/// <see cref="StallProbeCollectorAdapter"/>. Owns its 60-s idle-check timer (the
|
||||
/// collector itself has no timer — it's a pure probe invoked from outside).</item>
|
||||
/// <item><b>EspPolicyProviderStallHost</b> — always-on 60-s wall-clock tick for the
|
||||
/// <c>esp_policy_provider_stalled</c> tripwire (registered EnrollmentStatusTracking CSP
|
||||
/// provider continuously incomplete ≥ 15 min — the no-timeout ESP wait).</item>
|
||||
/// two-arm <c>esp_policy_provider_stalled</c> tripwire (EnrollmentStatusTracking CSP
|
||||
/// provider continuously incomplete ≥ 15 min, OR Setup\Apps providers registered
|
||||
/// without the expected <c>Sidecar</c>/IME provider — the no-timeout ESP wait).</item>
|
||||
/// </list>
|
||||
/// Optional peripheral hosts (driven by <see cref="CollectorConfiguration"/> toggles):
|
||||
/// <list type="bullet">
|
||||
@@ -307,11 +308,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
|
||||
}
|
||||
|
||||
// ESP policy-provider stall tripwire — always-on kernel host (no config gate, like
|
||||
// disk_space_low): a registered EnrollmentStatusTracking CSP provider that never
|
||||
// completes (e.g. co-management "ConfigMgr" without "Sidecar" on a pre-provisioned
|
||||
// device) parks the user ESP at "Apps (Identifying)" WITHOUT any OS timeout. Wall-clock
|
||||
// dwell, deliberately outside the StallProbe gate: the idle clock there resets on any
|
||||
// session activity, and the stall must accrue regardless of it.
|
||||
// disk_space_low): a provider that never completes, or co-management "ConfigMgr"
|
||||
// registered without the expected "Sidecar"/IME provider (issue #106 — even with
|
||||
// TrackingPoliciesCreated=1 the ESP ignores foreign provider names), parks the user
|
||||
// ESP at "Apps (Identifying)" WITHOUT any OS timeout. Wall-clock dwell, deliberately
|
||||
// outside the StallProbe gate: the idle clock there resets on any session activity,
|
||||
// and the stall must accrue regardless of it.
|
||||
hosts.Add(new EspPolicyProviderStallHost(
|
||||
sessionId: sessionId,
|
||||
tenantId: tenantId,
|
||||
|
||||
@@ -59,7 +59,7 @@ export const KNOWN_EVENT_TYPES: EventTypeEntry[] = [
|
||||
{ value: "stall_probe_check", label: "stall_probe_check", category: "stall",
|
||||
description: "Trace-level heartbeat from a stall probe — proves the logic ran (default: only the 15 min probe)." },
|
||||
{ value: "esp_policy_provider_stalled", label: "esp_policy_provider_stalled", category: "stall",
|
||||
description: "A registered ESP policy provider (EnrollmentStatusTracking CSP) stayed incomplete for 15+ minutes — e.g. co-management leaves only ConfigMgr registered without the Intune Sidecar provider, which parks the user ESP at 'Apps (Identifying)' with no OS timeout. One-shot Warning; payload carries the full provider table, per-provider state, and sidecarRegistered." },
|
||||
description: "The ESP's app-tracking wait (EnrollmentStatusTracking CSP) made no progress for 15+ minutes: a registered policy provider stayed incomplete (reason=provider_incomplete), or Setup/Apps providers exist without the expected Intune 'Sidecar' provider (reason=sidecar_provider_missing) — the co-management signature where only ConfigMgr is registered (even with TrackingPoliciesCreated=1) and the user ESP parks at 'Apps (Identifying)' with no OS timeout. One-shot Warning; payload carries reason, the full provider table, per-provider state, and sidecarRegistered." },
|
||||
{ value: "agent_late_start", label: "agent_late_start", category: "stall",
|
||||
description: "Low observation coverage: the agent started long after device boot (≥10 min) AND lived only briefly (≤5 min) before a terminal outcome — it arrived after the enrollment had already decided, so its diagnosis is a post-mortem of the end-state, not a live observation. Carries bootToAgentStartSeconds, agentUptimeSeconds and outcome. Often paired with script_timeout_suspected (a script that hung ahead of the agent bootstrap)." },
|
||||
|
||||
|
||||
Reference in New Issue
Block a user