Compare commits

...

10 Commits

Author SHA1 Message Date
Oliver Kieselbach fd7d3973e7 fix(agent): esp_policy_provider_stalled gains sidecar-missing arm (issue #106)
Customer follow-up corrected the field-case mechanics: TrackingPoliciesCreated=1
WAS set under the foreign ConfigMgr provider - the ESP ignores a foreign
provider's completion values and waits on the Intune-registered "Sidecar"
provider by name (renaming the key unblocks instantly without creating any
tracking entries). The completeness-keyed detector would have missed exactly
the incident it was built for.

- Arm 2: >=1 Setup\Apps provider registered without Sidecar, continuously for
  the 15-min dwell -> reason=sidecar_provider_missing; clock resets when
  Sidecar registers later (legitimate startup ordering); DevicePreparation-only
  snapshots don't activate it
- Arm 1 (continuously-incomplete providers) unchanged -> reason=provider_incomplete;
  sidecar-missing wins when both arms cross in one round (single event)
- Cross-restart fingerprint gains a "sidecarMissing" sentinel; payload adds
  reason + sidecarMissingForMinutes
- Regression test: ConfigMgr complete + Sidecar absent -> fires (previously
  classified healthy); probe/doc headers and OKF doc corrected, web event-type
  description updated

Evidence (research session c7e3db08): Intune's ESP-policy WAP payload always
registers Sidecar (ConfigMgr only via co-management settings policy, per
Niehaus' documented payload); Microsoft's ESP troubleshooting models app
tracking exclusively through the Sidecar subkey.

Agent tests 2043/2043, web tsc clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 16:48:50 +02:00
Oliver Kieselbach fd2b5c2da2 fix(ci): drop invalid SWA input, gate deploy on token via env guard
Azure/static-web-apps-deploy@v1 has no `skip_deploy_on_missing_secrets`
input (confirmed against its action.yml), so passing it did nothing but
emit an "Unexpected input(s)" warning — the intended "skip green on
Dependabot PRs" behaviour was never actually in effect.

Replace it with the documented mechanism: surface the deploy token as a
job-level env var and gate the deploy/close step with `if: env.DEPLOY_TOKEN
!= ''` (secrets cannot be referenced directly in an `if:`). On Dependabot
PRs the token is absent, so the step skips and the job stays green; real
deploys on main and internal PRs still run. Applied to both jobs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 16:38:58 +02:00
Oliver Kieselbach a44e37a7c2 fix(metrics): keep full failure-reason text for fleet-health display
BuildFleetFailureReasons truncated each reason to a 50-char prefix + "..."
and used that as both the grouping key AND the display value. The full text
never reached the UI, so its tooltip/tap-to-expand had nothing to reveal and
near-identical rows were indistinguishable past the cut.

Group by the 50-char prefix as before (variants still collapse into one row),
but carry the longest full reason as the display value. The UI truncates and
expands it on demand, so the complete text is now available end-to-end.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 16:25:11 +02:00
Oliver Kieselbach d8a252562d fix(web): pin engines.node to 22.x so SWA/Oryx does not grab Node 24
The prior ">=22" made Oryx build with the highest available Node (24.13),
which the Static Web Apps runtime rejects: "Node version 24.13.0 is not
supported. Please use one of the following versions 18, 20, 22." Pin to
22.x so Oryx selects the SWA-supported major. CI setup-node '22' already
resolves to latest 22.x, so build and deploy now agree.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 16:23:17 +02:00
Oliver Kieselbach a5d12ba1d0 chore(ci): unify Node toolchain on 22 across CI + engines pins
Node 20 reached EOL (2026-04-30); Node 22 is the current maintenance LTS
and already the runtime the MCP container ships (node:22-slim). Straighten
the version skew so per-component CI major == deploy runtime == engines floor:

- ci.yml: web, mcp and guardrails setup-node 20 -> 22
- combine-rules.yml: setup-node 20 -> 22
- mcp package.json: add engines.node ">=22" (matches Dockerfile)
- web package.json: add engines.node ">=22" (first explicit pin; SWA/Oryx
  no longer picks a Node version implicitly)

engine-strict stays off (no .npmrc), so the pins are advisory and cannot
hard-break local installs on other Node versions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 16:15:54 +02:00
Oliver Kieselbach 530c978bf8 fix(web): TruncatedLabel expands to full text, not just two lines
The expanded state used line-clamp-2, which re-truncated long values with an
ellipsis after two lines — unreadable on touch where the hover tooltip is
unavailable. Expand now wraps the entire string over as many lines as needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 15:11:54 +02:00
Oliver Kieselbach 09898d7ba0 feat(web): consistent truncate-with-tooltip + tap-to-expand across dashboard cards
Add reusable TruncatedLabel component: a single-line label that truncates
with an ellipsis, shows the full text as a native hover tooltip, and on
click/Enter/Space expands to two wrapped lines (touch-friendly). An
interactive={false} mode provides tooltip-only behaviour for use inside
already-clickable rows/links where an inner onClick would hijack navigation.

Apply it consistently to the long, dynamic label columns across:
- fleet-health: failure reasons, model + app ranked lists
- sla: top failing apps, violators failure reason
- geographic-performance: fastest/slowest location cards
- vulnerability exposure panel: CVE sample software
- agent metrics: top exception types
- progress + download/install: app-name labels
- admin software tabs: software names
- session reports (comment/adminNote) + ops events (message): tooltip-only

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 15:03:34 +02:00
Oliver Kieselbach 71eda93e8b fix(metrics): app-install failure rate over finished installs only
Second half of the terminal-only rate convention (follows 1cfc32c9, which
converted the enrollment success rates): app-install failureRate divided
by ALL install rows including "InProgress" ones — installs still running,
or orphaned when a session died mid-install — diluting every failure rate
customers see on the Apps dashboard and Fleet Health.

- New MetricsMath.TerminalFailureRatePct (Failed / (Failed + Succeeded),
  0 when nothing finished) shared by both aggregation sites
- MetricsMath.BuildAppMetricsPayload: per-app failureRate terminal-based
  (Fleet Health "Top Failing Apps"; subtitle now "% of N finished")
- AppsAnalyticsHelper: list, analytics summary, versionBreakdown,
  timeSeries and deviceModelBreakdown rates terminal-based; the
  duplicated trend block is deduplicated into ComputeFailureTrend, whose
  rates and >= 5-sample gate now run on finished installs per half; the
  device-model >= 5 sample floor likewise counts finished installs
- Web: InstallsTab "Avg Failure Rate" card computed over summed finished
  installs instead of totalInstalls

Tests: new AppsAnalyticsFailureRateTests (list/trend/analytics/model
cases incl. in-flight dilution and gate regressions) and a terminal-rate
case in MetricsMathAppPayloadTests. Backend 3223/3223, tsc clean,
vitest 602/602.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 10:01:28 +02:00
Oliver Kieselbach 1cfc32c9aa fix(metrics): success rate over finished enrollments only (geo + fleet health)
Customer feedback: the geographic performance pane showed a poor success
rate for a site mid-rollout because in-progress/pending sessions counted
against the rate. Success rate is an outcome quota — align Geo and Fleet
Health with the SLA convention already used by dashboard stats, usage
snapshots and the fleet rollup: Succeeded / (Succeeded + Failed).
Incomplete (terminal, non-failure) stays excluded.

- Geo: per-location SuccessRate over finished sessions; table badge and
  map tooltip render an em dash instead of a red 0% when nothing has
  finished yet
- Fleet Health: Stats.SuccessRate terminal-based; card shows "X of Y
  finished enrollments" and a neutral em dash state; per-model health
  gets a Failed count (additive wire field) so the client rate excludes
  in-flight devices; TopFailingModels FailureRate now failed/finished
- Tests: FleetHealthPayloadTests flipped to the new convention plus
  zero-finished, model-failed and in-flight-dilution cases; new
  GeographicMetricsSuccessRateTests pin the geo semantics

Backend 3216/3216, web tsc clean, vitest 602/602.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 09:47:31 +02:00
Oliver Kieselbach 11339c9868 fix(test): de-flake SessionDeletionWorkerTests via condition-gated runs
CI run 30047512186 failed Worker_drops_envelope_missing_required_fields_
without_invoking_handler: RunForAsync starts the BackgroundService poll loop
and waits a fixed 500ms wall-clock window — on a loaded CI runner (xUnit runs
test classes in parallel) the loop may not have processed the message before
the positive assertions (DeleteMessageAsync at least once) run. All ten
positive tests in the file shared that latent race; this run it hit the
malformed-envelope drop test.

Fix mirrors the watchdog event-gate de-flake: new Harness.RunUntilAsync
(condition, 10s ceiling, 25ms poll) runs the worker until the awaited side
effect is observable, then stops. Gates: ops-event recorded (last step of the
poison path, so send+delete+CAS are already done), main-queue delete for the
drop tests, >=3 visibility heartbeats for the heartbeat test. The kill-switch
test keeps the fixed window — a Times.Never expectation has no event to wait
for. OpsEventCalls capture is now lock-guarded so the condition poll can read
it while the worker thread appends.

16/16 worker tests green across 4 consecutive runs; full backend suite
3211/3211.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 00:03:10 +02:00
38 changed files with 1055 additions and 249 deletions
+3 -3
View File
@@ -63,7 +63,7 @@ jobs:
- uses: actions/setup-node@v7
with:
node-version: '20'
node-version: '22'
cache: npm
cache-dependency-path: src/Web/autopilot-monitor-web/package-lock.json
@@ -86,7 +86,7 @@ jobs:
- uses: actions/setup-node@v7
with:
node-version: '20'
node-version: '22'
cache: npm
cache-dependency-path: src/McpServer/autopilot-monitor-mcp/package-lock.json
@@ -106,7 +106,7 @@ jobs:
- uses: actions/setup-node@v7
with:
node-version: '20'
node-version: '22'
- name: Regenerate
run: node rules/scripts/combine.js
+1 -1
View File
@@ -26,7 +26,7 @@ jobs:
- uses: actions/setup-node@v7
with:
node-version: '20'
node-version: '22'
- name: Install ajv-cli
run: npm install -g ajv-cli ajv-formats
+11 -3
View File
@@ -21,6 +21,11 @@ jobs:
if: github.event_name == 'push' || github.event_name == 'workflow_dispatch' || (github.event_name == 'pull_request' && github.event.action != 'closed')
runs-on: ubuntu-latest
name: Build and Deploy Job
# Dependabot PRs run without repo secrets, so the deploy token is absent.
# Surface it as an env var so the deploy step can gate on it — secrets
# cannot be referenced directly in an `if:` expression.
env:
DEPLOY_TOKEN: ${{ secrets.AZURE_SWA_DEPLOY_TOKEN }}
steps:
- uses: actions/checkout@v7
with:
@@ -28,6 +33,9 @@ jobs:
fetch-depth: 0 # Full history needed for git-based sitemap lastmod dates
- name: Build And Deploy
id: builddeploy
# Skip (green) instead of failing when the token is missing (Dependabot
# PRs); real deploys on main and internal PRs still have it.
if: env.DEPLOY_TOKEN != ''
uses: Azure/static-web-apps-deploy@v1
env:
NEXT_PUBLIC_ENTRA_CLIENT_ID: ${{ vars.NEXT_PUBLIC_ENTRA_CLIENT_ID }}
@@ -39,9 +47,6 @@ jobs:
with:
azure_static_web_apps_api_token: ${{ secrets.AZURE_SWA_DEPLOY_TOKEN }}
repo_token: ${{ secrets.GITHUB_TOKEN }} # Used for Github integrations (i.e. PR comments)
# Dependabot PRs run without repo secrets, so the deploy token is absent.
# Skip (green) instead of failing the check; real deploys on main still have the token.
skip_deploy_on_missing_secrets: true
action: "upload"
###### Repository/Build Configurations - These values can be configured to match your app requirements. ######
# For more information regarding Static Web App workflow configurations, please visit: https://aka.ms/swaworkflowconfig
@@ -54,9 +59,12 @@ jobs:
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
name: Close Pull Request Job
env:
DEPLOY_TOKEN: ${{ secrets.AZURE_SWA_DEPLOY_TOKEN }}
steps:
- name: Close Pull Request
id: closepullrequest
if: env.DEPLOY_TOKEN != ''
uses: Azure/static-web-apps-deploy@v1
with:
azure_static_web_apps_api_token: ${{ secrets.AZURE_SWA_DEPLOY_TOKEN }}
+55 -29
View File
@@ -1,7 +1,7 @@
---
type: Concept
title: ESP Policy-Provider Stall Detection — the no-timeout Setup/Apps wait and the co-management Sidecar gap
description: Why a registered EnrollmentStatusTracking CSP policy provider that never sets TrackingPoliciesCreated parks the user ESP at "Apps (Identifying)" indefinitely, the pre-provisioning + ConfigMgr field case, and how the always-on esp_policy_provider_stalled tripwire (15-min wall-clock dwell, StartupEventGate cross-restart dedup) surfaces it.
description: Why the ESP's Apps wait is keyed to the Intune-registered "Sidecar" provider by name — a foreign provider (ConfigMgr) with TrackingPoliciesCreated=1 does not satisfy it — plus the classic never-completes stall, and how the always-on two-arm esp_policy_provider_stalled tripwire (reason=provider_incomplete | sidecar_provider_missing, 15-min wall-clock dwell, StartupEventGate cross-restart dedup) surfaces both.
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/EspPolicyProviderProbe.cs
tags:
- agent
@@ -9,7 +9,7 @@ tags:
- enrollment-status-tracking
- co-management
- stall-detection
timestamp: 2026-07-23T00:00:00+02:00
timestamp: 2026-07-24T00:00:00+02:00
---
# ESP Policy-Provider Stall Detection
@@ -42,37 +42,59 @@ exactly these registry paths. The same document states that ESP policy providers
aware of each other** and that **pre-provisioning is unsupported** with Autopilot into
co-management.
## The field case
## The field case (issue #106)
A tenant pre-provisions devices and installs the ConfigMgr client as a blocking app.
Occasionally the user-phase ESP reaches "Apps (Identifying)" and never leaves it — for
hours to days. On those devices `Device\Setup\Apps\PolicyProviders` contains **only a
`ConfigMgr` key (no `Sidecar`)** and `TrackingPoliciesCreated` is never set. Manually
renaming the key to `Sidecar` unblocks the ESP immediately (the IME then finds its
provider node and reports tracking-policies-created). The mechanism is exactly the
documented no-timeout wait; the unsupported pre-provisioning + co-management combination
explains why only some devices are affected. Why the Sidecar registration is missing or
displaced on those devices is not publicly documented — irrelevant for detection, since
the signature is deterministic.
`ConfigMgr` key (no `Sidecar`)** — and, per the customer's follow-up, **`TrackingPoliciesCreated=1`
IS set under it**. The provider is "complete" by the CSP value contract, yet the ESP
still waits. Manually renaming the key to `Sidecar` — which creates no tracking entries,
only satisfies the name — unblocks the ESP instantly.
The naive CSP-doc reading ("any provider with TrackingPoliciesCreated=1 satisfies the
wait") is therefore wrong for this case. What the evidence supports instead: provider
nodes are registered **by the Intune service at enrollment**, and its ESP-policy WAP
payload always registers `Sidecar` (ConfigMgr joins only via a co-management settings
policy — see the payload documented by Niehaus). The ESP waits on the provider it
expects **by name**; a foreign provider occupying the slot does not satisfy it,
regardless of its completion values. On the affected devices the CM client (installed
as a plain Win32 app, no co-management settings policy) evidently displaced the
Sidecar node — consistent with Microsoft's warning that providers "aren't currently
aware of others" and that pre-provisioning is unsupported with co-management. The exact
CloudExperienceHost mechanism is not publicly documented; the registry signature —
setupApps providers present, `Sidecar` absent — is deterministic either way.
## Detection
`EspPolicyProviderProbe` (fail-soft, static, `ScopedOverride` test seam — structural clone
of `EspTrackingInfoProbe`) enumerates all providers in both kinds and scopes.
`EspPolicyProviderStallDetector` evaluates a pure transition function on a 60-second tick
from the always-on `EspPolicyProviderStallHost`:
from the always-on `EspPolicyProviderStallHost` — **two arms**, same 15-minute wall-clock
dwell, one shared event type discriminated by `reason`:
* A provider **continuously incomplete for ≥ 15 minutes** (wall-clock) is stalled. The
dwell clock starts at first incomplete observation and resets on completion or
disappearance. Missing values parse as incomplete (fail-toward-observation).
* **Arm 1, `reason=provider_incomplete`** — a provider **continuously incomplete for
≥ 15 minutes**. The dwell clock starts at first incomplete observation and resets on
completion or disappearance. Missing values parse as incomplete
(fail-toward-observation). Catches e.g. Sidecar itself never reporting.
* **Arm 2, `reason=sidecar_provider_missing`** — **≥ 1 `Setup\Apps` provider registered
but none named `Sidecar`**, continuously for ≥ 15 minutes. Deliberately ignores
provider completeness (the field case had `TrackingPoliciesCreated=1`). Sidecar
registering later — the legitimate startup ordering — clears the condition and resets
the clock. DevicePreparation-only snapshots don't activate it.
* When both arms cross in the same round, ONE event is emitted and
`sidecar_provider_missing` wins as `reason` (it explains the hang regardless of the
foreign provider's state).
* Missing root key / empty provider list = normal early enrollment: clears all dwell
state, never fires.
* One `esp_policy_provider_stalled` **Warning** per newly stalled provider set
(`ImmediateUpload`), one-shot per provider in-process, cross-restart deduped via a
`StartupEventGate` fingerprint over the sorted currently-stalled key set — a restart
into the same stall stays silent, a new provider joining the stall re-reports.
* Payload: `providers[]` (full table: name, scope, kind, raw values, complete flag),
`stalledProviders[]` with `stalledForMinutes`, `sidecarRegistered`, `dwellMinutes`.
state (both arms), never fires.
* One `esp_policy_provider_stalled` **Warning** per newly stalled condition set
(`ImmediateUpload`), one-shot per condition in-process, cross-restart deduped via a
`StartupEventGate` fingerprint over the sorted currently-active fired conditions
(provider keys + a `sidecarMissing` sentinel) — a restart into the same stall stays
silent, a new condition joining re-reports.
* Payload: `reason`, `providers[]` (full table: name, scope, kind, raw values, complete
flag), `stalledProviders[]` with `stalledForMinutes`, `sidecarMissingForMinutes`
(arm 2 only), `sidecarRegistered`, `dwellMinutes`.
* **Purely observational**: emitted via `InformationalEventPost` (dispatch-guard exempt
pass-through) — never touches decision state. No config toggle, no ConfigVersion bump
(precedent: `disk_space_low`).
@@ -93,33 +115,37 @@ resource), so both the detector and the guardrail change ship with the agent rel
## Examples
Customer-case event (abbreviated):
Customer-case event (abbreviated) — note the provider is "complete" by CSP values, the
hang is carried entirely by `reason=sidecar_provider_missing`:
```json
{
"eventType": "esp_policy_provider_stalled",
"severity": "Warning",
"data": {
"reason": "sidecar_provider_missing",
"dwellMinutes": 15,
"sidecarRegistered": false,
"sidecarMissingForMinutes": 15.0,
"providers": [
{ "name": "ConfigMgr", "scope": "device", "kind": "setupApps",
"trackingPoliciesCreated": null, "installationState": null, "complete": false }
"trackingPoliciesCreated": 1, "installationState": null, "complete": true }
],
"stalledProviders": [
{ "name": "ConfigMgr", "scope": "device", "kind": "setupApps", "stalledForMinutes": 15.0 }
]
"stalledProviders": []
}
}
```
`sidecarRegistered=false` while `setupApps` providers exist is the co-management
signature; `sidecarRegistered=true` with a stalled third-party provider generalizes the
same contract violation.
`reason=sidecar_provider_missing` is the co-management signature;
`reason=provider_incomplete` with `sidecarRegistered=true` covers a stalled Sidecar or
third-party provider — the generalized contract violation.
# Citations
* EnrollmentStatusTracking CSP — <https://learn.microsoft.com/en-us/windows/client-management/mdm/enrollmentstatustracking-csp>
* Windows Autopilot into co-management (requirements, provider-conflict warning, troubleshooting registry paths) — <https://learn.microsoft.com/en-us/intune/configmgr/comanage/autopilot-enrollment>
* Troubleshoot the ESP (models app tracking exclusively via the `Sidecar` subkey) — <https://learn.microsoft.com/en-us/troubleshoot/mem/intune/device-enrollment/understand-troubleshoot-esp>
* Niehaus: the Intune ESP-policy WAP payload registering the provider nodes (`Sidecar` always, `ConfigMgr` only with a co-management settings policy) — <https://oofhours.com/2023/08/25/run-an-sccm-task-sequence-during-autopilot/>
* Niehaus: ESP tracks arbitrarily named providers when they are properly registered — <https://oofhours.com/2023/09/18/track-anything-using-esp/>
* Agent code: `EspPolicyProviderProbe.cs`, `EspPolicyProviderStallDetector.cs`, `EspPolicyProviderStallHost.cs` (SystemSignals / Orchestration.Hosts)
* Related: [MDM Reboot Coalescing](mdm-reboot-coalescing.md), [Gather-Rule Guardrails](../rules/gather-rule-guardrails.md)
+1 -1
View File
@@ -22,7 +22,7 @@ https://docs.autopilotmonitor.com (separate repository).
* [Agent Endpoint Migration](agent/endpoint-migration.md) - How the backend re-homes agents to a new API base URL via the config channel (backend move or per-tenant region move): live-fetch-only, allowlist-validated on both sides, one hop, kill wins.
* [MDM Reboot Coalescing](agent/mdm-reboot-coalescing.md) - Attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: aggregated DM-Enterprise 2800 observations (neutral, flush-time watermark) + the ANALYZE-ESP-005 advisory gated on an actually-observed reboot.
* [Secure Boot CA-2023 Detection](agent/secureboot-ca2023-detection.md) - Why ANALYZE-SEC-001 v3 reads the UEFI db/KEK variables directly instead of trusting the SecureBoot\Servicing registry, and how the one-sided uefiCa2023FirmwareConfirmed marker plus a not_exists precondition give absence-tolerant suppression across old and new agents.
* [ESP Policy-Provider Stall Detection](agent/esp-policy-provider-stall.md) - The EnrollmentStatusTracking CSP's no-timeout Setup/Apps wait: a registered provider that never sets TrackingPoliciesCreated (co-management "ConfigMgr" without the IME's "Sidecar") parks the user ESP at "Apps (Identifying)" for days; the always-on esp_policy_provider_stalled tripwire and the SOFTWARE\Microsoft\Windows\Autopilot gather-rule allowlist entry that surface it.
* [ESP Policy-Provider Stall Detection](agent/esp-policy-provider-stall.md) - The EnrollmentStatusTracking CSP's no-timeout Setup/Apps wait is keyed to the Intune-registered "Sidecar" provider by NAME: co-management's "ConfigMgr" parks the user ESP at "Apps (Identifying)" for days even with TrackingPoliciesCreated=1 (issue #106); the always-on two-arm esp_policy_provider_stalled tripwire (provider_incomplete | sidecar_provider_missing) and the SOFTWARE\Microsoft\Windows\Autopilot gather-rule allowlist entry that surface it.
# Rules
+3 -1
View File
@@ -1,6 +1,8 @@
# Log
## 2026-07-23
## 2026-07-24
* **Update**: `agent/esp-policy-provider-stall.md` — the customer's follow-up on issue #106 corrected the field-case mechanics: `TrackingPoliciesCreated=1` WAS set under the foreign `ConfigMgr` provider, so the ESP ignores a foreign provider's completion values and waits on the Intune-registered `Sidecar` provider by name (the rename to `Sidecar` creates no tracking entries yet unblocks instantly). The completeness-keyed detector would have missed exactly this — the incident it was built for. The tripwire is now two-armed under one event type discriminated by `reason`: `provider_incomplete` (unchanged dwell on continuously-incomplete providers) and `sidecar_provider_missing` (≥ 1 Setup/Apps provider registered without `Sidecar`, completeness deliberately ignored, clock reset when Sidecar registers later). Sidecar-missing wins the `reason` when both arms cross in one round; the cross-restart fingerprint gains a `sidecarMissing` sentinel; payload adds `reason` and `sidecarMissingForMinutes`. Evidence for the name-keyed wait: Niehaus' documented Intune WAP payload always registers `Sidecar` (ConfigMgr only via a co-management settings policy) and Microsoft's ESP troubleshooting models app tracking exclusively through the Sidecar subkey.
* **Creation**: Added `agent/esp-policy-provider-stall.md` — a customer diagnosed that pre-provisioned devices with the ConfigMgr client as a blocking app occasionally park the user ESP at "Apps (Identifying)" for days: only a `ConfigMgr` provider is registered under `EnrollmentStatusTracking\Device\Setup\Apps\PolicyProviders` (no `Sidecar` = IME) and `TrackingPoliciesCreated` never becomes 1 — a wait the CSP defines without any timeout; renaming the key to `Sidecar` unblocks immediately. The agent now ships the always-on `esp_policy_provider_stalled` tripwire (60-s tick, 15-min continuous-incompleteness dwell, one-shot per provider, `StartupEventGate` fingerprint across restarts, purely observational via the InformationalEvent rail — no toggle, no ConfigVersion bump), generalized to any provider in both `Setup/Apps` and `DevicePreparation` kinds rather than hardcoding the ConfigMgr case. Companion change: `SOFTWARE\Microsoft\Windows\Autopilot` joins the gather-rule registry allowlist so tenants can target the subtree with custom rules (e.g. `not_exists` on the Sidecar key).
@@ -74,14 +74,17 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
var previous = new Dictionary<string, DateTime> { ["setupApps|device|ConfigMgr"] = T0 };
var missing = EspPolicyProviderStallDetector.EvaluateStallTransition(
EspPolicyProviderSnapshot.Empty, previous, new HashSet<string>(), T0.AddHours(2), Dwell);
EspPolicyProviderSnapshot.Empty, previous, new HashSet<string>(), T0, false, T0.AddHours(2), Dwell);
Assert.Empty(missing.UpdatedFirstSeenUtc);
Assert.Empty(missing.NewlyStalled);
Assert.Null(missing.UpdatedSidecarMissingSinceUtc); // arm 2 resets too
Assert.Null(missing.SidecarMissingStalledForMinutes);
var registeredButEmpty = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(), previous, new HashSet<string>(), T0.AddHours(2), Dwell);
Snapshot(), previous, new HashSet<string>(), T0, false, T0.AddHours(2), Dwell);
Assert.Empty(registeredButEmpty.UpdatedFirstSeenUtc);
Assert.Empty(registeredButEmpty.NewlyStalled);
Assert.Null(registeredButEmpty.UpdatedSidecarMissingSinceUtc);
}
[Fact]
@@ -90,14 +93,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: null));
var round1 = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, new Dictionary<string, DateTime>(), new HashSet<string>(), T0, Dwell);
snapshot, new Dictionary<string, DateTime>(), new HashSet<string>(), null, false, T0, Dwell);
Assert.Equal(T0, round1.UpdatedFirstSeenUtc["setupApps|device|ConfigMgr"]);
Assert.Empty(round1.NewlyStalled);
var round2 = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, round1.UpdatedFirstSeenUtc, new HashSet<string>(), T0.AddMinutes(14), Dwell);
snapshot, round1.UpdatedFirstSeenUtc, new HashSet<string>(),
round1.UpdatedSidecarMissingSinceUtc, false, T0.AddMinutes(14), Dwell);
Assert.Equal(T0, round2.UpdatedFirstSeenUtc["setupApps|device|ConfigMgr"]); // first-seen preserved
Assert.Empty(round2.NewlyStalled);
Assert.Null(round2.SidecarMissingStalledForMinutes); // 14 min < dwell on arm 2 as well
}
[Fact]
@@ -107,7 +112,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
var seen = new Dictionary<string, DateTime> { ["setupApps|device|ConfigMgr"] = T0 };
var eval = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, seen, new HashSet<string>(), T0.AddMinutes(15), Dwell);
snapshot, seen, new HashSet<string>(), null, false, T0.AddMinutes(15), Dwell);
var stalled = Assert.Single(eval.NewlyStalled);
Assert.Equal("ConfigMgr", stalled.Provider.Name);
@@ -122,20 +127,20 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
// Completes mid-dwell → dropped from the map.
var completed = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("ConfigMgr", tracking: 1)), seen, new HashSet<string>(), T0.AddMinutes(10), Dwell);
Snapshot(SetupApps("ConfigMgr", tracking: 1)), seen, new HashSet<string>(), null, false, T0.AddMinutes(10), Dwell);
Assert.Empty(completed.UpdatedFirstSeenUtc);
// Re-appears incomplete much later → dwell restarts from the new now, no stall yet.
var reappeared = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("ConfigMgr", tracking: 0)), completed.UpdatedFirstSeenUtc,
new HashSet<string>(), T0.AddHours(3), Dwell);
new HashSet<string>(), null, false, T0.AddHours(3), Dwell);
Assert.Equal(T0.AddHours(3), reappeared.UpdatedFirstSeenUtc[key]);
Assert.Empty(reappeared.NewlyStalled);
// Disappearance behaves the same as completion.
var disappeared = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("Sidecar", tracking: 1)), new Dictionary<string, DateTime> { [key] = T0 },
new HashSet<string>(), T0.AddMinutes(20), Dwell);
new HashSet<string>(), null, false, T0.AddMinutes(20), Dwell);
Assert.False(disappeared.UpdatedFirstSeenUtc.ContainsKey(key));
}
@@ -147,7 +152,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
var fired = new HashSet<string>(StringComparer.OrdinalIgnoreCase) { "setupApps|device|ConfigMgr" };
var eval = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, seen, fired, T0.AddHours(5), Dwell);
snapshot, seen, fired, null, false, T0.AddHours(5), Dwell);
Assert.Empty(eval.NewlyStalled);
Assert.True(eval.UpdatedFirstSeenUtc.ContainsKey("setupApps|device|ConfigMgr")); // still tracked
@@ -162,7 +167,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
DevicePrep("ConfigMgr", installState: 1));
var eval = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, new Dictionary<string, DateTime>(), new HashSet<string>(), T0, Dwell);
snapshot, new Dictionary<string, DateTime>(), new HashSet<string>(), null, false, T0, Dwell);
Assert.Equal(3, eval.UpdatedFirstSeenUtc.Count);
Assert.Contains("setupApps|device|ConfigMgr", eval.UpdatedFirstSeenUtc.Keys);
@@ -179,6 +184,71 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
Assert.False(Snapshot(DevicePrep("Sidecar", installState: 3)).SidecarRegistered); // wrong kind
Assert.False(EspPolicyProviderSnapshot.Empty.SidecarRegistered);
}
// ------------------------------------------------- arm 2: sidecar-missing transition
[Fact]
public void Issue106_foreign_provider_complete_but_sidecar_missing_stalls_after_dwell()
{
// The actual field case: ConfigMgr with TrackingPoliciesCreated=1 — "complete" by the
// CSP value contract, arm 1 never fires — yet the ESP waits on the absent Sidecar.
var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: 1));
var round1 = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, new Dictionary<string, DateTime>(), new HashSet<string>(), null, false, T0, Dwell);
Assert.Empty(round1.NewlyStalled); // arm 1 stays silent...
Assert.Equal(T0, round1.UpdatedSidecarMissingSinceUtc); // ...but arm 2 starts its clock
Assert.Null(round1.SidecarMissingStalledForMinutes);
var round2 = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, round1.UpdatedFirstSeenUtc, new HashSet<string>(),
round1.UpdatedSidecarMissingSinceUtc, false, T0.AddMinutes(15), Dwell);
Assert.Empty(round2.NewlyStalled);
Assert.Equal(15.0, round2.SidecarMissingStalledForMinutes!.Value, precision: 3);
}
[Fact]
public void Sidecar_registering_resets_the_sidecar_missing_clock()
{
// ConfigMgr first (legitimate startup ordering) → clock runs...
var round1 = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("ConfigMgr", tracking: 1)),
new Dictionary<string, DateTime>(), new HashSet<string>(), null, false, T0, Dwell);
Assert.Equal(T0, round1.UpdatedSidecarMissingSinceUtc);
// ...Sidecar joins → condition clears, state resets.
var round2 = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("ConfigMgr", tracking: 1), SetupApps("Sidecar", tracking: 0)),
round1.UpdatedFirstSeenUtc, new HashSet<string>(),
round1.UpdatedSidecarMissingSinceUtc, false, T0.AddMinutes(10), Dwell);
Assert.Null(round2.UpdatedSidecarMissingSinceUtc);
Assert.Null(round2.SidecarMissingStalledForMinutes);
// Sidecar gone again much later → clock restarts from the new now, no instant stall.
var round3 = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("ConfigMgr", tracking: 1)),
round2.UpdatedFirstSeenUtc, new HashSet<string>(),
round2.UpdatedSidecarMissingSinceUtc, false, T0.AddHours(3), Dwell);
Assert.Equal(T0.AddHours(3), round3.UpdatedSidecarMissingSinceUtc);
Assert.Null(round3.SidecarMissingStalledForMinutes);
}
[Fact]
public void Sidecar_missing_one_shot_and_deviceprep_only_scoping()
{
// Already fired → never re-reported, but the clock keeps running.
var latched = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("ConfigMgr", tracking: 1)),
new Dictionary<string, DateTime>(), new HashSet<string>(), T0, true, T0.AddHours(5), Dwell);
Assert.Null(latched.SidecarMissingStalledForMinutes);
Assert.Equal(T0, latched.UpdatedSidecarMissingSinceUtc);
// DevicePreparation providers alone don't constitute the Setup/Apps wait — arm 2 off.
var devicePrepOnly = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(DevicePrep("ConfigMgr", installState: 1)),
new Dictionary<string, DateTime>(), new HashSet<string>(), null, false, T0, Dwell);
Assert.Null(devicePrepOnly.UpdatedSidecarMissingSinceUtc);
}
}
/// <summary>
@@ -243,7 +313,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
Assert.Equal(EspPolicyProviderStallDetector.SourceLabel, evt.Payload[SignalPayloadKeys.Source]);
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(evt.TypedPayload);
// Both arms cross in the same round (ConfigMgr incomplete AND Sidecar absent) →
// ONE event; sidecar-missing is the dominant reason.
Assert.Equal(EspPolicyProviderStallDetector.ReasonSidecarMissing, data["reason"]);
Assert.Equal(false, data["sidecarRegistered"]);
Assert.Equal(15.0, data["sidecarMissingForMinutes"]);
Assert.Equal(EspPolicyProviderStallDetector.DwellMinutes, data["dwellMinutes"]);
var stalled = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object>>>(data["stalledProviders"]);
var entry = Assert.Single(stalled);
@@ -252,6 +326,36 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
Assert.Equal(15.0, entry["stalledForMinutes"]);
}
[Fact]
public void Issue106_configmgr_complete_without_sidecar_still_emits_sidecar_missing()
{
// Regression test for the customer's follow-up on issue #106: TrackingPoliciesCreated=1
// WAS present under ConfigMgr — the provider looks "complete", arm 1 never fires, yet
// the ESP hangs because Sidecar is absent. Arm 2 must catch exactly this.
var configMgrComplete = new EspPolicyProviderSnapshot(new[]
{
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindSetupApps,
trackingPoliciesCreated: 1, installationState: null),
});
using var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => configMgrComplete);
var (sut, sink, clock) = BuildDetector();
sut.Tick();
Assert.Empty(StalledEvents(sink));
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
var evt = Assert.Single(StalledEvents(sink));
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(evt.TypedPayload);
Assert.Equal(EspPolicyProviderStallDetector.ReasonSidecarMissing, data["reason"]);
Assert.Equal(false, data["sidecarRegistered"]);
Assert.Equal(15.0, data["sidecarMissingForMinutes"]);
Assert.Empty(Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object>>>(data["stalledProviders"]));
var providers = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object?>>>(data["providers"]);
Assert.Equal(true, Assert.Single(providers)["complete"]); // the misleading "complete" is visible
}
[Fact]
public void Sidecar_complete_configmgr_incomplete_reports_only_configmgr_with_sidecar_flag()
{
@@ -264,7 +368,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
var evt = Assert.Single(StalledEvents(sink));
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(evt.TypedPayload);
Assert.Equal(EspPolicyProviderStallDetector.ReasonProviderIncomplete, data["reason"]); // arm 2 off — Sidecar is there
Assert.Equal(true, data["sidecarRegistered"]);
Assert.False(data.ContainsKey("sidecarMissingForMinutes"));
var stalled = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object>>>(data["stalledProviders"]);
Assert.Equal("ConfigMgr", Assert.Single(stalled)["name"]);
var providers = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object?>>>(data["providers"]);
@@ -35,13 +35,19 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
/// (registered during Autopilot-into-co-management, see the troubleshooting registry paths in
/// https://learn.microsoft.com/en-us/intune/configmgr/comanage/autopilot-enrollment). Microsoft
/// warns that ESP policy providers are not aware of each other and that pre-provisioning is
/// unsupported with co-management. Field case that motivated this probe: a pre-provisioned
/// device with the ConfigMgr client installed as a blocking app ended up with ONLY
/// <c>ConfigMgr</c> registered under <c>Device\Setup\Apps\PolicyProviders</c> (no
/// <c>Sidecar</c>) and <c>TrackingPoliciesCreated</c> never set — the user ESP hung at
/// "Apps (Identifying)" for hours to days; manually renaming the key to <c>Sidecar</c>
/// unblocked it immediately. <see cref="EspPolicyProviderStallDetector"/> turns this contract
/// into the <c>esp_policy_provider_stalled</c> tripwire.
/// unsupported with co-management. Field case that motivated this probe (issue #106): a
/// pre-provisioned device with the ConfigMgr client installed as a blocking app ended up with
/// ONLY <c>ConfigMgr</c> registered under <c>Device\Setup\Apps\PolicyProviders</c> (no
/// <c>Sidecar</c>) — and although <c>TrackingPoliciesCreated=1</c> WAS set under it, the user
/// ESP hung at "Apps (Identifying)" for hours to days; manually renaming the key to
/// <c>Sidecar</c> unblocked it immediately. So the ESP's Apps wait is keyed to the provider
/// the Intune service registered at enrollment (always <c>Sidecar</c>; ConfigMgr only joins
/// via a co-management settings policy — see the WAP payload documented in
/// https://oofhours.com/2023/08/25/run-an-sccm-task-sequence-during-autopilot/), and a
/// foreign provider's completion values do not satisfy it. Detection therefore keys on the
/// provider NAME topology, not only on completeness:
/// <see cref="EspPolicyProviderStallDetector"/> turns this contract into the two-arm
/// <c>esp_policy_provider_stalled</c> tripwire.
/// </para>
/// <para>
/// Fail-soft like <see cref="EspTrackingInfoProbe"/>: any failure is Debug-logged and yields
@@ -13,18 +13,30 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
/// <summary>
/// Turns the ESP policy-provider contract read by <see cref="EspPolicyProviderProbe"/> into
/// the one-shot <c>esp_policy_provider_stalled</c> Warning: a registered provider that stays
/// continuously incomplete for <see cref="DwellMinutes"/> minutes is reported once, with the
/// full provider table in the payload. See the probe's header for the CSP contract, the
/// official sources, and the co-management field case (ConfigMgr registered without Sidecar
/// → user ESP parked at "Apps (Identifying)" with no OS timeout) this tripwire exists for.
/// the one-shot <c>esp_policy_provider_stalled</c> Warning via two independent arms, each
/// with the same <see cref="DwellMinutes"/>-minute continuous-dwell rule and the full
/// provider table in the payload:
/// <list type="bullet">
/// <item><b>Arm 1 — <c>reason=provider_incomplete</c></b>: a registered provider stays
/// continuously incomplete (e.g. Sidecar itself never sets TrackingPoliciesCreated).</item>
/// <item><b>Arm 2 — <c>reason=sidecar_provider_missing</c></b>: at least one
/// <c>Setup\Apps</c> provider is registered but none of them is <c>Sidecar</c> (the IME).
/// This is the actual co-management field-case signature (issue #106): the foreign
/// <c>ConfigMgr</c> provider had <c>TrackingPoliciesCreated=1</c> — i.e. was "complete"
/// by the CSP value contract — yet the user ESP stayed parked at "Apps (Identifying)"
/// until the key was renamed to <c>Sidecar</c>. The ESP's Apps wait is keyed to the
/// Intune-registered provider by NAME, so provider completeness alone must not gate this
/// detection. See the probe's header for sources.</item>
/// </list>
/// <para>
/// Dwell semantics: the dwell clock per provider starts when it is first observed incomplete
/// and RESETS when the provider completes or disappears — only <i>continuously</i> incomplete
/// providers fire. A missing root key / empty provider list is normal early enrollment and
/// clears all dwell state (probe D4). Purely observational: emission goes through
/// <see cref="InformationalEventPost"/> (InformationalEvent pass-through, dispatch-guard
/// exempt), never into decision state.
/// Dwell semantics: arm 1's clock per provider starts when it is first observed incomplete
/// and RESETS when the provider completes or disappears; arm 2's clock starts when the
/// sidecar-missing condition is first observed and RESETS when Sidecar registers (covers the
/// legitimate "ConfigMgr registered before Sidecar" startup ordering) — only
/// <i>continuously</i> bad states fire. A missing root key / empty provider list is normal
/// early enrollment and clears all dwell state (probe D4). Purely observational: emission
/// goes through <see cref="InformationalEventPost"/> (InformationalEvent pass-through,
/// dispatch-guard exempt), never into decision state.
/// </para>
/// <para>
/// Duplicate suppression: in-process one-shot per provider key, plus a
@@ -44,6 +56,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
internal const string SourceLabel = "EspPolicyProviderStallDetector";
/// <summary>Payload <c>reason</c>: a registered provider stayed continuously incomplete (arm 1).</summary>
internal const string ReasonProviderIncomplete = "provider_incomplete";
/// <summary>Payload <c>reason</c>: Setup\Apps providers registered but Sidecar absent (arm 2 — dominant when both arms fire in one round).</summary>
internal const string ReasonSidecarMissing = "sidecar_provider_missing";
/// <summary>
/// Sentinel token representing the arm-2 condition in the cross-restart fingerprint.
/// Cannot collide with provider keys — those always contain <c>|</c> separators.
/// </summary>
private const string SidecarMissingFingerprintToken = "sidecarMissing";
private readonly string _sessionId;
private readonly string _tenantId;
private readonly InformationalEventPost _post;
@@ -57,6 +81,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
private readonly HashSet<string> _firedKeys =
new HashSet<string>(StringComparer.OrdinalIgnoreCase);
/// <summary>Arm-2 dwell state: first observation of "setupApps providers without Sidecar"; null = condition not active.</summary>
private DateTime? _sidecarMissingSinceUtc;
/// <summary>Arm-2 in-process one-shot latch (mirror of <see cref="_firedKeys"/>).</summary>
private bool _sidecarMissingFired;
public EspPolicyProviderStallDetector(
string sessionId,
string tenantId,
@@ -79,13 +109,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
internal void Tick()
{
var snapshot = EspPolicyProviderProbe.Read(_logger);
var eval = EvaluateStallTransition(snapshot, _firstSeenIncompleteUtc, _firedKeys, _clock.UtcNow, _dwell);
var eval = EvaluateStallTransition(
snapshot, _firstSeenIncompleteUtc, _firedKeys,
_sidecarMissingSinceUtc, _sidecarMissingFired, _clock.UtcNow, _dwell);
_firstSeenIncompleteUtc.Clear();
foreach (var kv in eval.UpdatedFirstSeenUtc)
_firstSeenIncompleteUtc[kv.Key] = kv.Value;
_sidecarMissingSinceUtc = eval.UpdatedSidecarMissingSinceUtc;
if (eval.NewlyStalled.Count > 0)
if (eval.NewlyStalled.Count > 0 || eval.SidecarMissingStalledForMinutes != null)
EmitStalled(snapshot, eval);
}
@@ -94,24 +127,29 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
/// registry, timer, or ingress (pattern: <c>PerformanceCollector.EvaluateDiskLowTransition</c>).
/// </summary>
/// <param name="snapshot">Current provider snapshot.</param>
/// <param name="firstSeenIncompleteUtc">Dwell state from the previous round (provider key → first-seen-incomplete).</param>
/// <param name="firstSeenIncompleteUtc">Arm-1 dwell state from the previous round (provider key → first-seen-incomplete).</param>
/// <param name="alreadyFired">Provider keys already reported this run (never re-report).</param>
/// <param name="sidecarMissingSinceUtc">Arm-2 dwell state from the previous round; null = condition was not active.</param>
/// <param name="sidecarMissingAlreadyFired">Arm-2 one-shot latch (never re-report).</param>
/// <param name="nowUtc">Evaluation timestamp.</param>
/// <param name="dwell">Continuous-incompleteness threshold.</param>
/// <param name="dwell">Continuous-condition threshold (shared by both arms).</param>
internal static StallEvaluation EvaluateStallTransition(
EspPolicyProviderSnapshot snapshot,
IReadOnlyDictionary<string, DateTime> firstSeenIncompleteUtc,
ISet<string> alreadyFired,
DateTime? sidecarMissingSinceUtc,
bool sidecarMissingAlreadyFired,
DateTime nowUtc,
TimeSpan dwell)
{
var result = new StallEvaluation();
// Missing root key or no registered providers = normal early enrollment: clear all
// dwell state (a provider must be continuously OBSERVED incomplete to accrue dwell).
// dwell state (a condition must be continuously OBSERVED to accrue dwell).
if (!snapshot.HasData || snapshot.Providers.Count == 0)
return result;
// Arm 1 — a registered provider continuously incomplete.
foreach (var provider in snapshot.Providers)
{
if (provider.IsComplete)
@@ -127,22 +165,44 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
result.NewlyStalled.Add(new StalledProvider(provider, stalledFor.TotalMinutes));
}
// Arm 2 — Setup\Apps providers registered, but none of them is Sidecar. Deliberately
// ignores provider completeness: the field case (issue #106) had ConfigMgr with
// TrackingPoliciesCreated=1 and the ESP still waited. Sidecar registering later
// (legitimate startup ordering) clears the condition and resets the clock.
if (HasSetupAppsProvider(snapshot) && !snapshot.SidecarRegistered)
{
var since = sidecarMissingSinceUtc ?? nowUtc;
result.UpdatedSidecarMissingSinceUtc = since;
var missingFor = nowUtc - since;
if (missingFor >= dwell && !sidecarMissingAlreadyFired)
result.SidecarMissingStalledForMinutes = missingFor.TotalMinutes;
}
return result;
}
private static bool HasSetupAppsProvider(EspPolicyProviderSnapshot snapshot)
=> snapshot.Providers.Any(p => p.Kind == EspPolicyProviderProbe.KindSetupApps);
private void EmitStalled(EspPolicyProviderSnapshot snapshot, StallEvaluation eval)
{
foreach (var stalled in eval.NewlyStalled)
_firedKeys.Add(stalled.Provider.Key);
if (eval.SidecarMissingStalledForMinutes != null)
_sidecarMissingFired = true;
// Cross-restart dedup: fingerprint = sorted set of all fired keys that are still
// incomplete. A restart into the same stall re-accrues dwell and lands on the same
// fingerprint → suppressed; a new provider joining the stall changes it → re-emitted.
// Cross-restart dedup: fingerprint = sorted set of all fired conditions that are
// still active (incomplete providers + the sidecar-missing sentinel). A restart into
// the same stall re-accrues dwell and lands on the same fingerprint → suppressed; a
// new condition joining the stall changes it → re-emitted.
var currentlyStalledKeys = snapshot.Providers
.Where(p => !p.IsComplete && _firedKeys.Contains(p.Key))
.Select(p => p.Key)
.OrderBy(k => k, StringComparer.OrdinalIgnoreCase)
.ToList();
if (_sidecarMissingFired && HasSetupAppsProvider(snapshot) && !snapshot.SidecarRegistered)
currentlyStalledKeys.Add(SidecarMissingFingerprintToken);
currentlyStalledKeys.Sort(StringComparer.OrdinalIgnoreCase);
var fingerprint = StartupEventGate.HashString(string.Join(";", currentlyStalledKeys));
if (_startupGate != null &&
!_startupGate.ShouldEmit(Constants.EventTypes.EspPolicyProviderStalled, fingerprint))
@@ -153,11 +213,30 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
return;
}
// Sidecar-missing is the dominant reason when both arms fire in one round: it
// explains the ESP hang regardless of the foreign providers' completeness.
var reason = eval.SidecarMissingStalledForMinutes != null
? ReasonSidecarMissing
: ReasonProviderIncomplete;
var messageParts = new List<string>();
if (eval.SidecarMissingStalledForMinutes != null)
{
var registeredNames = string.Join(", ", snapshot.Providers
.Where(p => p.Kind == EspPolicyProviderProbe.KindSetupApps)
.Select(p => $"{p.Name} ({p.Scope})"));
messageParts.Add(
$"expected IME provider '{EspPolicyProviderProbe.SidecarProviderName}' not registered for " +
$">= {DwellMinutes} min while Setup/Apps providers exist: {registeredNames}");
}
var stalledNames = string.Join(", ",
eval.NewlyStalled.Select(s => $"{s.Provider.Name} ({s.Provider.Scope}/{s.Provider.Kind})"));
if (eval.NewlyStalled.Count > 0)
messageParts.Add($"provider(s) incomplete for >= {DwellMinutes} min: {stalledNames}");
var data = new Dictionary<string, object>(StringComparer.Ordinal)
{
["reason"] = reason,
["dwellMinutes"] = DwellMinutes,
["sidecarRegistered"] = snapshot.SidecarRegistered,
["providers"] = snapshot.Providers.Select(p => new Dictionary<string, object?>(StringComparer.Ordinal)
@@ -177,6 +256,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
["stalledForMinutes"] = Math.Round(s.StalledForMinutes, 1),
}).ToList(),
};
if (eval.SidecarMissingStalledForMinutes != null)
data["sidecarMissingForMinutes"] = Math.Round(eval.SidecarMissingStalledForMinutes.Value, 1);
var message =
"ESP Setup/Apps tracking stalled: " + string.Join("; ", messageParts) +
" — the ESP waits on its app-tracking providers without any timeout";
_post.Emit(new EnrollmentEvent
{
@@ -188,27 +273,31 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
Source = SourceLabel,
Phase = EnrollmentPhase.Unknown,
ImmediateUpload = true,
Message = $"ESP policy provider(s) incomplete for >= {DwellMinutes} min: {stalledNames} " +
$"(sidecarRegistered={snapshot.SidecarRegistered}) — the ESP waits on " +
"Setup/Apps providers without any timeout",
Message = message,
Data = data
});
_startupGate?.MarkEmitted(Constants.EventTypes.EspPolicyProviderStalled);
_logger?.Warning(
$"EspPolicyProviderStallDetector: {stalledNames} incomplete for >= {DwellMinutes} min " +
$"(sidecarRegistered={snapshot.SidecarRegistered}).");
$"EspPolicyProviderStallDetector: {string.Join("; ", messageParts)} " +
$"(reason={reason}, sidecarRegistered={snapshot.SidecarRegistered}).");
}
/// <summary>Result of one <see cref="EvaluateStallTransition"/> round.</summary>
internal sealed class StallEvaluation
{
/// <summary>Next round's dwell state — only currently-incomplete providers survive.</summary>
/// <summary>Next round's arm-1 dwell state — only currently-incomplete providers survive.</summary>
public Dictionary<string, DateTime> UpdatedFirstSeenUtc { get; } =
new Dictionary<string, DateTime>(StringComparer.OrdinalIgnoreCase);
/// <summary>Providers that crossed the dwell threshold this round and were not yet reported.</summary>
/// <summary>Providers that crossed the dwell threshold this round and were not yet reported (arm 1).</summary>
public List<StalledProvider> NewlyStalled { get; } = new List<StalledProvider>();
/// <summary>Next round's arm-2 dwell state; null = sidecar-missing condition not active (reset).</summary>
public DateTime? UpdatedSidecarMissingSinceUtc { get; set; }
/// <summary>Non-null when the sidecar-missing condition crossed the dwell threshold this round and was not yet reported (arm 2).</summary>
public double? SidecarMissingStalledForMinutes { get; set; }
}
internal readonly struct StalledProvider
@@ -37,8 +37,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
/// <see cref="StallProbeCollectorAdapter"/>. Owns its 60-s idle-check timer (the
/// collector itself has no timer — it's a pure probe invoked from outside).</item>
/// <item><b>EspPolicyProviderStallHost</b> — always-on 60-s wall-clock tick for the
/// <c>esp_policy_provider_stalled</c> tripwire (registered EnrollmentStatusTracking CSP
/// provider continuously incomplete ≥ 15 min — the no-timeout ESP wait).</item>
/// two-arm <c>esp_policy_provider_stalled</c> tripwire (EnrollmentStatusTracking CSP
/// provider continuously incomplete ≥ 15 min, OR Setup\Apps providers registered
/// without the expected <c>Sidecar</c>/IME provider — the no-timeout ESP wait).</item>
/// </list>
/// Optional peripheral hosts (driven by <see cref="CollectorConfiguration"/> toggles):
/// <list type="bullet">
@@ -307,11 +308,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
}
// ESP policy-provider stall tripwire — always-on kernel host (no config gate, like
// disk_space_low): a registered EnrollmentStatusTracking CSP provider that never
// completes (e.g. co-management "ConfigMgr" without "Sidecar" on a pre-provisioned
// device) parks the user ESP at "Apps (Identifying)" WITHOUT any OS timeout. Wall-clock
// dwell, deliberately outside the StallProbe gate: the idle clock there resets on any
// session activity, and the stall must accrue regardless of it.
// disk_space_low): a provider that never completes, or co-management "ConfigMgr"
// registered without the expected "Sidecar"/IME provider (issue #106 — even with
// TrackingPoliciesCreated=1 the ESP ignores foreign provider names), parks the user
// ESP at "Apps (Identifying)" WITHOUT any OS timeout. Wall-clock dwell, deliberately
// outside the StallProbe gate: the idle clock there resets on any session activity,
// and the stall must accrue regardless of it.
hosts.Add(new EspPolicyProviderStallHost(
sessionId: sessionId,
tenantId: tenantId,
@@ -0,0 +1,195 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text.Json;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Functions.Apps;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Moq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Pins the terminal-only failure-rate convention on the Apps dashboard aggregations
/// (<see cref="AppsAnalyticsHelper"/>): every failureRate is Failed / (Failed + Succeeded) —
/// finished installs only. "InProgress" rows (still installing, or orphaned by a session that
/// died mid-install) sit in the install counts but never dilute a rate, matching the enrollment
/// success-rate convention. Responses are anonymous projections, so tests assert on the
/// JSON wire shape like the sibling suites.
/// </summary>
public class AppsAnalyticsFailureRateTests
{
private const string TenantId = "00000000-0000-0000-0000-000000000fa1";
private static AppInstallSummary A(
string status,
string appName = "Contoso App",
DateTime? startedAt = null,
string sessionId = "s-1",
string appVersion = "")
=> new()
{
TenantId = TenantId,
SessionId = sessionId,
AppName = appName,
AppVersion = appVersion,
Status = status,
StartedAt = startedAt ?? DateTime.UtcNow.AddHours(-2),
DurationSeconds = status == "Succeeded" ? 30 : 0,
};
// ── /apps/list ──────────────────────────────────────────────────────────
[Fact]
public void List_FailureRate_IsOverFinishedInstalls_InProgressExcluded()
{
var summaries = new List<AppInstallSummary>
{
A("Failed"),
A("Succeeded"),
A("InProgress"),
A("InProgress"),
};
var root = JsonSerializer.SerializeToElement(
AppsAnalyticsHelper.BuildAppsListResponse(summaries, days: 30));
var app = root.GetProperty("apps")[0];
Assert.Equal(4, app.GetProperty("totalInstalls").GetInt32());
// 1 / (1 + 1) = 50% (over finished), NOT 1 / 4 = 25% (over all installs).
Assert.Equal(50d, app.GetProperty("failureRate").GetDouble());
}
[Fact]
public void List_FailureRate_IsZero_WhenNothingFinishedYet()
{
var summaries = new List<AppInstallSummary> { A("InProgress"), A("InProgress") };
var root = JsonSerializer.SerializeToElement(
AppsAnalyticsHelper.BuildAppsListResponse(summaries, days: 30));
Assert.Equal(0d, root.GetProperty("apps")[0].GetProperty("failureRate").GetDouble());
}
[Fact]
public void List_Trend_UsesFinishedRates_AndGatesOnFinishedSamples()
{
var now = DateTime.UtcNow;
var firstHalf = now.AddDays(-20); // days: 30 → midpoint at now - 15d
var secondHalf = now.AddDays(-5);
var summaries = new List<AppInstallSummary>();
// First half: 1 failed / 5 finished = 20%.
summaries.Add(A("Failed", startedAt: firstHalf));
for (var i = 0; i < 4; i++) summaries.Add(A("Succeeded", startedAt: firstHalf));
// Second half: 3 failed / 5 finished = 60% — plus 10 in-flight installs that
// previously dragged the half's rate down to 20% and masked the regression.
for (var i = 0; i < 3; i++) summaries.Add(A("Failed", startedAt: secondHalf));
for (var i = 0; i < 2; i++) summaries.Add(A("Succeeded", startedAt: secondHalf));
for (var i = 0; i < 10; i++) summaries.Add(A("InProgress", startedAt: secondHalf));
var root = JsonSerializer.SerializeToElement(
AppsAnalyticsHelper.BuildAppsListResponse(summaries, days: 30));
var app = root.GetProperty("apps")[0];
Assert.Equal("worsening", app.GetProperty("trend").GetString());
Assert.Equal(40d, app.GetProperty("trendDelta").GetDouble());
}
[Fact]
public void List_Trend_StaysStable_WhenAHalfLacksFiveFinishedInstalls()
{
var now = DateTime.UtcNow;
var summaries = new List<AppInstallSummary>();
// First half: 5 finished. Second half: 4 finished + many in-flight — the old
// install-count gate would have passed; the finished-sample gate must not.
for (var i = 0; i < 5; i++) summaries.Add(A("Succeeded", startedAt: now.AddDays(-20)));
for (var i = 0; i < 4; i++) summaries.Add(A("Failed", startedAt: now.AddDays(-5)));
for (var i = 0; i < 10; i++) summaries.Add(A("InProgress", startedAt: now.AddDays(-5)));
var root = JsonSerializer.SerializeToElement(
AppsAnalyticsHelper.BuildAppsListResponse(summaries, days: 30));
var app = root.GetProperty("apps")[0];
Assert.Equal("stable", app.GetProperty("trend").GetString());
Assert.Equal(JsonValueKind.Null, app.GetProperty("trendDelta").ValueKind);
}
// ── /apps/{appName}/analytics ───────────────────────────────────────────
private static async Task<JsonElement> BuildAnalyticsAsync(
List<AppInstallSummary> summaries, ISessionRepository? sessionRepo = null)
{
if (sessionRepo == null)
{
var mock = new Mock<ISessionRepository>();
mock.Setup(r => r.GetSessionAsync(It.IsAny<string>(), It.IsAny<string>()))
.ReturnsAsync((SessionSummary?)null);
sessionRepo = mock.Object;
}
var result = await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
summaries, sessionRepo, "Contoso App", days: 30);
return JsonSerializer.SerializeToElement(result);
}
[Fact]
public async Task Analytics_SummaryVersionAndTimeSeriesRates_AreOverFinishedInstalls()
{
var summaries = new List<AppInstallSummary>
{
A("Failed", appVersion: "1.0"),
A("Succeeded", appVersion: "1.0"),
A("InProgress", appVersion: "1.0"),
A("InProgress", appVersion: "1.0"),
};
var root = await BuildAnalyticsAsync(summaries);
// Summary: 1 / (1 + 1) = 50%, NOT 1 / 4 = 25%.
var summary = root.GetProperty("summary");
Assert.Equal(4, summary.GetProperty("totalInstalls").GetInt32());
Assert.Equal(50d, summary.GetProperty("failureRate").GetDouble());
var version = root.GetProperty("versionBreakdown")[0];
Assert.Equal(4, version.GetProperty("installs").GetInt32());
Assert.Equal(50d, version.GetProperty("failureRate").GetDouble());
// All four installs land in today's bucket → same terminal-only rate there.
var buckets = root.GetProperty("timeSeries").EnumerateArray().ToList();
var todayBucket = buckets.Single(b => b.GetProperty("installs").GetInt32() > 0);
Assert.Equal(50d, todayBucket.GetProperty("failureRate").GetDouble());
}
[Fact]
public async Task Analytics_DeviceModelBreakdown_RatesAndSampleFloor_UseFinishedInstalls()
{
var sessionRepo = new Mock<ISessionRepository>();
sessionRepo.Setup(r => r.GetSessionAsync(TenantId, It.IsAny<string>()))
.ReturnsAsync((string _, string sessionId) => new SessionSummary
{
TenantId = TenantId,
SessionId = sessionId,
Manufacturer = "Contoso",
Model = sessionId.StartsWith("s-a") ? "ModelA" : "ModelB",
});
var summaries = new List<AppInstallSummary>();
// ModelA: 5 finished (1 failed → 20%) + 3 in-flight → ranked, rate undiluted.
summaries.Add(A("Failed", sessionId: "s-a-0"));
for (var i = 1; i <= 4; i++) summaries.Add(A("Succeeded", sessionId: $"s-a-{i}"));
for (var i = 5; i <= 7; i++) summaries.Add(A("InProgress", sessionId: $"s-a-{i}"));
// ModelB: 4 finished + 6 in-flight — 10 installs, but below the 5-finished floor.
for (var i = 0; i <= 3; i++) summaries.Add(A("Failed", sessionId: $"s-b-{i}"));
for (var i = 4; i <= 9; i++) summaries.Add(A("InProgress", sessionId: $"s-b-{i}"));
var root = await BuildAnalyticsAsync(summaries, sessionRepo.Object);
var breakdown = root.GetProperty("deviceModelBreakdown");
var model = Assert.Single(breakdown.EnumerateArray());
Assert.Equal("ModelA", model.GetProperty("model").GetString());
Assert.Equal(8, model.GetProperty("installs").GetInt32());
Assert.Equal(20d, model.GetProperty("failureRate").GetDouble());
}
}
@@ -10,9 +10,10 @@ namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Covers the server-side Fleet Health aggregation in
/// <see cref="MetricsMath.BuildFleetHealthPayload"/>, the single source of truth shared by the
/// tenant (metrics/fleet-health) and global (global/metrics/fleet-health) functions. These assert
/// the semantics ported verbatim from the previous client-side aggregation — notably success rate
/// over ALL sessions and average duration over non-in-progress sessions (failures included).
/// tenant (metrics/fleet-health) and global (global/metrics/fleet-health) functions. Success and
/// failure rates follow the SLA convention — over finished enrollments (Succeeded + Failed) only,
/// so in-flight sessions never dilute them. Average duration stays over non-in-progress sessions
/// that carry a duration (failures included).
/// </summary>
public class FleetHealthPayloadTests
{
@@ -55,7 +56,7 @@ public class FleetHealthPayloadTests
}
[Fact]
public void SuccessRate_IsOverAllSessions_NotJustTerminal()
public void SuccessRate_IsOverFinishedSessions_InFlightAndIncompleteExcluded()
{
var sessions = new List<SessionSummary>
{
@@ -63,16 +64,34 @@ public class FleetHealthPayloadTests
S(SessionStatus.Succeeded),
S(SessionStatus.Failed),
S(SessionStatus.InProgress),
S(SessionStatus.Pending),
S(SessionStatus.Incomplete), // terminal but non-failure — stays out of the rate
};
var payload = MetricsMath.BuildFleetHealthPayload(sessions, days: 7);
Assert.Equal(4, payload.Stats.Total);
Assert.Equal(6, payload.Stats.Total);
Assert.Equal(2, payload.Stats.Succeeded);
Assert.Equal(1, payload.Stats.Failed);
Assert.Equal(1, payload.Stats.InProgress);
// 2 / 4 = 50% (over total), NOT 2 / 3 = 66.7% (over terminal).
Assert.Equal(50d, payload.Stats.SuccessRate);
Assert.Equal(1, payload.Stats.Incomplete);
// 2 / (2 + 1) = 66.7% (over finished), NOT 2 / 6 = 33.3% (over total).
Assert.Equal(66.7d, payload.Stats.SuccessRate);
}
[Fact]
public void SuccessRate_IsZero_WhenNothingFinishedYet()
{
var sessions = new List<SessionSummary>
{
S(SessionStatus.InProgress),
S(SessionStatus.Pending),
};
var payload = MetricsMath.BuildFleetHealthPayload(sessions, days: 7);
// No finished enrollments → no rate; clients render "—" off Succeeded + Failed == 0.
Assert.Equal(0d, payload.Stats.SuccessRate);
}
[Fact]
@@ -111,6 +130,27 @@ public class FleetHealthPayloadTests
Assert.DoesNotContain(payload.ModelHealth, m => m.Model == "Contoso Model1");
}
[Fact]
public void ModelHealth_CountsFailedSeparately_InFlightInTotalOnly()
{
var sessions = new List<SessionSummary>
{
S(SessionStatus.Succeeded, model: "X"),
S(SessionStatus.Failed, model: "X"),
S(SessionStatus.InProgress, model: "X"),
};
var payload = MetricsMath.BuildFleetHealthPayload(sessions, days: 7);
var m = Assert.Single(payload.ModelHealth);
// Total keeps the full device count; the client derives the rate from
// Succeeded / (Succeeded + Failed), so the in-flight session must land
// in Total but in neither outcome bucket.
Assert.Equal(3, m.Total);
Assert.Equal(1, m.Succeeded);
Assert.Equal(1, m.Failed);
}
[Fact]
public void ModelKey_FallsBackToUnknown_WhenManufacturerAndModelBlank()
{
@@ -147,10 +187,13 @@ public class FleetHealthPayloadTests
{
var sessions = new List<SessionSummary>
{
// Model A: 1 failed of 2 → 50%
// Model A: 1 failed / 2 finished → 50%; the 2 in-flight sessions widen Total
// but must not dilute the rate (over-total it would read 25%).
S(SessionStatus.Failed, model: "A"),
S(SessionStatus.Succeeded, model: "A"),
// Model B: 2 failed of 2 → 100%
S(SessionStatus.InProgress, model: "A"),
S(SessionStatus.Pending, model: "A"),
// Model B: 2 failed of 2 finished → 100%
S(SessionStatus.Failed, model: "B"),
S(SessionStatus.Failed, model: "B"),
// Model C: no failures → excluded entirely
@@ -164,35 +207,37 @@ public class FleetHealthPayloadTests
Assert.Equal(2, payload.TopFailingModels[0].Failed);
Assert.Equal(100, payload.TopFailingModels[0].FailureRate);
Assert.Equal("Contoso A", payload.TopFailingModels[1].Model);
Assert.Equal(4, payload.TopFailingModels[1].Total);
Assert.Equal(50, payload.TopFailingModels[1].FailureRate);
Assert.DoesNotContain(payload.TopFailingModels, m => m.Model == "Contoso C");
}
[Fact]
public void FailureReasons_TruncateLongReasons_AndTakeTopFive()
public void FailureReasons_GroupByPrefix_KeepFullestText_AndTakeTopFive()
{
var sessions = new List<SessionSummary>();
// 6 distinct reasons with descending frequency 6..1 → only top 5 kept.
for (int freq = 6; freq >= 1; freq--)
// 5 distinct short reasons with descending frequency 5..1.
for (int freq = 5; freq >= 1; freq--)
for (int i = 0; i < freq; i++)
sessions.Add(S(SessionStatus.Failed, failureReason: $"Reason {freq}"));
// A null reason folds into "Unknown".
sessions.Add(S(SessionStatus.Failed, failureReason: null));
// An over-long reason is collapsed to a 50-char prefix + ellipsis.
var longReason = new string('x', 60);
sessions.Add(S(SessionStatus.Failed, failureReason: longReason));
// Two variants that share the first 50 characters collapse into one group;
// the longest variant is kept for display so the UI can show the full text.
var prefix = new string('x', 50);
var shortVariant = prefix + " short";
var longVariant = prefix + " a much longer tail that must survive intact";
for (int i = 0; i < 6; i++)
sessions.Add(S(SessionStatus.Failed, failureReason: shortVariant));
sessions.Add(S(SessionStatus.Failed, failureReason: longVariant));
var payload = MetricsMath.BuildFleetHealthPayload(sessions, days: 30);
Assert.Equal(5, payload.FailureReasons.Count);
Assert.Equal("Reason 6", payload.FailureReasons[0].Reason);
Assert.Equal(6, payload.FailureReasons[0].Count);
// The collapsed long reason exists in the tally (50 chars + "...").
var collapsed = new string('x', 50) + "...";
Assert.True(
sessions.Count(s => s.FailureReason == longReason) == 1,
"sanity: exactly one long-reason session");
Assert.Equal(53, collapsed.Length);
// The grouped cluster (6 + 1 = 7) outranks every "Reason N".
Assert.Equal(7, payload.FailureReasons[0].Count);
// The full, longest text survives end-to-end — no 50-char truncation.
Assert.Equal(longVariant, payload.FailureReasons[0].Reason);
Assert.DoesNotContain("...", payload.FailureReasons[0].Reason);
}
[Fact]
@@ -0,0 +1,99 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Functions.Functions.Metrics;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Pins the success-rate semantics of the geographic aggregation
/// (<see cref="GetGeographicMetricsFunction.ComputeGeographicMetrics"/>): the rate is an outcome
/// quota over finished enrollments (Succeeded + Failed) following the SLA convention. In-flight
/// sessions (InProgress/Pending/Stalled/AwaitingUser) and Incomplete (terminal, non-failure) sit
/// in SessionCount but never dilute the rate — the customer-visible symptom this pins against was
/// a site mid-rollout showing a poor rate despite zero actual failures.
/// </summary>
public class GeographicMetricsSuccessRateTests
{
private static SessionSummary S(SessionStatus status, string city = "Frankfurt")
=> new()
{
TenantId = "00000000-0000-0000-0000-000000000fe0",
SessionId = Guid.NewGuid().ToString(),
Status = status,
StartedAt = DateTime.UtcNow.AddHours(-2),
GeoCountry = "DE",
GeoRegion = "Hessen",
GeoCity = city,
GeoLoc = "50.11,8.68",
};
[Fact]
public void SuccessRate_IsOverFinishedSessions_InFlightAndIncompleteExcluded()
{
var sessions = new List<SessionSummary>
{
S(SessionStatus.Succeeded),
S(SessionStatus.Succeeded),
S(SessionStatus.Succeeded),
S(SessionStatus.Failed),
S(SessionStatus.InProgress),
S(SessionStatus.Pending),
S(SessionStatus.AwaitingUser),
S(SessionStatus.Incomplete), // terminal but non-failure — stays out of the rate
};
var result = GetGeographicMetricsFunction.ComputeGeographicMetrics(
sessions, new List<AppInstallSummary>(), "city");
var loc = Assert.Single(result.Locations);
Assert.Equal(8, loc.SessionCount);
Assert.Equal(3, loc.Succeeded);
Assert.Equal(1, loc.Failed);
// 3 / (3 + 1) = 75% (over finished), NOT 3 / 8 = 37.5% (over all sessions).
Assert.Equal(75d, loc.SuccessRate);
}
[Fact]
public void SuccessRate_IsZero_WhenNothingFinishedYet()
{
var sessions = new List<SessionSummary>
{
S(SessionStatus.InProgress),
S(SessionStatus.Pending),
};
var result = GetGeographicMetricsFunction.ComputeGeographicMetrics(
sessions, new List<AppInstallSummary>(), "city");
var loc = Assert.Single(result.Locations);
// No finished enrollments → no rate; the frontend renders "—" off Succeeded + Failed == 0.
Assert.Equal(0d, loc.SuccessRate);
Assert.Equal(0, loc.Succeeded + loc.Failed);
Assert.Equal(2, loc.SessionCount);
}
[Fact]
public void SuccessRate_IsPerLocation_NotGlobal()
{
var sessions = new List<SessionSummary>
{
// Frankfurt: 1 / 1 finished → 100% despite the in-flight session.
S(SessionStatus.Succeeded, city: "Frankfurt"),
S(SessionStatus.InProgress, city: "Frankfurt"),
// Berlin: 1 / 2 finished → 50%.
S(SessionStatus.Succeeded, city: "Berlin"),
S(SessionStatus.Failed, city: "Berlin"),
};
var result = GetGeographicMetricsFunction.ComputeGeographicMetrics(
sessions, new List<AppInstallSummary>(), "city");
var frankfurt = result.Locations.Single(l => l.City == "Frankfurt");
var berlin = result.Locations.Single(l => l.City == "Berlin");
Assert.Equal(100d, frankfurt.SuccessRate);
Assert.Equal(50d, berlin.SuccessRate);
}
}
@@ -119,6 +119,24 @@ public class MetricsMathAppPayloadTests
Assert.Equal(2, topCode.GetProperty("count").GetInt32());
}
[Fact]
public void FailureRate_IsOverFinishedInstalls_InProgressExcluded()
{
var summaries = new List<AppInstallSummary>
{
new() { AppName = "App A", Status = "Failed" },
new() { AppName = "App A", Status = "Succeeded", DurationSeconds = 10 },
new() { AppName = "App A", Status = "InProgress" },
new() { AppName = "App A", Status = "InProgress" },
};
var app = Build(summaries).GetProperty("topFailingApps")[0];
Assert.Equal(4, app.GetProperty("totalInstalls").GetInt32());
// 1 / (1 + 1) = 50% (over finished), NOT 1 / 4 = 25% (over all installs).
Assert.Equal(50d, app.GetProperty("failureRate").GetDouble());
}
[Fact]
public void SlowestApps_DropsAppsBelowSampleFloor()
{
@@ -62,7 +62,7 @@ public class SessionDeletionWorkerTests
};
harness.EnqueueMessage(JsonConvert.SerializeObject(envelope), dequeueCount: QueuePollingWorkerBase.DefaultMaxDequeueCount + 1);
await harness.RunForAsync(TimeSpan.FromMilliseconds(500));
await harness.RunUntilAsync(() => harness.OpsEventRecorded());
// Poison queue received the body and main queue deleted the original.
harness.PoisonQueue.Verify(q => q.SendMessageAsync(
@@ -121,7 +121,7 @@ public class SessionDeletionWorkerTests
},
"etag-mock"));
await harness.RunForAsync(TimeSpan.FromMilliseconds(500));
await harness.RunUntilAsync(() => harness.OpsEventRecorded());
var poisoned = Assert.Single(harness.OpsEventCalls, e => e.EventType == "SessionDeletionPoisoned");
Assert.Contains($"\"observedResidualCount\":{capObservation}", poisoned.RawDetails);
@@ -168,7 +168,7 @@ public class SessionDeletionWorkerTests
},
"etag-mock"));
await harness.RunForAsync(TimeSpan.FromMilliseconds(500));
await harness.RunUntilAsync(() => harness.OpsEventRecorded());
var poisoned = Assert.Single(harness.OpsEventCalls, e => e.EventType == "SessionDeletionPoisoned");
Assert.Contains("\"observedResidualCount\":3", poisoned.RawDetails);
@@ -307,7 +307,7 @@ public class SessionDeletionWorkerTests
},
"etag-mock"));
await harness.RunForAsync(TimeSpan.FromMilliseconds(500));
await harness.RunUntilAsync(() => harness.OpsEventRecorded());
var poisoned = Assert.Single(harness.OpsEventCalls, e => e.EventType == "SessionDeletionPoisoned");
Assert.True(poisoned.RawDetails.Length < 4096,
@@ -357,7 +357,7 @@ public class SessionDeletionWorkerTests
return Task.FromResult(Response.FromValue(receipt, new Mock<Response>().Object));
});
await harness.RunForAsync(TimeSpan.FromMilliseconds(500));
await harness.RunUntilAsync(() => harness.OpsEventRecorded());
var casIdx = callOrder.IndexOf("cas-running-to-poisoned");
var poisonIdx = callOrder.IndexOf("poison-queue-send");
@@ -402,7 +402,7 @@ public class SessionDeletionWorkerTests
CurrentState = SessionDeletionState.Poisoned,
});
await harness.RunForAsync(TimeSpan.FromMilliseconds(500));
await harness.RunUntilAsync(() => harness.OpsEventRecorded());
// Both CAS attempts fired, and the poison-queue send still completed.
harness.StorageMock.Verify(s => s.CasSetSessionDeletionStateAsync(
@@ -444,7 +444,7 @@ public class SessionDeletionWorkerTests
CurrentState = "Unknown",
});
await harness.RunForAsync(TimeSpan.FromMilliseconds(500));
await harness.RunUntilAsync(() => harness.OpsEventRecorded());
harness.PoisonQueue.Verify(q => q.SendMessageAsync(It.IsAny<string>(), It.IsAny<CancellationToken>()),
Times.AtLeastOnce);
@@ -457,7 +457,7 @@ public class SessionDeletionWorkerTests
var harness = new Harness();
harness.EnqueueMessage("{ \"this is not valid JSON for an envelope: garbage", dequeueCount: 1);
await harness.RunForAsync(TimeSpan.FromMilliseconds(500));
await harness.RunUntilAsync(() => harness.MainQueueDeleted());
harness.HandlerMock.Verify(h => h.HandleAsync(
It.IsAny<SessionDeletionEnvelope>(), It.IsAny<CancellationToken>()),
@@ -491,7 +491,7 @@ public class SessionDeletionWorkerTests
await Task.Delay(TimeSpan.FromMilliseconds(350), ct);
});
await harness.RunForAsync(TimeSpan.FromMilliseconds(800));
await harness.RunUntilAsync(() => harness.HeartbeatCount() >= 3);
// Heartbeat called at least 3× while the handler was busy (350ms / 50ms ≈ 7, minus a few
// due to scheduling jitter). One call is enough to prove the heartbeat task is wired.
@@ -512,7 +512,7 @@ public class SessionDeletionWorkerTests
};
harness.EnqueueMessage(JsonConvert.SerializeObject(envelope), dequeueCount: 1);
await harness.RunForAsync(TimeSpan.FromMilliseconds(500));
await harness.RunUntilAsync(() => harness.MainQueueDeleted());
harness.HandlerMock.Verify(h => h.HandleAsync(
It.IsAny<SessionDeletionEnvelope>(), It.IsAny<CancellationToken>()),
@@ -663,7 +663,9 @@ public class SessionDeletionWorkerTests
opsRepo.Setup(r => r.SaveOpsEventAsync(It.IsAny<OpsEventEntry>()))
.Returns<OpsEventEntry>(e =>
{
OpsEventCalls.Add(CapturedOpsEvent.From(e));
// Locked so RunUntilAsync's condition poll can read Count from the test
// thread while the worker thread appends.
lock (OpsEventCalls) OpsEventCalls.Add(CapturedOpsEvent.From(e));
return Task.CompletedTask;
});
var alertDispatch = new OpsAlertDispatchService(
@@ -721,6 +723,41 @@ public class SessionDeletionWorkerTests
try { await Sut.StopAsync(CancellationToken.None); }
catch (OperationCanceledException) { /* expected */ }
}
/// <summary>
/// De-flake (CI run 30047512186): <see cref="RunForAsync"/>'s fixed wall-clock window
/// raced the worker's background poll loop on loaded CI runners — positive assertions
/// (delete issued, ops event recorded) could fire before the loop ever processed the
/// message. Runs the worker until <paramref name="condition"/> observes the awaited
/// side effect (polled every 25 ms); the ceiling only bounds a genuinely broken worker,
/// the asserts after this call remain the source of truth. Same pattern as the watchdog
/// event-gate de-flake. Only for POSITIVE expectations — Times.Never-style tests keep
/// the fixed window (there is no event to wait for).
/// </summary>
public async Task RunUntilAsync(Func<bool> condition, TimeSpan? timeout = null)
{
var deadline = DateTime.UtcNow + (timeout ?? TimeSpan.FromSeconds(10));
using var cts = new CancellationTokenSource();
try { await Sut.StartAsync(cts.Token); }
catch (OperationCanceledException) { /* expected */ }
while (!condition() && DateTime.UtcNow < deadline)
await Task.Delay(25);
cts.Cancel();
try { await Sut.StopAsync(CancellationToken.None); }
catch (OperationCanceledException) { /* expected */ }
}
// Condition probes for RunUntilAsync. Moq's Invocations list snapshots under its
// internal lock, so polling it from the test thread while the worker invokes the
// mock is safe. OpsEventCalls is guarded by its own lock (see the repo callback).
public bool MainQueueDeleted() =>
MainQueue.Invocations.Any(i => i.Method.Name == nameof(QueueClient.DeleteMessageAsync));
public int HeartbeatCount() =>
MainQueue.Invocations.Count(i => i.Method.Name == nameof(QueueClient.UpdateMessageAsync));
public bool OpsEventRecorded()
{
lock (OpsEventCalls) return OpsEventCalls.Count > 0;
}
}
private sealed record AuditEntry(
@@ -1,5 +1,6 @@
using System.Collections.Specialized;
using System.Threading;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
@@ -146,24 +147,11 @@ namespace AutopilotMonitor.Functions.Functions.Apps
var succeeded = g.Count(s => s.Status == "Succeeded");
var failed = g.Count(s => s.Status == "Failed");
var completed = g.Where(s => s.Status == "Succeeded").ToList();
var failureRate = total > 0 ? Math.Round((double)failed / total * 100, 1) : 0;
var failureRate = MetricsMath.TerminalFailureRatePct(failed, succeeded);
// Trend: compare failure rate in first half vs second half of window.
// Only emit non-stable trend if BOTH halves have >= 5 installs (too noisy otherwise).
var firstHalf = g.Where(s => s.StartedAt < midpoint).ToList();
var secondHalf = g.Where(s => s.StartedAt >= midpoint).ToList();
double? trendDelta = null;
string trend = "stable";
if (firstHalf.Count >= 5 && secondHalf.Count >= 5)
{
var fhRate = (double)firstHalf.Count(s => s.Status == "Failed") / firstHalf.Count * 100;
var shRate = (double)secondHalf.Count(s => s.Status == "Failed") / secondHalf.Count * 100;
var delta = Math.Round(shRate - fhRate, 1);
trendDelta = delta;
if (delta < -1) trend = "improving";
else if (delta > 1) trend = "worsening";
}
var (trend, trendDelta) = ComputeFailureTrend(firstHalf, secondHalf);
return new
{
@@ -275,7 +263,7 @@ namespace AutopilotMonitor.Functions.Functions.Apps
var succeeded = summaries.Count(s => s.Status == "Succeeded");
var failed = summaries.Count(s => s.Status == "Failed");
var completed = summaries.Where(s => s.Status == "Succeeded").ToList();
var failureRate = Math.Round((double)failed / total * 100, 1);
var failureRate = MetricsMath.TerminalFailureRatePct(failed, succeeded);
var avgDurationSeconds = completed.Count > 0 ? Math.Round(completed.Average(s => s.DurationSeconds), 0) : 0;
var p95DurationSeconds = Percentile(completed.Select(s => s.DurationSeconds).ToList(), 0.95);
var avgDownloadBytes = completed.Count > 0 ? (long)completed.Average(s => s.DownloadBytes) : 0;
@@ -283,16 +271,7 @@ namespace AutopilotMonitor.Functions.Functions.Apps
// Trend (same rule as list endpoint).
var firstHalf = summaries.Where(s => s.StartedAt < midpoint).ToList();
var secondHalf = summaries.Where(s => s.StartedAt >= midpoint).ToList();
double? trendDelta = null;
string trend = "stable";
if (firstHalf.Count >= 5 && secondHalf.Count >= 5)
{
var fhRate = (double)firstHalf.Count(s => s.Status == "Failed") / firstHalf.Count * 100;
var shRate = (double)secondHalf.Count(s => s.Status == "Failed") / secondHalf.Count * 100;
trendDelta = Math.Round(shRate - fhRate, 1);
if (trendDelta < -1) trend = "improving";
else if (trendDelta > 1) trend = "worsening";
}
var (trend, trendDelta) = ComputeFailureTrend(firstHalf, secondHalf);
var flakinessScore = total > 0
? Math.Round((double)summaries.Count(s => s.AttemptNumber > 1) / total, 3)
@@ -308,12 +287,13 @@ namespace AutopilotMonitor.Functions.Functions.Apps
{
var vTotal = g.Count();
var vFailed = g.Count(s => s.Status == "Failed");
var vSucceeded = g.Count(s => s.Status == "Succeeded");
return new
{
appVersion = g.Key,
installs = vTotal,
failed = vFailed,
failureRate = vTotal > 0 ? Math.Round((double)vFailed / vTotal * 100, 1) : 0
failureRate = MetricsMath.TerminalFailureRatePct(vFailed, vSucceeded)
};
})
.OrderByDescending(v => v.installs)
@@ -359,12 +339,16 @@ namespace AutopilotMonitor.Functions.Functions.Apps
Model = sessionLookup[$"{s.TenantId}|{s.SessionId}"].Model ?? "Unknown"
})
.GroupBy(x => new { x.Manufacturer, x.Model })
.Where(g => g.Count() >= 5)
// Sample floor on FINISHED installs — the displayed metric is the terminal-only
// failure rate, so a model with many in-flight installs but few outcomes is still
// too noisy to rank.
.Where(g => g.Count(x => x.Summary.Status == "Failed" || x.Summary.Status == "Succeeded") >= 5)
.Select(g =>
{
var modelTotal = g.Count();
var modelFailed = g.Count(x => x.Summary.Status == "Failed");
var modelFailureRate = Math.Round((double)modelFailed / modelTotal * 100, 1);
var modelSucceeded = g.Count(x => x.Summary.Status == "Succeeded");
var modelFailureRate = MetricsMath.TerminalFailureRatePct(modelFailed, modelSucceeded);
var lift = failureRate > 0
? Math.Round(modelFailureRate / failureRate, 2)
: 0;
@@ -496,6 +480,30 @@ namespace AutopilotMonitor.Functions.Functions.Apps
// ── Helpers ─────────────────────────────────────────────────────────
/// <summary>
/// Failure-rate trend between the two halves of the window, shared by the list and
/// analytics endpoints. Rates follow the terminal-only convention
/// (<see cref="MetricsMath.TerminalFailureRatePct"/>), and a non-stable trend requires
/// at least 5 FINISHED installs in BOTH halves — too noisy otherwise, and a half made
/// up of in-flight installs has no outcome to trend on.
/// </summary>
private static (string Trend, double? TrendDelta) ComputeFailureTrend(
List<AppInstallSummary> firstHalf, List<AppInstallSummary> secondHalf)
{
var fhFailed = firstHalf.Count(s => s.Status == "Failed");
var fhSucceeded = firstHalf.Count(s => s.Status == "Succeeded");
var shFailed = secondHalf.Count(s => s.Status == "Failed");
var shSucceeded = secondHalf.Count(s => s.Status == "Succeeded");
if (fhFailed + fhSucceeded < 5 || shFailed + shSucceeded < 5)
return ("stable", null);
var delta = Math.Round(
MetricsMath.TerminalFailureRatePct(shFailed, shSucceeded)
- MetricsMath.TerminalFailureRatePct(fhFailed, fhSucceeded), 1);
var trend = delta < -1 ? "improving" : delta > 1 ? "worsening" : "stable";
return (trend, delta);
}
private static List<object> BuildTimeSeries(List<AppInstallSummary> summaries, DateTime cutoff, DateTime now, string bucket)
{
var start = bucket == "week" ? StartOfWeek(cutoff) : cutoff.Date;
@@ -531,7 +539,7 @@ namespace AutopilotMonitor.Functions.Functions.Apps
installs = bTotal,
succeeded = bSucceeded,
failed = bFailed,
failureRate = bTotal > 0 ? Math.Round((double)bFailed / bTotal * 100, 1) : 0,
failureRate = MetricsMath.TerminalFailureRatePct(bFailed, bSucceeded),
avgDurationSeconds = bCompleted.Count > 0 ? Math.Round(bCompleted.Average(s => s.DurationSeconds), 0) : 0
};
})
@@ -106,6 +106,13 @@ namespace AutopilotMonitor.Functions.Functions.Metrics
var first = group.First();
var succeeded = group.Where(s => s.Status == SessionStatus.Succeeded).ToList();
var failed = group.Where(s => s.Status == SessionStatus.Failed).ToList();
// Success rate is an outcome quota over finished enrollments only
// (Succeeded + Failed), matching the SLA convention. In-flight sessions
// (InProgress/Pending/Stalled/AwaitingUser) and Incomplete (terminal,
// non-failure) must not dilute it — a site mid-rollout would otherwise
// show a poor rate despite zero actual failures. 0 finished → 0; the
// frontend renders "—" off Succeeded+Failed instead of a 0% badge.
var finished = succeeded.Count + failed.Count;
// Duration stats from succeeded sessions with valid duration
var durations = succeeded
@@ -156,7 +163,7 @@ namespace AutopilotMonitor.Functions.Functions.Metrics
SessionCount = group.Count(),
Succeeded = succeeded.Count,
Failed = failed.Count,
SuccessRate = group.Count() > 0 ? Math.Round((double)succeeded.Count / group.Count() * 100, 1) : 0,
SuccessRate = finished > 0 ? Math.Round((double)succeeded.Count / finished * 100, 1) : 0,
AvgDurationMinutes = Math.Round(avgDuration, 1),
MedianDurationMinutes = Math.Round(medianDuration, 1),
P95DurationMinutes = Math.Round(p95Duration, 1),
@@ -7,6 +7,19 @@ namespace AutopilotMonitor.Functions.Helpers;
/// </summary>
public static class MetricsMath
{
/// <summary>
/// App-install failure rate over finished installs only: Failed / (Failed + Succeeded), one
/// decimal, 0 when nothing finished. The same outcome-quota convention as the enrollment
/// success rate — "InProgress" rows (still installing, or orphaned by a session that died
/// mid-install) never dilute the rate. Shared by the app-metrics payload and the Apps
/// dashboard aggregations so the definition can't drift between panes.
/// </summary>
public static double TerminalFailureRatePct(int failed, int succeeded)
{
var finished = failed + succeeded;
return finished > 0 ? Math.Round((double)failed / finished * 100, 1) : 0;
}
/// <summary>
/// Builds the complete app-metrics response object from a (pre-time-filtered) set of app
/// install summaries. Single source of truth for both the tenant (<c>metrics/app</c>) and
@@ -40,7 +53,7 @@ public static class MetricsMath
totalInstalls = total,
succeeded = completed.Count,
failed = failed.Count,
failureRate = total > 0 ? Math.Round((double)failed.Count / total * 100, 1) : 0,
failureRate = TerminalFailureRatePct(failed.Count, completed.Count),
avgDurationSeconds = completed.Count > 0 ? Math.Round(completed.Average(s => s.DurationSeconds), 0) : 0,
maxDurationSeconds = completed.Count > 0 ? completed.Max(s => s.DurationSeconds) : 0,
avgDownloadBytes = completed.Count > 0 ? (long)completed.Average(s => s.DownloadBytes) : 0,
@@ -93,9 +106,10 @@ public static class MetricsMath
/// Single source of truth for the tenant (<c>metrics/fleet-health</c>) and global
/// (<c>global/metrics/fleet-health</c>) functions. Replaces the previous client-side path that
/// drained up to 200k raw sessions into the browser and ran these aggregations on the main
/// thread. Semantics are preserved verbatim from that client code, notably: success rate is
/// over ALL sessions (not just terminal ones), and average duration counts every
/// non-in-progress session that carries a positive duration (failures included).
/// thread. Success rate follows the SLA convention: Succeeded / (Succeeded + Failed) — finished
/// enrollments only, so in-flight sessions and Incomplete (terminal, non-failure) never dilute
/// it. Average duration counts every non-in-progress session that carries a positive duration
/// (failures included).
/// </summary>
public static FleetHealthMetrics BuildFleetHealthPayload(IReadOnlyList<SessionSummary> sessions, int days)
{
@@ -121,6 +135,7 @@ public static class MetricsMath
}
int total = sessions.Count;
int finished = succeeded + failed;
var stats = new FleetHealthStats
{
Total = total,
@@ -128,7 +143,7 @@ public static class MetricsMath
Failed = failed,
InProgress = inProgress,
Incomplete = incomplete,
SuccessRate = total > 0 ? Math.Round((double)succeeded / total * 100, 1) : 0,
SuccessRate = finished > 0 ? Math.Round((double)succeeded / finished * 100, 1) : 0,
AvgDurationMinutes = completedWithDurationCount > 0
? (int)Math.Round((double)completedDurationSeconds / completedWithDurationCount / 60.0, MidpointRounding.AwayFromZero)
: 0,
@@ -179,19 +194,29 @@ public static class MetricsMath
private static List<FleetFailureReason> BuildFleetFailureReasons(IReadOnlyList<SessionSummary> sessions)
{
var reasons = new Dictionary<string, int>();
// Group near-identical messages by a 50-char prefix so variants collapse
// into one row, but keep the longest full reason as the display value. The
// UI truncates and expands it on demand, so the complete text must survive.
var groups = new Dictionary<string, (string Display, int Count)>();
foreach (var s in sessions)
{
if (s.Status != SessionStatus.Failed) continue;
var reason = string.IsNullOrEmpty(s.FailureReason) ? "Unknown" : s.FailureReason;
// Collapse very long reasons to a 50-char prefix so near-identical messages group.
if (reason.Length > 50) reason = reason.Substring(0, 50) + "...";
reasons[reason] = reasons.GetValueOrDefault(reason) + 1;
var key = reason.Length > 50 ? reason.Substring(0, 50) : reason;
if (groups.TryGetValue(key, out var g))
{
var display = reason.Length > g.Display.Length ? reason : g.Display;
groups[key] = (display, g.Count + 1);
}
else
{
groups[key] = (reason, 1);
}
}
return reasons
.OrderByDescending(kv => kv.Value)
return groups
.OrderByDescending(kv => kv.Value.Count)
.Take(5)
.Select(kv => new FleetFailureReason { Reason = kv.Key, Count = kv.Value })
.Select(kv => new FleetFailureReason { Reason = kv.Value.Display, Count = kv.Value.Count })
.ToList();
}
@@ -208,6 +233,7 @@ public static class MetricsMath
}
m.Total++;
if (s.Status == SessionStatus.Succeeded) m.Succeeded++;
else if (s.Status == SessionStatus.Failed) m.Failed++;
}
return models.Values
.OrderByDescending(m => m.Total)
@@ -240,15 +266,18 @@ public static class MetricsMath
private static List<FleetFailingModel> BuildFleetTopFailingModels(IReadOnlyList<SessionSummary> sessions)
{
var acc = new Dictionary<string, (int Failed, int Total)>();
var acc = new Dictionary<string, (int Failed, int Succeeded, int Total)>();
foreach (var s in sessions)
{
var key = FleetModelKey(s);
acc.TryGetValue(key, out var cur);
cur.Total++;
if (s.Status == SessionStatus.Failed) cur.Failed++;
else if (s.Status == SessionStatus.Succeeded) cur.Succeeded++;
acc[key] = cur;
}
// FailureRate over finished enrollments only (mirror of the success-rate convention);
// the Where(Failed > 0) guard also keeps the denominator non-zero.
return acc
.Where(kv => kv.Value.Failed > 0)
.Select(kv => new FleetFailingModel
@@ -256,7 +285,8 @@ public static class MetricsMath
Model = kv.Key,
Failed = kv.Value.Failed,
Total = kv.Value.Total,
FailureRate = (int)Math.Round((double)kv.Value.Failed / kv.Value.Total * 100, MidpointRounding.AwayFromZero),
FailureRate = (int)Math.Round(
(double)kv.Value.Failed / (kv.Value.Failed + kv.Value.Succeeded) * 100, MidpointRounding.AwayFromZero),
})
.OrderByDescending(m => m.Failed)
.Take(5)
@@ -3,6 +3,9 @@
"version": "1.5.1",
"description": "MCP server for Autopilot-Monitor — enables AI agents to query enrollment sessions, device properties, events and vulnerabilities",
"type": "module",
"engines": {
"node": ">=22"
},
"bin": {
"autopilot-monitor-mcp": "./dist/index.js"
},
@@ -36,7 +36,8 @@ namespace AutopilotMonitor.Shared.Models
public int InProgress { get; set; }
/// <summary>Terminal, non-failure sessions (timeout reclassification). Surfaced as its own count; not a failure.</summary>
public int Incomplete { get; set; }
/// <summary>Succeeded / total * 100 (one decimal). Note: over all sessions, not just terminal.</summary>
/// <summary>Succeeded / (Succeeded + Failed) * 100 (one decimal) — finished enrollments only,
/// matching the SLA convention. 0 when nothing has finished yet (clients render "—").</summary>
public double SuccessRate { get; set; }
/// <summary>Average duration in minutes over non-in-progress sessions that carry a duration.</summary>
public int AvgDurationMinutes { get; set; }
@@ -58,8 +59,11 @@ namespace AutopilotMonitor.Shared.Models
public sealed class FleetModelHealth
{
public string Model { get; set; } = default!;
/// <summary>All sessions on this model in the window, including in-flight ones.</summary>
public int Total { get; set; }
public int Succeeded { get; set; }
/// <summary>Clients derive the per-model rate as Succeeded / (Succeeded + Failed).</summary>
public int Failed { get; set; }
}
public sealed class FleetSlowModel
@@ -73,7 +77,9 @@ namespace AutopilotMonitor.Shared.Models
{
public string Model { get; set; } = default!;
public int Failed { get; set; }
/// <summary>All sessions on this model in the window, including in-flight ones.</summary>
public int Total { get; set; }
/// <summary>Failed / (Failed + Succeeded) * 100, rounded — finished enrollments only.</summary>
public int FailureRate { get; set; }
}
}
@@ -3,6 +3,7 @@
import { useCallback, useEffect, useState } from "react";
import { api } from "@/lib/api";
import { authenticatedFetch } from "@/lib/authenticatedFetch";
import TruncatedLabel from "@/components/TruncatedLabel";
import { trackEvent } from "@/lib/appInsights";
import type { AutoResolveResult, IgnoredSoftwareEntry } from "./SoftwareMappingTypes";
@@ -211,8 +212,8 @@ export function IgnoredSoftwareTab({
return (
<tr key={key} className="hover:bg-gray-50 dark:hover:bg-gray-700/30 transition-colors">
<td className="px-3 py-3 text-sm text-gray-900 dark:text-gray-100 truncate">
{entry.softwareName}
<td className="px-3 py-3">
<TruncatedLabel text={entry.softwareName} className="block text-sm text-gray-900 dark:text-gray-100" />
</td>
<td className="px-3 py-3 text-sm text-gray-600 dark:text-gray-400 truncate">
{entry.publisher || <span className="text-gray-300 dark:text-gray-600 italic">unknown</span>}
@@ -3,6 +3,7 @@
import { useCallback, useEffect, useState } from "react";
import Link from "next/link";
import { api } from "@/lib/api";
import TruncatedLabel from "@/components/TruncatedLabel";
import { authenticatedFetch, TokenExpiredError } from "@/lib/authenticatedFetch";
import { extractContinuation } from "@/lib/paginationLink";
import { extractSessionId, buildAutoReason } from "./opsEventSessionHelpers";
@@ -445,8 +446,8 @@ export function OpsEventsSection({
<td className="px-3 py-2.5 text-sm font-mono text-gray-700 dark:text-gray-300 whitespace-nowrap">
{evt.eventType}
</td>
<td className="px-3 py-2.5 text-sm text-gray-700 dark:text-gray-300 max-w-[300px] truncate">
{evt.message}
<td className="px-3 py-2.5 max-w-[300px]">
<TruncatedLabel interactive={false} text={evt.message ?? ""} className="block text-sm text-gray-700 dark:text-gray-300" />
</td>
<td className="px-3 py-2.5 text-sm font-mono text-gray-700 dark:text-gray-300">
{evt.tenantId ? (evt.tenantId.length > 8 ? `${evt.tenantId.slice(0, 8)}...` : evt.tenantId) : <span className="text-gray-300 dark:text-gray-600">-</span>}
@@ -3,6 +3,7 @@
import { useCallback, useEffect, useState } from "react";
import { api } from "@/lib/api";
import { authenticatedFetch, TokenExpiredError } from "@/lib/authenticatedFetch";
import TruncatedLabel from "@/components/TruncatedLabel";
import { extractContinuation } from "@/lib/paginationLink";
import { isGuid } from "@/utils/inputValidation";
import { trackEvent } from "@/lib/appInsights";
@@ -355,8 +356,10 @@ export function SessionReportsSection({
<td className="px-4 py-3 text-sm text-gray-700 dark:text-gray-300">
{r.submittedBy}
</td>
<td className="px-4 py-3 text-sm text-gray-700 dark:text-gray-300 truncate max-w-xs">
{r.comment || <span className="text-gray-400 italic">no comment</span>}
<td className="px-4 py-3 max-w-xs">
{r.comment
? <TruncatedLabel interactive={false} text={r.comment} className="block text-sm text-gray-700 dark:text-gray-300" />
: <span className="text-sm text-gray-400 italic">no comment</span>}
</td>
<td className="px-4 py-3 text-sm text-gray-700 dark:text-gray-300">
{r.adminNote ? (
@@ -364,7 +367,7 @@ export function SessionReportsSection({
<svg className="w-4 h-4 text-indigo-500 dark:text-indigo-400 shrink-0" fill="currentColor" viewBox="0 0 24 24">
<path d="M3 17.25V21h3.75L17.81 9.94l-3.75-3.75L3 17.25zM20.71 7.04a1 1 0 000-1.41l-2.34-2.34a1 1 0 00-1.41 0l-1.83 1.83 3.75 3.75 1.83-1.83z" />
</svg>
<span className="truncate max-w-[140px]">{r.adminNote}</span>
<TruncatedLabel interactive={false} text={r.adminNote} className="max-w-[140px]" />
</div>
) : (
<span className="text-gray-300 dark:text-gray-600">—</span>
@@ -3,6 +3,7 @@
import { useCallback, useEffect, useState } from "react";
import { api } from "@/lib/api";
import { authenticatedFetch, TokenExpiredError } from "@/lib/authenticatedFetch";
import TruncatedLabel from "@/components/TruncatedLabel";
import { trackEvent } from "@/lib/appInsights";
import type { UnmatchedSoftwareEntry, AutoResolveResult } from "./SoftwareMappingTypes";
@@ -564,9 +565,7 @@ export function UnmappedSoftwareTab({
/>
)}
</div>
<div className="px-3 py-3 text-sm text-gray-900 dark:text-gray-100 flex-1 min-w-0 truncate">
{entry.softwareName}
</div>
<TruncatedLabel text={entry.softwareName} className="px-3 py-3 text-sm text-gray-900 dark:text-gray-100 flex-1" />
<div className="px-3 py-3 text-sm text-gray-600 dark:text-gray-400 truncate" style={{ width: "18%", flexShrink: 0 }}>
{entry.publisher || <span className="text-gray-300 dark:text-gray-600 italic">unknown</span>}
</div>
@@ -3,6 +3,7 @@
import { useEffect, useState, useMemo, useCallback } from 'react';
import { useRouter } from 'next/navigation';
import { api } from '@/lib/api';
import TruncatedLabel from '@/components/TruncatedLabel';
import { useAuth } from '../../../../contexts/AuthContext';
import { useNotifications } from '../../../../contexts/NotificationContext';
import { authenticatedFetch, TokenExpiredError } from "@/lib/authenticatedFetch";
@@ -705,7 +706,7 @@ export function SectionAgentMetrics() {
<p className="text-xs text-gray-500 mb-1">Top exceptions:</p>
{crashRate.topExceptions.map((e) => (
<div key={e.exceptionType} className="flex justify-between text-xs">
<span className="text-red-600 font-mono truncate mr-2">{e.exceptionType}</span>
<TruncatedLabel text={e.exceptionType} className="text-red-600 font-mono mr-2" />
<span className="text-gray-500">{e.count}x</span>
</div>
))}
@@ -170,11 +170,14 @@ export default function InstallsTab({ scope, timeRange }: InstallsTabProps) {
if (!data) return { totalApps: 0, totalInstalls: 0, avgFailureRate: 0 };
const apps = data.apps ?? [];
const totalFailed = apps.reduce((acc, a) => acc + a.failed, 0);
const totalInstalls = data.totalInstalls;
// Terminal-only convention: rate over finished installs (succeeded + failed), so
// in-flight/orphaned InProgress rows don't dilute it. Matches the per-app failureRate.
const totalSucceeded = apps.reduce((acc, a) => acc + a.succeeded, 0);
const totalFinished = totalFailed + totalSucceeded;
return {
totalApps: data.totalApps,
totalInstalls,
avgFailureRate: totalInstalls > 0 ? Math.round((totalFailed / totalInstalls) * 1000) / 10 : 0,
totalInstalls: data.totalInstalls,
avgFailureRate: totalFinished > 0 ? Math.round((totalFailed / totalFinished) * 1000) / 10 : 0,
};
}, [data]);
@@ -16,6 +16,7 @@ export interface FleetHealthStats {
inProgress: number;
/** Terminal, non-failure (timeout reclassification). Surfaced separately; not counted as a failure. */
incomplete: number;
/** Succeeded / (succeeded + failed) — finished enrollments only. 0 when nothing finished yet. */
successRate: number;
avgDurationMinutes: number;
}
@@ -30,8 +31,10 @@ export interface FleetFailureReason {
}
export interface FleetModelHealth {
model: string;
/** All sessions on this model in the window, including in-flight ones. */
total: number;
succeeded: number;
failed: number;
}
export interface FleetSlowModel {
model: string;
@@ -10,6 +10,7 @@ import { useNotifications } from "../../contexts/NotificationContext";
import { api } from "@/lib/api";
import { authenticatedFetch, TokenExpiredError } from "@/lib/authenticatedFetch";
import FleetStatCard from "./components/FleetStatCard";
import TruncatedLabel from "../../components/TruncatedLabel";
import { useFleetHealth } from "./hooks/useFleetHealth";
import { useAggregatedAdminScope } from "@/hooks";
import { GlobalAdminBanner, globalAdminSubtitle } from "@/components/GlobalAdminBanner";
@@ -129,6 +130,9 @@ export default function FleetHealthPage() {
successRate: s?.successRate ?? 0,
avgDuration: s?.avgDurationMinutes ?? 0,
};
// Success rate is an outcome quota over finished enrollments (succeeded + failed);
// with none finished there is no rate to show — the card renders "—" instead of 0%.
const finishedCount = stats.succeeded + stats.failed;
const modelHealth = data?.modelHealth ?? [];
const slowestModels = data?.slowestModels ?? [];
@@ -238,10 +242,16 @@ export default function FleetHealthPage() {
<div className="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-5 gap-5 mb-8">
<FleetStatCard
title="Success Rate"
value={`${stats.successRate.toFixed(1)}%`}
subtitle={`${stats.succeeded} of ${stats.total} enrollments`}
value={finishedCount > 0 ? `${stats.successRate.toFixed(1)}%` : "—"}
subtitle={
finishedCount > 0
? `${stats.succeeded} of ${finishedCount} finished enrollments`
: "No finished enrollments yet"
}
color={
stats.successRate >= 95
finishedCount === 0
? "slate"
: stats.successRate >= 95
? "green"
: stats.successRate >= 80
? "yellow"
@@ -395,9 +405,10 @@ export default function FleetHealthPage() {
</span>
<div className="flex-1 min-w-0">
<div className="flex items-center justify-between mb-1">
<span className="text-sm text-gray-700 truncate pr-2">
{fr.reason}
</span>
<TruncatedLabel
text={fr.reason}
className="text-sm text-gray-700 pr-2"
/>
<span className="text-sm font-medium text-gray-900 flex-shrink-0">
({fr.count})
</span>
@@ -435,9 +446,10 @@ export default function FleetHealthPage() {
</span>
<div className="flex-1 min-w-0">
<div className="flex items-center justify-between mb-1">
<span className="text-sm text-gray-700 truncate pr-2">
{m.model}
</span>
<TruncatedLabel
text={m.model}
className="text-sm text-gray-700 pr-2"
/>
<span className="text-sm font-medium text-gray-900 flex-shrink-0">
{m.avgMinutes} min
</span>
@@ -470,9 +482,10 @@ export default function FleetHealthPage() {
</span>
<div className="flex-1 min-w-0">
<div className="flex items-center justify-between mb-1">
<span className="text-sm text-gray-700 truncate pr-2">
{m.model}
</span>
<TruncatedLabel
text={m.model}
className="text-sm text-gray-700 pr-2"
/>
<span className="text-sm font-medium text-red-600 flex-shrink-0">
{m.failed} failed
</span>
@@ -536,9 +549,10 @@ export default function FleetHealthPage() {
</span>
<div className="flex-1 min-w-0">
<div className="flex items-center justify-between mb-1">
<span className="text-sm text-gray-700 truncate pr-2">
{app.appName}
</span>
<TruncatedLabel
text={app.appName}
className="text-sm text-gray-700 pr-2"
/>
<span className="text-sm font-medium text-gray-900 flex-shrink-0">
{avgLabel} avg
</span>
@@ -586,9 +600,10 @@ export default function FleetHealthPage() {
</span>
<div className="flex-1 min-w-0">
<div className="flex items-center justify-between mb-1">
<span className="text-sm text-gray-700 truncate pr-2">
{app.appName}
</span>
<TruncatedLabel
text={app.appName}
className="text-sm text-gray-700 pr-2"
/>
<span className="text-sm font-medium text-red-600 flex-shrink-0">
{app.failed} failed
</span>
@@ -601,7 +616,7 @@ export default function FleetHealthPage() {
/>
</div>
<span className="text-xs text-gray-400 flex-shrink-0">
{app.failureRate}% of {app.totalInstalls}
{app.failureRate}% of {app.succeeded + app.failed} finished
</span>
</div>
{app.topFailureCodes.length > 0 && (
@@ -631,10 +646,13 @@ export default function FleetHealthPage() {
) : (
<div className="space-y-3">
{modelHealth.map((m) => {
// Rate over finished enrollments only; devices still enrolling don't count
// against the model. No finished enrollments → no rate ("—", neutral bar).
const finished = m.succeeded + m.failed;
const successRate =
m.total > 0
? Math.round((m.succeeded / m.total) * 100)
: 0;
finished > 0
? Math.round((m.succeeded / finished) * 100)
: null;
return (
<Link
key={m.model}
@@ -645,19 +663,21 @@ export default function FleetHealthPage() {
<div className="flex items-baseline justify-between mb-1">
<span className="text-sm text-gray-700 group-hover:text-blue-600 break-words leading-snug">{m.model}</span>
<span className="ml-3 flex-shrink-0 text-sm font-medium text-gray-900">
{successRate}% <span className="text-xs font-normal text-gray-400">({m.total} devices)</span>
{successRate !== null ? `${successRate}%` : "—"} <span className="text-xs font-normal text-gray-400">({m.total} devices)</span>
</span>
</div>
<div className="w-full h-2 bg-gray-100 rounded-full overflow-hidden">
<div
className={`h-full rounded-full transition-all ${
successRate >= 95
successRate === null
? "bg-gray-300"
: successRate >= 95
? "bg-green-500"
: successRate >= 80
? "bg-yellow-500"
: "bg-red-500"
}`}
style={{ width: `${successRate}%` }}
style={{ width: `${successRate ?? 0}%` }}
/>
</div>
</Link>
@@ -59,7 +59,7 @@ export const KNOWN_EVENT_TYPES: EventTypeEntry[] = [
{ value: "stall_probe_check", label: "stall_probe_check", category: "stall",
description: "Trace-level heartbeat from a stall probe — proves the logic ran (default: only the 15 min probe)." },
{ value: "esp_policy_provider_stalled", label: "esp_policy_provider_stalled", category: "stall",
description: "A registered ESP policy provider (EnrollmentStatusTracking CSP) stayed incomplete for 15+ minutes — e.g. co-management leaves only ConfigMgr registered without the Intune Sidecar provider, which parks the user ESP at 'Apps (Identifying)' with no OS timeout. One-shot Warning; payload carries the full provider table, per-provider state, and sidecarRegistered." },
description: "The ESP's app-tracking wait (EnrollmentStatusTracking CSP) made no progress for 15+ minutes: a registered policy provider stayed incomplete (reason=provider_incomplete), or Setup/Apps providers exist without the expected Intune 'Sidecar' provider (reason=sidecar_provider_missing) — the co-management signature where only ConfigMgr is registered (even with TrackingPoliciesCreated=1) and the user ESP parks at 'Apps (Identifying)' with no OS timeout. One-shot Warning; payload carries reason, the full provider table, per-provider state, and sidecarRegistered." },
{ value: "agent_late_start", label: "agent_late_start", category: "stall",
description: "Low observation coverage: the agent started long after device boot (≥10 min) AND lived only briefly (≤5 min) before a terminal outcome — it arrived after the enrollment had already decided, so its diagnosis is a post-mortem of the end-state, not a live observation. Carries bootToAgentStartSeconds, agentUptimeSeconds and outcome. Often paired with script_timeout_suspected (a script that hung ahead of the agent bootstrap)." },
@@ -11,8 +11,11 @@ interface LocationMetric {
city: string;
loc: string;
sessionCount: number;
succeeded: number;
failed: number;
avgDurationMinutes: number;
appLoadScore: number;
/** Over finished enrollments (succeeded + failed) only; 0 when nothing finished yet. */
successRate: number;
avgThroughputBytesPerSec: number;
isOutlier: boolean;
@@ -160,7 +163,10 @@ export default function GeoMap({ locations, globalAvgDuration, selectedLocation,
<Tooltip direction="top" offset={[0, -10]} opacity={0.95}>
<div className="text-xs">
<div className="font-bold text-sm mb-1">{loc.locationKey}</div>
<div><strong>Sessions:</strong> {loc.sessionCount} ({loc.successRate}% success)</div>
<div>
<strong>Sessions:</strong> {loc.sessionCount}
{loc.succeeded + loc.failed > 0 ? ` (${loc.successRate}% success)` : " (none finished yet)"}
</div>
<div><strong>Avg Duration:</strong> {Math.round(loc.avgDurationMinutes)} min</div>
{loc.appLoadScore > 0 && (
<div><strong>App-Load-Score:</strong> {Math.round(loc.appLoadScore)}</div>
@@ -4,6 +4,7 @@ import { useEffect, useState, useRef, useMemo, useCallback } from "react";
import { useRouter } from "next/navigation";
import dynamic from "next/dynamic";
import { ProtectedRoute } from "../../components/ProtectedRoute";
import TruncatedLabel from "@/components/TruncatedLabel";
import { useTenant } from "../../contexts/TenantContext";
import { useAuth } from "../../contexts/AuthContext";
import { api } from "@/lib/api";
@@ -323,18 +324,14 @@ export default function GeographicPerformancePage() {
</div>
<div className="bg-white rounded-lg shadow p-4">
<div className="text-sm font-medium text-gray-500">Fastest Location</div>
<div className="text-lg font-bold text-green-600 truncate" title={stats.fastest?.locationKey}>
{stats.fastest?.locationKey || "—"}
</div>
<TruncatedLabel text={stats.fastest?.locationKey || "—"} className="block text-lg font-bold text-green-600" />
<div className="text-xs text-gray-400">
{stats.fastest ? `${Math.round(stats.fastest.avgDurationMinutes)} min avg` : "No data"}
</div>
</div>
<div className="bg-white rounded-lg shadow p-4">
<div className="text-sm font-medium text-gray-500">Slowest Location</div>
<div className="text-lg font-bold text-red-600 truncate" title={stats.slowest?.locationKey}>
{stats.slowest?.locationKey || "—"}
</div>
<TruncatedLabel text={stats.slowest?.locationKey || "—"} className="block text-lg font-bold text-red-600" />
<div className="text-xs text-gray-400">
{stats.slowest ? `${Math.round(stats.slowest.avgDurationMinutes)} min avg` : "No data"}
</div>
@@ -507,17 +504,28 @@ export default function GeographicPerformancePage() {
</td>
<td className="px-4 py-3 text-sm text-gray-700 font-medium">{loc.sessionCount}</td>
<td className="px-4 py-3 text-sm">
<span
className={`inline-flex items-center px-2 py-0.5 rounded text-xs font-medium ${
loc.successRate >= 90
? "bg-green-100 text-green-800"
: loc.successRate >= 70
? "bg-yellow-100 text-yellow-800"
: "bg-red-100 text-red-800"
}`}
>
{loc.successRate}%
</span>
{/* Rate is over finished enrollments (succeeded + failed) only; a
location where everything is still in flight has no rate yet. */}
{loc.succeeded + loc.failed > 0 ? (
<span
className={`inline-flex items-center px-2 py-0.5 rounded text-xs font-medium ${
loc.successRate >= 90
? "bg-green-100 text-green-800"
: loc.successRate >= 70
? "bg-yellow-100 text-yellow-800"
: "bg-red-100 text-red-800"
}`}
>
{loc.successRate}%
</span>
) : (
<span
className="inline-flex items-center px-2 py-0.5 rounded text-xs font-medium bg-gray-100 text-gray-500"
title="No finished enrollments yet"
>
—
</span>
)}
</td>
<td className="px-4 py-3 text-sm">
<span
@@ -3,6 +3,7 @@
import { useTenant } from "../../contexts/TenantContext";
import { useAuth } from "../../contexts/AuthContext";
import { useNotifications } from "../../contexts/NotificationContext";
import TruncatedLabel from "@/components/TruncatedLabel";
import { useSignalR } from "../../contexts/SignalRContext";
import { ProtectedRoute } from "../../components/ProtectedRoute";
import { useProgressSearch } from "./hooks/useProgressSearch";
@@ -374,13 +375,16 @@ export default function ProgressPortalPage() {
{isCurrent && (step.id === 3 || step.id === 5) && (
<div className="flex items-center space-x-1.5 mt-0.5">
<div className="w-1.5 h-1.5 bg-blue-400 rounded-full animate-pulse flex-shrink-0" />
<span className="text-xs text-blue-500 truncate">
{currentDownload?.active && currentDownload.appName
? `Downloading ${currentDownload.appName}...`
: currentInstall?.active && currentInstall.appName
? `Installing ${currentInstall.appName}...`
: "Processing..."}
</span>
<TruncatedLabel
className="text-xs text-blue-500"
text={
currentDownload?.active && currentDownload.appName
? `Downloading ${currentDownload.appName}...`
: currentInstall?.active && currentInstall.appName
? `Installing ${currentInstall.appName}...`
: "Processing..."
}
/>
</div>
)}
</div>
@@ -398,9 +402,7 @@ export default function ProgressPortalPage() {
<div>
<p className="text-xs text-blue-500 mb-1 font-medium">Downloading</p>
<div className="flex items-center justify-between mb-1">
<span className="text-sm text-blue-700 font-medium truncate pr-2">
{currentDownload.appName}
</span>
<TruncatedLabel text={currentDownload.appName} className="text-sm text-blue-700 font-medium pr-2" />
{currentDownload.downloadRateBps > 0 && (
<span className="text-xs text-blue-500 flex-shrink-0">
{currentDownload.downloadRateBps >= 1024 * 1024
@@ -443,9 +445,7 @@ export default function ProgressPortalPage() {
<div className="flex items-center justify-between">
<div className="flex items-center space-x-1.5 min-w-0">
<div className="w-1.5 h-1.5 bg-blue-500 rounded-full animate-pulse flex-shrink-0" />
<span className="text-sm text-blue-700 font-medium truncate">
{currentInstall.appName}
</span>
<TruncatedLabel text={currentInstall.appName} className="text-sm text-blue-700 font-medium" />
</div>
{installElapsedMs != null && installElapsedMs > 0 && (
<span className="text-xs text-blue-600 font-medium tabular-nums flex-shrink-0 ml-2">
@@ -4,6 +4,7 @@ import { useEffect, useState, useCallback } from 'react';
import { useTenant } from '../../contexts/TenantContext';
import { useAuth } from '../../contexts/AuthContext';
import { useNotifications } from '../../contexts/NotificationContext';
import TruncatedLabel from '@/components/TruncatedLabel';
import { ProtectedRoute } from '../../components/ProtectedRoute';
import { api } from "@/lib/api";
import { authenticatedFetch, TokenExpiredError } from "@/lib/authenticatedFetch";
@@ -333,7 +334,7 @@ export default function SlaPage() {
<span className="text-xs text-gray-400 w-4">{i + 1}</span>
<div className="flex-1 min-w-0">
<div className="flex items-center justify-between mb-1">
<span className="text-sm text-gray-700 dark:text-gray-300 truncate pr-2">{app.appName}</span>
<TruncatedLabel text={app.appName} className="text-sm text-gray-700 dark:text-gray-300 pr-2" />
<span className="text-sm font-medium text-red-600 dark:text-red-400 flex-shrink-0">{app.failCount} failed</span>
</div>
<div className="w-full h-2 bg-gray-100 dark:bg-gray-700 rounded-full overflow-hidden">
@@ -429,7 +430,9 @@ export default function SlaPage() {
{v.violationType}
</span>
</td>
<td className="px-6 py-4 text-sm text-gray-500 dark:text-gray-400 max-w-xs truncate">{v.failureReason || "-"}</td>
<td className="px-6 py-4 max-w-xs">
<TruncatedLabel text={v.failureReason || "-"} className="block text-sm text-gray-500 dark:text-gray-400" />
</td>
</tr>
))}
</tbody>
@@ -2,6 +2,7 @@
import { useMemo, useState } from "react";
import { partitionHistoricReplayEvents } from "@/lib/historicReplay";
import TruncatedLabel from "@/components/TruncatedLabel";
import { shouldSkipLowBytesTotal, shouldSkipNoActivity } from "@/lib/downloadProgressFilters";
import { formatBytes, formatThroughput, formatDuration } from "@/lib/formatting";
import DoBreakdownBar from "./DoBreakdownBar";
@@ -353,7 +354,7 @@ function DownloadItem({ download: dl, progressPercent }: { download: DownloadIte
<path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M4 16v1a3 3 0 003 3h10a3 3 0 003-3v-1m-4-4l-4 4m0 0l-4-4m4 4V4" />
</svg>
)}
<span className={`text-sm font-medium truncate ${dl.isSkipped ? "text-gray-500" : "text-gray-900"}`}>{dl.appName}</span>
<TruncatedLabel text={dl.appName} className={`text-sm font-medium ${dl.isSkipped ? "text-gray-500" : "text-gray-900"}`} />
{dl.isSkipped && (
<span className="text-xs px-2 py-0.5 rounded-full bg-gray-200 text-gray-600 font-medium">Skipped</span>
)}
@@ -3,6 +3,7 @@
import { useEffect, useMemo, useState } from "react";
import { getErrorCodeEntry, formatErrorCode } from "@/utils/errorCodeMap";
import { partitionHistoricReplayEvents } from "@/lib/historicReplay";
import TruncatedLabel from "@/components/TruncatedLabel";
interface InstallEvent {
timestamp: string;
@@ -465,9 +466,7 @@ function InstallItemRow({ item }: { item: InstallItem }) {
<path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M9 3v2m6-2v2M9 19v2m6-2v2M5 9H3m2 6H3m18-6h-2m2 6h-2M7 19h10a2 2 0 002-2V7a2 2 0 00-2-2H7a2 2 0 00-2 2v10a2 2 0 002 2zM9 9h6v6H9V9z" />
</svg>
)}
<span className={`text-sm font-medium truncate ${item.state === "Skipped" ? "text-gray-500" : "text-gray-900"}`}>
{item.appName}
</span>
<TruncatedLabel text={item.appName} className={`text-sm font-medium ${item.state === "Skipped" ? "text-gray-500" : "text-gray-900"}`} />
{item.state === "Skipped" && (
<span className="text-xs px-2 py-0.5 rounded-full bg-gray-200 text-gray-600 font-medium">Skipped</span>
)}
@@ -0,0 +1,67 @@
"use client";
import { useState } from "react";
interface TruncatedLabelProps {
/** The full text to display. Also used verbatim as the hover tooltip. */
text: string;
/** Extra classes for the text styling (font size, colour, padding). */
className?: string;
/**
* When false, only the hover tooltip is provided — no click-to-expand and no
* button semantics. Use this inside an already-clickable row/link/button,
* where an inner onClick would hijack the outer navigation.
*/
interactive?: boolean;
}
/**
* A single-line label that truncates with an ellipsis by default.
*
* - Hovering always shows the full text in a native browser tooltip (standard
* tooltip, no special cursor).
* - When `interactive` (the default), clicking — or pressing Enter/Space when
* focused — toggles a fully expanded view that wraps the entire text over as
* many lines as needed. This keeps long values readable on touch devices
* where hover is unavailable.
* - When `interactive={false}`, it is a plain truncated span with only the
* tooltip, safe to nest inside a clickable row or link.
*/
export default function TruncatedLabel({
text,
className = "",
interactive = true,
}: TruncatedLabelProps) {
const [expanded, setExpanded] = useState(false);
if (!interactive) {
return (
<span title={text} className={`${className} min-w-0 truncate`}>
{text}
</span>
);
}
const toggle = () => setExpanded((v) => !v);
return (
<span
title={text}
role="button"
tabIndex={0}
aria-expanded={expanded}
onClick={toggle}
onKeyDown={(e) => {
if (e.key === "Enter" || e.key === " ") {
e.preventDefault();
toggle();
}
}}
className={`${className} min-w-0 cursor-pointer ${
expanded ? "whitespace-normal break-words" : "truncate"
}`}
>
{text}
</span>
);
}
@@ -1,6 +1,7 @@
"use client";
import { severityChipClass, KEV_BADGE_CLASS } from "@/lib/vulnerabilitySeverity";
import TruncatedLabel from "@/components/TruncatedLabel";
export interface VulnerabilityTopCve {
cveId: string;
@@ -136,7 +137,7 @@ export default function VulnerabilityExposurePanel({
)}
</div>
{c.sampleSoftware.length > 0 && (
<div className="text-[11px] text-gray-500 dark:text-gray-400 truncate mt-0.5">{c.sampleSoftware.join(", ")}</div>
<TruncatedLabel text={c.sampleSoftware.join(", ")} className="block text-[11px] text-gray-500 dark:text-gray-400 mt-0.5" />
)}
</div>
<div className="text-right flex-shrink-0">
@@ -2,6 +2,9 @@
"name": "autopilot-monitor-web",
"version": "1.0.0",
"private": true,
"engines": {
"node": "22.x"
},
"scripts": {
"dev": "next dev",
"prebuild": "node scripts/sync-error-codes.js && node scripts/generate-lastmod.js",