Compare commits

..

8 Commits

Author SHA1 Message Date
Oliver Kieselbach 540f543834 feat(agent): extend historic IME-replay suppression to app events + app aggregation
Part 2 of the eaf3d8c4 fix (follows 088c3b04). In that session 147 of 157 app
events were replayed from a previous enrollment's IME log — all 10 "installed
apps" were phantoms, polluting the timeline, AppInstallSummaries, MCP counts,
the app_tracking_summary header, culprit lists and final-status.json.

Rename (nothing deployed yet): historic_script_replay_detected →
historic_ime_replay_detected; SuppressIfHistoricReplay / IsHistoricImeReplay.

Agent:
- EmitAppState suppresses stale-replay transitions BEFORE the sub-phase
  declaration, timing bookkeeping and terminal-signal dedup — covers the
  informational event, the AppInstallCompleted/Failed DecisionSignal, the
  shadow download_progress and the summary snapshot. Behavior note: phantom
  terminal signals can no longer spuriously re-arm the AdvisoryCompletion
  window (app signals are otherwise observation-only — verified no completion
  gate reads them). EmitDoTelemetry gets the same guard.
- Tracker-level guard at the HandlePatternMatch choke point skips the 23
  app-mutating actions (incl. espphasedetected — a stale AccountSetup line
  would make fresh DeviceSetup lines bounce as "backward") for source lines
  >24h stale, so replayed apps never enter PackageStates, persistence,
  app_tracking_summary, culprit lists or final-status. New internal
  UtcNowProvider seam (no ctor change), threshold single-sourced as
  ImeLogTracker.HistoricReplayThreshold.
- SimulationMode now bypasses all replay suppression — 088c3b04 had broken
  the --replay-log-dir dev tool, which feeds historic lines on purpose.

Backend: IsHistoricImeReplay guard in AggregateAppInstallEvent keeps replayed
events out of AppInstallSummaries (SLA/dashboards/metrics inherit).

Web: generic lib/historicReplay.ts partition (scriptExecutions becomes a
wrapper); InstallProgress hides replayed installs with a muted note;
DownloadProgress filters silently.

MCP: get_session_summary filters replayed events — cleans appInstalls stats,
errorCount (replayed app_install_failed are Error-severity) and keyEvents.

Tests: +13 agent (app suppression, tracker guard), +1 backend, +7 web,
+6 MCP. Agent 2039, backend 3211, web 602, MCP 455 all green; drift test
validates the rename.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 23:43:59 +02:00
Oliver Kieselbach 088c3b0479 fix(agent): suppress historic IME-log script replay + timeline-consistent durations
Session eaf3d8c4: a previous enrollment's IME logs survived on disk (dated
2026-07-16, 7 days old) and the agent replayed them at startup — 156 phantom
script_completed events in a 7-minute session, each showing ~170h durations
(clock-clamped completion minus RAW stale start), plus an immediateUpload
flood (3x ingress_backpressure) and RemediationScriptCount inflated to 156.

Agent:
- Script events whose source line is >24h stale are suppressed entirely; a
  one-shot historic_script_replay_detected Info event (new EventType) dates
  the replayed window via earliestRejectedSourceTimestamp. Future-skew
  rejections and clock-only fallbacks still emit.
- durationSeconds now uses a timeline-consistent pair: on rejected-timestamp
  emissions the raw source completion pairs with the raw start (true runtime
  instead of cross-timeline garbage). 24h plausibility backstop omits
  implausible values; same consistency for script_timeout_suspected.
- Stale pending platform-script slots (>24h older than the current start
  line) are discarded so an old orphaned start can't pair with a fresh run
  (extracted HandlePlatformScriptStarted as internal test seam).

Web: partitionHistoricScriptEvents hides legacy-agent replay events (muted
"N historic executions hidden" note) and the reducer blanks >24h durations —
covers sessions recorded before the agent fix rolls out.

Backend: IsHistoricScriptReplay guard keeps replayed events from inflating
Platform-/RemediationScriptCount during the rollout window.

Registered historic_script_replay_detected in Constants.EventTypes, MCP
resource-catalog (script_events) + KEY_EVENT_TYPES, gather-rules eventTypes.

Tests: 8 agent (suppression/one-shot/raw-pair/backstop/slot-hardening),
8 web (partition/blanking), 10 backend (predicate). Agent 2026, backend
3210, web 595, MCP 449 all green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 18:39:11 +02:00
Oliver Kieselbach 3224774752 fix(test): EDR-safe straggler replay — no more renamed system executable
The straggler integration test copied powershell.exe under a synthetic
agent name and launched it — textbook MITRE T1036.003 ("Rename Legitimate
Utilities"). Defender for Endpoint flagged the run ("System executable
renamed and launched", initiating process testhost.net48.exe) and ISOLATED
the dev machine.

Replaced with function shadowing: a harness script defines Get-Process /
Stop-Process stub functions (PowerShell resolves functions before cmdlets)
and dot-sources the cleanup script. The stubs report a live agent-named
straggler on every probe and record the kill, pinning the identical
contract — poll by NAME until the deadline, force-kill by NAME, then
delete — with zero process spawns, zero PE copies, nothing EDR-visible.

The production routine never renames or copies executables; the flagged
technique existed only in this test. Lesson recorded in tasks/lessons.md.

22/22 cleanup tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 17:24:07 +02:00
Oliver Kieselbach e7ea431ccf feat(esp): recovery story for terminally reported ESP failures (session 4910a5a5)
DeviceSetup/Apps failed, the user pressed "Try again" ~23 min later, apps
recovered to 10/10 and DeviceSetup resolved - yet the advisory_completion
window un-defanged the stale failure and failed the live enrollment,
because the advisory variant was exempt from all re-arm/rebase guards.

Engine fix (DecisionCore):
- New facts EspAdvisoryFailureResolvedUtc + EspAdvisoryFailureCategory
  (state-borne; deadline payload would be lost on re-arm/rebase).
- TryResolveAdvisoryOnCategoryRecovery in both ProvisioningComplete
  handlers: category match sets the fact + emits
  esp_failure_advisory_resolved.
- Guards now (!hasAdvisoryAnchor || advisoryResolved): enforcement-progress
  re-arm, hello-never-observed promote and the reboot rebase apply again
  once the failure demonstrably un-happened; unresolved advisories keep
  the legacy un-defang exactly.
- Truthful terminal reason esp_recovered_without_completion_evidence
  (+ failureRecovered=true, LastFailureTrigger=DeadlineFired so the
  likely-stuck app promotion stays off).

Story events (agent tracker):
- esp_failure_retry_detected: failed subcategory leaves "failed" after the
  terminal fire - consistent with the user pressing "Try again".
- esp_failure_recovered: previously failed category reaches success.
- EspFailureDetectedEventArgs.LikelyCulpritApps (never-started first,
  cap 3) rides the EspTerminalFailure payload so esp_failure_advisory /
  enrollment_failed name the culprit app directly.

Surfacing: MCP resource-catalog, web timeline badges (Recovered /
User retry pills, culprit chip on advisory), gather-rule autocomplete,
ANALYZE-ESP-006 "ESP Failure Recovered After User Retry".

Tests: AdvisoryRecoveryTests (13, incl. the 527/611/635/671 field-ordinal
replay that no longer fails the session), EspFailureRecoveryStoryTests (5),
snapshot-builder guards 18 -> 20. All suites green (DecisionCore 493,
V2.Core 2018, Backend 3201, web vitest 589, MCP 449 incl. drift).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 17:03:44 +02:00
Oliver Kieselbach 9e24b92fbf feat(agent): ESP policy-provider stall detector + Autopilot registry guardrail
Customer field case: pre-provisioned devices with the ConfigMgr client as a
blocking app occasionally park the user ESP at "Apps (Identifying)" for days.
Root cause (verified against the EnrollmentStatusTracking CSP doc): only a
"ConfigMgr" provider is registered under Device\Setup\Apps\PolicyProviders
(no "Sidecar" = IME) and TrackingPoliciesCreated never becomes 1 - a wait the
CSP defines WITHOUT any timeout. Renaming the key to Sidecar unblocks
immediately. Microsoft documents that ESP providers are not aware of each
other and that pre-provisioning is unsupported with co-management.

New always-on esp_policy_provider_stalled tripwire, generalized to any
provider in both kinds (Setup/Apps TrackingPoliciesCreated != 1,
DevicePreparation InstallationState not in {2,3}):

- EspPolicyProviderProbe: fail-soft registry read of the
  EnrollmentStatusTracking mirror (device + user-SID scopes), defensive
  value parsing (missing/unparseable = incomplete), ScopedOverride test seam.
  Research and CSP contract quotes captured in the XML doc header.
- EspPolicyProviderStallDetector: pure EvaluateStallTransition on a 15-min
  continuous-incompleteness dwell (resets on completion/disappearance;
  absent key = normal early enrollment), one-shot per provider,
  StartupEventGate fingerprint over the sorted stalled key set for
  cross-restart dedup. Warning + ImmediateUpload via InformationalEventPost;
  purely observational, no decision-engine coupling, no config toggle,
  no ConfigVersion bump (disk_space_low precedent).
- EspPolicyProviderStallHost: 60-s wall-clock tick, deliberately outside the
  StallProbe idle gate (idle clock resets on session activity) and the
  idle-stopped periodic collectors.
- Registered in Constants.EventTypes, MCP resource-catalog (health_events),
  web gather-rule autocomplete (stall category).

Guardrail companion: SOFTWARE\Microsoft\Windows\Autopilot joins the
gather-rule registry allowlist ("Autopilot / OOBE / Setup") so tenants can
build custom rules on the subtree (e.g. not_exists on the Sidecar provider
key); generated web mirror regenerated via combine.js; positive + sibling-
spoof guard tests added.

Docs: OKF concept doc agent/esp-policy-provider-stall.md (+ index/log).

Tests: 22 new detector tests (pure transition matrix + emission surface incl.
the ConfigMgr-only customer scenario and gate-across-restart dedup); agent
suite 2013/2013, web vitest 589/589 + tsc clean, MCP 449 passed incl.
event-types-drift.

Both the detector and the embedded allowlist take effect with the next agent
release.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 15:20:47 +02:00
Oliver Kieselbach 45d7c03ccf fix(web): WhiteGlove per-block durations use sequence-canonical edges, not timestamp min/max
Customer report (session fcb44595): dashboard said 115 min, session detail
said 5h 2m for the same pre-provisioned enrollment. The detail's per-block
duration calc took min/max over ALL event timestamps inside each
sequence-block, which two real-world timestamp shapes break:

- an app_install_started re-emitted by the ImeLogTracker after the Part-2
  resume legitimately carries its historical Part-1 event time -> the User
  Enrollment block stretched back across the sealed pause (56m -> 3h18m)
- a do_telemetry with a +1h source timestamp (IME log line with a wrong
  timezone bias, just under the agent's 1h future-skew clamp) stretched
  Pre-Provisioning (58m -> 1h44m)

Sum: exactly the "5h 2m" the customer saw.

Block spans are now anchored on the first/last event BY SEQUENCE (sequence
is the canonical order), with Part 1 starting at session.startedAt for
backend DurationSeconds parity. Extracted as pure helper
computeWhiteGloveDurations with tests modeled on the real session; replay
against the customer's 880 events yields 1h 54m == backend 6872s.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 14:26:33 +02:00
Oliver Kieselbach 90012d9a66 feat(agent): verify Secure Boot CA-2023 state against firmware, not just servicing registry
Customer report: ANALYZE-SEC-001 warned on devices whose firmware already
carried the Windows UEFI CA 2023 certificate — the agent only read the
SecureBoot\Servicing registry value, which mirrors the Windows Update
rollout and lags behind reality (OEM factory image, WinCS, WU not run yet).

Agent: new UefiSecureBootCertReader reads the UEFI db/KEK variables
natively (GetFirmwareEnvironmentVariableExW; SeSystemEnvironmentPrivilege
enabled on the process token and restored — first privilege-adjustment
code in the repo). Fail-soft, never throws; legacy BIOS maps to not_uefi.
secureboot_status gains per-certificate booleans, uefiFirmwareReadStatus,
and the one-sided marker uefiCa2023FirmwareConfirmed (emitted only on
positive confirmation, never as false).

Rule v3.0.0: not_exists precondition on the marker suppresses the rule
when the firmware confirms the certificate; the marker's absence keeps
old agents on exact v2 behavior, so the rule is safe to seed before the
agent rollout. Firmware-verified-missing raises confidence (+20 db,
+10 KEK 2K CA 2023).

Verified: full solution build; backend 3201, agent 1988, web 582 tests
green; tsc/ajv clean; elevated cross-check on real firmware matches
Get-SecureBootUEFI on all seven certificate probes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 14:07:01 +02:00
Oliver Kieselbach e8ef4f53ae feat(agent): OOBE-state observability via WinRT SystemSetupInfo
Sample Windows.System.Profile.SystemSetupInfo.OutOfBoxExperienceState
(build >= 17763, SYSTEM-callable, CLR-native WinRT projection — no NuGet)
as a purely observational signal, mirroring the bootstrap guard marker:

- oobeStateAtAgentStart on agent_started and agent_late_start payloads:
  makes late starts provable instead of heuristic ("completed" on a
  first_run = OOBE was already over when the agent arrived) and validates
  the bootstrap OOBE guard fleet-wide.
- One-shot oobe_state_completed trace event on the in_progress->completed
  flip, piggybacked on the DesktopArrivalDetector 30s poll: an
  owner-independent desktop corroboration for DAD-starved sessions
  (4d5a0b78 class — WTS+WMI owner resolution dead, desktop_arrived never
  fires). Often absent in healthy sessions since the DAD timer is
  disposed at desktop arrival, which is when the flip lands.

OobeStateReader is fail-soft (never throws, "unavailable" on missing
contract); the flip check has its own try/catch so it cannot pollute the
DAD wmiErrorCount liveness diagnostics, "unavailable" never overwrites
the baseline, and the one-shot flag survives ResetForRealUserSwitch.
No decision-engine involvement, no config/backend changes. Event type
registered in Constants.EventTypes, MCP resource-catalog (phase_events)
and web gather-rule autocomplete.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:39:10 +02:00
68 changed files with 5267 additions and 192 deletions
+125
View File
@@ -0,0 +1,125 @@
---
type: Concept
title: ESP Policy-Provider Stall Detection — the no-timeout Setup/Apps wait and the co-management Sidecar gap
description: Why a registered EnrollmentStatusTracking CSP policy provider that never sets TrackingPoliciesCreated parks the user ESP at "Apps (Identifying)" indefinitely, the pre-provisioning + ConfigMgr field case, and how the always-on esp_policy_provider_stalled tripwire (15-min wall-clock dwell, StartupEventGate cross-restart dedup) surfaces it.
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/EspPolicyProviderProbe.cs
tags:
- agent
- esp
- enrollment-status-tracking
- co-management
- stall-detection
timestamp: 2026-07-23T00:00:00+02:00
---
# ESP Policy-Provider Stall Detection
## Schema
The Enrollment Status Page decides when its "Apps" category may progress from
"Identifying" using the **EnrollmentStatusTracking CSP**
(<https://learn.microsoft.com/en-us/windows/client-management/mdm/enrollmentstatustracking-csp>),
mirrored in the registry under
`HKLM\SOFTWARE\Microsoft\Windows\Autopilot\EnrollmentStatusTracking\{Device|<UserSID>}\…`.
Two contracts matter:
1. **`Setup/Apps/PolicyProviders/<Name>`** — per the CSP doc: *"Existence of this node
indicates to the ESP that it shouldn't show the tracking status message until the
TrackingPoliciesCreated node has been set to true."* This wait has **no timeout**. A
registered provider that never sets `TrackingPoliciesCreated=1` parks the ESP at
"Apps (Identifying)" indefinitely — observed for days in the field. Exists in device
scope (`Device`) and user scope (`S-<SID>`, appears at sign-in).
2. **`DevicePreparation/PolicyProviders/<Name>/InstallationState`** — must reach 2
(NotRequired) or 3 (Completed); 1=NotInstalled, 4=Error. Device scope only. The ESP
applies a default 15-minute timeout here, so these stalls self-surface as ESP errors —
the detector still records them for the diagnostic trail.
Known provider names: **`Sidecar` is the Intune Management Extension** (the CSP doc names
"Intune's agents, such as SideCar"); **`ConfigMgr`** is the Configuration Manager client,
registered during Autopilot-into-co-management — the troubleshooting section of
<https://learn.microsoft.com/en-us/intune/configmgr/comanage/autopilot-enrollment> queries
exactly these registry paths. The same document states that ESP policy providers **are not
aware of each other** and that **pre-provisioning is unsupported** with Autopilot into
co-management.
## The field case
A tenant pre-provisions devices and installs the ConfigMgr client as a blocking app.
Occasionally the user-phase ESP reaches "Apps (Identifying)" and never leaves it — for
hours to days. On those devices `Device\Setup\Apps\PolicyProviders` contains **only a
`ConfigMgr` key (no `Sidecar`)** and `TrackingPoliciesCreated` is never set. Manually
renaming the key to `Sidecar` unblocks the ESP immediately (the IME then finds its
provider node and reports tracking-policies-created). The mechanism is exactly the
documented no-timeout wait; the unsupported pre-provisioning + co-management combination
explains why only some devices are affected. Why the Sidecar registration is missing or
displaced on those devices is not publicly documented — irrelevant for detection, since
the signature is deterministic.
## Detection
`EspPolicyProviderProbe` (fail-soft, static, `ScopedOverride` test seam — structural clone
of `EspTrackingInfoProbe`) enumerates all providers in both kinds and scopes.
`EspPolicyProviderStallDetector` evaluates a pure transition function on a 60-second tick
from the always-on `EspPolicyProviderStallHost`:
* A provider **continuously incomplete for ≥ 15 minutes** (wall-clock) is stalled. The
dwell clock starts at first incomplete observation and resets on completion or
disappearance. Missing values parse as incomplete (fail-toward-observation).
* Missing root key / empty provider list = normal early enrollment: clears all dwell
state, never fires.
* One `esp_policy_provider_stalled` **Warning** per newly stalled provider set
(`ImmediateUpload`), one-shot per provider in-process, cross-restart deduped via a
`StartupEventGate` fingerprint over the sorted currently-stalled key set — a restart
into the same stall stays silent, a new provider joining the stall re-reports.
* Payload: `providers[]` (full table: name, scope, kind, raw values, complete flag),
`stalledProviders[]` with `stalledForMinutes`, `sidecarRegistered`, `dwellMinutes`.
* **Purely observational**: emitted via `InformationalEventPost` (dispatch-guard exempt
pass-through) — never touches decision state. No config toggle, no ConfigVersion bump
(precedent: `disk_space_low`).
Deliberately **not** built on `StallProbeCollector` (its idle clock resets on any session
activity — a provider stall coexists with a chatty session) and not on the periodic
collectors (idle-stopped after 15 minutes — exactly the dormant window the dwell must
observe).
## Guardrail companion
`rules/guardrails.json` allow-lists `SOFTWARE\Microsoft\Windows\Autopilot` (category
"Autopilot / OOBE / Setup") so tenants can build their own gather rules on the subtree —
e.g. a `not_exists` registry rule on the `Sidecar` provider key. The subtree carries
deployment state only (provider registrations, tracking lists, app IDs, per-user SIDs);
sensitivity is at or below existing entries. Enforcement is agent-side (embedded
resource), so both the detector and the guardrail change ship with the agent release.
## Examples
Customer-case event (abbreviated):
```json
{
"eventType": "esp_policy_provider_stalled",
"severity": "Warning",
"data": {
"dwellMinutes": 15,
"sidecarRegistered": false,
"providers": [
{ "name": "ConfigMgr", "scope": "device", "kind": "setupApps",
"trackingPoliciesCreated": null, "installationState": null, "complete": false }
],
"stalledProviders": [
{ "name": "ConfigMgr", "scope": "device", "kind": "setupApps", "stalledForMinutes": 15.0 }
]
}
}
```
`sidecarRegistered=false` while `setupApps` providers exist is the co-management
signature; `sidecarRegistered=true` with a stalled third-party provider generalizes the
same contract violation.
# Citations
* EnrollmentStatusTracking CSP — <https://learn.microsoft.com/en-us/windows/client-management/mdm/enrollmentstatustracking-csp>
* Windows Autopilot into co-management (requirements, provider-conflict warning, troubleshooting registry paths) — <https://learn.microsoft.com/en-us/intune/configmgr/comanage/autopilot-enrollment>
* Agent code: `EspPolicyProviderProbe.cs`, `EspPolicyProviderStallDetector.cs`, `EspPolicyProviderStallHost.cs` (SystemSignals / Orchestration.Hosts)
* Related: [MDM Reboot Coalescing](mdm-reboot-coalescing.md), [Gather-Rule Guardrails](../rules/gather-rule-guardrails.md)
+85
View File
@@ -0,0 +1,85 @@
---
type: Concept
title: Secure Boot CA-2023 Detection — firmware evidence vs. Windows servicing status
description: Why ANALYZE-SEC-001 v3 reads the UEFI db/KEK variables directly (GetFirmwareEnvironmentVariableExW + SeSystemEnvironmentPrivilege) instead of trusting the SecureBoot\Servicing registry key, and how the one-sided uefiCa2023FirmwareConfirmed marker plus a not_exists precondition give absence-tolerant rule suppression across old and new agents.
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Interop/UefiSecureBootCertReader.cs
tags:
- agent
- secureboot
- ca2023
- analyze-rules
- backward-compatibility
timestamp: 2026-07-23T00:00:00+02:00
---
# Secure Boot CA-2023 Detection
## Schema
Two independent evidence sources describe the Windows UEFI CA 2023 rollout state:
1. **Servicing registry** (`HKLM\SYSTEM\CurrentControlSet\Control\SecureBoot\Servicing\UEFICA2023Status`)
— reflects whether the *Windows Update* rollout ran and self-verified. Lags reality:
devices whose firmware received the certificate another way (OEM factory image, WinCS
CLI, manual deployment) or where WU simply has not run yet show `unknown`/`notfound`.
2. **Firmware itself** — the UEFI variables `db` (signature database, GUID
`{d719b2cb-3d3a-4596-a3bc-dad00e67656f}`) and `KEK` (GUID
`{8be4df61-93ca-11d2-aa0d-00e098032b8c}`). Authoritative. Certificate subject CNs are
ASCII bytes inside the DER signature lists, so a byte-substring search for
`Windows UEFI CA 2023` / `Microsoft Corporation KEK 2K CA 2023` decides presence —
the same check `Get-SecureBootUEFI` enables from PowerShell.
`UefiSecureBootCertReader` (agent, `Monitoring/Interop`) reads both variables natively via
`GetFirmwareEnvironmentVariableExW` — no PowerShell runspace during OOBE. The call needs
`SeSystemEnvironmentPrivilege`: SYSTEM holds it default-disabled, so the reader enables it
on the process token (`AdjustTokenPrivileges`) and restores the previous state afterwards.
Fail-soft contract like `OobeStateReader`: never throws; failures map to status strings
(`not_uefi` = legacy BIOS via `ERROR_INVALID_FUNCTION`, `variable_not_found`,
`privilege_denied`, `error_<code>`).
Fields land on the existing `secureboot_status` event (once per agent start,
restart-deduped by the StartupEventGate): `uefiFirmwareReadStatus`, per-cert booleans
(`uefiDbHasWindowsUefiCa2023`, `uefiDbHasWindowsProductionPca2011`,
`uefiKekHasMicrosoftKek2kCa2023`, …) — emitted only when the respective read succeeded —
plus the marker below.
## The one-sided marker (absence-tolerant suppression)
Rule-engine constraints (pinned by `RuleEnginePreconditionTests`):
* **Conditions** never match on an absent field (null guard) — new fields must stay
`required: false` or the rule dies for every old agent.
* **Preconditions**: `not_exists` passes while the field is absent/empty and fails as soon
as the field is present with *any* value; comparison operators fail closed on absence.
A two-sided boolean therefore cannot drive suppression: `not_exists` on
`uefiDbHasWindowsUefiCa2023` would also suppress the rule when the field is present with
`false`. Hence the **one-sided marker** `uefiCa2023FirmwareConfirmed = true`, emitted
ONLY when the db read succeeded AND the 2023 certificate was found — never as `false`.
ANALYZE-SEC-001 v3 adds a `not_exists` precondition on it:
| Device state | Marker | Rule |
|---|---|---|
| Old agent (no firmware fields) | absent | evaluates exactly like v2 |
| New agent, cert missing or firmware unreadable | absent | evaluates; optional conditions `uefiDbHasWindowsUefiCa2023=false` (+20) / `uefiKekHasMicrosoftKek2kCa2023=false` (+10) raise confidence |
| New agent, firmware confirms cert | present | silently suppressed (the customer-reported false positive) |
Because the marker exists nowhere until the new agent ships, the v3 rule is safe to seed
**before** the agent rollout.
## Examples
Customer-equivalent verification from PowerShell (authoritative, matches the reader):
```powershell
[Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI -Name db).Bytes).Contains('Windows UEFI CA 2023')
[Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI -Name KEK).Bytes).Contains('Microsoft Corporation KEK 2K CA 2023')
```
## Citations
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Interop/UefiSecureBootCertReader.cs` — native reader
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Telemetry/DeviceInfo/DeviceInfoCollector.NetworkAndSecurity.cs` — `CollectSecureBootStatus` / `AppendFirmwareCertFields`
* `rules/analyze/ANALYZE-SEC-001.json` — v3.0.0 rule definition
* `src/Backend/AutopilotMonitor.Functions.Tests/RuleEnginePreconditionTests.cs` — pinned precondition semantics + v3 mirror tests
* Microsoft: [Act now — Secure Boot certificates expire June 2026](https://techcommunity.microsoft.com/blog/windows-itpro-blog/act-now-secure-boot-certificates-expire-in-june-2026/4426856)
+2
View File
@@ -21,6 +21,8 @@ https://docs.autopilotmonitor.com (separate repository).
* [Build Counter Blob](agent/build-counter-blob.md) - How agent build numbers are minted: shared counter blob with ETag-CAS so local and CI builds never collide; CI additionally attests build provenance (Sigstore).
* [Agent Endpoint Migration](agent/endpoint-migration.md) - How the backend re-homes agents to a new API base URL via the config channel (backend move or per-tenant region move): live-fetch-only, allowlist-validated on both sides, one hop, kill wins.
* [MDM Reboot Coalescing](agent/mdm-reboot-coalescing.md) - Attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: aggregated DM-Enterprise 2800 observations (neutral, flush-time watermark) + the ANALYZE-ESP-005 advisory gated on an actually-observed reboot.
* [Secure Boot CA-2023 Detection](agent/secureboot-ca2023-detection.md) - Why ANALYZE-SEC-001 v3 reads the UEFI db/KEK variables directly instead of trusting the SecureBoot\Servicing registry, and how the one-sided uefiCa2023FirmwareConfirmed marker plus a not_exists precondition give absence-tolerant suppression across old and new agents.
* [ESP Policy-Provider Stall Detection](agent/esp-policy-provider-stall.md) - The EnrollmentStatusTracking CSP's no-timeout Setup/Apps wait: a registered provider that never sets TrackingPoliciesCreated (co-management "ConfigMgr" without the IME's "Sidecar") parks the user ESP at "Apps (Identifying)" for days; the always-on esp_policy_provider_stalled tripwire and the SOFTWARE\Microsoft\Windows\Autopilot gather-rule allowlist entry that surface it.
# Rules
+4
View File
@@ -2,6 +2,10 @@
## 2026-07-23
* **Creation**: Added `agent/esp-policy-provider-stall.md` — a customer diagnosed that pre-provisioned devices with the ConfigMgr client as a blocking app occasionally park the user ESP at "Apps (Identifying)" for days: only a `ConfigMgr` provider is registered under `EnrollmentStatusTracking\Device\Setup\Apps\PolicyProviders` (no `Sidecar` = IME) and `TrackingPoliciesCreated` never becomes 1 — a wait the CSP defines without any timeout; renaming the key to `Sidecar` unblocks immediately. The agent now ships the always-on `esp_policy_provider_stalled` tripwire (60-s tick, 15-min continuous-incompleteness dwell, one-shot per provider, `StartupEventGate` fingerprint across restarts, purely observational via the InformationalEvent rail — no toggle, no ConfigVersion bump), generalized to any provider in both `Setup/Apps` and `DevicePreparation` kinds rather than hardcoding the ConfigMgr case. Companion change: `SOFTWARE\Microsoft\Windows\Autopilot` joins the gather-rule registry allowlist so tenants can target the subtree with custom rules (e.g. `not_exists` on the Sidecar key).
* **Creation**: Added `agent/secureboot-ca2023-detection.md` — ANALYZE-SEC-001 warned on devices whose firmware already carried the 2023 certificate, because the agent only read the `SecureBoot\Servicing\UEFICA2023Status` registry value, which mirrors the Windows Update rollout rather than the firmware (customer report). The agent now reads the UEFI `db`/`KEK` variables natively (`GetFirmwareEnvironmentVariableExW` with `SeSystemEnvironmentPrivilege` enabled on the process token — first privilege-adjustment code in the repo) and emits per-certificate booleans on `secureboot_status`. Suppression rides on a one-sided marker, `uefiCa2023FirmwareConfirmed`, emitted only on positive confirmation: the engine's `not_exists` precondition fails on any present value, so a two-sided boolean could not distinguish "confirmed" from "checked and missing", while the marker's absence keeps old agents and firmware-missing devices on the v2 path. Rule v3.0.0 is therefore safe to seed before the agent release.
* **Update**: `url-registry.md` — the first release after the alias migration surfaced Front Door caching as a correctness hazard: the alias served the previous `version.json` (`X-Cache: TCP_REMOTE_HIT`) after the origin had rotated, and since manifest and ZIP are independent cache entries, a stale manifest with a fresh ZIP fails the bootstrap SHA-256 check. Route caching on `/agent/*` is now disabled and every blob upload stamps `Cache-Control: no-cache` (CI `Send-Blob`, local `build.ps1`, `versions/*.json` manifests) — all agent-container blobs rotate in place, so nothing there is safely cacheable.
* **Update**: `agent/decision-engine.md` — the RealmJoin completion gate has a third terminal, `FirstDeploymentIncomplete`. Session 224b2087 showed why: an interactive logon 17 s after DeploymentPhase 101 makes RealmJoin reclassify the run as a secondary-user deployment, whose branch writes 200/210 and never 110 — the gate stayed closed until the 60-min hard timeout although Hello and desktop were long resolved. `realmjoin_phase_changed` is now a typed `RealmJoinPhaseChanged` signal, so `RealmJoinFacts.LastDeploymentPhase` is persisted across agent restarts (this also fixes every `realmjoin_timeout` event claiming "last phase: 0"); a transition out of 100/101 into 200/210 releases the gate with the `realmjoin_first_deployment_incomplete` Warning, and the `RealmJoinDetected` replay path evaluates the same rule for restarts that swallow the transition. A first observation of 210 deliberately does not release — session 6f1959c0 started at 210 and only deployed afterwards.
+90
View File
@@ -0,0 +1,90 @@
{
"$schema": "../schema/analyze-rule.schema.json",
"ruleId": "ANALYZE-ESP-006",
"title": "ESP Failure Recovered After User Retry",
"description": "Detects the recovery story: the ESP reported a terminal failure (typically DeviceSetup/Apps), the user most likely pressed 'Try again' on the failure page, the failed step re-ran and completed successfully. The earlier failure did not stick — the enrollment continued past it.",
"severity": "info",
"category": "esp",
"version": "1.0.0",
"enabled": true,
"baseConfidence": 85,
"conditions": [
{
"signal": "advisory",
"source": "event_type",
"eventType": "esp_failure_advisory",
"operator": "exists",
"value": "",
"required": true
},
{
"signal": "recovered",
"source": "event_type",
"eventType": "esp_failure_recovered",
"operator": "exists",
"value": "",
"required": true
},
{
"signal": "retry",
"source": "event_type",
"eventType": "esp_failure_retry_detected",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "category",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "category",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "failedSubcategory",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "failedSubcategory",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "likelyCulpritApps",
"source": "event_type",
"eventType": "esp_failure_advisory",
"dataField": "likelyCulpritApps",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "minutesSinceFailure",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "minutesSinceFailure",
"operator": "exists",
"value": "",
"required": false
}
],
"confidenceFactors": [
{ "signal": "retry", "condition": "exists", "weight": 10 },
{ "signal": "likelyCulpritApps", "condition": "exists", "weight": 5 }
],
"explanation": "The Enrollment Status Page reported a **failure** in the `{{category}}` phase (`{{failedSubcategory}}` subcategory, most likely app(s): `{{likelyCulpritApps}}`) — and then **recovered**: the failed step left its failed state and completed successfully about `{{minutesSinceFailure}}` min later.\n\nThat pattern is consistent with the user pressing **'Try again'** on the ESP failure page (the profile allows failure-page actions; the agent cannot observe the click itself). The failure the user saw was real, but it did not stick — the ESP re-ran the step, it succeeded, and the enrollment continued.\n\nRead the session timeline as one arc: `esp_failure_advisory` (the failure, with the likely culprit app) → `esp_failure_retry_detected` (the retry) → `esp_failure_recovered` (the recovery). If this pattern repeats across devices for the same app, that app's install is flaky on first attempt — fix the app rather than the devices. Any literal `{{…}}` placeholder means that detail was not recorded for this session.",
"remediation": [
{
"title": "Make the first install attempt stick",
"steps": [
"Check the likely culprit app(s) (`{{likelyCulpritApps}}`) in Intune: an app that fails on the first ESP pass but succeeds on retry usually has a dependency/ordering or detection-timing problem.",
"If the app shows 'never started installing' in esp_failure_settle_started, verify its assignment and applicability rules — ESP counted it as required, but IME never processed it.",
"If the pattern repeats across the tenant, consider removing the app from the ESP blocking list until the first-attempt install is reliable — users are seeing an avoidable failure screen.",
"No device-side action is needed for this session: the failure recovered and the enrollment continued."
]
}
],
"tags": ["esp", "recovery", "try-again", "retry", "continue-anyway", "enrollment"]
}
+33 -5
View File
@@ -2,10 +2,10 @@
"$schema": "../schema/analyze-rule.schema.json",
"ruleId": "ANALYZE-SEC-001",
"title": "Secure Boot UEFI CA 2023 certificate not deployed",
"description": "Detects devices where the Windows UEFI CA 2023 certificate has not been deployed to the Secure Boot DB. The 2011 certificates expired in June 2026 — affected devices no longer receive Secure Boot security updates until the 2023 certificate is deployed.",
"description": "Detects devices where the Windows UEFI CA 2023 certificate has not been deployed to the Secure Boot DB. The 2011 certificates expired in June 2026 — affected devices no longer receive Secure Boot security updates until the 2023 certificate is deployed. Since v3 the agent reads the firmware db/KEK variables directly: when the firmware itself confirms the 2023 certificate, the rule is suppressed even if the Windows servicing status has not caught up yet.",
"severity": "warning",
"category": "security",
"version": "2.0.0",
"version": "3.0.0",
"enabled": true,
"baseConfidence": 70,
"preconditions": [
@@ -16,6 +16,14 @@
"operator": "equals",
"value": "false",
"description": "Skip on virtual machines — Secure Boot DB is host-controlled (Hyper-V/VMware/etc), the inside-the-VM UEFICA2023Status registry value is not the right indicator and not actionable for the VM admin."
},
{
"source": "event_data",
"eventType": "secureboot_status",
"dataField": "uefiCa2023FirmwareConfirmed",
"operator": "not_exists",
"value": "",
"description": "Skip when the firmware db itself confirms the Windows UEFI CA 2023 certificate — the marker is only emitted on positive firmware confirmation, so old agents (field absent) and unconfirmed devices keep evaluating the rule. The servicing registry status lags behind actual firmware state (OEM factory image, WinCS, manual deployment, or WU simply not run yet)."
}
],
"conditions": [
@@ -54,13 +62,33 @@
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "db_ca2023_firmware_missing",
"source": "event_data",
"eventType": "secureboot_status",
"dataField": "uefiDbHasWindowsUefiCa2023",
"operator": "equals",
"value": "false",
"required": false
},
{
"signal": "kek_2023_missing",
"source": "event_data",
"eventType": "secureboot_status",
"dataField": "uefiKekHasMicrosoftKek2kCa2023",
"operator": "equals",
"value": "false",
"required": false
}
],
"confidenceFactors": [
{ "signal": "ca2023_notfound", "condition": "exists", "weight": 20 },
{ "signal": "ca2023_error", "condition": "exists", "weight": 10 }
{ "signal": "ca2023_error", "condition": "exists", "weight": 10 },
{ "signal": "db_ca2023_firmware_missing", "condition": "exists", "weight": 20 },
{ "signal": "kek_2023_missing", "condition": "exists", "weight": 10 }
],
"explanation": "This device has Secure Boot enabled but the **Windows UEFI CA 2023** certificate has not been deployed to the Secure Boot DB.\n\nThe original Windows UEFI CA 2011 certificates **expired on June 27, 2026 — this deadline has passed**. The automatic pre-expiry rollout window is over: this device no longer receives Secure Boot security updates (DB/DBX) and cannot trust 2023-signed boot managers until the 2023 certificate is deployed. Deploying it now requires active remediation (Windows servicing opt-in and, on some models, an OEM firmware update) rather than waiting for an automatic update.\n\nCurrent UEFI CA 2023 status: `{{uefiCA2023Status}}`",
"explanation": "This device has Secure Boot enabled but the **Windows UEFI CA 2023** certificate has not been deployed to the Secure Boot DB.\n\nThe original Windows UEFI CA 2011 certificates **expired on June 27, 2026 — this deadline has passed**. The automatic pre-expiry rollout window is over: this device no longer receives Secure Boot security updates (DB/DBX) and cannot trust 2023-signed boot managers until the 2023 certificate is deployed. Deploying it now requires active remediation (Windows servicing opt-in and, on some models, an OEM firmware update) rather than waiting for an automatic update.\n\nOn devices running a current agent this finding is verified against the firmware itself: the agent reads the UEFI `db` signature database directly, and this warning is automatically suppressed when the firmware already contains the 2023 certificate — regardless of the Windows servicing status below.\n\nCurrent UEFI CA 2023 servicing status: `{{uefiCA2023Status}}`",
"remediation": [
{
"title": "Deploy UEFI CA 2023 certificate now",
@@ -69,7 +97,7 @@
"Use Intune policy to enable Secure Boot DB update (Windows UEFI CA 2023)",
"Check with the device OEM whether a firmware update is required for the DB update to apply on this model",
"Alternatively, use the WinCS CLI tool from Microsoft for manual deployment",
"Verify deployment: check registry HKLM\\SYSTEM\\CurrentControlSet\\Control\\SecureBoot\\Servicing\\UEFICA2023Status = 'Updated'",
"Verify deployment against the firmware directly: [Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI -Name db).Bytes).Contains('Windows UEFI CA 2023') — this is authoritative; the registry value HKLM\\SYSTEM\\CurrentControlSet\\Control\\SecureBoot\\Servicing\\UEFICA2023Status = 'Updated' only reflects the Windows Update rollout",
"After DB update: ensure the boot manager is updated to a 2023-signed version via monthly cumulative update"
]
}
+144 -4
View File
@@ -1884,6 +1884,110 @@
"user-experience"
]
},
{
"ruleId": "ANALYZE-ESP-006",
"title": "ESP Failure Recovered After User Retry",
"description": "Detects the recovery story: the ESP reported a terminal failure (typically DeviceSetup/Apps), the user most likely pressed 'Try again' on the failure page, the failed step re-ran and completed successfully. The earlier failure did not stick — the enrollment continued past it.",
"severity": "info",
"category": "esp",
"version": "1.0.0",
"enabled": true,
"baseConfidence": 85,
"conditions": [
{
"signal": "advisory",
"source": "event_type",
"eventType": "esp_failure_advisory",
"operator": "exists",
"value": "",
"required": true
},
{
"signal": "recovered",
"source": "event_type",
"eventType": "esp_failure_recovered",
"operator": "exists",
"value": "",
"required": true
},
{
"signal": "retry",
"source": "event_type",
"eventType": "esp_failure_retry_detected",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "category",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "category",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "failedSubcategory",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "failedSubcategory",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "likelyCulpritApps",
"source": "event_type",
"eventType": "esp_failure_advisory",
"dataField": "likelyCulpritApps",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "minutesSinceFailure",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "minutesSinceFailure",
"operator": "exists",
"value": "",
"required": false
}
],
"confidenceFactors": [
{
"signal": "retry",
"condition": "exists",
"weight": 10
},
{
"signal": "likelyCulpritApps",
"condition": "exists",
"weight": 5
}
],
"explanation": "The Enrollment Status Page reported a **failure** in the `{{category}}` phase (`{{failedSubcategory}}` subcategory, most likely app(s): `{{likelyCulpritApps}}`) — and then **recovered**: the failed step left its failed state and completed successfully about `{{minutesSinceFailure}}` min later.\n\nThat pattern is consistent with the user pressing **'Try again'** on the ESP failure page (the profile allows failure-page actions; the agent cannot observe the click itself). The failure the user saw was real, but it did not stick — the ESP re-ran the step, it succeeded, and the enrollment continued.\n\nRead the session timeline as one arc: `esp_failure_advisory` (the failure, with the likely culprit app) → `esp_failure_retry_detected` (the retry) → `esp_failure_recovered` (the recovery). If this pattern repeats across devices for the same app, that app's install is flaky on first attempt — fix the app rather than the devices. Any literal `{{…}}` placeholder means that detail was not recorded for this session.",
"remediation": [
{
"title": "Make the first install attempt stick",
"steps": [
"Check the likely culprit app(s) (`{{likelyCulpritApps}}`) in Intune: an app that fails on the first ESP pass but succeeds on retry usually has a dependency/ordering or detection-timing problem.",
"If the app shows 'never started installing' in esp_failure_settle_started, verify its assignment and applicability rules — ESP counted it as required, but IME never processed it.",
"If the pattern repeats across the tenant, consider removing the app from the ESP blocking list until the first-attempt install is reliable — users are seeing an avoidable failure screen.",
"No device-side action is needed for this session: the failure recovered and the enrollment continued."
]
}
],
"tags": [
"esp",
"recovery",
"try-again",
"retry",
"continue-anyway",
"enrollment"
]
},
{
"ruleId": "ANALYZE-ID-001",
"title": "Expected Machine Certificate Not Found",
@@ -2178,10 +2282,10 @@
{
"ruleId": "ANALYZE-SEC-001",
"title": "Secure Boot UEFI CA 2023 certificate not deployed",
"description": "Detects devices where the Windows UEFI CA 2023 certificate has not been deployed to the Secure Boot DB. The 2011 certificates expired in June 2026 — affected devices no longer receive Secure Boot security updates until the 2023 certificate is deployed.",
"description": "Detects devices where the Windows UEFI CA 2023 certificate has not been deployed to the Secure Boot DB. The 2011 certificates expired in June 2026 — affected devices no longer receive Secure Boot security updates until the 2023 certificate is deployed. Since v3 the agent reads the firmware db/KEK variables directly: when the firmware itself confirms the 2023 certificate, the rule is suppressed even if the Windows servicing status has not caught up yet.",
"severity": "warning",
"category": "security",
"version": "2.0.0",
"version": "3.0.0",
"enabled": true,
"baseConfidence": 70,
"preconditions": [
@@ -2192,6 +2296,14 @@
"operator": "equals",
"value": "false",
"description": "Skip on virtual machines — Secure Boot DB is host-controlled (Hyper-V/VMware/etc), the inside-the-VM UEFICA2023Status registry value is not the right indicator and not actionable for the VM admin."
},
{
"source": "event_data",
"eventType": "secureboot_status",
"dataField": "uefiCa2023FirmwareConfirmed",
"operator": "not_exists",
"value": "",
"description": "Skip when the firmware db itself confirms the Windows UEFI CA 2023 certificate — the marker is only emitted on positive firmware confirmation, so old agents (field absent) and unconfirmed devices keep evaluating the rule. The servicing registry status lags behind actual firmware state (OEM factory image, WinCS, manual deployment, or WU simply not run yet)."
}
],
"conditions": [
@@ -2230,6 +2342,24 @@
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "db_ca2023_firmware_missing",
"source": "event_data",
"eventType": "secureboot_status",
"dataField": "uefiDbHasWindowsUefiCa2023",
"operator": "equals",
"value": "false",
"required": false
},
{
"signal": "kek_2023_missing",
"source": "event_data",
"eventType": "secureboot_status",
"dataField": "uefiKekHasMicrosoftKek2kCa2023",
"operator": "equals",
"value": "false",
"required": false
}
],
"confidenceFactors": [
@@ -2242,9 +2372,19 @@
"signal": "ca2023_error",
"condition": "exists",
"weight": 10
},
{
"signal": "db_ca2023_firmware_missing",
"condition": "exists",
"weight": 20
},
{
"signal": "kek_2023_missing",
"condition": "exists",
"weight": 10
}
],
"explanation": "This device has Secure Boot enabled but the **Windows UEFI CA 2023** certificate has not been deployed to the Secure Boot DB.\n\nThe original Windows UEFI CA 2011 certificates **expired on June 27, 2026 — this deadline has passed**. The automatic pre-expiry rollout window is over: this device no longer receives Secure Boot security updates (DB/DBX) and cannot trust 2023-signed boot managers until the 2023 certificate is deployed. Deploying it now requires active remediation (Windows servicing opt-in and, on some models, an OEM firmware update) rather than waiting for an automatic update.\n\nCurrent UEFI CA 2023 status: `{{uefiCA2023Status}}`",
"explanation": "This device has Secure Boot enabled but the **Windows UEFI CA 2023** certificate has not been deployed to the Secure Boot DB.\n\nThe original Windows UEFI CA 2011 certificates **expired on June 27, 2026 — this deadline has passed**. The automatic pre-expiry rollout window is over: this device no longer receives Secure Boot security updates (DB/DBX) and cannot trust 2023-signed boot managers until the 2023 certificate is deployed. Deploying it now requires active remediation (Windows servicing opt-in and, on some models, an OEM firmware update) rather than waiting for an automatic update.\n\nOn devices running a current agent this finding is verified against the firmware itself: the agent reads the UEFI `db` signature database directly, and this warning is automatically suppressed when the firmware already contains the 2023 certificate — regardless of the Windows servicing status below.\n\nCurrent UEFI CA 2023 servicing status: `{{uefiCA2023Status}}`",
"remediation": [
{
"title": "Deploy UEFI CA 2023 certificate now",
@@ -2253,7 +2393,7 @@
"Use Intune policy to enable Secure Boot DB update (Windows UEFI CA 2023)",
"Check with the device OEM whether a firmware update is required for the DB update to apply on this model",
"Alternatively, use the WinCS CLI tool from Microsoft for manual deployment",
"Verify deployment: check registry HKLM\\SYSTEM\\CurrentControlSet\\Control\\SecureBoot\\Servicing\\UEFICA2023Status = 'Updated'",
"Verify deployment against the firmware directly: [Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI -Name db).Bytes).Contains('Windows UEFI CA 2023') — this is authoritative; the registry value HKLM\\SYSTEM\\CurrentControlSet\\Control\\SecureBoot\\Servicing\\UEFICA2023Status = 'Updated' only reflects the Windows Update rollout",
"After DB update: ensure the boot manager is updated to a 2023-signed version via monthly cumulative update"
]
}
+2 -1
View File
@@ -43,7 +43,8 @@
"prefixes": [
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Setup",
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\OOBE",
"SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Winlogon"
"SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Winlogon",
"SOFTWARE\\Microsoft\\Windows\\Autopilot"
]
},
{
@@ -39,6 +39,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Harness
logFolder: Tmp.Path,
patterns: new List<ImeLogPattern>(),
logger: Logger);
// Mirror ImeLogHost wiring: the historic-replay guard judges staleness against the
// (virtual) agent clock so tests stay deterministic.
Tracker.UtcNowProvider = () => Clock.UtcNow;
}
public IReadOnlyList<FakeSignalIngressSink.PostedSignal> DecisionSignals(DecisionSignalKind kind) =>
@@ -405,5 +405,144 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
var started = trace.Events.Find(e => e.EventType == SharedEventTypes.DesktopDetectorStarted);
Assert.Equal(0, started.Data["noCandidateThresholdMinutes"]);
}
// ============================================================================
// OOBE-state flip (oobe_state_completed) — observational one-shot on the poll
// ============================================================================
// The WinRT OutOfBoxExperienceState flip in_progress->completed is an owner-
// independent desktop corroboration for DAD-starved sessions (4d5a0b78 class).
// Tests drive polls synchronously via PollOnceForTest after Stop() so the real
// timer cannot race, and pin the owner seams to an excluded user so a real
// explorer.exe on the test machine can never set _desktopArrived (which would
// short-circuit PollForDesktop before the flip check).
private static DesktopArrivalDetector BuildFlipDetector(TempDirectory tmp, TraceCapture trace, System.Func<string> oobeProvider)
{
var detector = new DesktopArrivalDetector(new AgentLogger(tmp.Path, AgentLogLevel.Info), noCandidateTimeoutMinutes: 0)
{
OnTraceEvent = trace.Sink,
OobeStateProvider = oobeProvider,
SessionOwnerResolver = _ => "SYSTEM",
ProcessOwnerResolver = _ => "SYSTEM",
};
return detector;
}
/// <summary>Returns the queued states in order, then repeats the last one forever.</summary>
private static System.Func<string> StateQueue(params string[] states)
{
var queue = new Queue<string>(states);
var last = states[states.Length - 1];
return () => queue.Count > 0 ? queue.Dequeue() : last;
}
[Fact]
public void OobeFlip_emits_exactly_once_with_payload()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
// Baseline sample happens synchronously inside Start()/ArmTimer.
using var detector = BuildFlipDetector(tmp, trace, StateQueue("in_progress", "completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest(); // reads "completed" -> flip
detector.PollOnceForTest(); // still "completed" -> no second emit
Assert.Equal(1, trace.CountOf(SharedEventTypes.OobeStateCompleted));
var flip = trace.Events.Find(e => e.EventType == SharedEventTypes.OobeStateCompleted);
Assert.Equal("in_progress", flip.Data["previousState"]);
Assert.True(flip.Data.ContainsKey("pollCount"));
Assert.True(flip.Data.ContainsKey("minutesSinceStart"));
Assert.False((bool)flip.Data["desktopArrived"]);
Assert.True(flip.Data.ContainsKey("wmiErrorCount"));
Assert.True(flip.Data.ContainsKey("wtsErrorCount"));
}
[Fact]
public void OobeFlip_does_not_emit_when_state_is_already_completed()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace, StateQueue("completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest();
detector.PollOnceForTest();
// Productive machines read "completed" from the very first sample — the flip
// event must never fire there (the agent_started field already records the state).
Assert.Equal(0, trace.CountOf(SharedEventTypes.OobeStateCompleted));
}
[Fact]
public void OobeFlip_does_not_emit_when_in_progress_was_never_seen()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace, StateQueue("unavailable", "completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest();
// Old builds without the WinRT contract report "unavailable"; a later
// "completed" without a positively observed "in_progress" is not a flip.
Assert.Equal(0, trace.CountOf(SharedEventTypes.OobeStateCompleted));
}
[Fact]
public void OobeFlip_survives_transient_unavailable_between_polls()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace, StateQueue("in_progress", "unavailable", "completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest(); // transient reflection failure — must NOT clear the baseline
detector.PollOnceForTest(); // "completed" -> flip still detected
Assert.Equal(1, trace.CountOf(SharedEventTypes.OobeStateCompleted));
}
[Fact]
public void OobeFlip_provider_throwing_never_emits_and_never_pollutes_wmi_error_count()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace, () => throw new System.InvalidOperationException("boom"));
detector.Start(); // baseline sample throws -> swallowed
detector.Stop();
detector.PollOnceForTest();
detector.PollOnceForTest();
Assert.Equal(0, trace.CountOf(SharedEventTypes.OobeStateCompleted));
// The flip check has its own try/catch — a throw must not fall into
// PollForDesktop's outer catch, which increments the WMI error counter the
// session-4d5a0b78 liveness diagnostics rely on. first_poll carries that counter.
var firstPoll = trace.Events.Find(e => e.EventType == SharedEventTypes.DesktopDetectorFirstPoll);
Assert.Equal(0, firstPoll.Data["wmiErrorCount"]);
}
[Fact]
public void OobeFlip_one_shot_flag_survives_ResetForRealUserSwitch()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace,
StateQueue("in_progress", "completed", "in_progress", "completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest(); // flip #1
detector.ResetForRealUserSwitch(); // re-arms + re-samples baseline ("in_progress")
detector.Stop();
detector.PollOnceForTest(); // "completed" again — must NOT re-emit
Assert.Equal(1, trace.CountOf(SharedEventTypes.OobeStateCompleted));
}
}
}
@@ -290,9 +290,20 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
[InlineData(@"SAM\SAM\Domains\Account\Users")]
[InlineData(@"SECURITY\Policy\Secrets")]
[InlineData(@"HKLM\SOFTWARE\Microsoft\EnrollmentsSomethingElse")] // prefix spoofing
[InlineData(@"HKLM\SOFTWARE\Microsoft\Windows\AutopilotSomethingElse")] // sibling spoof of the Autopilot prefix
public void RegistryCollector_OffAllowlistPath_IsBlockedBeforeTheHiveIsOpened(string target)
=> AssertBlocked(new RegistryCollector().Execute(Rule("registry", target), Context()), target);
// Session 5d735290: the EnrollmentStatusTracking subtree (ESP policy providers / tracking
// state) is allow-listed so tenants can build gather rules on the co-management stall
// signature the esp_policy_provider_stalled detector reports (e.g. not_exists on the
// Sidecar provider key).
[Theory]
[InlineData(@"SOFTWARE\Microsoft\Windows\Autopilot")]
[InlineData(@"SOFTWARE\Microsoft\Windows\Autopilot\EnrollmentStatusTracking\Device\Setup\Apps\PolicyProviders\Sidecar")]
public void RegistryGuard_AutopilotEnrollmentTrackingSubtree_IsAllowed(string subPath)
=> Assert.True(GatherRuleGuards.IsRegistryPathAllowed(subPath));
[Theory]
[InlineData("SELECT * FROM Win32_UserAccount")]
[InlineData("SELECT * FROM Win32_ShadowCopy")]
@@ -0,0 +1,166 @@
using System;
using System.Collections.Generic;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
{
/// <summary>
/// Tracker-level historic-replay guard (session eaf3d8c4 part 2): source lines > 24 h older
/// than now are content from a previous enrollment whose IME log survived on disk. The guard
/// in <c>HandlePatternMatch</c> skips app-mutating actions for such lines so replayed apps
/// never enter <c>_packageStates</c> / phase snapshots / persistence — keeping
/// app_tracking_summary, culprit lists and final-status clean. Script actions pass through
/// (the adapter suppresses their emissions), and SimulationMode bypasses the guard entirely.
/// </summary>
public sealed class ImeLogTrackerHistoricReplayGuardTests
{
private static readonly DateTime Now = new DateTime(2026, 7, 23, 15, 42, 0, DateTimeKind.Utc);
private static List<ImeLogPattern> TestPatterns() => new List<ImeLogPattern>
{
new ImeLogPattern
{
PatternId = "T-POLICIES", Category = "always", Enabled = true,
Pattern = @"policies discovered = (?<policies>\[.*\])",
Action = "policiesDiscovered",
Parameters = new Dictionary<string, string>(),
},
new ImeLogPattern
{
PatternId = "T-INSTALLED", Category = "always", Enabled = true,
Pattern = @"app (?<id>[\w-]+) installed",
Action = "updateStateInstalled",
Parameters = new Dictionary<string, string>(),
},
new ImeLogPattern
{
PatternId = "T-ESP-PHASE", Category = "always", Enabled = true,
Pattern = @"In EspPhase: (?<espPhase>\w+)",
Action = "espPhaseDetected",
Parameters = new Dictionary<string, string>(),
},
new ImeLogPattern
{
PatternId = "T-SCRIPT-START", Category = "always", Enabled = true,
Pattern = @"script (?<id>[\w-]+) started",
Action = "scriptStarted",
Parameters = new Dictionary<string, string> { ["scriptType"] = "platform" },
},
};
private static ImeLogTracker BuildTracker(TempDirectory tmp)
{
var tracker = new ImeLogTracker(
logFolder: tmp.Path,
patterns: TestPatterns(),
logger: new AgentLogger(tmp.Path, AgentLogLevel.Info));
tracker.UtcNowProvider = () => Now;
return tracker;
}
private const string PoliciesLine = @"policies discovered = [{""Id"":""app-1"",""Name"":""App One""}]";
[Fact]
public void Stale_policies_line_does_not_populate_package_states()
{
using var tmp = new TempDirectory();
using var tracker = BuildTracker(tmp);
tracker.ProcessLogMessageForTest(PoliciesLine, Now.AddDays(-7));
Assert.Empty(tracker.GetAllKnownPackageStates());
}
[Fact]
public void Fresh_relog_of_same_policies_is_tracked_after_stale_skip()
{
// IME re-logs the full policy JSON on every app-policy check-in — skipping the
// stale copy cannot starve the current enrollment.
using var tmp = new TempDirectory();
using var tracker = BuildTracker(tmp);
tracker.ProcessLogMessageForTest(PoliciesLine, Now.AddDays(-7));
tracker.ProcessLogMessageForTest(PoliciesLine, Now.AddMinutes(-2));
var all = tracker.GetAllKnownPackageStates();
var pkg = Assert.Single(all);
Assert.Equal("app-1", pkg.Id);
}
[Fact]
public void Stale_state_update_against_fresh_app_is_skipped()
{
using var tmp = new TempDirectory();
using var tracker = BuildTracker(tmp);
var stateChanges = 0;
tracker.OnAppStateChanged = (_, __, ___) => stateChanges++;
tracker.ProcessLogMessageForTest(PoliciesLine, Now.AddMinutes(-5)); // fresh discovery
tracker.ProcessLogMessageForTest("app app-1 installed", Now.AddDays(-7)); // stale mutation
Assert.Equal(0, stateChanges);
var pkg = Assert.Single(tracker.GetAllKnownPackageStates());
Assert.NotEqual(AppInstallationState.Installed, pkg.InstallationState);
}
[Fact]
public void Stale_esp_phase_line_does_not_block_fresh_phase_detection()
{
// A stale "AccountSetup" would advance _currentPhaseOrder and make the fresh
// enrollment's DeviceSetup bounce as "backward" — the guard must skip it.
using var tmp = new TempDirectory();
using var tracker = BuildTracker(tmp);
var phases = new List<string>();
tracker.OnEspPhaseChanged = p => phases.Add(p);
tracker.ProcessLogMessageForTest("In EspPhase: AccountSetup", Now.AddDays(-7));
tracker.ProcessLogMessageForTest("In EspPhase: DeviceSetup", Now.AddMinutes(-1));
Assert.Equal(new[] { "DeviceSetup" }, phases);
}
[Fact]
public void Stale_script_line_still_reaches_script_handlers()
{
// Script tracker state is harmless for stale lines (adapter suppresses the
// emissions; the stale-slot hardening covers leftovers) — the guard must not
// swallow script actions.
using var tmp = new TempDirectory();
using var tracker = BuildTracker(tmp);
var started = new List<ScriptStartedInfo>();
tracker.OnScriptStarted = i => started.Add(i);
tracker.ProcessLogMessageForTest("script aaaa1111-0000-0000-0000-000000000000 started", Now.AddDays(-7));
Assert.Single(started);
}
[Fact]
public void SimulationMode_bypasses_the_guard()
{
using var tmp = new TempDirectory();
using var tracker = BuildTracker(tmp);
tracker.SimulationMode = true;
tracker.ProcessLogMessageForTest(PoliciesLine, Now.AddDays(-7));
Assert.Single(tracker.GetAllKnownPackageStates());
}
[Fact]
public void Lines_without_timestamp_are_not_treated_as_stale()
{
// Non-CMTrace lines carry no source timestamp (entry == null) — never stale.
using var tmp = new TempDirectory();
using var tracker = BuildTracker(tmp);
tracker.ProcessLogMessageForTest(PoliciesLine, sourceTimestampUtc: null);
Assert.Single(tracker.GetAllKnownPackageStates());
}
}
}
@@ -0,0 +1,40 @@
#nullable enable
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
{
/// <summary>
/// Smoke tests for the WinRT <c>SystemSetupInfo.OutOfBoxExperienceState</c> reader.
/// The actual state on the test machine is unknown (CI runners are post-OOBE; Server
/// SKUs may lack the UniversalApiContract entirely), so only the fail-soft contract
/// is pinned: never throw, always return a member of the known value set.
/// </summary>
public sealed class OobeStateReaderTests
{
[Fact]
public void Read_never_throws_and_returns_known_value()
{
var value = OobeStateReader.Read();
Assert.False(string.IsNullOrEmpty(value));
Assert.True(
value == "not_started"
|| value == "in_progress"
|| value == "completed"
|| value == OobeStateReader.Unavailable
|| value.StartsWith("unknown_"),
$"unexpected reader value '{value}'");
}
[Fact]
public void Read_is_stable_across_repeated_calls()
{
// Exercises the cached-PropertyInfo path (second call skips resolution).
var first = OobeStateReader.Read();
var second = OobeStateReader.Read();
Assert.Equal(first, second);
}
}
}
@@ -15,7 +15,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
/// <item>a locked tree must abort WITHOUT deleting anything (old script force-deleted
/// marker + state around the locks and orphaned the binaries),</item>
/// <item>a straggler agent process (the incident's second instance) must be found by
/// NAME, killed, and the cleanup must then complete,</item>
/// NAME, killed, and the cleanup must then complete — simulated via function
/// shadowing, NEVER by launching a renamed executable (EDR isolation, see the test),</item>
/// <item>concurrent cleanup scripts must be serialized by the lock file.</item>
/// </list>
/// Timings are dialed down via the test-only BuildCleanupScript parameters so each case
@@ -111,60 +112,55 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
}
[Fact]
public void Straggler_process_holding_the_tree_is_killed_and_cleanup_completes()
public void Straggler_agent_process_is_waited_on_and_force_killed_via_name_backstop()
{
// Full incident replay: a SECOND agent-named process (here: a renamed
// powershell.exe, matching how only the process NAME identifies agents) holds a
// file in the tree open. The old script waited on the launcher PID only and ran
// its force-delete against the live locks. The hardened script must find the
// straggler by name, kill it after the wait deadline, and then delete everything.
var fakeExe = Path.Combine(_root, _fakeProcessName + ".exe");
File.Copy(SystemPaths.PowerShell, fakeExe);
// Incident replay of the second-instance mechanics WITHOUT spawning a process:
// an earlier version of this test copied powershell.exe under the fake agent name
// and launched it — textbook MITRE T1036.003 ("Rename Legitimate Utilities"),
// which Defender for Endpoint answered with a machine ISOLATION on the dev box
// (see tasks/lessons.md). Instead, a harness script shadows Get-Process /
// Stop-Process with functions (PowerShell resolves functions before cmdlets, so
// the dot-sourced cleanup script binds to the stubs): Get-Process reports a live
// agent-named straggler on every probe; Stop-Process records the kill. This pins
// the same contract — the script polls by NAME until the deadline, then
// force-kills by NAME before deleting — with zero EDR-visible behavior.
var probeLog = Path.Combine(_root, "probe.log");
var cleanupScriptPath = Path.Combine(_root, $"cleanup-{Guid.NewGuid():N}.ps1");
File.WriteAllText(cleanupScriptPath, BuildScript(processWaitSeconds: 2));
var lockTarget = Path.Combine(_basePath, "Agent", "fake.exe").Replace("'", "''");
Process straggler = null;
try
{
straggler = Process.Start(new ProcessStartInfo
{
FileName = fakeExe,
Arguments = "-NoProfile -Command \"$f = [IO.File]::Open('" + lockTarget +
"', 'Open', 'Read', 'None'); Start-Sleep -Seconds 120\"",
UseShellExecute = false,
CreateNoWindow = true,
});
var harness = @"
$probeLog = 'LOG_PATH'
$global:AmProbeCount = 0
function Get-Process {
[CmdletBinding()]
param([string[]]$Name)
if ($Name -contains 'FAKE_NAME') {
$global:AmProbeCount++
Add-Content -Path $probeLog -Value ('probe ' + $global:AmProbeCount)
return [pscustomobject]@{ Name = 'FAKE_NAME'; Id = 424242 }
}
Microsoft.PowerShell.Management\Get-Process @PSBoundParameters
}
function Stop-Process {
[CmdletBinding()]
param([Parameter(ValueFromPipeline = $true)]$InputObject, [switch]$Force)
process { Add-Content -Path $probeLog -Value ('kill ' + $InputObject.Id + ' force=' + $Force) }
}
. 'SCRIPT_PATH'
"
.Replace("LOG_PATH", probeLog)
.Replace("FAKE_NAME", _fakeProcessName)
.Replace("SCRIPT_PATH", cleanupScriptPath);
// Give the straggler time to acquire its file handle before cleanup starts.
var acquireDeadline = DateTime.UtcNow.AddSeconds(20);
var acquired = false;
while (!acquired && DateTime.UtcNow < acquireDeadline)
{
try
{
using (File.Open(Path.Combine(_basePath, "Agent", "fake.exe"),
FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
}
}
catch (IOException)
{
acquired = true; // share violation ⇒ straggler holds the file
}
if (!acquired) System.Threading.Thread.Sleep(200);
}
Assert.True(acquired, "straggler never acquired its file lock");
RunScript(harness);
RunScript(BuildScript(processWaitSeconds: 2));
Assert.True(straggler.HasExited, "straggler must be force-killed by the name-based backstop");
Assert.False(Directory.Exists(_basePath), "after the kill the tree must be fully deleted");
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
}
finally
{
try { if (straggler != null && !straggler.HasExited) straggler.Kill(); } catch { }
try { straggler?.Dispose(); } catch { }
}
var log = File.ReadAllLines(probeLog);
Assert.Contains("kill 424242 force=True", log);
Assert.True(Array.FindAll(log, l => l.StartsWith("probe")).Length >= 2,
"script must poll by name in the wait loop AND re-probe for the kill backstop");
Assert.False(Directory.Exists(_basePath), "after the kill the tree must be fully deleted");
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
Assert.False(File.Exists(LockPath));
}
[Fact]
@@ -0,0 +1,256 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
{
/// <summary>
/// Session 4910a5a5 (2026-07-23) — post-terminal recovery story. Field sequence:
/// DeviceSetup/Apps failed (culprit app never started installing) → settle window expired
/// → terminal EspFailureDetected → the user pressed "Try again" ~23 min later → Apps left
/// the failed state → 10/10 installed → DeviceSetup resolved. Pre-fix, that recovery only
/// appeared as a generic <c>subcategory_state_change</c>. Coverage:
/// <list type="bullet">
/// <item><c>esp_failure_retry_detected</c> fires once when the failed subcategory leaves
/// the failed state AFTER the terminal fire.</item>
/// <item><c>esp_failure_recovered</c> fires once when the category then resolves to
/// success.</item>
/// <item>An in-settle-window recovery keeps emitting only
/// <c>esp_failure_settle_recovered</c> — the post-terminal pair stays silent.</item>
/// <item>Apps failures carry <c>LikelyCulpritApps</c> on the EspFailureDetected args,
/// never-started apps ranked first.</item>
/// </list>
/// </summary>
public sealed class EspFailureRecoveryStoryTests
{
private static readonly DateTime Fixed = new DateTime(2026, 7, 23, 4, 30, 0, DateTimeKind.Utc);
private sealed class Fixture : IDisposable
{
public TempDirectory Tmp { get; } = new TempDirectory();
public FakeSignalIngressSink Sink { get; } = new FakeSignalIngressSink();
public List<EspFailureDetectedEventArgs> EspFailures { get; } = new List<EspFailureDetectedEventArgs>();
public ProvisioningStatusTracker Tracker { get; }
public Fixture(Func<IReadOnlyList<AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppPackageState>>? packageStatesProbe = null)
{
var clock = new VirtualClock(Fixed);
var post = new InformationalEventPost(Sink, clock);
var logger = new AgentLogger(Tmp.Path, AgentLogLevel.Info);
Tracker = new ProvisioningStatusTracker(
sessionId: "S1",
tenantId: "T1",
post: post,
logger: logger,
appxScanner: new FakeAppxDeploymentFailureScanner(),
backgroundDispatcher: action => action(),
packageStatesProbe: packageStatesProbe);
Tracker.EspFailureDetected += (_, args) => EspFailures.Add(args);
}
public IReadOnlyList<(string EventType, IReadOnlyDictionary<string, object> Data)> CapturedEvents()
{
return Sink.Posted
.Where(p => p.Kind == DecisionSignalKind.InformationalEvent && p.Payload != null)
.Select(p =>
{
var eventType = p.Payload!.TryGetValue(SignalPayloadKeys.EventType, out var et) ? et : "";
var data = p.TypedPayload as IReadOnlyDictionary<string, object>
?? (IReadOnlyDictionary<string, object>)new Dictionary<string, object>();
return (eventType, data);
})
.ToList();
}
/// <summary>Drive the session-4910a5a5 shape up to the terminal fire.</summary>
public void DriveToTerminalDeviceSetupAppsFailure()
{
Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Apps (9 of 10 installed)""}
}");
Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""failed"",""subcategoryStatusText"":""Apps (Error)""}
}");
Tracker.TriggerSettleTimerForTest("DeviceSetupCategory.Status");
}
public void Dispose()
{
Tracker.Dispose();
Tmp.Dispose();
}
}
[Fact]
public void PostTerminalRetry_EmitsRetryDetected_Once()
{
using var f = new Fixture();
f.DriveToTerminalDeviceSetupAppsFailure();
Assert.Single(f.EspFailures);
// "Try again": Apps leaves the failed state well after the settle window.
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Apps (Working on it...)""}
}");
var retry = Assert.Single(f.CapturedEvents(), e => e.EventType == "esp_failure_retry_detected");
Assert.Equal("DeviceSetup", (string)retry.Data["category"]);
Assert.Equal("Apps", (string)retry.Data["subcategory"]);
Assert.Equal("failed", (string)retry.Data["previousState"]);
Assert.Equal("inProgress", (string)retry.Data["newState"]);
Assert.Equal("Provisioning_DeviceSetup_Apps_Failed", (string)retry.Data["failureType"]);
Assert.True(retry.Data.ContainsKey("minutesSinceFailure"));
// A later unrelated transition does not re-emit.
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Apps (10 of 10 installed)""}
}");
Assert.Single(f.CapturedEvents(), e => e.EventType == "esp_failure_retry_detected");
}
[Fact]
public void PostTerminalRecovery_EmitsRecovered_OnCategorySuccess()
{
using var f = new Fixture();
f.DriveToTerminalDeviceSetupAppsFailure();
// Retry → all apps installed → Windows resolves the category boolean.
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Apps (Working on it...)""}
}");
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": true,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Apps (10 of 10 installed)""}
}");
var recovered = Assert.Single(f.CapturedEvents(), e => e.EventType == "esp_failure_recovered");
Assert.Equal("DeviceSetup", (string)recovered.Data["category"]);
Assert.Equal("Apps", (string)recovered.Data["failedSubcategory"]);
Assert.Equal("Provisioning_DeviceSetup_Apps_Failed", (string)recovered.Data["failureType"]);
Assert.True(recovered.Data.ContainsKey("minutesSinceFailure"));
// Duplicate category-success writes do not re-emit (fire-once).
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": true,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Apps (10 of 10 installed, take 2)""}
}");
Assert.Single(f.CapturedEvents(), e => e.EventType == "esp_failure_recovered");
}
[Fact]
public void InSettleWindowRecovery_KeepsPostTerminalPairSilent()
{
using var f = new Fixture();
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""AppsSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Apps (0 of 1 installed)""}
}");
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""AppsSubcategory"": {""subcategoryState"":""failed"",""subcategoryStatusText"":""Apps (Error)""}
}");
// Recovery INSIDE the settle window — terminal fire is suppressed.
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""AppsSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Apps (Working on it...)""}
}");
f.Tracker.TriggerSettleTimerForTest("DeviceSetupCategory.Status");
Assert.Empty(f.EspFailures);
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": true,
""AppsSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Apps (1 of 1 installed)""}
}");
var events = f.CapturedEvents();
Assert.Contains(events, e => e.EventType == "esp_failure_settle_recovered");
Assert.DoesNotContain(events, e => e.EventType == "esp_failure_retry_detected");
Assert.DoesNotContain(events, e => e.EventType == "esp_failure_recovered");
}
[Fact]
public void AppsFailure_TerminalArgs_CarryLikelyCulprits_NeverStartedFirst()
{
var neverStarted = BuildApp("app-bg", "Teams Backgrounds v2",
AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Unknown,
downloadingSeen: false);
var startedButPending = BuildApp("app-slow", "Slow Installer",
AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Unknown,
downloadingSeen: true);
var installed = BuildApp("app-ok", "Falcon Sensor",
AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Installed,
downloadingSeen: true);
using var f = new Fixture(() => new[] { startedButPending, neverStarted, installed });
f.DriveToTerminalDeviceSetupAppsFailure();
var fired = Assert.Single(f.EspFailures);
Assert.Equal(2, fired.LikelyCulpritApps.Count);
// Never-started app ranked first — it starved the ESP gate without any IME event.
Assert.Equal("Teams Backgrounds v2", fired.LikelyCulpritApps[0]);
Assert.Equal("Slow Installer", fired.LikelyCulpritApps[1]);
}
[Fact]
public void NonAppsFailure_TerminalArgs_CarryNoCulprits()
{
var neverStarted = BuildApp("app-bg", "Teams Backgrounds v2",
AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Unknown,
downloadingSeen: false);
using var f = new Fixture(() => new[] { neverStarted });
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Working""}
}");
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""failed"",""subcategoryStatusText"":""Failed""}
}");
f.Tracker.TriggerSettleTimerForTest("DeviceSetupCategory.Status");
var fired = Assert.Single(f.EspFailures);
Assert.Empty(fired.LikelyCulpritApps);
}
private static AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppPackageState BuildApp(
string id, string name,
AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState state,
bool downloadingSeen)
{
var pkg = new AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppPackageState(id, 0);
pkg.UpdateName(name);
pkg.UpdateIntent(AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppIntent.Install);
pkg.UpdateTargeted(AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppTargeted.Device);
// DownloadingOrInstallingSeen is derived inside UpdateState (Downloading..Installing
// range) — drive it through a Downloading transition instead of setting it directly.
if (downloadingSeen)
pkg.UpdateState(AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Downloading);
if (state != AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Unknown
&& state != AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Downloading)
pkg.UpdateState(state);
return pkg;
}
}
}
@@ -0,0 +1,359 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Persistence;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
{
/// <summary>
/// Session 5d735290 — pure detection logic of <see cref="EspPolicyProviderStallDetector"/>:
/// the CSP completion rule and the dwell/one-shot transition. No registry, no timers.
/// </summary>
public sealed class EspPolicyProviderStallLogicTests
{
private static readonly DateTime T0 = new DateTime(2026, 7, 23, 10, 0, 0, DateTimeKind.Utc);
private static readonly TimeSpan Dwell = TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes);
private static PolicyProviderState SetupApps(string name, int? tracking, string scope = "device") =>
new PolicyProviderState(name, scope, EspPolicyProviderProbe.KindSetupApps, tracking, null);
private static PolicyProviderState DevicePrep(string name, int? installState) =>
new PolicyProviderState(name, "device", EspPolicyProviderProbe.KindDevicePreparation, null, installState);
private static EspPolicyProviderSnapshot Snapshot(params PolicyProviderState[] providers) =>
new EspPolicyProviderSnapshot(providers);
// ------------------------------------------------- completion rule (CSP contract)
[Theory]
[InlineData(1, true)] // TrackingPoliciesCreated=true
[InlineData(0, false)] // explicit false (the CSP default)
[InlineData(null, false)] // value missing — the field case
[InlineData(2, false)] // anything else is not "true"
public void SetupApps_provider_complete_only_at_tracking_1(int? tracking, bool expected)
=> Assert.Equal(expected, EspPolicyProviderProbe.IsProviderComplete(
EspPolicyProviderProbe.KindSetupApps, tracking, null));
[Theory]
[InlineData(2, true)] // NotRequired
[InlineData(3, true)] // Completed
[InlineData(1, false)] // NotInstalled
[InlineData(4, false)] // Error
[InlineData(null, false)]
public void DevicePreparation_provider_complete_at_state_2_or_3(int? state, bool expected)
=> Assert.Equal(expected, EspPolicyProviderProbe.IsProviderComplete(
EspPolicyProviderProbe.KindDevicePreparation, null, state));
[Fact]
public void TryReadInt_tolerates_dword_string_and_bool_forms()
{
Assert.Equal(1, EspPolicyProviderProbe.TryReadInt(1));
Assert.Equal(3, EspPolicyProviderProbe.TryReadInt(3L));
Assert.Equal(1, EspPolicyProviderProbe.TryReadInt(" 1 "));
Assert.Equal(1, EspPolicyProviderProbe.TryReadInt(true));
Assert.Equal(0, EspPolicyProviderProbe.TryReadInt(false));
Assert.Null(EspPolicyProviderProbe.TryReadInt(null));
Assert.Null(EspPolicyProviderProbe.TryReadInt("not-a-number"));
Assert.Null(EspPolicyProviderProbe.TryReadInt(new byte[] { 1 }));
}
// ------------------------------------------------- dwell transition
[Fact]
public void Empty_or_missing_snapshot_clears_dwell_state_and_never_stalls()
{
var previous = new Dictionary<string, DateTime> { ["setupApps|device|ConfigMgr"] = T0 };
var missing = EspPolicyProviderStallDetector.EvaluateStallTransition(
EspPolicyProviderSnapshot.Empty, previous, new HashSet<string>(), T0.AddHours(2), Dwell);
Assert.Empty(missing.UpdatedFirstSeenUtc);
Assert.Empty(missing.NewlyStalled);
var registeredButEmpty = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(), previous, new HashSet<string>(), T0.AddHours(2), Dwell);
Assert.Empty(registeredButEmpty.UpdatedFirstSeenUtc);
Assert.Empty(registeredButEmpty.NewlyStalled);
}
[Fact]
public void Incomplete_provider_accrues_dwell_but_does_not_stall_before_threshold()
{
var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: null));
var round1 = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, new Dictionary<string, DateTime>(), new HashSet<string>(), T0, Dwell);
Assert.Equal(T0, round1.UpdatedFirstSeenUtc["setupApps|device|ConfigMgr"]);
Assert.Empty(round1.NewlyStalled);
var round2 = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, round1.UpdatedFirstSeenUtc, new HashSet<string>(), T0.AddMinutes(14), Dwell);
Assert.Equal(T0, round2.UpdatedFirstSeenUtc["setupApps|device|ConfigMgr"]); // first-seen preserved
Assert.Empty(round2.NewlyStalled);
}
[Fact]
public void Provider_stalls_at_dwell_threshold_with_correct_minutes()
{
var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: 0));
var seen = new Dictionary<string, DateTime> { ["setupApps|device|ConfigMgr"] = T0 };
var eval = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, seen, new HashSet<string>(), T0.AddMinutes(15), Dwell);
var stalled = Assert.Single(eval.NewlyStalled);
Assert.Equal("ConfigMgr", stalled.Provider.Name);
Assert.Equal(15.0, stalled.StalledForMinutes, precision: 3);
}
[Fact]
public void Completion_or_disappearance_resets_the_dwell_clock()
{
var key = "setupApps|device|ConfigMgr";
var seen = new Dictionary<string, DateTime> { [key] = T0 };
// Completes mid-dwell → dropped from the map.
var completed = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("ConfigMgr", tracking: 1)), seen, new HashSet<string>(), T0.AddMinutes(10), Dwell);
Assert.Empty(completed.UpdatedFirstSeenUtc);
// Re-appears incomplete much later → dwell restarts from the new now, no stall yet.
var reappeared = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("ConfigMgr", tracking: 0)), completed.UpdatedFirstSeenUtc,
new HashSet<string>(), T0.AddHours(3), Dwell);
Assert.Equal(T0.AddHours(3), reappeared.UpdatedFirstSeenUtc[key]);
Assert.Empty(reappeared.NewlyStalled);
// Disappearance behaves the same as completion.
var disappeared = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("Sidecar", tracking: 1)), new Dictionary<string, DateTime> { [key] = T0 },
new HashSet<string>(), T0.AddMinutes(20), Dwell);
Assert.False(disappeared.UpdatedFirstSeenUtc.ContainsKey(key));
}
[Fact]
public void Already_fired_provider_is_never_reported_again()
{
var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: null));
var seen = new Dictionary<string, DateTime> { ["setupApps|device|ConfigMgr"] = T0 };
var fired = new HashSet<string>(StringComparer.OrdinalIgnoreCase) { "setupApps|device|ConfigMgr" };
var eval = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, seen, fired, T0.AddHours(5), Dwell);
Assert.Empty(eval.NewlyStalled);
Assert.True(eval.UpdatedFirstSeenUtc.ContainsKey("setupApps|device|ConfigMgr")); // still tracked
}
[Fact]
public void Mixed_kinds_and_scopes_are_tracked_under_distinct_keys()
{
var snapshot = Snapshot(
SetupApps("ConfigMgr", tracking: null),
SetupApps("ConfigMgr", tracking: null, scope: "user:S-1-5-21-1111"),
DevicePrep("ConfigMgr", installState: 1));
var eval = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, new Dictionary<string, DateTime>(), new HashSet<string>(), T0, Dwell);
Assert.Equal(3, eval.UpdatedFirstSeenUtc.Count);
Assert.Contains("setupApps|device|ConfigMgr", eval.UpdatedFirstSeenUtc.Keys);
Assert.Contains("setupApps|user:S-1-5-21-1111|ConfigMgr", eval.UpdatedFirstSeenUtc.Keys);
Assert.Contains("devicePreparation|device|ConfigMgr", eval.UpdatedFirstSeenUtc.Keys);
}
[Fact]
public void SidecarRegistered_matches_setupApps_sidecar_only()
{
Assert.True(Snapshot(SetupApps("Sidecar", tracking: 0)).SidecarRegistered);
Assert.True(Snapshot(SetupApps("sidecar", tracking: 1, scope: "user:S-1-5-21-1")).SidecarRegistered);
Assert.False(Snapshot(SetupApps("ConfigMgr", tracking: 1)).SidecarRegistered);
Assert.False(Snapshot(DevicePrep("Sidecar", installState: 3)).SidecarRegistered); // wrong kind
Assert.False(EspPolicyProviderSnapshot.Empty.SidecarRegistered);
}
}
/// <summary>
/// Emission surface of the detector via the real <see cref="InformationalEventPost"/> rail.
/// Static probe override → serial collection (same contract as
/// <c>EspConfigDetectedTrackingFieldsTests</c>).
/// </summary>
[Collection("SerialThreading")]
public sealed class EspPolicyProviderStallDetectorEmissionTests : IDisposable
{
private static readonly DateTime At = new DateTime(2026, 7, 23, 12, 0, 0, DateTimeKind.Utc);
private readonly TempDirectory _tmp = new TempDirectory();
public void Dispose() => _tmp.Dispose();
private (EspPolicyProviderStallDetector sut, FakeSignalIngressSink sink, VirtualClock clock)
BuildDetector(StartupEventGate? gate = null)
{
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
var sink = new FakeSignalIngressSink();
var clock = new VirtualClock(At);
var post = new InformationalEventPost(sink, clock);
var sut = new EspPolicyProviderStallDetector("S1", "T1", post, logger, clock, gate);
return (sut, sink, clock);
}
private static EspPolicyProviderSnapshot ConfigMgrOnly() => new EspPolicyProviderSnapshot(new[]
{
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindSetupApps,
trackingPoliciesCreated: null, installationState: null),
});
private static EspPolicyProviderSnapshot SidecarCompleteConfigMgrIncomplete() => new EspPolicyProviderSnapshot(new[]
{
new PolicyProviderState("Sidecar", "device", EspPolicyProviderProbe.KindSetupApps, 1, null),
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindSetupApps, 0, null),
});
private static IReadOnlyList<FakeSignalIngressSink.PostedSignal> StalledEvents(FakeSignalIngressSink sink) =>
sink.Posted.Where(p =>
p.Payload != null
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
&& et == "esp_policy_provider_stalled").ToList();
[Fact]
public void CustomerScenario_configmgr_without_sidecar_emits_one_warning_after_dwell()
{
using var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => ConfigMgrOnly());
var (sut, sink, clock) = BuildDetector();
sut.Tick(); // first sighting — dwell starts
Assert.Empty(StalledEvents(sink));
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick(); // threshold crossed → emit
var evt = Assert.Single(StalledEvents(sink));
Assert.Equal(DecisionSignalKind.InformationalEvent, evt.Kind);
Assert.Equal("Warning", evt.Payload![SignalPayloadKeys.Severity]);
Assert.Equal("true", evt.Payload[SignalPayloadKeys.ImmediateUpload]);
Assert.Equal(EspPolicyProviderStallDetector.SourceLabel, evt.Payload[SignalPayloadKeys.Source]);
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(evt.TypedPayload);
Assert.Equal(false, data["sidecarRegistered"]);
Assert.Equal(EspPolicyProviderStallDetector.DwellMinutes, data["dwellMinutes"]);
var stalled = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object>>>(data["stalledProviders"]);
var entry = Assert.Single(stalled);
Assert.Equal("ConfigMgr", entry["name"]);
Assert.Equal(EspPolicyProviderProbe.KindSetupApps, entry["kind"]);
Assert.Equal(15.0, entry["stalledForMinutes"]);
}
[Fact]
public void Sidecar_complete_configmgr_incomplete_reports_only_configmgr_with_sidecar_flag()
{
using var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => SidecarCompleteConfigMgrIncomplete());
var (sut, sink, clock) = BuildDetector();
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
var evt = Assert.Single(StalledEvents(sink));
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(evt.TypedPayload);
Assert.Equal(true, data["sidecarRegistered"]);
var stalled = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object>>>(data["stalledProviders"]);
Assert.Equal("ConfigMgr", Assert.Single(stalled)["name"]);
var providers = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object?>>>(data["providers"]);
Assert.Equal(2, providers.Count); // full table always attached
}
[Fact]
public void Latch_prevents_second_emission_for_the_same_provider()
{
using var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => ConfigMgrOnly());
var (sut, sink, clock) = BuildDetector();
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(30));
sut.Tick();
sut.Tick();
Assert.Single(StalledEvents(sink));
}
[Fact]
public void Startup_gate_suppresses_same_stall_across_restart_but_reports_new_provider_set()
{
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
// Run 1: fires and persists the fingerprint.
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => ConfigMgrOnly()))
{
var (sut, sink, clock) = BuildDetector(new StartupEventGate(_tmp.Path, logger));
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
Assert.Single(StalledEvents(sink));
}
// Run 2 (simulated restart — fresh detector, gate reloaded from the same state dir):
// identical stall re-accrues dwell but lands on the same fingerprint → suppressed.
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => ConfigMgrOnly()))
{
var (sut, sink, clock) = BuildDetector(new StartupEventGate(_tmp.Path, logger));
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
Assert.Empty(StalledEvents(sink));
}
// Run 3: a SECOND provider joins the stall → fingerprint changes → reported.
var widened = new EspPolicyProviderSnapshot(new[]
{
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindSetupApps, null, null),
new PolicyProviderState("Contoso", "device", EspPolicyProviderProbe.KindSetupApps, 0, null),
});
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => widened))
{
var (sut, sink, clock) = BuildDetector(new StartupEventGate(_tmp.Path, logger));
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
Assert.Single(StalledEvents(sink));
}
}
[Fact]
public void Healthy_or_absent_tracking_never_emits()
{
var healthy = new EspPolicyProviderSnapshot(new[]
{
new PolicyProviderState("Sidecar", "device", EspPolicyProviderProbe.KindSetupApps, 1, null),
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindDevicePreparation, null, 3),
});
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => healthy))
{
var (sut, sink, clock) = BuildDetector();
sut.Tick();
clock.Advance(TimeSpan.FromHours(4));
sut.Tick();
Assert.Empty(StalledEvents(sink));
}
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => EspPolicyProviderSnapshot.Empty))
{
var (sut, sink, clock) = BuildDetector();
sut.Tick();
clock.Advance(TimeSpan.FromHours(4));
sut.Tick();
Assert.Empty(StalledEvents(sink));
}
}
}
}
@@ -0,0 +1,178 @@
#nullable enable
using System.Collections.Generic;
using System.Text;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
{
/// <summary>
/// Pins the pure parts of the UEFI Secure Boot certificate probe: the ASCII substring
/// search over raw signature-list bytes and the <c>secureboot_status</c> payload shaping
/// (in particular the one-sided <c>uefiCa2023FirmwareConfirmed</c> marker whose ABSENCE
/// is load-bearing for ANALYZE-SEC-001's not_exists precondition). The actual firmware
/// read is only smoke-tested: test machines may be VMs, non-elevated, or legacy BIOS,
/// so only the fail-soft contract is asserted.
/// </summary>
public sealed class UefiSecureBootCertReaderTests
{
// --- ContainsAscii ---
[Fact]
public void ContainsAscii_finds_needle_embedded_in_binary_noise()
{
var bytes = Concat(new byte[] { 0x00, 0xA1, 0x30, 0x82 },
Encoding.ASCII.GetBytes("Windows UEFI CA 2023"),
new byte[] { 0x00, 0xFF });
Assert.True(UefiSecureBootCertReader.ContainsAscii(bytes, "Windows UEFI CA 2023"));
}
[Fact]
public void ContainsAscii_finds_needle_at_the_very_end()
{
var bytes = Concat(new byte[] { 0x01, 0x02 }, Encoding.ASCII.GetBytes("KEK 2K CA 2023"));
Assert.True(UefiSecureBootCertReader.ContainsAscii(bytes, "KEK 2K CA 2023"));
}
[Fact]
public void ContainsAscii_rejects_partial_overlap_and_absent_needle()
{
var bytes = Encoding.ASCII.GetBytes("Windows UEFI CA 2011 something else");
Assert.False(UefiSecureBootCertReader.ContainsAscii(bytes, "Windows UEFI CA 2023"));
// Repeated prefix bytes must not confuse the scan.
var tricky = Encoding.ASCII.GetBytes("ababac");
Assert.True(UefiSecureBootCertReader.ContainsAscii(tricky, "abac"));
}
[Fact]
public void ContainsAscii_handles_null_empty_and_oversized_needle()
{
Assert.False(UefiSecureBootCertReader.ContainsAscii(null, "x"));
Assert.False(UefiSecureBootCertReader.ContainsAscii(new byte[0], "x"));
Assert.False(UefiSecureBootCertReader.ContainsAscii(new byte[] { 1, 2 }, ""));
Assert.False(UefiSecureBootCertReader.ContainsAscii(new byte[] { 1, 2 }, "abc"));
}
// --- Payload shaping ---
[Fact]
public void Marker_emitted_only_when_db_ok_and_ca2023_present()
{
var data = new Dictionary<string, object>();
DeviceInfoCollector.AppendFirmwareCertFields(data, new UefiSecureBootCertSnapshot
{
DbStatus = UefiSecureBootCertReader.StatusOk,
KekStatus = UefiSecureBootCertReader.StatusOk,
DbHasWindowsUefiCa2023 = true,
KekHasMicrosoftKek2kCa2023 = true
});
Assert.Equal(true, data["uefiCa2023FirmwareConfirmed"]);
Assert.Equal("ok", data["uefiFirmwareReadStatus"]);
Assert.Equal(true, data["uefiDbHasWindowsUefiCa2023"]);
Assert.Equal(true, data["uefiKekHasMicrosoftKek2kCa2023"]);
}
[Fact]
public void Marker_absent_when_ca2023_missing_even_though_read_succeeded()
{
var data = new Dictionary<string, object>();
DeviceInfoCollector.AppendFirmwareCertFields(data, new UefiSecureBootCertSnapshot
{
DbStatus = UefiSecureBootCertReader.StatusOk,
KekStatus = UefiSecureBootCertReader.StatusOk,
DbHasWindowsUefiCa2023 = false,
DbHasWindowsProductionPca2011 = true
});
// The marker must be OMITTED, not false — not_exists fails on any present value.
Assert.False(data.ContainsKey("uefiCa2023FirmwareConfirmed"));
Assert.Equal(false, data["uefiDbHasWindowsUefiCa2023"]);
Assert.Equal(true, data["uefiDbHasWindowsProductionPca2011"]);
}
[Fact]
public void Failed_db_read_emits_status_but_no_cert_booleans_and_no_marker()
{
var data = new Dictionary<string, object>();
DeviceInfoCollector.AppendFirmwareCertFields(data, new UefiSecureBootCertSnapshot
{
DbStatus = UefiSecureBootCertReader.StatusPrivilegeDenied,
KekStatus = UefiSecureBootCertReader.StatusPrivilegeDenied
});
Assert.Equal(UefiSecureBootCertReader.StatusPrivilegeDenied, data["uefiFirmwareReadStatus"]);
Assert.False(data.ContainsKey("uefiDbHasWindowsUefiCa2023"));
Assert.False(data.ContainsKey("uefiKekHasMicrosoftKek2kCa2023"));
Assert.False(data.ContainsKey("uefiCa2023FirmwareConfirmed"));
}
[Fact]
public void Db_ok_but_kek_failed_keeps_db_booleans_and_reports_kek_status()
{
var data = new Dictionary<string, object>();
DeviceInfoCollector.AppendFirmwareCertFields(data, new UefiSecureBootCertSnapshot
{
DbStatus = UefiSecureBootCertReader.StatusOk,
KekStatus = UefiSecureBootCertReader.StatusVariableNotFound,
DbHasWindowsUefiCa2023 = true
});
Assert.Equal(UefiSecureBootCertReader.StatusVariableNotFound, data["uefiFirmwareReadStatus"]);
Assert.Equal(true, data["uefiDbHasWindowsUefiCa2023"]);
Assert.Equal(true, data["uefiCa2023FirmwareConfirmed"]);
Assert.False(data.ContainsKey("uefiKekHasMicrosoftKek2kCa2023"));
}
[Fact]
public void Null_snapshot_leaves_payload_untouched()
{
var data = new Dictionary<string, object> { { "uefiSecureBootEnabled", true } };
DeviceInfoCollector.AppendFirmwareCertFields(data, null);
Assert.Single(data);
}
[Fact]
public void DescribeFirmwareVerdict_covers_all_shapes()
{
Assert.Equal("unavailable", DeviceInfoCollector.DescribeFirmwareVerdict(null));
Assert.Equal(UefiSecureBootCertReader.StatusNotUefi, DeviceInfoCollector.DescribeFirmwareVerdict(
new UefiSecureBootCertSnapshot { DbStatus = UefiSecureBootCertReader.StatusNotUefi }));
Assert.Equal("CA2023 confirmed", DeviceInfoCollector.DescribeFirmwareVerdict(
new UefiSecureBootCertSnapshot { DbStatus = UefiSecureBootCertReader.StatusOk, DbHasWindowsUefiCa2023 = true }));
Assert.Equal("CA2023 missing", DeviceInfoCollector.DescribeFirmwareVerdict(
new UefiSecureBootCertSnapshot { DbStatus = UefiSecureBootCertReader.StatusOk, DbHasWindowsUefiCa2023 = false }));
}
// --- Reader smoke (real firmware; machine state unknown) ---
[Fact]
public void Read_never_throws_and_reports_a_status_for_both_variables()
{
var snapshot = UefiSecureBootCertReader.Read();
Assert.NotNull(snapshot);
Assert.False(string.IsNullOrEmpty(snapshot.DbStatus));
Assert.False(string.IsNullOrEmpty(snapshot.KekStatus));
}
private static byte[] Concat(params byte[][] parts)
{
var total = 0;
foreach (var part in parts) total += part.Length;
var result = new byte[total];
var offset = 0;
foreach (var part in parts)
{
System.Array.Copy(part, 0, result, offset, part.Length);
offset += part.Length;
}
return result;
}
}
}
@@ -0,0 +1,157 @@
using System;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
using AutopilotMonitor.Agent.V2.Core.SignalAdapters;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.Shared.Models;
using SharedEventTypes = AutopilotMonitor.Shared.Constants.EventTypes;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
{
/// <summary>
/// Historic-replay suppression for APP events (session eaf3d8c4 part 2): 147 of 157 app
/// events in that session were replayed from a previous enrollment's IME log — all 10
/// "installed apps" were phantoms. Stale app-state transitions and DO telemetry must be
/// suppressed BEFORE the sub-phase declaration, the timing bookkeeping and the terminal
/// DecisionSignal dedup, so fire-once flags stay unconsumed and no phantom
/// AppInstallCompleted/Failed reaches the engine (which could spuriously re-arm the
/// AdvisoryCompletion window). The script twin lives in
/// <see cref="ImeLogTrackerAdapterHistoricReplayTests"/>.
/// </summary>
public sealed class ImeLogTrackerAdapterHistoricAppReplayTests
{
private static readonly DateTime ClockNow = new DateTime(2026, 7, 23, 15, 42, 0, DateTimeKind.Utc);
[Fact]
public void Stale_terminal_app_transition_is_fully_suppressed()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var app = new AppPackageState("app-old", 0);
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7);
adapter.TriggerAppStateFromTest(app, AppInstallationState.Installing, AppInstallationState.Installed);
Assert.Empty(f.InfoEvents(SharedEventTypes.AppInstallComplete));
Assert.Empty(f.DecisionSignals(DecisionSignalKind.AppInstallCompleted));
// Shadow download_progress (status=completed) and the summary snapshot ride the
// same emit — both must be gone too.
Assert.Empty(f.InfoEvents(SharedEventTypes.DownloadProgress));
Assert.Empty(f.InfoEvents(SharedEventTypes.AppTrackingSummary));
// Timing bookkeeping must not be poisoned (feeds FinalStatusBuilder).
Assert.False(adapter.AppTimings.ContainsKey("app-old"));
Assert.Single(f.InfoEvents(SharedEventTypes.HistoricImeReplayDetected));
}
[Fact]
public void Fresh_run_after_suppressed_stale_run_fires_signal_timing_and_subphase()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var app = new AppPackageState("app-X", 0);
// Fresh ESP phase so the sub-phase declaration is armed.
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddMinutes(-5);
adapter.TriggerEspPhaseFromTest("DeviceSetup");
// Stale replay of the same app — suppressed, must consume NEITHER the sub-phase
// fire-once NOR the terminal-signal dedup for this appId.
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7);
adapter.TriggerAppStateFromTest(app, AppInstallationState.Unknown, AppInstallationState.Installed);
Assert.Empty(f.InfoEvents(SharedEventTypes.PhaseTransition));
Assert.Empty(f.DecisionSignals(DecisionSignalKind.AppInstallCompleted));
// The app genuinely runs in THIS enrollment.
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddMinutes(-2);
adapter.TriggerAppStateFromTest(app, AppInstallationState.Unknown, AppInstallationState.Installing);
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddMinutes(-1);
adapter.TriggerAppStateFromTest(app, AppInstallationState.Installing, AppInstallationState.Installed);
Assert.Single(f.InfoEvents(SharedEventTypes.PhaseTransition)); // sub-phase fired fresh
Assert.Single(f.DecisionSignals(DecisionSignalKind.AppInstallCompleted)); // signal fired fresh
var timing = adapter.AppTimings["app-X"];
Assert.Equal(ClockNow.AddMinutes(-2), timing.StartedAtUtc); // fresh stamps, not stale
Assert.Equal(ClockNow.AddMinutes(-1), timing.CompletedAtUtc);
}
[Fact]
public void Stale_do_telemetry_is_suppressed_fresh_emits()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var app = new AppPackageState("app-do", 0);
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7);
adapter.TriggerDoTelemetryFromTest(app);
Assert.Empty(f.InfoEvents(SharedEventTypes.DoTelemetry));
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddMinutes(-1);
adapter.TriggerDoTelemetryFromTest(app);
Assert.Single(f.InfoEvents(SharedEventTypes.DoTelemetry));
}
[Fact]
public void Oneshot_summary_is_shared_between_script_and_app_suppressions()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "aaa11111",
ScriptType = "platform",
ExitCode = 0,
Result = "Success",
});
adapter.TriggerAppStateFromTest(new AppPackageState("app-old", 0),
AppInstallationState.Installing, AppInstallationState.Installed);
Assert.Single(f.InfoEvents(SharedEventTypes.HistoricImeReplayDetected));
}
[Fact]
public void FutureSkew_app_transition_still_emits()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var app = new AppPackageState("app-skew", 0);
// Mid-enrollment clock jump — genuine current activity, stamped to the clock.
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddHours(2);
adapter.TriggerAppStateFromTest(app, AppInstallationState.Installing, AppInstallationState.Installed);
var info = Assert.Single(f.InfoEvents(SharedEventTypes.AppInstallComplete));
Assert.Equal(ClockNow, info.OccurredAtUtc);
Assert.Equal("true", info.Payload!["derivedTimestamp"]);
Assert.Single(f.DecisionSignals(DecisionSignalKind.AppInstallCompleted));
Assert.Empty(f.InfoEvents(SharedEventTypes.HistoricImeReplayDetected));
}
[Fact]
public void SimulationMode_bypasses_suppression_for_apps_and_scripts()
{
// The --replay-log-dir dev tool intentionally feeds historic lines.
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
f.Tracker.SimulationMode = true;
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7);
adapter.TriggerAppStateFromTest(new AppPackageState("app-sim", 0),
AppInstallationState.Installing, AppInstallationState.Installed);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "bbb22222",
ScriptType = "platform",
ExitCode = 0,
Result = "Success",
});
Assert.Single(f.InfoEvents(SharedEventTypes.AppInstallComplete));
Assert.Single(f.InfoEvents(SharedEventTypes.ScriptCompleted));
Assert.Empty(f.InfoEvents(SharedEventTypes.HistoricImeReplayDetected));
}
}
}
@@ -0,0 +1,257 @@
using System;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
using AutopilotMonitor.Agent.V2.Core.SignalAdapters;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using SharedEventTypes = AutopilotMonitor.Shared.Constants.EventTypes;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
{
/// <summary>
/// Historic-replay handling (session eaf3d8c4): a previous enrollment's IME log surviving
/// on disk made the agent replay week-old script activity — 156 phantom script_completed
/// events with ~170 h durations (clock-clamped completion minus RAW stale start) and an
/// immediate-upload flood. Script events whose source line is &gt; 24 h stale are now
/// suppressed entirely (one-shot <c>historic_ime_replay_detected</c> summary instead),
/// and durations are computed from a timeline-consistent timestamp pair with a 24 h
/// plausibility backstop.
/// </summary>
public sealed class ImeLogTrackerAdapterHistoricReplayTests
{
private static readonly DateTime ClockNow = new DateTime(2026, 7, 23, 15, 42, 0, DateTimeKind.Utc);
// ---------------------------------------------------------------------
// Suppression: stale (> 24 h past) source lines are a previous enrollment
// ---------------------------------------------------------------------
[Fact]
public void Stale_completion_is_suppressed_and_emits_oneshot_summary()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var ancient = ClockNow.AddDays(-7);
f.Tracker.LastMatchedLogTimestamp = ancient;
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "d78c1822",
ScriptType = "platform",
ExitCode = 0,
Result = "Failed",
StartedAtUtc = ancient.AddMinutes(-30),
// Bootstrap marker in stdout — week-old evidence must not drive the
// bootstrap-detected trace either.
Stdout = "Bootstrap script version: v2.0",
});
Assert.Empty(f.InfoEvents(SharedEventTypes.ScriptCompleted));
Assert.Empty(f.InfoEvents(SharedEventTypes.ScriptFailed));
Assert.Empty(f.InfoEvents(SharedEventTypes.ScriptTimeoutSuspected));
Assert.Empty(f.InfoEvents(SharedEventTypes.AgentTrace));
var summary = f.InfoEvent(SharedEventTypes.HistoricImeReplayDetected);
Assert.Equal(ancient.ToString("o"), summary.Payload!["earliestRejectedSourceTimestamp"]);
}
[Fact]
public void Summary_is_oneshot_across_started_and_completed_suppressions()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7);
adapter.TriggerScriptStartedFromTest(new ScriptStartedInfo { PolicyId = "aaa11111", ScriptType = "platform" });
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7).AddMinutes(1);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "aaa11111",
ScriptType = "platform",
ExitCode = 0,
Result = "Success",
});
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7).AddMinutes(2);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "bbb22222",
ScriptType = "remediation",
ScriptPart = "detection",
ExitCode = 0,
ComplianceResult = "True",
});
Assert.Empty(f.InfoEvents(SharedEventTypes.ScriptStarted));
Assert.Empty(f.InfoEvents(SharedEventTypes.ScriptCompleted));
// Exactly ONE summary regardless of how many replayed lines followed — and it dates
// the replay window from the FIRST (earliest) suppressed line.
var summary = Assert.Single(f.InfoEvents(SharedEventTypes.HistoricImeReplayDetected));
Assert.Equal(ClockNow.AddDays(-7).ToString("o"), summary.Payload!["earliestRejectedSourceTimestamp"]);
}
[Fact]
public void Fresh_events_after_suppression_emit_normally()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "aaa11111",
ScriptType = "platform",
ExitCode = 0,
Result = "Success",
});
Assert.Empty(f.InfoEvents(SharedEventTypes.ScriptCompleted));
// The agent catches up to current log content — normal emission resumes.
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddMinutes(-2);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "ccc33333",
ScriptType = "platform",
ExitCode = 0,
Result = "Success",
StartedAtUtc = ClockNow.AddMinutes(-3),
});
var info = Assert.Single(f.InfoEvents(SharedEventTypes.ScriptCompleted));
Assert.Equal("ccc33333", info.Payload!["policyId"]);
Assert.Equal("60.00", info.Payload["durationSeconds"]);
}
[Fact]
public void Clock_fallback_without_source_timestamp_still_emits()
{
// derivedFromClock with NO rejected source ts (synthetic/callback path) is not a
// replay — must emit, and must not trigger the summary.
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "ddd44444",
ScriptType = "platform",
ExitCode = 0,
Result = "Success",
});
var info = Assert.Single(f.InfoEvents(SharedEventTypes.ScriptCompleted));
Assert.Equal("true", info.Payload!["derivedTimestamp"]);
Assert.False(info.Payload.ContainsKey("rejectedSourceTimestamp"));
Assert.Empty(f.InfoEvents(SharedEventTypes.HistoricImeReplayDetected));
}
// ---------------------------------------------------------------------
// Duration: timeline-consistent pair + plausibility backstop
// ---------------------------------------------------------------------
[Fact]
public void FutureSkew_completion_emits_with_clamped_stamp_and_raw_pair_duration()
{
// Mid-enrollment clock jump (WhiteGlove +1h CMTrace skew case): the source ts is
// rejected as future-skewed — NOT suppressed — and the duration must come from the
// raw source pair (both ends on the same skewed timeline), not clamped-now − raw-start.
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var skewed = ClockNow.AddHours(2);
f.Tracker.LastMatchedLogTimestamp = skewed;
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "eee55555",
ScriptType = "platform",
ExitCode = 0,
Result = "Success",
StartedAtUtc = skewed.AddSeconds(-360),
});
var info = Assert.Single(f.InfoEvents(SharedEventTypes.ScriptCompleted));
Assert.Equal(ClockNow, info.OccurredAtUtc); // stamp: clamped to clock
Assert.Equal("true", info.Payload!["derivedTimestamp"]);
Assert.Equal(skewed.ToString("o"), info.Payload["rejectedSourceTimestamp"]);
Assert.Equal("360.00", info.Payload["durationSeconds"]); // duration: raw pair
Assert.Empty(f.InfoEvents(SharedEventTypes.HistoricImeReplayDetected));
}
[Fact]
public void FutureSkew_failed_platform_script_timeout_heuristic_uses_raw_pair()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var skewed = ClockNow.AddHours(2);
f.Tracker.LastMatchedLogTimestamp = skewed;
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "fff66666",
ScriptType = "platform",
ExitCode = 0,
Result = "Failed",
StartedAtUtc = skewed.AddMinutes(-26), // past the 25-min suspicion threshold
});
var suspected = Assert.Single(f.InfoEvents(SharedEventTypes.ScriptTimeoutSuspected));
Assert.Equal("1560.00", suspected.Payload!["durationSeconds"]);
Assert.Equal(ClockNow, suspected.OccurredAtUtc);
}
[Fact]
public void Implausible_duration_above_24h_is_omitted()
{
// Fresh (accepted) completion paired with an ancient start — e.g. a stale pending
// slot a legacy path left behind. The duration would be a cross-run lie; omit it.
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddMinutes(-2);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "abc12345",
ScriptType = "platform",
ExitCode = 0,
Result = "Failed",
StartedAtUtc = ClockNow.AddHours(-30),
});
var info = Assert.Single(f.InfoEvents(SharedEventTypes.ScriptFailed));
Assert.False(info.Payload!.ContainsKey("durationSeconds"));
Assert.False(info.Payload.ContainsKey("durationBasis"));
// The 30 h "duration" must not read as a hung script either.
Assert.Empty(f.InfoEvents(SharedEventTypes.ScriptTimeoutSuspected));
}
// ---------------------------------------------------------------------
// Tracker slot hardening: stale pending start must not pair with a fresh run
// ---------------------------------------------------------------------
[Fact]
public void Fresh_start_discards_stale_pending_slot_from_previous_enrollment()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
// Previous enrollment: replayed start line whose matching result never appeared
// (the script was mid-run when that enrollment ended).
f.Tracker.SeedPendingPlatformScriptForTesting("beef7777", exitCode: null, exitObservedAtUtc: null,
startedAtUtc: ClockNow.AddDays(-7));
// Same policy genuinely re-runs now — the fresh start must begin a NEW execution.
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddSeconds(-90);
f.Tracker.HandlePlatformScriptStarted("beef7777");
f.Tracker.LastMatchedLogTimestamp = ClockNow;
f.Tracker.CompletePlatformScriptFromImeResultForTesting("beef7777", "Success", ClockNow);
var info = f.InfoEvent(SharedEventTypes.ScriptCompleted);
// Without the hardening the slot kept the 7-day-old start: the duration would have
// been dropped by the 24 h backstop (or, before that, shown as ~168 h).
Assert.Equal("90.00", info.Payload!["durationSeconds"]);
}
}
}
@@ -208,39 +208,60 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
}
else
{
// Platform script: AgentExecutor.log entries create/enrich, IME log entries also create
if (!_pendingPlatformScripts.ContainsKey(id))
{
// A fresh start for this policy (no pending entry) begins a NEW execution.
// Clear any emitted-marker from a prior run so IME re-evaluations / retries of
// the same platform-script policy within one agent lifetime are not silently
// deduped away. Within a single run the start always precedes exit/result, so
// this never clears the current run's own marker.
_platformScriptResultEmitted.Remove(id);
_pendingPlatformScripts[id] = new ScriptExecutionState
{
PolicyId = id,
ScriptType = "platform",
// Prefer the source CMTrace timestamp so a script that started before the
// agent launched (replayed log content) is dated to its real start, not now.
// Set only at slot creation: the start line fires twice (agentexecutor + ime
// source) but the earliest wins.
StartedAtUtc = LastMatchedLogTimestamp ?? DateTime.UtcNow,
};
// Live "running" indicator — same signal health scripts emit. Gated on slot
// creation so the duplicate start line (agentexecutor + ime source) doesn't
// double-emit for the same execution.
try { OnScriptStarted?.Invoke(new ScriptStartedInfo { PolicyId = id, ScriptType = "platform" }); }
catch (Exception ex) { _logger.Warning($"ImeLogTracker: OnScriptStarted handler threw: {ex.Message}"); }
}
_lastPlatformScriptPolicyId = id;
// Started lines fire twice per script (agentexecutor + ime source) and carry no
// outcome — the matching `platform script completed` line below carries result+exit
// and stays on Info. Keep starts on Debug so Info reflects script outcomes only.
_logger.Debug($"ImeLogTracker: platform script started: {id} (source: {source ?? "ime"})");
HandlePlatformScriptStarted(id, source);
}
}
// Platform script: AgentExecutor.log entries create/enrich, IME log entries also create.
// Internal so the stale-slot hardening is directly testable (the public path arrives via
// a regex Match).
internal void HandlePlatformScriptStarted(string id, string source = null)
{
// Stale-slot hardening (session eaf3d8c4): a replayed start line from a previous
// enrollment whose matching result never appeared (script was mid-run when that
// enrollment ended) leaves a pending slot with a days-old StartedAtUtc. When the
// same policy genuinely re-runs now, the fresh start must begin a NEW execution —
// otherwise the ancient start pairs with the fresh completion and the duration
// lies. > 24 h mirrors the adapter's source-timestamp staleness clamp.
var currentStartTs = LastMatchedLogTimestamp ?? DateTime.UtcNow;
if (_pendingPlatformScripts.TryGetValue(id, out var pendingSlot)
&& pendingSlot.StartedAtUtc.HasValue
&& currentStartTs - pendingSlot.StartedAtUtc.Value > TimeSpan.FromHours(24))
{
_logger.Debug($"ImeLogTracker: discarding stale pending platform-script slot for {id} (started {pendingSlot.StartedAtUtc:o})");
_pendingPlatformScripts.Remove(id);
}
if (!_pendingPlatformScripts.ContainsKey(id))
{
// A fresh start for this policy (no pending entry) begins a NEW execution.
// Clear any emitted-marker from a prior run so IME re-evaluations / retries of
// the same platform-script policy within one agent lifetime are not silently
// deduped away. Within a single run the start always precedes exit/result, so
// this never clears the current run's own marker.
_platformScriptResultEmitted.Remove(id);
_pendingPlatformScripts[id] = new ScriptExecutionState
{
PolicyId = id,
ScriptType = "platform",
// Prefer the source CMTrace timestamp so a script that started before the
// agent launched (replayed log content) is dated to its real start, not now.
// Set only at slot creation: the start line fires twice (agentexecutor + ime
// source) but the earliest wins.
StartedAtUtc = currentStartTs,
};
// Live "running" indicator — same signal health scripts emit. Gated on slot
// creation so the duplicate start line (agentexecutor + ime source) doesn't
// double-emit for the same execution.
try { OnScriptStarted?.Invoke(new ScriptStartedInfo { PolicyId = id, ScriptType = "platform" }); }
catch (Exception ex) { _logger.Warning($"ImeLogTracker: OnScriptStarted handler threw: {ex.Message}"); }
}
_lastPlatformScriptPolicyId = id;
// Started lines fire twice per script (agentexecutor + ime source) and carry no
// outcome — the matching `platform script completed` line below carries result+exit
// and stays on Info. Keep starts on Debug so Info reflects script outcomes only.
_logger.Debug($"ImeLogTracker: platform script started: {id} (source: {source ?? "ime"})");
}
// Routes line-by-line script-data handlers (context / exitCode / output) to the right
// accumulator based on the pattern's scriptType parameter. Health-script lines feed the
// single-slot _pendingHealthScript; platform-script lines feed _pendingPlatformScripts.
@@ -194,6 +194,27 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
}
}
// Actions that mutate app/phase tracking state (_packageStates, _phasePackageSnapshots,
// _currentPhaseOrder, DO telemetry) — the historic-replay guard skips these for source
// lines from a previous enrollment so replayed apps never enter tracked state,
// persistence, app_tracking_summary, culprit lists or final-status. Script actions are
// deliberately NOT here: their tracker state is harmless (stale-slot hardening covers
// leftovers) and the adapter suppresses their emissions. espphasedetected IS here — a
// stale "In EspPhase: AccountSetup" would advance _currentPhaseOrder and make the fresh
// enrollment's DeviceSetup lines bounce as "backward"; IME re-logs the phase constantly,
// so fresh lines re-deliver it immediately.
private static readonly HashSet<string> AppMutatingActions = new HashSet<string>(StringComparer.Ordinal)
{
"imeshutdown", "espphasedetected", "setcurrentapp",
"updatestateinstalled", "updatestatedownloading", "updatestateinstalling",
"updatestateskipped", "updatestateerror", "updatestatepostponed",
"captureexitcode", "capturehresult", "captureappversion",
"captureapptypewinget", "captureapptypemsi", "captureattemptnumber",
"capturedetectionresult", "esptrackstatus", "policiesdiscovered",
"ignorecompletedapp", "updatename", "updatewin32appstate",
"cancelstuckandsetcurrent", "updatedotelemetry",
};
private void HandlePatternMatch(CompiledPattern pattern, Match match, string message, CmTraceLogEntry entry)
{
LastMatchedPatternId = pattern.PatternId;
@@ -206,6 +227,19 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
try { OnPatternMatched?.Invoke(pattern.PatternId); }
catch (Exception ex) { _logger?.Warning($"OnPatternMatched handler threw: {ex.Message}"); }
// Historic-replay guard (session eaf3d8c4): a source line > 24 h older than now is
// content from a previous enrollment whose IME log survived on disk. App-mutating
// actions are skipped BEFORE _seenAppIds so replayed apps poison neither the tracked
// state nor the phase-change ignore list. SimulationMode (--replay-log-dir dev tool)
// replays historic logs on purpose and bypasses the guard.
var isStaleReplayLine = !SimulationMode && entry != null
&& (UtcNowProvider() - NormalizeUtc(entry.Timestamp)) > HistoricReplayThreshold;
if (isStaleReplayLine && AppMutatingActions.Contains(pattern.Action?.ToLower() ?? string.Empty))
{
_logger.Debug($"ImeLogTracker: skipped app action '{pattern.Action}' for historic line ({entry.Timestamp:o})");
return;
}
try
{
var id = match.Groups["id"]?.Value;
@@ -71,6 +71,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
public double SpeedFactor { get; set; } = 50;
private DateTime _lastLogTimestamp = DateTime.MinValue;
// Source log lines older than this relative to now are content from a previous
// enrollment (IME logs surviving a re-enrollment, session eaf3d8c4). Single source of
// truth shared with the adapter's ResolveOccurredAt clamp and replay suppression.
internal static readonly TimeSpan HistoricReplayThreshold = TimeSpan.FromHours(24);
// Deterministic-time seam for the historic-replay guard; production default is the
// system clock, ImeLogHost rewires it to the agent IClock, tests to their VirtualClock.
internal Func<DateTime> UtcNowProvider { get; set; } = () => DateTime.UtcNow;
/// <summary>
/// CMTrace timestamps from IME are emitted in UTC, but DateTimeKind may arrive as
/// Unspecified depending on the parser path. Normalize via SpecifyKind=Utc rather than
/// ToUniversalTime() to avoid double-conversion when Kind is already UTC.
/// </summary>
internal static DateTime NormalizeUtc(DateTime value) =>
value.Kind == DateTimeKind.Utc ? value : DateTime.SpecifyKind(value, DateTimeKind.Utc);
// Background task
private Task _pollingTask;
private CancellationTokenSource _cts;
@@ -61,6 +61,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
internal Func<int, string> SessionOwnerResolver { get; set; }
internal Func<int, string> ProcessOwnerResolver { get; set; }
// Test seam for the WinRT OOBE-state sample (null = OobeStateReader.Read).
internal Func<string> OobeStateProvider { get; set; }
// OOBE-state flip tracking (observational only — see OobeStateReader contract).
// _oobeCompletedEmitted is one-shot per agent lifetime and deliberately NOT reset
// by ResetForRealUserSwitch: the OOBE flip is a global OS transition, not per-user.
private string _lastOobeState;
private bool _oobeCompletedEmitted;
/// <summary>
/// System/service account names that should NOT be considered real users.
/// </summary>
@@ -161,6 +170,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
_startedAtUtc = DateTime.UtcNow;
// Baseline OOBE-state sample so the first poll can already detect a flip
// (self-contained try/catch inside, like the Emit* helpers).
CheckOobeStateFlip();
_pollingTimer?.Dispose();
_pollingTimer = new Timer(
PollForDesktop,
@@ -186,6 +199,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
_pollCount++;
CheckOobeStateFlip();
try
{
var explorerProcesses = Process.GetProcessesByName("explorer");
@@ -381,6 +396,66 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
catch (Exception ex) { _logger.Verbose($"DesktopArrivalDetector: detector-started emit failed: {ex.Message}"); }
}
/// <summary>
/// Samples the WinRT OOBE state (<see cref="OobeStateReader"/>) and emits the one-shot
/// <c>oobe_state_completed</c> trace event on the in_progress-&gt;completed flip.
/// Observational only — an owner-independent desktop corroboration for sessions where
/// WTS+WMI owner resolution starves and <c>desktop_arrived</c> never fires. Often absent
/// in healthy sessions: the polling timer is disposed at desktop arrival, which is when
/// the flip lands (empirical probe, session 9c404ae9).
/// <para>
/// Self-contained try/catch: a throw here must NOT reach <see cref="PollForDesktop"/>'s
/// outer catch, which would inflate <see cref="_wmiErrorCountSinceStart"/> and skew the
/// session-4d5a0b78 liveness diagnostics. "unavailable" readings never overwrite the last
/// known state, so a transient reflection failure between two polls cannot mask the flip.
/// One-shot flag survives <see cref="ResetForRealUserSwitch"/> — the flip is a global OS
/// transition, not per-user.
/// </para>
/// </summary>
private void CheckOobeStateFlip()
{
try
{
var current = (OobeStateProvider ?? OobeStateReader.Read)();
if (string.IsNullOrEmpty(current) || current == OobeStateReader.Unavailable)
return;
if (!_oobeCompletedEmitted
&& _lastOobeState == "in_progress"
&& current == "completed")
{
_oobeCompletedEmitted = true;
var minutesSinceStart = Math.Round((DateTime.UtcNow - _startedAtUtc).TotalMinutes, 1);
_logger.Info("DesktopArrivalDetector: OOBE state flipped in_progress->completed (WinRT SystemSetupInfo)");
OnTraceEvent?.Invoke(
SharedEventTypes.OobeStateCompleted,
$"OOBE state flipped in_progress->completed (WinRT SystemSetupInfo, poll {_pollCount}) — owner-independent desktop corroboration.",
new Dictionary<string, object>
{
{ "previousState", "in_progress" },
{ "pollCount", _pollCount },
{ "minutesSinceStart", minutesSinceStart },
{ "desktopArrived", _desktopArrived },
{ "wmiErrorCount", _wmiErrorCountSinceStart },
{ "wtsErrorCount", _wtsErrorCountSinceStart },
});
}
_lastOobeState = current;
}
catch (Exception ex)
{
_logger.Verbose($"DesktopArrivalDetector: OOBE state check failed: {ex.Message}");
}
}
/// <summary>
/// Drives a single synchronous poll for unit tests — the production poll is private and
/// timer-driven with a fixed 30 s interval (precedent: <see cref="ResolveOwner"/> is
/// likewise internal for direct testing).
/// </summary>
internal void PollOnceForTest() => PollForDesktop(null);
/// <summary>
/// Resolves the owning user of an explorer.exe candidate. Session 4d5a0b78 fix
/// (2026-06-11): the WMI <c>Win32_Process.GetOwner</c> call failed on every poll on
@@ -1,4 +1,5 @@
using System;
using System.Collections.Generic;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
@@ -38,16 +39,30 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
/// </summary>
public string Category { get; }
/// <summary>
/// Session 4910a5a5: names of the tracked apps most likely behind an Apps-subcategory
/// failure (capped, never-started apps ranked first — those starve ESP's app gate
/// without ever producing an IME install event). Snapshot from
/// <c>ProvisioningStatusTracker.SnapshotTrackedAppsNotCompleted</c> at failure-arming
/// time; empty for non-Apps failures and event-log-derived failures. Propagated onto
/// the EspTerminalFailure signal payload so the DecisionEngine's
/// <c>esp_failure_advisory</c> / <c>enrollment_failed</c> events can name the culprit
/// directly.
/// </summary>
public IReadOnlyList<string> LikelyCulpritApps { get; }
public EspFailureDetectedEventArgs(
string failureType,
string errorCode = null,
string failedSubcategory = null,
string category = null)
string category = null,
IReadOnlyList<string> likelyCulpritApps = null)
{
FailureType = string.IsNullOrEmpty(failureType) ? "unknown" : failureType;
ErrorCode = string.IsNullOrEmpty(errorCode) ? null : errorCode;
FailedSubcategory = string.IsNullOrEmpty(failedSubcategory) ? null : failedSubcategory;
Category = string.IsNullOrEmpty(category) ? null : category;
LikelyCulpritApps = likelyCulpritApps ?? Array.Empty<string>();
}
}
}
@@ -0,0 +1,301 @@
using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using Microsoft.Win32;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
/// <summary>
/// Session 5d735290 (2026-07-23) — reads the ESP's registered policy providers from
/// <c>HKLM\SOFTWARE\Microsoft\Windows\Autopilot\EnrollmentStatusTracking</c>, the registry
/// mirror of the EnrollmentStatusTracking CSP
/// (https://learn.microsoft.com/en-us/windows/client-management/mdm/enrollmentstatustracking-csp).
/// <para>
/// CSP contract the ESP enforces (quotes from the doc above):
/// <list type="bullet">
/// <item><c>Setup/Apps/PolicyProviders/&lt;Name&gt;</c> — "Existence of this node indicates
/// to the ESP that it shouldn't show the tracking status message until the
/// TrackingPoliciesCreated node has been set to true." This wait has NO timeout: a
/// registered provider that never sets <c>TrackingPoliciesCreated=1</c> parks the ESP at
/// "Apps (Identifying)" indefinitely (observed for days in the field).</item>
/// <item><c>DevicePreparation/PolicyProviders/&lt;Name&gt;/InstallationState</c> — must reach
/// 2 (NotRequired) or 3 (Completed); 1=NotInstalled, 4=Error. ESP applies a default
/// 15-minute timeout here, so stalls in this stage are self-surfacing — still reported
/// for the diagnostic trail.</item>
/// </list>
/// Registry layout: <c>Device</c> subkey for device scope, <c>S-&lt;SID&gt;</c> subkeys for user
/// scope (appear at sign-in), each containing <c>Setup\Apps\PolicyProviders\&lt;Name&gt;</c>;
/// <c>DevicePreparation\PolicyProviders</c> exists in device scope only.
/// </para>
/// <para>
/// Known provider names: <c>Sidecar</c> is the Intune Management Extension ("Intune's agents,
/// such as SideCar" per the CSP doc); <c>ConfigMgr</c> is the Configuration Manager client
/// (registered during Autopilot-into-co-management, see the troubleshooting registry paths in
/// https://learn.microsoft.com/en-us/intune/configmgr/comanage/autopilot-enrollment). Microsoft
/// warns that ESP policy providers are not aware of each other and that pre-provisioning is
/// unsupported with co-management. Field case that motivated this probe: a pre-provisioned
/// device with the ConfigMgr client installed as a blocking app ended up with ONLY
/// <c>ConfigMgr</c> registered under <c>Device\Setup\Apps\PolicyProviders</c> (no
/// <c>Sidecar</c>) and <c>TrackingPoliciesCreated</c> never set — the user ESP hung at
/// "Apps (Identifying)" for hours to days; manually renaming the key to <c>Sidecar</c>
/// unblocked it immediately. <see cref="EspPolicyProviderStallDetector"/> turns this contract
/// into the <c>esp_policy_provider_stalled</c> tripwire.
/// </para>
/// <para>
/// Fail-soft like <see cref="EspTrackingInfoProbe"/>: any failure is Debug-logged and yields
/// <see cref="EspPolicyProviderSnapshot.Empty"/>; a missing root key (early enrollment,
/// non-Autopilot device) is not an error. Values are parsed defensively — a missing or
/// unparseable value counts as incomplete (fail-toward-observation).
/// </para>
/// </summary>
internal static class EspPolicyProviderProbe
{
internal const string TrackingRootKeyPath =
@"SOFTWARE\Microsoft\Windows\Autopilot\EnrollmentStatusTracking";
/// <summary>Provider kind: <c>Setup\Apps\PolicyProviders</c> (TrackingPoliciesCreated wait, no timeout).</summary>
internal const string KindSetupApps = "setupApps";
/// <summary>Provider kind: <c>DevicePreparation\PolicyProviders</c> (InstallationState wait, 15-min OS timeout).</summary>
internal const string KindDevicePreparation = "devicePreparation";
/// <summary>The IME's provider name — its absence while other providers are registered is the field-case signature.</summary>
internal const string SidecarProviderName = "Sidecar";
internal const string DeviceScope = "device";
private const string TrackingPoliciesCreatedValueName = "TrackingPoliciesCreated";
private const string InstallationStateValueName = "InstallationState";
/// <summary>
/// Test seam — when non-null, <see cref="Read"/> delegates to this func instead of
/// touching the live registry. Use <see cref="ScopedOverride"/> to guarantee cleanup.
/// </summary>
internal static Func<AgentLogger, EspPolicyProviderSnapshot> TestOverride;
/// <summary>
/// Reads all registered ESP policy providers. <see cref="EspPolicyProviderSnapshot.HasData"/>
/// is <c>true</c> when the EnrollmentStatusTracking root key existed and was readable
/// (even with no providers registered); <see cref="EspPolicyProviderSnapshot.Empty"/> otherwise.
/// </summary>
/// <param name="logger">Optional logger — Debug-level trace only; no warn/error.</param>
public static EspPolicyProviderSnapshot Read(AgentLogger logger = null)
{
var probe = TestOverride;
if (probe != null) return probe(logger);
try
{
using (var rootKey = Registry.LocalMachine.OpenSubKey(TrackingRootKeyPath))
{
if (rootKey == null)
{
logger?.Debug("EspPolicyProviderProbe: EnrollmentStatusTracking registry key not found");
return EspPolicyProviderSnapshot.Empty;
}
var providers = new List<PolicyProviderState>();
foreach (var scopeName in rootKey.GetSubKeyNames())
{
var isDevice = string.Equals(scopeName, "Device", StringComparison.OrdinalIgnoreCase);
// User scope repeats the Setup structure under the signed-in user's SID.
var isUserSid = scopeName.StartsWith("S-", StringComparison.OrdinalIgnoreCase);
if (!isDevice && !isUserSid) continue;
var scopeLabel = isDevice ? DeviceScope : "user:" + scopeName;
using (var scopeKey = rootKey.OpenSubKey(scopeName))
{
if (scopeKey == null) continue;
CollectProviders(
scopeKey, @"Setup\Apps\PolicyProviders", scopeLabel, KindSetupApps,
TrackingPoliciesCreatedValueName, providers, logger);
if (isDevice)
{
CollectProviders(
scopeKey, @"DevicePreparation\PolicyProviders", scopeLabel, KindDevicePreparation,
InstallationStateValueName, providers, logger);
}
}
}
return new EspPolicyProviderSnapshot(providers);
}
}
catch (Exception ex)
{
logger?.Debug($"EspPolicyProviderProbe: read threw: {ex.Message}");
return EspPolicyProviderSnapshot.Empty;
}
}
private static void CollectProviders(
RegistryKey scopeKey,
string providersSubPath,
string scopeLabel,
string kind,
string stateValueName,
List<PolicyProviderState> providers,
AgentLogger logger)
{
try
{
using (var providersKey = scopeKey.OpenSubKey(providersSubPath))
{
if (providersKey == null) return;
foreach (var providerName in providersKey.GetSubKeyNames())
{
using (var providerKey = providersKey.OpenSubKey(providerName))
{
if (providerKey == null) continue;
var value = TryReadInt(providerKey.GetValue(stateValueName));
providers.Add(new PolicyProviderState(
name: providerName,
scope: scopeLabel,
kind: kind,
trackingPoliciesCreated: kind == KindSetupApps ? value : null,
installationState: kind == KindDevicePreparation ? value : null));
}
}
}
}
catch (Exception ex)
{
logger?.Debug($"EspPolicyProviderProbe: '{providersSubPath}' ({scopeLabel}) read threw: {ex.Message}");
}
}
/// <summary>
/// Defensive value parse: REG_DWORD expected per CSP (booleans stored as 0/1), but string
/// forms are tolerated. Anything missing or unparseable is <c>null</c> — which the
/// completion rule treats as incomplete.
/// </summary>
internal static int? TryReadInt(object rawValue)
{
switch (rawValue)
{
case null: return null;
case int i: return i;
case long l when l >= int.MinValue && l <= int.MaxValue: return (int)l;
case string s when int.TryParse(s.Trim(), NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsed):
return parsed;
case bool b: return b ? 1 : 0;
default: return null;
}
}
/// <summary>
/// Pure completion rule per the CSP contract: <c>setupApps</c> providers are complete when
/// <c>TrackingPoliciesCreated == 1</c>; <c>devicePreparation</c> providers when
/// <c>InstallationState</c> is 2 (NotRequired) or 3 (Completed).
/// </summary>
internal static bool IsProviderComplete(string kind, int? trackingPoliciesCreated, int? installationState)
=> kind == KindSetupApps
? trackingPoliciesCreated == 1
: installationState == 2 || installationState == 3;
/// <summary>
/// Disposable scope that sets <see cref="TestOverride"/> for the lifetime of the scope
/// and restores the previous value on Dispose. Nestable. Internal test-only helper.
/// </summary>
internal sealed class ScopedOverride : IDisposable
{
private readonly Func<AgentLogger, EspPolicyProviderSnapshot> _previous;
private int _disposed;
public ScopedOverride(Func<AgentLogger, EspPolicyProviderSnapshot> probe)
{
_previous = TestOverride;
TestOverride = probe;
}
public void Dispose()
{
if (System.Threading.Interlocked.Exchange(ref _disposed, 1) == 1) return;
TestOverride = _previous;
}
}
}
/// <summary>
/// One registered ESP policy provider with its completion-relevant state.
/// <see cref="Key"/> is the stable identity used for dwell tracking and one-shot latching.
/// </summary>
internal readonly struct PolicyProviderState
{
public PolicyProviderState(string name, string scope, string kind, int? trackingPoliciesCreated, int? installationState)
{
Name = name ?? string.Empty;
Scope = scope ?? string.Empty;
Kind = kind ?? string.Empty;
TrackingPoliciesCreated = trackingPoliciesCreated;
InstallationState = installationState;
IsComplete = EspPolicyProviderProbe.IsProviderComplete(kind, trackingPoliciesCreated, installationState);
}
/// <summary>Provider key name, e.g. "Sidecar" or "ConfigMgr".</summary>
public string Name { get; }
/// <summary><c>"device"</c> or <c>"user:S-1-5-21-…"</c>.</summary>
public string Scope { get; }
/// <summary><see cref="EspPolicyProviderProbe.KindSetupApps"/> or <see cref="EspPolicyProviderProbe.KindDevicePreparation"/>.</summary>
public string Kind { get; }
/// <summary>Raw <c>TrackingPoliciesCreated</c> value (setupApps only); <c>null</c> when missing/unparseable.</summary>
public int? TrackingPoliciesCreated { get; }
/// <summary>Raw <c>InstallationState</c> value (devicePreparation only); <c>null</c> when missing/unparseable.</summary>
public int? InstallationState { get; }
/// <summary>Completion per <see cref="EspPolicyProviderProbe.IsProviderComplete"/>.</summary>
public bool IsComplete { get; }
/// <summary>Stable dwell/latch identity: <c>kind|scope|name</c>.</summary>
public string Key => Kind + "|" + Scope + "|" + Name;
}
/// <summary>
/// Snapshot of all registered ESP policy providers. <see cref="HasData"/> is <c>false</c> only
/// when the EnrollmentStatusTracking root key was missing or unreadable — an empty provider
/// list with <see cref="HasData"/> = <c>true</c> is valid data (nothing registered yet).
/// </summary>
internal readonly struct EspPolicyProviderSnapshot
{
public static readonly EspPolicyProviderSnapshot Empty = default;
public EspPolicyProviderSnapshot(IReadOnlyList<PolicyProviderState> providers)
{
HasData = true;
Providers = providers ?? Array.Empty<PolicyProviderState>();
}
/// <summary><c>true</c> when the EnrollmentStatusTracking root key existed and was readable.</summary>
public bool HasData { get; }
/// <summary>All registered providers across scopes and kinds; empty when none registered.</summary>
public IReadOnlyList<PolicyProviderState> Providers { get; }
/// <summary>
/// <c>true</c> when a <c>Setup\Apps</c> provider named "Sidecar" (the IME) is registered in
/// any scope — its absence while other providers are registered is the co-management
/// field-case signature.
/// </summary>
public bool SidecarRegistered
{
get
{
var providers = Providers;
if (providers == null) return false;
return providers.Any(p =>
p.Kind == EspPolicyProviderProbe.KindSetupApps &&
string.Equals(p.Name, EspPolicyProviderProbe.SidecarProviderName, StringComparison.OrdinalIgnoreCase));
}
}
}
}
@@ -0,0 +1,226 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Persistence;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
/// <summary>
/// Turns the ESP policy-provider contract read by <see cref="EspPolicyProviderProbe"/> into
/// the one-shot <c>esp_policy_provider_stalled</c> Warning: a registered provider that stays
/// continuously incomplete for <see cref="DwellMinutes"/> minutes is reported once, with the
/// full provider table in the payload. See the probe's header for the CSP contract, the
/// official sources, and the co-management field case (ConfigMgr registered without Sidecar
/// → user ESP parked at "Apps (Identifying)" with no OS timeout) this tripwire exists for.
/// <para>
/// Dwell semantics: the dwell clock per provider starts when it is first observed incomplete
/// and RESETS when the provider completes or disappears — only <i>continuously</i> incomplete
/// providers fire. A missing root key / empty provider list is normal early enrollment and
/// clears all dwell state (probe D4). Purely observational: emission goes through
/// <see cref="InformationalEventPost"/> (InformationalEvent pass-through, dispatch-guard
/// exempt), never into decision state.
/// </para>
/// <para>
/// Duplicate suppression: in-process one-shot per provider key, plus a
/// <see cref="StartupEventGate"/> fingerprint over the sorted set of currently-stalled keys so
/// an agent restart into the same stall does not re-report it — while a NEW provider joining
/// the stall changes the fingerprint and is reported (the event carries the full list).
/// </para>
/// </summary>
internal sealed class EspPolicyProviderStallDetector
{
/// <summary>
/// Wall-clock minutes a provider must stay continuously incomplete before it is reported.
/// Mirrors the CSP's own DevicePreparation default timeout; the Setup/Apps wait this
/// tripwire mainly targets has no OS timeout at all.
/// </summary>
internal const int DwellMinutes = 15;
internal const string SourceLabel = "EspPolicyProviderStallDetector";
private readonly string _sessionId;
private readonly string _tenantId;
private readonly InformationalEventPost _post;
private readonly AgentLogger _logger;
private readonly IClock _clock;
private readonly StartupEventGate? _startupGate;
private readonly TimeSpan _dwell;
private readonly Dictionary<string, DateTime> _firstSeenIncompleteUtc =
new Dictionary<string, DateTime>(StringComparer.OrdinalIgnoreCase);
private readonly HashSet<string> _firedKeys =
new HashSet<string>(StringComparer.OrdinalIgnoreCase);
public EspPolicyProviderStallDetector(
string sessionId,
string tenantId,
InformationalEventPost post,
AgentLogger logger,
IClock clock,
StartupEventGate? startupGate = null,
TimeSpan? dwell = null)
{
_sessionId = sessionId;
_tenantId = tenantId;
_post = post ?? throw new ArgumentNullException(nameof(post));
_logger = logger;
_clock = clock ?? throw new ArgumentNullException(nameof(clock));
_startupGate = startupGate;
_dwell = dwell ?? TimeSpan.FromMinutes(DwellMinutes);
}
/// <summary>Host timer callback and test seam — one probe + evaluation round.</summary>
internal void Tick()
{
var snapshot = EspPolicyProviderProbe.Read(_logger);
var eval = EvaluateStallTransition(snapshot, _firstSeenIncompleteUtc, _firedKeys, _clock.UtcNow, _dwell);
_firstSeenIncompleteUtc.Clear();
foreach (var kv in eval.UpdatedFirstSeenUtc)
_firstSeenIncompleteUtc[kv.Key] = kv.Value;
if (eval.NewlyStalled.Count > 0)
EmitStalled(snapshot, eval);
}
/// <summary>
/// Pure evaluation of one probe round — the entire detection decision, testable without
/// registry, timer, or ingress (pattern: <c>PerformanceCollector.EvaluateDiskLowTransition</c>).
/// </summary>
/// <param name="snapshot">Current provider snapshot.</param>
/// <param name="firstSeenIncompleteUtc">Dwell state from the previous round (provider key → first-seen-incomplete).</param>
/// <param name="alreadyFired">Provider keys already reported this run (never re-report).</param>
/// <param name="nowUtc">Evaluation timestamp.</param>
/// <param name="dwell">Continuous-incompleteness threshold.</param>
internal static StallEvaluation EvaluateStallTransition(
EspPolicyProviderSnapshot snapshot,
IReadOnlyDictionary<string, DateTime> firstSeenIncompleteUtc,
ISet<string> alreadyFired,
DateTime nowUtc,
TimeSpan dwell)
{
var result = new StallEvaluation();
// Missing root key or no registered providers = normal early enrollment: clear all
// dwell state (a provider must be continuously OBSERVED incomplete to accrue dwell).
if (!snapshot.HasData || snapshot.Providers.Count == 0)
return result;
foreach (var provider in snapshot.Providers)
{
if (provider.IsComplete)
continue; // completed → falls out of the dwell map (reset)
var firstSeen = firstSeenIncompleteUtc.TryGetValue(provider.Key, out var seen)
? seen
: nowUtc;
result.UpdatedFirstSeenUtc[provider.Key] = firstSeen;
var stalledFor = nowUtc - firstSeen;
if (stalledFor >= dwell && !alreadyFired.Contains(provider.Key))
result.NewlyStalled.Add(new StalledProvider(provider, stalledFor.TotalMinutes));
}
return result;
}
private void EmitStalled(EspPolicyProviderSnapshot snapshot, StallEvaluation eval)
{
foreach (var stalled in eval.NewlyStalled)
_firedKeys.Add(stalled.Provider.Key);
// Cross-restart dedup: fingerprint = sorted set of all fired keys that are still
// incomplete. A restart into the same stall re-accrues dwell and lands on the same
// fingerprint → suppressed; a new provider joining the stall changes it → re-emitted.
var currentlyStalledKeys = snapshot.Providers
.Where(p => !p.IsComplete && _firedKeys.Contains(p.Key))
.Select(p => p.Key)
.OrderBy(k => k, StringComparer.OrdinalIgnoreCase)
.ToList();
var fingerprint = StartupEventGate.HashString(string.Join(";", currentlyStalledKeys));
if (_startupGate != null &&
!_startupGate.ShouldEmit(Constants.EventTypes.EspPolicyProviderStalled, fingerprint))
{
_logger?.Debug(
"EspPolicyProviderStallDetector: stalled provider set already reported by a " +
"previous agent run (fingerprint match) — suppressed.");
return;
}
var stalledNames = string.Join(", ",
eval.NewlyStalled.Select(s => $"{s.Provider.Name} ({s.Provider.Scope}/{s.Provider.Kind})"));
var data = new Dictionary<string, object>(StringComparer.Ordinal)
{
["dwellMinutes"] = DwellMinutes,
["sidecarRegistered"] = snapshot.SidecarRegistered,
["providers"] = snapshot.Providers.Select(p => new Dictionary<string, object?>(StringComparer.Ordinal)
{
["name"] = p.Name,
["scope"] = p.Scope,
["kind"] = p.Kind,
["trackingPoliciesCreated"] = p.TrackingPoliciesCreated,
["installationState"] = p.InstallationState,
["complete"] = p.IsComplete,
}).ToList(),
["stalledProviders"] = eval.NewlyStalled.Select(s => new Dictionary<string, object>(StringComparer.Ordinal)
{
["name"] = s.Provider.Name,
["scope"] = s.Provider.Scope,
["kind"] = s.Provider.Kind,
["stalledForMinutes"] = Math.Round(s.StalledForMinutes, 1),
}).ToList(),
};
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
Timestamp = _clock.UtcNow,
EventType = Constants.EventTypes.EspPolicyProviderStalled,
Severity = EventSeverity.Warning,
Source = SourceLabel,
Phase = EnrollmentPhase.Unknown,
ImmediateUpload = true,
Message = $"ESP policy provider(s) incomplete for >= {DwellMinutes} min: {stalledNames} " +
$"(sidecarRegistered={snapshot.SidecarRegistered}) — the ESP waits on " +
"Setup/Apps providers without any timeout",
Data = data
});
_startupGate?.MarkEmitted(Constants.EventTypes.EspPolicyProviderStalled);
_logger?.Warning(
$"EspPolicyProviderStallDetector: {stalledNames} incomplete for >= {DwellMinutes} min " +
$"(sidecarRegistered={snapshot.SidecarRegistered}).");
}
/// <summary>Result of one <see cref="EvaluateStallTransition"/> round.</summary>
internal sealed class StallEvaluation
{
/// <summary>Next round's dwell state — only currently-incomplete providers survive.</summary>
public Dictionary<string, DateTime> UpdatedFirstSeenUtc { get; } =
new Dictionary<string, DateTime>(StringComparer.OrdinalIgnoreCase);
/// <summary>Providers that crossed the dwell threshold this round and were not yet reported.</summary>
public List<StalledProvider> NewlyStalled { get; } = new List<StalledProvider>();
}
internal readonly struct StalledProvider
{
public StalledProvider(PolicyProviderState provider, double stalledForMinutes)
{
Provider = provider;
StalledForMinutes = stalledForMinutes;
}
public PolicyProviderState Provider { get; }
public double StalledForMinutes { get; }
}
}
}
@@ -116,6 +116,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
// failure details that fire after the settle window expires.
private Dictionary<string, Timer> _provisioningFailureSettleTimers;
private Dictionary<string, EspFailureDetectedEventArgs> _provisioningFailureSettleArgs;
// Session 4910a5a5 — post-terminal recovery story. Populated when the settle window
// expires and EspFailureDetected actually fires (terminal path); consumed by the
// esp_failure_retry_detected / esp_failure_recovered emitters so a late "Try again"
// recovery (23 min after the failure in the field session) is visible on the timeline
// instead of hiding inside a generic subcategory_state_change. Keyed by category-name.
private Dictionary<string, EspFailureDetectedEventArgs> _provisioningFailureTerminalArgs;
private Dictionary<string, DateTime> _provisioningFailureTerminalUtc;
private HashSet<string> _postFailureRetryEmitted;
private HashSet<string> _postFailureRecoveredEmitted;
// Session 2bc884b6: AppX enrichment scan seams + fire-once guard. The scanner and the
// dispatcher are injectable so tests never touch the real event log or the threadpool.
private readonly IAppxDeploymentFailureScanner _appxScanner;
@@ -285,6 +294,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
_lastSubcategoryStates = new Dictionary<string, Dictionary<string, string>>(StringComparer.OrdinalIgnoreCase);
_provisioningFailureSettleTimers = new Dictionary<string, Timer>(StringComparer.OrdinalIgnoreCase);
_provisioningFailureSettleArgs = new Dictionary<string, EspFailureDetectedEventArgs>(StringComparer.OrdinalIgnoreCase);
_provisioningFailureTerminalArgs = new Dictionary<string, EspFailureDetectedEventArgs>(StringComparer.OrdinalIgnoreCase);
_provisioningFailureTerminalUtc = new Dictionary<string, DateTime>(StringComparer.OrdinalIgnoreCase);
_postFailureRetryEmitted = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
_postFailureRecoveredEmitted = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
_deviceSetupProvisioningCompleteFired = false;
_deviceSetupFallbackTimer = null;
_accountSetupProvisioningCompleteFired = false;
@@ -665,6 +678,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
// null-ing here is correctness-neutral on the start-condition side.
ResetAccountSetupFallbackTimer();
_accountSetupProvisioningCompleteFired = true;
MaybeEmitPostFailureRecovered("AccountSetupCategory.Status", "AccountSetup");
try { AccountSetupProvisioningComplete?.Invoke(this, EventArgs.Empty); }
catch (Exception ex) { _logger.Error("AccountSetupProvisioningComplete handler failed (fallback path)", ex); }
}
@@ -833,6 +847,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
// null-ing here is correctness-neutral on the start-condition side.
ResetDeviceSetupFallbackTimer();
_deviceSetupProvisioningCompleteFired = true;
MaybeEmitPostFailureRecovered("DeviceSetupCategory.Status", "DeviceSetup");
try { DeviceSetupProvisioningComplete?.Invoke(this, EventArgs.Empty); }
catch (Exception ex) { _logger.Error("DeviceSetupProvisioningComplete handler failed (fallback path)", ex); }
}
@@ -946,12 +961,25 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
var severity = categorySucceeded == false ? EventSeverity.Warning : EventSeverity.Info;
EmitProvisioningEvent(categoryLabel, categorySucceeded, statusText, subcategories, severity);
EmitRawRegistryDump(categoryLabel, jsonValue, $"category_resolved_{(categorySucceeded == true ? "success" : "failed")}");
// Session 4910a5a5: category resolving to success after a terminally
// fired failure = the failure recovered (fallback-path successes call
// this from their own timer handlers).
if (categorySucceeded == true)
{
MaybeEmitPostFailureRecovered(categoryName, categoryLabel);
}
}
else
{
var transitions = DetectSubcategoryTransitions(categoryName, subcategories);
StoreSubcategoryStates(categoryName, subcategories);
// Session 4910a5a5: after a terminally fired failure, the failed
// subcategory leaving "failed" is the ESP re-running the step —
// consistent with the user pressing "Try again" on the failure page.
MaybeEmitPostFailureRetry(categoryName, categoryLabel, transitions);
var failureTransitions = transitions.Where(t => t.IsFailure).ToList();
if (transitions.Count > 0)
{
@@ -1025,6 +1053,21 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
$"escalating as {failureTypeName} (errorCode={errorCode ?? "n/a"})");
_logger.Warning($"Provisioning failure detected: {failureTypeName}");
// Session 4910a5a5: name the likely culprit app(s) directly on the failure args so
// the DecisionEngine's advisory/terminal events carry them — never-started apps
// first (they starve ESP's app gate without ever producing an IME install event),
// capped at 3 names.
IReadOnlyList<string> likelyCulpritApps = null;
if (string.Equals(failedSubcategoryName, "Apps", StringComparison.OrdinalIgnoreCase))
{
likelyCulpritApps = SnapshotTrackedAppsNotCompleted()
.OrderBy(a => a["everStartedInstalling"] is bool started && started ? 1 : 0)
.Select(a => a["appName"]?.ToString())
.Where(n => !string.IsNullOrEmpty(n))
.Take(3)
.ToList();
}
// Caller already holds _stateLock.
ArmFailureSettleTimer(
categoryName: categoryName,
@@ -1032,7 +1075,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
failureType: failureTypeName,
errorCode: errorCode,
failedSubcategory: failedSubcategoryName,
category: categoryLabel));
category: categoryLabel,
likelyCulpritApps: likelyCulpritApps));
// Settle-window owns the actual EspFailureDetected fire. The legacy "fire-after-emit"
// path in CheckProvisioningStatus is now a no-op for this code-path; it stays in place
@@ -1292,6 +1336,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
_provisioningCategoriesResolved.Remove(categoryName);
}
}
else
{
// Session 4910a5a5: remember the terminally fired failure so a LATE
// recovery (outside the settle window — e.g. user pressing "Try again"
// minutes later) can be told as esp_failure_retry_detected /
// esp_failure_recovered instead of hiding in a generic state-change.
_provisioningFailureTerminalArgs[categoryName] = args;
_provisioningFailureTerminalUtc[categoryName] = DateTime.UtcNow;
}
}
if (recovered)
@@ -1382,6 +1435,133 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
});
}
/// <summary>
/// Session 4910a5a5: after a terminally fired failure (settle window expired,
/// EspFailureDetected raised), the failed subcategory leaving the <c>failed</c> state
/// means the ESP re-ran the step — consistent with the user pressing "Try again" on
/// the ESP failure page (the agent cannot observe the click itself). Emits
/// <c>esp_failure_retry_detected</c> once per terminally fired category. Caller holds
/// <see cref="_stateLock"/>.
/// </summary>
private void MaybeEmitPostFailureRetry(string categoryName, string categoryLabel, List<SubcategoryTransition> transitions)
{
if (transitions == null || transitions.Count == 0) return;
if (_postFailureRetryEmitted.Contains(categoryName)) return;
if (_provisioningFailureTerminalArgs == null
|| !_provisioningFailureTerminalArgs.TryGetValue(categoryName, out var terminalArgs))
{
return; // No terminal fire on record — in-window recovery is settle_recovered's job.
}
SubcategoryTransition? retryTransition = null;
foreach (var t in transitions)
{
if (string.Equals(t.OldState, "failed", StringComparison.OrdinalIgnoreCase)
&& !string.Equals(t.NewState, "failed", StringComparison.OrdinalIgnoreCase))
{
retryTransition = t;
break;
}
}
if (retryTransition == null) return;
var retry = retryTransition.Value;
_postFailureRetryEmitted.Add(categoryName);
var minutesSinceFailure = MinutesSinceTerminalFailure(categoryName);
var data = new Dictionary<string, object>
{
{ "category", categoryLabel },
{ "subcategory", retry.SubcategoryName },
{ "previousState", retry.OldState },
{ "newState", retry.NewState },
{ "failureType", terminalArgs.FailureType },
{ "minutesSinceFailure", minutesSinceFailure },
};
if (!string.IsNullOrEmpty(terminalArgs.ErrorCode))
data["errorCode"] = terminalArgs.ErrorCode;
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
EventType = Constants.EventTypes.EspFailureRetryDetected,
Severity = EventSeverity.Info,
Source = "EspAndHelloTracker",
Phase = EnrollmentPhase.Unknown,
Message = $"ESP retry detected — {categoryLabel}/{retry.SubcategoryName} left the failed state " +
$"({retry.OldState} → {retry.NewState}) {minutesSinceFailure} min after the failure — " +
"consistent with the user pressing 'Try again' on the ESP failure page.",
Data = data,
ImmediateUpload = true
});
}
/// <summary>
/// Session 4910a5a5: the category behind a terminally fired failure reached success
/// (categorySucceeded=true or the all-subcategories-succeeded fallback) — the failure
/// demonstrably recovered. Emits <c>esp_failure_recovered</c> once per terminally
/// fired category so the timeline tells "failed → retried → recovered" instead of
/// leaving the stale failure as the last word. Caller holds <see cref="_stateLock"/>.
/// </summary>
private void MaybeEmitPostFailureRecovered(string categoryName, string categoryLabel)
{
if (_postFailureRecoveredEmitted == null || _postFailureRecoveredEmitted.Contains(categoryName)) return;
if (_provisioningFailureTerminalArgs == null
|| !_provisioningFailureTerminalArgs.TryGetValue(categoryName, out var terminalArgs))
{
return;
}
_postFailureRecoveredEmitted.Add(categoryName);
var minutesSinceFailure = MinutesSinceTerminalFailure(categoryName);
string subcategoryStatus = null;
if (terminalArgs.FailedSubcategory != null
&& _lastSubcategoryStates != null
&& _lastSubcategoryStates.TryGetValue(categoryName, out var subs)
&& subs.TryGetValue(terminalArgs.FailedSubcategory, out var currentState))
{
subcategoryStatus = currentState;
}
var data = new Dictionary<string, object>
{
{ "category", categoryLabel },
{ "failedSubcategory", terminalArgs.FailedSubcategory ?? "category-level" },
{ "failureType", terminalArgs.FailureType },
{ "minutesSinceFailure", minutesSinceFailure },
};
if (!string.IsNullOrEmpty(terminalArgs.ErrorCode))
data["errorCode"] = terminalArgs.ErrorCode;
if (subcategoryStatus != null)
data["subcategoryState"] = subcategoryStatus;
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
EventType = Constants.EventTypes.EspFailureRecovered,
Severity = EventSeverity.Info,
Source = "EspAndHelloTracker",
Phase = EnrollmentPhase.Unknown,
Message = $"ESP failure recovered — {categoryLabel}/{terminalArgs.FailedSubcategory ?? "category-level"} " +
$"completed successfully {minutesSinceFailure} min after the failure; the earlier ESP failure no longer applies.",
Data = data,
ImmediateUpload = true
});
}
private int MinutesSinceTerminalFailure(string categoryName)
{
if (_provisioningFailureTerminalUtc != null
&& _provisioningFailureTerminalUtc.TryGetValue(categoryName, out var firedAt))
{
return Math.Max(0, (int)Math.Round((DateTime.UtcNow - firedAt).TotalMinutes));
}
return 0;
}
private void EmitFailureSettleStarted(EspFailureDetectedEventArgs args)
{
var data = new Dictionary<string, object>
@@ -1863,6 +2043,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
_provisioningFailureSettleTimers = new Dictionary<string, Timer>(StringComparer.OrdinalIgnoreCase);
if (_provisioningFailureSettleArgs == null)
_provisioningFailureSettleArgs = new Dictionary<string, EspFailureDetectedEventArgs>(StringComparer.OrdinalIgnoreCase);
if (_provisioningFailureTerminalArgs == null)
_provisioningFailureTerminalArgs = new Dictionary<string, EspFailureDetectedEventArgs>(StringComparer.OrdinalIgnoreCase);
if (_provisioningFailureTerminalUtc == null)
_provisioningFailureTerminalUtc = new Dictionary<string, DateTime>(StringComparer.OrdinalIgnoreCase);
if (_postFailureRetryEmitted == null)
_postFailureRetryEmitted = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
if (_postFailureRecoveredEmitted == null)
_postFailureRecoveredEmitted = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
}
}
@@ -0,0 +1,72 @@
using System;
using System.Reflection;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Interop
{
/// <summary>
/// Reads WinRT <c>Windows.System.Profile.SystemSetupInfo.OutOfBoxExperienceState</c>
/// (NotStarted=0 / InProgress=1 / Completed=2) via the CLR's built-in WinRT projection —
/// no NuGet package, no winmd reference; the type is resolved at runtime with
/// <c>ContentType=WindowsRuntime</c>, the same mechanism the bootstrap script uses from
/// PowerShell 5.1.
/// <para>
/// <b>Contract:</b> observational only — never feed this into decision-engine logic.
/// Requires Windows 10 1809+ (build 17763, UniversalApiContract 7.0) and is callable
/// from the SYSTEM service context (empirically validated, probe session 9c404ae9).
/// On older builds or SKUs without the contract the type resolves to null and
/// <see cref="Read"/> returns <c>"unavailable"</c>. <see cref="Read"/> never throws.
/// </para>
/// <para>
/// Empirical timing (Win11 24H2 probe): the state flips InProgress→Completed at desktop
/// arrival — it is a corroboration signal, not an early one. WDP reseal and placeholder
/// (fooUser) flip semantics are unverified; another reason this must stay observational.
/// </para>
/// </summary>
internal static class OobeStateReader
{
public const string Unavailable = "unavailable";
// Resolved once per process. Benign race: concurrent first calls resolve the same
// PropertyInfo and the assignment is idempotent — no lock needed.
private static PropertyInfo _stateProperty;
private static bool _initTried;
/// <summary>
/// Returns <c>"not_started"</c> / <c>"in_progress"</c> / <c>"completed"</c>,
/// <c>"unknown_&lt;n&gt;"</c> for unexpected enum values, or <c>"unavailable"</c>
/// when the WinRT contract is absent or any part of the read fails. Never throws.
/// </summary>
public static string Read()
{
try
{
if (!_initTried)
{
var type = Type.GetType(
"Windows.System.Profile.SystemSetupInfo, Windows.System.Profile, ContentType=WindowsRuntime",
throwOnError: false);
_stateProperty = type?.GetProperty(
"OutOfBoxExperienceState",
BindingFlags.Public | BindingFlags.Static);
_initTried = true;
}
if (_stateProperty == null)
return Unavailable;
var value = Convert.ToInt32(_stateProperty.GetValue(null));
switch (value)
{
case 0: return "not_started";
case 1: return "in_progress";
case 2: return "completed";
default: return "unknown_" + value;
}
}
catch
{
return Unavailable;
}
}
}
}
@@ -0,0 +1,316 @@
using System;
using System.Runtime.InteropServices;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Interop
{
/// <summary>
/// Reads the UEFI Secure Boot signature databases <c>db</c> and <c>KEK</c> directly from
/// firmware via <c>GetFirmwareEnvironmentVariableExW</c> and checks which Microsoft
/// signing certificates they contain. This is the authoritative source for the 2023
/// certificate rollout state — the <c>SecureBoot\Servicing</c> registry key only reflects
/// whether the *Windows Update* rollout ran and verified, and lags behind when the
/// certificate reached the firmware another way (OEM factory image, WinCS, manual).
/// <para>
/// The certificate subject CNs are stored as ASCII bytes inside the DER-encoded
/// signature lists, so a plain byte-substring search is sufficient — the same check
/// Microsoft documents via <c>Get-SecureBootUEFI</c> (which wraps this exact API).
/// </para>
/// <para>
/// <b>Contract:</b> observational only — never feed this into decision-engine logic.
/// <see cref="Read"/> never throws. Reading firmware variables requires
/// <c>SeSystemEnvironmentPrivilege</c>, which SYSTEM holds but must be enabled on the
/// process token first; the previous privilege state is restored after the read.
/// On legacy BIOS the API fails with <c>ERROR_INVALID_FUNCTION</c> → status
/// <c>not_uefi</c>; every failure maps to a status string instead of an exception.
/// </para>
/// </summary>
internal static class UefiSecureBootCertReader
{
public const string StatusOk = "ok";
public const string StatusNotUefi = "not_uefi";
public const string StatusVariableNotFound = "variable_not_found";
public const string StatusPrivilegeDenied = "privilege_denied";
// EFI_IMAGE_SECURITY_DATABASE_GUID — namespace of the "db" variable.
private const string DbVariableGuid = "{d719b2cb-3d3a-4596-a3bc-dad00e67656f}";
// EFI_GLOBAL_VARIABLE — namespace of the "KEK" variable.
private const string KekVariableGuid = "{8be4df61-93ca-11d2-aa0d-00e098032b8c}";
// Subject CNs searched for in db. 2023 set per Microsoft's June-2026 expiry guidance;
// 2011 set kept for context (which legacy CAs are still present).
internal const string CertWindowsUefiCa2023 = "Windows UEFI CA 2023";
internal const string CertMicrosoftUefiCa2023 = "Microsoft UEFI CA 2023";
internal const string CertMicrosoftOptionRomCa2023 = "Microsoft Option ROM UEFI CA 2023";
internal const string CertWindowsProductionPca2011 = "Microsoft Windows Production PCA 2011";
internal const string CertMicrosoftUefiCa2011 = "Microsoft Corporation UEFI CA 2011";
// Subject CNs searched for in KEK.
internal const string CertMicrosoftKek2kCa2023 = "Microsoft Corporation KEK 2K CA 2023";
internal const string CertMicrosoftKekCa2011 = "Microsoft Corporation KEK CA 2011";
private const int ErrorInvalidFunction = 1; // legacy BIOS / no UEFI
private const int ErrorInsufficientBuffer = 122;
private const int ErrorEnvvarNotFound = 203;
private const int ErrorPrivilegeNotHeld = 1314;
private const int ErrorNotAllAssigned = 1300;
private const int InitialBufferSize = 64 * 1024;
private const int RetryBufferSize = 256 * 1024;
/// <summary>
/// Reads db and KEK and evaluates the certificate presence flags. Never throws;
/// per-variable failures surface as <see cref="UefiSecureBootCertSnapshot.DbStatus"/> /
/// <see cref="UefiSecureBootCertSnapshot.KekStatus"/>. Certificate flags are only
/// meaningful when the corresponding status is <see cref="StatusOk"/>.
/// </summary>
public static UefiSecureBootCertSnapshot Read()
{
var snapshot = new UefiSecureBootCertSnapshot();
try
{
using (var privilege = SystemEnvironmentPrivilege.TryEnable())
{
if (!privilege.Enabled)
{
snapshot.DbStatus = StatusPrivilegeDenied;
snapshot.KekStatus = StatusPrivilegeDenied;
return snapshot;
}
byte[] db;
snapshot.DbStatus = TryReadVariable("db", DbVariableGuid, out db);
if (snapshot.DbStatus == StatusOk)
{
snapshot.DbHasWindowsUefiCa2023 = ContainsAscii(db, CertWindowsUefiCa2023);
snapshot.DbHasMicrosoftUefiCa2023 = ContainsAscii(db, CertMicrosoftUefiCa2023);
snapshot.DbHasMicrosoftOptionRomCa2023 = ContainsAscii(db, CertMicrosoftOptionRomCa2023);
snapshot.DbHasWindowsProductionPca2011 = ContainsAscii(db, CertWindowsProductionPca2011);
snapshot.DbHasMicrosoftUefiCa2011 = ContainsAscii(db, CertMicrosoftUefiCa2011);
}
byte[] kek;
snapshot.KekStatus = TryReadVariable("KEK", KekVariableGuid, out kek);
if (snapshot.KekStatus == StatusOk)
{
snapshot.KekHasMicrosoftKek2kCa2023 = ContainsAscii(kek, CertMicrosoftKek2kCa2023);
snapshot.KekHasMicrosoftKekCa2011 = ContainsAscii(kek, CertMicrosoftKekCa2011);
}
}
}
catch (Exception ex)
{
var status = "error_" + ex.GetType().Name;
if (snapshot.DbStatus == null) snapshot.DbStatus = status;
if (snapshot.KekStatus == null) snapshot.KekStatus = status;
}
return snapshot;
}
/// <summary>
/// Reads one firmware variable into <paramref name="bytes"/> (trimmed to the actual
/// length). Returns a status string; <paramref name="bytes"/> is null unless the
/// status is <see cref="StatusOk"/>.
/// </summary>
private static string TryReadVariable(string name, string guid, out byte[] bytes)
{
bytes = null;
var buffer = new byte[InitialBufferSize];
var length = NativeMethods.GetFirmwareEnvironmentVariableExW(name, guid, buffer, (uint)buffer.Length, IntPtr.Zero);
if (length == 0)
{
var error = Marshal.GetLastWin32Error();
if (error == ErrorInsufficientBuffer)
{
buffer = new byte[RetryBufferSize];
length = NativeMethods.GetFirmwareEnvironmentVariableExW(name, guid, buffer, (uint)buffer.Length, IntPtr.Zero);
if (length == 0)
error = Marshal.GetLastWin32Error();
}
if (length == 0)
{
switch (error)
{
case ErrorInvalidFunction: return StatusNotUefi;
case ErrorEnvvarNotFound: return StatusVariableNotFound;
case ErrorPrivilegeNotHeld: return StatusPrivilegeDenied;
default: return "error_" + error;
}
}
}
bytes = new byte[length];
Array.Copy(buffer, bytes, (int)length);
return StatusOk;
}
/// <summary>
/// Byte-level ASCII substring search (ordinal, case-sensitive). The signature lists
/// are at most a few hundred KB and read once per agent start — a naive scan is fine.
/// </summary>
internal static bool ContainsAscii(byte[] haystack, string needle)
{
if (haystack == null || string.IsNullOrEmpty(needle) || needle.Length > haystack.Length)
return false;
for (var i = 0; i <= haystack.Length - needle.Length; i++)
{
var j = 0;
while (j < needle.Length && haystack[i + j] == (byte)needle[j])
j++;
if (j == needle.Length)
return true;
}
return false;
}
/// <summary>
/// Enables <c>SeSystemEnvironmentPrivilege</c> on the process token for the lifetime
/// of the instance and restores the previous privilege state on dispose. SYSTEM holds
/// the privilege (default-disabled); without elevation <see cref="Enabled"/> stays
/// false and the caller reports <see cref="StatusPrivilegeDenied"/>.
/// </summary>
private sealed class SystemEnvironmentPrivilege : IDisposable
{
public bool Enabled { get; private set; }
private IntPtr _token = IntPtr.Zero;
private NativeMethods.TOKEN_PRIVILEGES _previousState;
private bool _restoreNeeded;
public static SystemEnvironmentPrivilege TryEnable()
{
var instance = new SystemEnvironmentPrivilege();
try
{
if (!NativeMethods.OpenProcessToken(NativeMethods.GetCurrentProcess(),
NativeMethods.TOKEN_ADJUST_PRIVILEGES | NativeMethods.TOKEN_QUERY, out instance._token))
return instance;
NativeMethods.LUID luid;
if (!NativeMethods.LookupPrivilegeValueW(null, "SeSystemEnvironmentPrivilege", out luid))
return instance;
var newState = new NativeMethods.TOKEN_PRIVILEGES
{
PrivilegeCount = 1,
Luid = luid,
Attributes = NativeMethods.SE_PRIVILEGE_ENABLED
};
uint returnLength;
if (!NativeMethods.AdjustTokenPrivileges(instance._token, false, ref newState,
(uint)Marshal.SizeOf(typeof(NativeMethods.TOKEN_PRIVILEGES)),
out instance._previousState, out returnLength))
return instance;
// AdjustTokenPrivileges returns TRUE even when the privilege is not held —
// the real verdict is ERROR_NOT_ALL_ASSIGNED in the last error.
if (Marshal.GetLastWin32Error() == ErrorNotAllAssigned)
return instance;
// Restore only when the privilege was not already enabled before.
instance._restoreNeeded = instance._previousState.PrivilegeCount > 0
&& (instance._previousState.Attributes & NativeMethods.SE_PRIVILEGE_ENABLED) == 0;
instance.Enabled = true;
}
catch
{
// Fail-soft: Enabled stays false.
}
return instance;
}
public void Dispose()
{
if (_token != IntPtr.Zero)
{
try
{
if (_restoreNeeded)
{
NativeMethods.TOKEN_PRIVILEGES ignored;
uint returnLength;
NativeMethods.AdjustTokenPrivileges(_token, false, ref _previousState,
(uint)Marshal.SizeOf(typeof(NativeMethods.TOKEN_PRIVILEGES)),
out ignored, out returnLength);
}
}
catch
{
// Best effort — the process token privilege state is not load-bearing
// for anything else the agent does.
}
NativeMethods.CloseHandle(_token);
_token = IntPtr.Zero;
}
}
}
private static class NativeMethods
{
public const uint TOKEN_ADJUST_PRIVILEGES = 0x0020;
public const uint TOKEN_QUERY = 0x0008;
public const uint SE_PRIVILEGE_ENABLED = 0x0002;
[StructLayout(LayoutKind.Sequential)]
public struct LUID
{
public uint LowPart;
public int HighPart;
}
// Single-privilege variant of the variable-length native struct — sufficient here
// because exactly one privilege is adjusted per call.
[StructLayout(LayoutKind.Sequential)]
public struct TOKEN_PRIVILEGES
{
public uint PrivilegeCount;
public LUID Luid;
public uint Attributes;
}
[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
public static extern uint GetFirmwareEnvironmentVariableExW(
string lpName, string lpGuid, byte[] pBuffer, uint nSize, IntPtr pdwAttributes);
[DllImport("kernel32.dll")]
public static extern IntPtr GetCurrentProcess();
[DllImport("advapi32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool OpenProcessToken(IntPtr processHandle, uint desiredAccess, out IntPtr tokenHandle);
[DllImport("advapi32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool LookupPrivilegeValueW(string lpSystemName, string lpName, out LUID lpLuid);
[DllImport("advapi32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool AdjustTokenPrivileges(IntPtr tokenHandle, bool disableAllPrivileges,
ref TOKEN_PRIVILEGES newState, uint bufferLength, out TOKEN_PRIVILEGES previousState, out uint returnLength);
[DllImport("kernel32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool CloseHandle(IntPtr handle);
}
}
/// <summary>
/// Result of one firmware read. Certificate flags are only meaningful when the
/// corresponding status equals <see cref="UefiSecureBootCertReader.StatusOk"/>.
/// </summary>
internal sealed class UefiSecureBootCertSnapshot
{
public string DbStatus { get; set; }
public string KekStatus { get; set; }
public bool DbHasWindowsUefiCa2023 { get; set; }
public bool DbHasMicrosoftUefiCa2023 { get; set; }
public bool DbHasMicrosoftOptionRomCa2023 { get; set; }
public bool DbHasWindowsProductionPca2011 { get; set; }
public bool DbHasMicrosoftUefiCa2011 { get; set; }
public bool KekHasMicrosoftKek2kCa2023 { get; set; }
public bool KekHasMicrosoftKekCa2011 { get; set; }
}
}
@@ -8,6 +8,7 @@ using System.Text.Json;
using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Security;
using AutopilotMonitor.DecisionCore.Engine;
@@ -493,7 +494,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo
}
}
EmitDeviceInfoEvent(Constants.EventTypes.SecureBootStatus, $"SecureBoot: {data["uefiSecureBootEnabled"]}, CA2023: {data["uefiCA2023Status"]}", data);
// Direct firmware evidence (UEFI db/KEK variables) — authoritative for the
// CA-2023 rollout state. The Servicing key above only mirrors whether the
// Windows Update rollout ran; it lags behind when the certificate reached
// the firmware another way (OEM factory image, WinCS, manual deployment).
var firmware = UefiSecureBootCertReader.Read();
AppendFirmwareCertFields(data, firmware);
EmitDeviceInfoEvent(Constants.EventTypes.SecureBootStatus,
$"SecureBoot: {data["uefiSecureBootEnabled"]}, CA2023: {data["uefiCA2023Status"]}, firmware: {DescribeFirmwareVerdict(firmware)}",
data);
}
catch (Exception ex)
{
@@ -501,6 +511,50 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo
}
}
/// <summary>
/// Maps a <see cref="UefiSecureBootCertSnapshot"/> onto the <c>secureboot_status</c>
/// payload. Certificate booleans are only emitted when the respective variable read
/// succeeded — an absent boolean means "firmware state unknown", never "missing".
/// </summary>
internal static void AppendFirmwareCertFields(Dictionary<string, object> data, UefiSecureBootCertSnapshot firmware)
{
if (firmware == null) return;
data["uefiFirmwareReadStatus"] = firmware.DbStatus != UefiSecureBootCertReader.StatusOk
? firmware.DbStatus
: firmware.KekStatus;
if (firmware.DbStatus == UefiSecureBootCertReader.StatusOk)
{
data["uefiDbHasWindowsUefiCa2023"] = firmware.DbHasWindowsUefiCa2023;
data["uefiDbHasMicrosoftUefiCa2023"] = firmware.DbHasMicrosoftUefiCa2023;
data["uefiDbHasMicrosoftOptionRomCa2023"] = firmware.DbHasMicrosoftOptionRomCa2023;
data["uefiDbHasWindowsProductionPca2011"] = firmware.DbHasWindowsProductionPca2011;
data["uefiDbHasMicrosoftUefiCa2011"] = firmware.DbHasMicrosoftUefiCa2011;
// One-sided marker consumed by ANALYZE-SEC-001's not_exists precondition.
// The engine's not_exists passes only while the field is absent/empty, so the
// marker must be OMITTED (never emitted as false) unless the firmware db
// positively confirms the 2023 certificate — old agents and unconfirmed
// devices then keep evaluating the rule.
if (firmware.DbHasWindowsUefiCa2023)
data["uefiCa2023FirmwareConfirmed"] = true;
}
if (firmware.KekStatus == UefiSecureBootCertReader.StatusOk)
{
data["uefiKekHasMicrosoftKek2kCa2023"] = firmware.KekHasMicrosoftKek2kCa2023;
data["uefiKekHasMicrosoftKekCa2011"] = firmware.KekHasMicrosoftKekCa2011;
}
}
internal static string DescribeFirmwareVerdict(UefiSecureBootCertSnapshot firmware)
{
if (firmware == null) return "unavailable";
if (firmware.DbStatus != UefiSecureBootCertReader.StatusOk) return firmware.DbStatus ?? "unavailable";
return firmware.DbHasWindowsUefiCa2023 ? "CA2023 confirmed" : "CA2023 missing";
}
private void CollectBitLockerStatus()
{
try
@@ -36,6 +36,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
/// <item><b>StallProbeHost</b> — <see cref="StallProbeCollector"/> +
/// <see cref="StallProbeCollectorAdapter"/>. Owns its 60-s idle-check timer (the
/// collector itself has no timer — it's a pure probe invoked from outside).</item>
/// <item><b>EspPolicyProviderStallHost</b> — always-on 60-s wall-clock tick for the
/// <c>esp_policy_provider_stalled</c> tripwire (registered EnrollmentStatusTracking CSP
/// provider continuously incomplete ≥ 15 min — the no-timeout ESP wait).</item>
/// </list>
/// Optional peripheral hosts (driven by <see cref="CollectorConfiguration"/> toggles):
/// <list type="bullet">
@@ -303,6 +306,20 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
harmlessModernDeploymentEventIds: collectors.ModernDeploymentHarmlessEventIds));
}
// ESP policy-provider stall tripwire — always-on kernel host (no config gate, like
// disk_space_low): a registered EnrollmentStatusTracking CSP provider that never
// completes (e.g. co-management "ConfigMgr" without "Sidecar" on a pre-provisioned
// device) parks the user ESP at "Apps (Identifying)" WITHOUT any OS timeout. Wall-clock
// dwell, deliberately outside the StallProbe gate: the idle clock there resets on any
// session activity, and the stall must accrue regardless of it.
hosts.Add(new EspPolicyProviderStallHost(
sessionId: sessionId,
tenantId: tenantId,
logger: logger,
ingress: ingress,
clock: clock,
startupGate: _startupEventGate));
// Deterministic update corroboration — compares the persisted OS build (CurrentBuild.UBR)
// across agent restarts and emits os_build_changed when it differs (session 7443317c:
// build jump proved an OOBE quality update the WU channel never showed). Must be added
@@ -0,0 +1,94 @@
#nullable enable
using System;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Persistence;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
namespace AutopilotMonitor.Agent.V2.Core.Orchestration
{
/// <summary>
/// Drives the <see cref="EspPolicyProviderStallDetector"/> from a 60-s wall-clock tick.
/// Deliberately NOT idle-coupled (a policy-provider stall coexists with a chatty session —
/// <see cref="StallProbeHost"/>'s idle clock would keep resetting) and NOT idle-stopped
/// (<see cref="PeriodicCollectorLifecycleHost"/> would sleep through exactly the dormant
/// window the dwell must observe). Always-on kernel host, no config gate — precedent:
/// the <c>disk_space_low</c> tripwire.
/// <para>
/// First tick fires after 60 s, not at Start: the EnrollmentStatusTracking key legitimately
/// doesn't exist at agent start, and nothing can be 15 min stale at t=0.
/// </para>
/// </summary>
internal sealed class EspPolicyProviderStallHost : ICollectorHost
{
public string Name => "EspPolicyProviderStallDetector";
private static readonly TimeSpan PollInterval = TimeSpan.FromSeconds(60);
private readonly EspPolicyProviderStallDetector _detector;
private readonly AgentLogger _logger;
private Timer? _tickTimer;
private int _disposed;
public EspPolicyProviderStallHost(
string sessionId,
string tenantId,
AgentLogger logger,
ISignalIngressSink ingress,
IClock clock,
StartupEventGate? startupGate)
{
if (ingress == null) throw new ArgumentNullException(nameof(ingress));
if (clock == null) throw new ArgumentNullException(nameof(clock));
_logger = logger;
var post = new InformationalEventPost(ingress, clock);
_detector = new EspPolicyProviderStallDetector(
sessionId: sessionId,
tenantId: tenantId,
post: post,
logger: logger,
clock: clock,
startupGate: startupGate);
}
public void Start()
{
_tickTimer = new Timer(
_ => SafeTick(),
state: null,
dueTime: PollInterval,
period: PollInterval);
_logger.Info($"EspPolicyProviderStallHost: started (tick every {PollInterval.TotalSeconds}s).");
}
public void Stop()
{
try
{
_tickTimer?.Dispose();
_tickTimer = null;
}
catch (Exception ex) { _logger.Warning($"EspPolicyProviderStallHost: timer dispose failed: {ex.Message}"); }
}
public void Dispose()
{
if (Interlocked.Exchange(ref _disposed, 1) == 1) return;
Stop();
}
private void SafeTick()
{
try
{
_detector.Tick();
}
catch (Exception ex)
{
_logger.Error("EspPolicyProviderStallHost: tick failed.", ex);
}
}
}
}
@@ -111,6 +111,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
logger: logger,
matchLogPath: expandedMatchLogPath,
stateDirectory: stateDirectory);
// Historic-replay guard runs against the agent clock, not the raw system clock —
// keeps the tracker's staleness verdicts consistent with the adapter's clamp.
_tracker.UtcNowProvider = () => clock.UtcNow;
if (simulationMode)
{
@@ -1,6 +1,7 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Globalization;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
@@ -267,6 +268,14 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
payload["failedSubcategory"] = failedSubcategory!;
if (!string.IsNullOrEmpty(category))
payload["category"] = category!;
// Session 4910a5a5: culprit-app names ride the signal so the DecisionEngine's
// esp_failure_advisory / enrollment_failed events can name them directly.
var culprits = args?.LikelyCulpritApps;
if (culprits != null && culprits.Count > 0)
{
payload["likelyCulpritApps"] = string.Join(", ", culprits);
payload["likelyCulpritAppCount"] = culprits.Count.ToString(CultureInfo.InvariantCulture);
}
_ingress.Post(
kind: DecisionSignalKind.EspTerminalFailure,
@@ -97,12 +97,24 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
// bootstrap marker line. Lets MCP report "which device runs which bootstrap version".
private bool _bootstrapDetectedPosted;
// One-shot: historic script replay detected (a previous enrollment's IME log content
// surviving on disk) — its script events are suppressed, and a single
// historic_ime_replay_detected Info event explains the gap in the timeline.
private bool _historicReplayNoted;
private int _suppressedHistoricReplayEvents; // forensic count for the log line
// A platform script whose observed run duration reaches this is treated as having hit the
// IME script-execution timeout (default ~30 min). 25 min leaves headroom below 30 so clock
// jitter / the agent's late replay window can't miss it, while staying well above any
// legitimately long install wrapper that completes on its own.
private static readonly TimeSpan ScriptTimeoutSuspectedThreshold = TimeSpan.FromMinutes(25);
// Source log lines older than this relative to the clock are pathological — shared by
// ResolveOccurredAt's staleness clamp, the historic-replay suppression, the duration
// plausibility backstop AND the tracker's app-mutation guard (single source of truth
// in ImeLogTracker so the layers can never drift apart).
private static readonly TimeSpan StaleSourceThreshold = ImeLogTracker.HistoricReplayThreshold;
// Matches the deterministic marker line the bootstrap script writes via Write-Log,
// e.g. "Bootstrap script version: v2.0" — see scripts/Bootstrap/Install-AutopilotMonitor.ps1.
// Same shape as the web-side regex in utils/bootstrapVersion.ts; kept in sync intentionally.
@@ -289,17 +301,11 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
rawSourceTimestamp = sourceTsOverride ?? _tracker.LastMatchedLogTimestamp;
if (rawSourceTimestamp.HasValue)
{
// CMTrace log timestamps from IME are emitted in UTC, but DateTimeKind may
// arrive as Unspecified depending on the parser path. Normalize via
// SpecifyKind=Utc rather than ToUniversalTime() to avoid double-conversion
// when Kind is already UTC.
var asUtc = rawSourceTimestamp.Value.Kind == DateTimeKind.Utc
? rawSourceTimestamp.Value
: DateTime.SpecifyKind(rawSourceTimestamp.Value, DateTimeKind.Utc);
var asUtc = NormalizeUtc(rawSourceTimestamp.Value);
var clockNow = _clock.UtcNow;
var ageHours = (clockNow - asUtc).TotalHours;
if (ageHours > 24.0 || ageHours < -1.0)
if (ageHours > StaleSourceThreshold.TotalHours || ageHours < -1.0)
{
// Pathologically stale (>24h old) or in the future (>1h skew) — log,
// fall back, and let the caller flag the event.
@@ -325,6 +331,67 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
data["rejectedSourceTimestamp"] = rawSourceTs.Value.ToString("o", System.Globalization.CultureInfo.InvariantCulture);
}
// CMTrace Kind normalization — shared implementation lives on the tracker (used by its
// historic-replay guard too).
private static DateTime NormalizeUtc(DateTime value) => ImeLogTracker.NormalizeUtc(value);
/// <summary>
/// True when an event's source log line is a historic replay: the source timestamp
/// was rejected as stale (> <see cref="StaleSourceThreshold"/> in the PAST). Those lines
/// are script runs and app installs from a previous enrollment whose IME log survived on
/// disk (session eaf3d8c4: 156 replayed script completions + 147 replayed app events
/// shown as current activity, immediate-upload flood → ingress_backpressure). Future-skew
/// rejections (negative age — mid-enrollment clock jump) and clock-only fallbacks (no
/// source timestamp at all) are NOT replay and must still emit. Emits the one-shot
/// <c>historic_ime_replay_detected</c> summary on the first suppression; replay is
/// chronological, so that first suppressed line carries the earliest rejected source
/// timestamp. Bypassed in <see cref="ImeLogTracker.SimulationMode"/> — the
/// --replay-log-dir dev tool feeds historic lines on purpose.
/// </summary>
private bool SuppressIfHistoricReplay(
bool derivedFromClock, DateTime? rawSourceTs, string eventLabel, string entityId)
{
if (_tracker.SimulationMode) return false;
if (!derivedFromClock || !rawSourceTs.HasValue) return false;
var asUtc = NormalizeUtc(rawSourceTs.Value);
var clockNow = _clock.UtcNow;
var age = clockNow - asUtc;
if (age <= StaleSourceThreshold) return false; // future-skew or fresh → emit
bool emitSummary;
int suppressedSoFar;
lock (_lock)
{
suppressedSoFar = ++_suppressedHistoricReplayEvents;
emitSummary = !_historicReplayNoted;
_historicReplayNoted = true;
}
if (emitSummary)
{
var culture = System.Globalization.CultureInfo.InvariantCulture;
// Standard batch upload on purpose — no immediateUpload; the whole point of the
// suppression is to stop the replayed batch from flooding the ingest.
_post.Emit(
eventType: SharedEventTypes.HistoricImeReplayDetected,
source: SourceLabel,
message: "Historic IME activity from a previous enrollment detected in the log "
+ $"(earliest source timestamp {asUtc:o}) — its script and app events are suppressed for this session.",
severity: EventSeverity.Info,
data: new Dictionary<string, string>(StringComparer.Ordinal)
{
["decision"] = "historic_ime_replay_suppressed",
["earliestRejectedSourceTimestamp"] = asUtc.ToString("o", culture),
["staleThresholdHours"] = StaleSourceThreshold.TotalHours.ToString("F0", culture),
});
}
_logger?.Debug(
$"ImeAdapter: suppressed historic {eventLabel} for {entityId} "
+ $"(source ts {asUtc:o}, age {age.TotalHours:F1}h, suppressed so far: {suppressedSoFar})");
return true;
}
private void EmitEspPhase(string phase)
{
if (string.IsNullOrEmpty(phase)) return;
@@ -504,6 +571,14 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
var now = ResolveOccurredAt(out var derivedFromClock, out var rawSourceTs);
// Historic replay (a previous enrollment's log) — do not emit at all. Placed BEFORE
// the sub-phase declaration, the timing bookkeeping and the terminal-signal dedup so
// a replayed app neither consumes the fire-once flags nor poisons _appTimings (which
// feeds FinalStatusBuilder) nor posts a phantom DecisionSignal. The early return also
// covers the shadow download_progress and the app_tracking_summary snapshot below.
if (SuppressIfHistoricReplay(derivedFromClock, rawSourceTs, $"app-state {oldState}->{newState}", app.Id))
return;
// Plan §5 Fix 2: first app activity in a given ESP phase opens the sub-phase on
// the UI timeline via a `phase_transition` declaration event. Fire-once per ESP
// phase; emission happens BEFORE the actual app_* event so the UI sees the phase
@@ -819,6 +894,11 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
if (!string.IsNullOrEmpty(patternId)) data["patternId"] = patternId!;
var now = ResolveOccurredAt(out var derivedFromClock, out var rawSourceTs);
// Historic replay — a week-old DO summary is not this enrollment's download.
if (SuppressIfHistoricReplay(derivedFromClock, rawSourceTs, SharedEventTypes.DoTelemetry, app.Id))
return;
TagDerivedTimestamp(data, derivedFromClock, rawSourceTs);
var label = string.IsNullOrEmpty(app.Name) ? app.Id : app.Name;
@@ -910,13 +990,27 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
now = ResolveOccurredAt(script.ExitObservedAtUtc, out derivedFromClock, out rawSourceTs);
else
now = ResolveOccurredAt(out derivedFromClock, out rawSourceTs);
// Historic replay (a previous enrollment's log) — do not emit at all. The early
// return also skips MaybeEmitBootstrapDetected / MaybeEmitScriptTimeoutSuspected,
// which would otherwise fire on week-old evidence.
if (SuppressIfHistoricReplay(derivedFromClock, rawSourceTs, eventType, script.PolicyId))
return;
TagDerivedTimestamp(data, derivedFromClock, rawSourceTs);
// Per-script run duration (start line → this completion, `now`). Surfaced on every
// Per-script run duration (start line → this completion). Surfaced on every
// script completion so the UI can flag slow / inefficient scripts, not just timed-out
// ones. Omitted when the start line was never observed (StartedAtUtc null); clamped at 0
// against clock skew between the (possibly clock-derived) completion stamp and start.
//
// When the completion timestamp was clock-derived because the source line's timestamp
// was rejected (future-skew CMTrace clock jump — the stale case never reaches here),
// `now` lives on the clock's timeline while StartedAtUtc was captured RAW from the
// source log. Mixing the two produced absurd durations (session eaf3d8c4: 170 h), so
// the duration uses the raw source pair — both ends on the same timeline, the
// difference is the true runtime. The event stamp stays `now` (timeline position).
//
// L19 (delta review 2026-07-02): the same field name carries two different semantics —
// start→own-end-signal (≈ script runtime) vs HS-SCRIPT-START→HS-NEW-RESULT (the whole
// cycle INCLUDING IME's 30 s–3 min batched reporting latency, i.e. systematically
@@ -924,10 +1018,16 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
// consumers never compare the two as equals. The emitting handler stamps the basis on
// the state (health-script early-signal completions are script_runtime too); the
// scriptType fallback covers states from paths that predate the field.
var completionForDuration = derivedFromClock && rawSourceTs.HasValue
? NormalizeUtc(rawSourceTs.Value)
: now;
if (script.StartedAtUtc.HasValue)
{
var durationSeconds = (now - script.StartedAtUtc.Value).TotalSeconds;
if (durationSeconds >= 0)
var durationSeconds = (completionForDuration - script.StartedAtUtc.Value).TotalSeconds;
// Plausibility backstop: IME's script execution timeout is ~30 min; anything above
// 24 h is a cross-run artifact (e.g. a stale pending start slot from a previous
// enrollment paired with a fresh completion) — omit rather than lie.
if (durationSeconds >= 0 && durationSeconds <= StaleSourceThreshold.TotalSeconds)
{
data["durationSeconds"] = durationSeconds.ToString("F2", culture);
data["durationBasis"] = script.DurationBasis
@@ -979,7 +1079,7 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
$"ImeAdapter: script completed policyId={shortId} type={script.ScriptType ?? "?"} result={script.Result ?? "?"} exit={(script.ExitCode.HasValue ? script.ExitCode.Value.ToString() : "n/a")}");
MaybeEmitBootstrapDetected(script, now);
MaybeEmitScriptTimeoutSuspected(script, now);
MaybeEmitScriptTimeoutSuspected(script, now, completionForDuration);
}
/// <summary>
@@ -991,15 +1091,19 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
/// late start / pre-failed ESP. Advisory only: no state mutation, the underlying
/// <c>script_failed</c> is still emitted separately. Health (remediation) scripts are
/// excluded — they re-run on their own cadence and have a different timeout regime.
/// <paramref name="durationEndUtc"/> is the timeline-consistent completion for duration
/// math (raw source ts on rejected-timestamp emissions); <paramref name="occurredAtUtc"/>
/// stamps the emitted event.
/// </summary>
private void MaybeEmitScriptTimeoutSuspected(ScriptExecutionState script, DateTime completionUtc)
private void MaybeEmitScriptTimeoutSuspected(ScriptExecutionState script, DateTime occurredAtUtc, DateTime durationEndUtc)
{
if (IsRemediation(script.ScriptType)) return; // platform scripts only
if (!script.StartedAtUtc.HasValue) return; // no start ⇒ no duration
if (!string.Equals(script.Result, "Failed", StringComparison.OrdinalIgnoreCase)) return;
var duration = completionUtc - script.StartedAtUtc.Value;
var duration = durationEndUtc - script.StartedAtUtc.Value;
if (duration < ScriptTimeoutSuspectedThreshold) return;
if (duration > StaleSourceThreshold) return; // implausible — cross-run artifact, not a hung script
if (!_tracker.TryClaimScriptTimeoutSuspected(script.PolicyId)) return; // restart-safe dedup per policyId (persisted tracker state)
@@ -1026,7 +1130,7 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
message: $"Platform script {shortId} ran {durationMinutes:F0} min and was marked Failed — suspected IME script timeout blocking the enrollment.",
severity: EventSeverity.Warning,
data: data,
occurredAtUtc: completionUtc);
occurredAtUtc: occurredAtUtc);
_logger?.Info(
$"ImeAdapter: script_timeout_suspected policyId={shortId} duration={durationMinutes:F1}min result={script.Result} espPhase={espPhase ?? "(none)"}");
@@ -1081,6 +1185,12 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
if (!string.IsNullOrEmpty(patternId)) data["patternId"] = patternId!;
var now = ResolveOccurredAt(out var derivedFromClock, out var rawSourceTs);
// Historic replay — suppressing here also kills the immediateUpload flood the
// replayed batch caused (session eaf3d8c4: 3× ingress_backpressure).
if (SuppressIfHistoricReplay(derivedFromClock, rawSourceTs, SharedEventTypes.ScriptStarted, info.PolicyId))
return;
TagDerivedTimestamp(data, derivedFromClock, rawSourceTs);
var label = IsRemediation(info.ScriptType) ? "Health script" : "Platform script";
@@ -6,6 +6,7 @@ using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Configuration;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Runtime;
@@ -93,6 +94,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
// correction between agent start and termination no longer skews bootToStartSeconds.
private readonly ulong _bootAnchorTickMs;
private readonly DateTime _bootAnchorUtc;
// WinRT OOBE state at construction (handler is built at agent start) — corroborates
// the late-start heuristic: "completed" here means OOBE was already over on arrival.
private readonly string _oobeStateAtAgentStart;
private int _handled;
public EnrollmentTerminationHandler(
@@ -129,6 +133,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
_bootAnchorTickMs = ObservationCoverage.CurrentTickMs();
_bootAnchorUtc = DateTime.UtcNow;
_oobeStateAtAgentStart = OobeStateReader.Read();
}
/// <summary>
@@ -658,6 +663,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
{ "agentUptimeSeconds", Math.Round(uptimeSeconds, 1) },
{ "deviceBootUtc", deviceBootUtc.ToString("o") },
{ "agentStartUtc", _agentStartTimeUtc.ToString("o") },
{ "oobeStateAtAgentStart", _oobeStateAtAgentStart },
{ "outcome", args.Outcome.ToString() },
{ "note", "The agent's bootstrap (an Intune platform script) ran only near the end of the enrollment, so the agent observed the already-decided end-state rather than the failure window. Treat the diagnosis as a post-mortem; check for a platform/remediation script that hung ahead of the bootstrap (see script_timeout_suspected)." },
},
@@ -2,6 +2,7 @@ using System;
using System.Collections.Generic;
using AutopilotMonitor.Agent.V2.Core.Configuration;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Security;
@@ -68,6 +69,10 @@ namespace AutopilotMonitor.Agent.V2.Runtime
{ "configVersion", configVersion },
{ "remoteConfigFetched", outcome == RemoteConfigFetchOutcome.Succeeded },
{ "remoteConfigOutcome", outcome.ToString() },
// WinRT OOBE state sampled at this start. Together with previousExitType
// this makes late starts provable: "completed" on a first_run means OOBE
// was already over when the agent arrived (e.g. IME queue blocked).
{ "oobeStateAtAgentStart", OobeStateReader.Read() },
};
if (!string.IsNullOrEmpty(previousExit?.CrashExceptionType))
@@ -0,0 +1,382 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Serialization;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.DecisionCore.State;
using Xunit;
namespace AutopilotMonitor.DecisionCore.Tests
{
/// <summary>
/// Session 4910a5a5 (2026-07-23) — advisory recovery. Field failure: DeviceSetup/Apps
/// failed (culprit app never started installing), the advisory defang armed the
/// <c>advisory_completion</c> window, the user pressed "Try again" ~23 min later, the apps
/// re-ran to 10/10 and <c>DeviceSetupProvisioningComplete</c> arrived — yet the window
/// fired 30 min after the advisory and un-defanged the stale failure
/// (<c>enrollment_failed: esp_terminal_failure</c>) on a live enrollment mid-AccountSetup,
/// because the advisory variant was exempt from ALL re-arm/rebase guards. Coverage:
/// <list type="bullet">
/// <item>The advisory records the failed category + culprit apps.</item>
/// <item>A ProvisioningComplete of the SAME category sets the resolved fact + emits
/// <c>esp_failure_advisory_resolved</c>; a different category does not.</item>
/// <item>Deadline fire after recovery + enforcement progress → re-arm, not Failed.</item>
/// <item>Deadline fire after recovery without progress → Failed with the truthful
/// <c>esp_recovered_without_completion_evidence</c> reason and
/// LastFailureTrigger=DeadlineFired (likely-stuck promotion off).</item>
/// <item>Unrecovered advisory keeps the legacy un-defang exactly (regression guard).</item>
/// <item>Reboot re-bases a recovered-advisory window (and still does NOT touch an
/// unrecovered one).</item>
/// <item>Snapshot round-trip + census surfacing of the new facts.</item>
/// </list>
/// </summary>
public sealed class AdvisoryRecoveryTests
{
private static readonly DateTime T0 = new DateTime(2026, 7, 23, 3, 30, 0, DateTimeKind.Utc);
private static DecisionSignal MakeSignal(
long ordinal,
DecisionSignalKind kind,
DateTime occurredAtUtc,
IReadOnlyDictionary<string, string>? payload = null)
{
return new DecisionSignal(
sessionSignalOrdinal: ordinal,
sessionTraceOrdinal: ordinal,
kind: kind,
kindSchemaVersion: 1,
occurredAtUtc: occurredAtUtc,
sourceOrigin: "test",
evidence: new Evidence(EvidenceKind.Synthetic, $"{kind}-{ordinal}", "test"),
payload: payload);
}
private static DecisionSignal DeadlineFired(long ordinal, DateTime occurredAtUtc) =>
MakeSignal(ordinal, DecisionSignalKind.DeadlineFired, occurredAtUtc,
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.AdvisoryCompletion });
private static DecisionEffect? FindTimelineEffect(DecisionStep step, string eventType) =>
step.Effects.FirstOrDefault(e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry
&& e.Parameters != null
&& e.Parameters.TryGetValue("eventType", out var et)
&& et == eventType);
private static ActiveDeadline? FindDeadline(DecisionState state, string name) =>
state.Deadlines.FirstOrDefault(d => d.Name == name);
/// <summary>
/// Replays the session-4910a5a5 shape up to (but excluding) the ESP terminal failure:
/// Classic UserDriven, ContinueAnyway profile, Hello disabled, DeviceSetup entered,
/// AccountSetup entered (04:18, BEFORE the 04:34 DeviceSetup/Apps failure — the IME
/// phase line landed while device apps were still at 9/10).
/// </summary>
private static DecisionState SetupSession(DecisionEngine engine)
{
var state = DecisionState.CreateInitial("sess-4910a5a5", "tenant-e46bc88e", T0);
state = engine.Reduce(state, MakeSignal(0, DecisionSignalKind.SessionStarted, T0)).NewState;
state = engine.Reduce(state, MakeSignal(
5, DecisionSignalKind.EspConfigDetected, T0.AddMinutes(1),
new Dictionary<string, string>
{
[SignalPayloadKeys.SkipUserEsp] = "false",
[SignalPayloadKeys.SkipDeviceEsp] = "false",
[SignalPayloadKeys.EspSyncFailureTimeoutMinutes] = "60",
[SignalPayloadKeys.EspAllowContinueAnyway] = "true",
})).NewState;
state = engine.Reduce(state, MakeSignal(
8, DecisionSignalKind.HelloPolicyDetected, T0.AddMinutes(2),
new Dictionary<string, string> { [SignalPayloadKeys.HelloEnabled] = "false" })).NewState;
state = engine.Reduce(state, MakeSignal(
10, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(6),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "DeviceSetup" })).NewState;
state = engine.Reduce(state, MakeSignal(
20, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(48),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState;
return state;
}
private static DecisionStep ApplyDeviceSetupAppsFailure(DecisionEngine engine, DecisionState state, long ordinal = 50) =>
engine.Reduce(state, MakeSignal(
ordinal, DecisionSignalKind.EspTerminalFailure, T0.AddMinutes(64),
new Dictionary<string, string>
{
["failureType"] = "Provisioning_DeviceSetup_Apps_Failed",
["failedSubcategory"] = "Apps",
["category"] = "DeviceSetup",
["likelyCulpritApps"] = "Oriflame Cosmectics - Teams Backgroundsv v2",
["likelyCulpritAppCount"] = "1",
}));
private static DecisionStep ApplyDeviceSetupResolved(DecisionEngine engine, DecisionState state, long ordinal = 60) =>
engine.Reduce(state, MakeSignal(
ordinal, DecisionSignalKind.DeviceSetupProvisioningComplete, T0.AddMinutes(87)));
// ================================================== advisory arming context ====
[Fact]
public void Advisory_RecordsFailedCategory_AndCulpritApps()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
var step = ApplyDeviceSetupAppsFailure(engine, state);
Assert.NotNull(step.NewState.EspAdvisoryFailureRecordedUtc);
Assert.Equal("DeviceSetup", step.NewState.EspAdvisoryFailureCategory?.Value);
Assert.Null(step.NewState.EspAdvisoryFailureResolvedUtc);
var advisory = FindTimelineEffect(step, "esp_failure_advisory");
Assert.NotNull(advisory);
Assert.Equal("Oriflame Cosmectics - Teams Backgroundsv v2", advisory!.Parameters!["likelyCulpritApps"]);
}
// ===================================================== recovery hook ====
[Fact]
public void DeviceSetupComplete_SameCategory_ResolvesAdvisory_AndEmitsStoryEvent()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
var step = ApplyDeviceSetupResolved(engine, state);
Assert.NotNull(step.NewState.EspAdvisoryFailureResolvedUtc);
Assert.Equal(60, step.NewState.EspAdvisoryFailureResolvedUtc!.SourceSignalOrdinal);
// The anchor itself stays — the advisory happened; it just no longer blocks.
Assert.NotNull(step.NewState.EspAdvisoryFailureRecordedUtc);
var resolved = FindTimelineEffect(step, "esp_failure_advisory_resolved");
Assert.NotNull(resolved);
Assert.Equal("DeviceSetup", resolved!.Parameters!["category"]);
Assert.Equal("23", resolved.Parameters["minutesSinceAdvisory"]);
}
[Fact]
public void DeviceSetupComplete_DifferentCategory_DoesNotResolve()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
// Advisory for AccountSetup — DeviceSetup resolving must not defuse it.
state = engine.Reduce(state, MakeSignal(
50, DecisionSignalKind.EspTerminalFailure, T0.AddMinutes(64),
new Dictionary<string, string>
{
["failureType"] = "Provisioning_AccountSetup_Apps_Failed",
["failedSubcategory"] = "Apps",
["category"] = "AccountSetup",
})).NewState;
var step = ApplyDeviceSetupResolved(engine, state);
Assert.Null(step.NewState.EspAdvisoryFailureResolvedUtc);
Assert.Null(FindTimelineEffect(step, "esp_failure_advisory_resolved"));
}
[Fact]
public void DeviceSetupComplete_Duplicate_ResolvesOnlyOnce()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
state = ApplyDeviceSetupResolved(engine, state).NewState;
// Post-reboot duplicate (AnchorAlreadySet passthrough branch).
var duplicate = engine.Reduce(state, MakeSignal(
65, DecisionSignalKind.DeviceSetupProvisioningComplete, T0.AddMinutes(92)));
Assert.Equal(60, duplicate.NewState.EspAdvisoryFailureResolvedUtc!.SourceSignalOrdinal);
Assert.Null(FindTimelineEffect(duplicate, "esp_failure_advisory_resolved"));
}
// ================================================ deadline fire semantics ====
[Fact]
public void DeadlineFire_AfterRecovery_WithEnforcementProgress_ReArmsInsteadOfFailing()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
state = ApplyDeviceSetupResolved(engine, state).NewState;
// Post-reboot IME re-assert of AccountSetup (session shape: 05:02:59, ordinal 654)
// — user-phase enforcement demonstrably progressed since the arming at ordinal 50.
state = engine.Reduce(state, MakeSignal(
70, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(92),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState;
var step = engine.Reduce(state, DeadlineFired(80, T0.AddMinutes(94)));
// Not failed — the window re-arms with fresh baselines instead.
Assert.Null(step.NewState.Outcome);
Assert.NotEqual(SessionStage.Failed, step.NewState.Stage);
var rearmed = FindDeadline(step.NewState, DeadlineNames.AdvisoryCompletion);
Assert.NotNull(rearmed);
Assert.Equal(T0.AddMinutes(94).AddMinutes(30), rearmed!.DueAtUtc);
Assert.Contains(step.Effects, e => e.Kind == DecisionEffectKind.ScheduleDeadline);
}
[Fact]
public void DeadlineFire_AfterRecovery_NoProgress_FailsWithRecoveredReason()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
state = ApplyDeviceSetupResolved(engine, state).NewState;
var step = engine.Reduce(state, DeadlineFired(80, T0.AddMinutes(94)));
Assert.Equal(SessionStage.Failed, step.NewState.Stage);
Assert.Equal(SessionOutcome.EnrollmentFailed, step.NewState.Outcome);
var failed = FindTimelineEffect(step, "enrollment_failed");
Assert.NotNull(failed);
// Truthful reason: the original failure recovered — do not blame it.
Assert.Equal("esp_recovered_without_completion_evidence", failed!.Parameters!["reason"]);
Assert.Equal("true", failed.Parameters["failureRecovered"]);
Assert.Equal("advisory_recovered_window_expired_without_completion_evidence", failed.Parameters["advisoryReason"]);
// Likely-stuck app promotion stays off: apps are not stuck, they succeeded.
Assert.Equal(nameof(DecisionSignalKind.DeadlineFired), step.NewState.LastFailureTrigger?.Value);
}
[Fact]
public void DeadlineFire_WithoutRecovery_KeepsLegacyUndefang()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
// No recovery — even with enforcement progress the advisory variant must
// un-defang exactly as before (its anchor failure never un-happened).
state = engine.Reduce(state, MakeSignal(
70, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(92),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState;
var step = engine.Reduce(state, DeadlineFired(80, T0.AddMinutes(94)));
Assert.Equal(SessionStage.Failed, step.NewState.Stage);
var failed = FindTimelineEffect(step, "enrollment_failed");
Assert.NotNull(failed);
Assert.Equal("esp_terminal_failure", failed!.Parameters!["reason"]);
Assert.Equal(nameof(DecisionSignalKind.EspTerminalFailure), step.NewState.LastFailureTrigger?.Value);
}
// ======================================================= reboot rebase ====
[Fact]
public void Reboot_AfterRecovery_RebasesAdvisoryWindow()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
state = ApplyDeviceSetupResolved(engine, state).NewState;
var step = engine.Reduce(state, MakeSignal(
70, DecisionSignalKind.SystemRebootObserved, T0.AddMinutes(92)));
Assert.Contains(step.Effects, e =>
e.Kind == DecisionEffectKind.CancelDeadline
&& e.CancelDeadlineName == DeadlineNames.AdvisoryCompletion);
var rebased = FindDeadline(step.NewState, DeadlineNames.AdvisoryCompletion);
Assert.NotNull(rebased);
Assert.Equal(T0.AddMinutes(92).AddMinutes(30), rebased!.DueAtUtc);
}
[Fact]
public void Reboot_WithoutRecovery_DoesNotTouchAdvisoryWindow()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
var originalDue = FindDeadline(state, DeadlineNames.AdvisoryCompletion)!.DueAtUtc;
var step = engine.Reduce(state, MakeSignal(
70, DecisionSignalKind.SystemRebootObserved, T0.AddMinutes(92)));
Assert.DoesNotContain(step.Effects, e => e.Kind == DecisionEffectKind.CancelDeadline);
Assert.Equal(originalDue, FindDeadline(step.NewState, DeadlineNames.AdvisoryCompletion)!.DueAtUtc);
}
// ============================================== full session-4910a5a5 replay ====
[Fact]
public void Session4910a5a5_FailureRecoveryRebootFire_DoesNotFailTheLiveEnrollment()
{
// Field ordinals: 527 EspTerminalFailure → 611 DeviceSetupProvisioningComplete
// (post "Try again", apps 10/10) → 635 SystemRebootObserved → 654 AccountSetup
// re-assert → 671 DeadlineFired. Pre-fix outcome: enrollment_failed at the fire.
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state, ordinal: 527).NewState;
state = ApplyDeviceSetupResolved(engine, state, ordinal: 611).NewState;
state = engine.Reduce(state, MakeSignal(
635, DecisionSignalKind.SystemRebootObserved, T0.AddMinutes(92))).NewState;
state = engine.Reduce(state, MakeSignal(
654, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(93),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState;
var step = engine.Reduce(state, DeadlineFired(671, T0.AddMinutes(94)));
Assert.Null(step.NewState.Outcome);
Assert.NotEqual(SessionStage.Failed, step.NewState.Stage);
Assert.NotNull(FindDeadline(step.NewState, DeadlineNames.AdvisoryCompletion));
}
// =============================================== serialization + census ====
[Fact]
public void SnapshotRoundTrip_PreservesRecoveryFacts()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
state = ApplyDeviceSetupResolved(engine, state).NewState;
var roundtripped = StateSerializer.Deserialize(StateSerializer.Serialize(state));
Assert.Equal("DeviceSetup", roundtripped.EspAdvisoryFailureCategory?.Value);
Assert.Equal(state.EspAdvisoryFailureResolvedUtc!.Value, roundtripped.EspAdvisoryFailureResolvedUtc?.Value);
Assert.Equal(state.EspAdvisoryFailureResolvedUtc.SourceSignalOrdinal, roundtripped.EspAdvisoryFailureResolvedUtc?.SourceSignalOrdinal);
}
[Fact]
public void LegacySnapshot_WithoutRecoveryFacts_DeserializesToNull()
{
// Simulate a snapshot written by a pre-fix agent: strip the new properties from
// the JSON entirely — deserialization must default them to null (additive compat).
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
state = ApplyDeviceSetupResolved(engine, state).NewState;
var legacy = Newtonsoft.Json.Linq.JObject.Parse(StateSerializer.Serialize(state));
foreach (var prop in legacy.Properties()
.Where(p => p.Name.IndexOf("EspAdvisoryFailureResolved", StringComparison.OrdinalIgnoreCase) >= 0
|| p.Name.IndexOf("EspAdvisoryFailureCategory", StringComparison.OrdinalIgnoreCase) >= 0)
.ToList())
{
prop.Remove();
}
var roundtripped = StateSerializer.Deserialize(legacy.ToString());
Assert.Null(roundtripped.EspAdvisoryFailureResolvedUtc);
Assert.Null(roundtripped.EspAdvisoryFailureCategory);
// The pre-existing advisory anchor still round-trips.
Assert.NotNull(roundtripped.EspAdvisoryFailureRecordedUtc);
}
[Fact]
public void Census_SurfacesAdvisoryResolved()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
state = ApplyDeviceSetupResolved(engine, state).NewState;
var census = DecisionStateSignalCensus.Build(state);
Assert.Contains("esp_advisory_failure_resolved", census.SignalsSeen);
Assert.True(census.SignalTimestamps.ContainsKey("espAdvisoryFailureResolved"));
}
}
}
@@ -114,7 +114,10 @@ namespace AutopilotMonitor.DecisionCore.Tests
// state-change-only dedupe anchor for the completion_waiting event → 17.
// Session 772fe502 (2026-07-13) added `helloWizardStartedUtc` — genuine
// Shell-Core 62404 wizard launch; vetoes the policy-disabled Hello skip → 18.
Assert.Equal(18, facts.Count);
// Session 4910a5a5 (2026-07-23) added `espAdvisoryFailureResolvedUtc` (the failed
// category recovered — lifts the advisory's re-arm/rebase guard exemption) and
// `espAdvisoryFailureCategory` (which category the advisory was for) → 20.
Assert.Equal(20, facts.Count);
Assert.All(facts.Values, v => Assert.Null(v));
}
@@ -236,10 +239,12 @@ namespace AutopilotMonitor.DecisionCore.Tests
// `completionWaitingFingerprint` as the state-change-only dedupe anchor for
// the completion_waiting event → 17. Session 772fe502 (2026-07-13) added
// `helloWizardStartedUtc` — the genuine Shell-Core 62404 wizard-launch fact
// that vetoes/retracts the policy-disabled Hello skip → 18. If this number
// ever changes, both the count expectation AND the actual snapshot output
// need to evolve in lockstep.
Assert.Equal(18, expectedFactKeys.Count);
// that vetoes/retracts the policy-disabled Hello skip → 18. Session 4910a5a5
// (2026-07-23) added `espAdvisoryFailureResolvedUtc` + `espAdvisoryFailureCategory`
// — advisory-recovery facts that lift the AdvisoryCompletion guard exemption → 20.
// If this number ever changes, both the count expectation AND the actual snapshot
// output need to evolve in lockstep.
Assert.Equal(20, expectedFactKeys.Count);
var snapshot = DecisionStateSnapshotBuilder.Build(DecisionState.CreateInitial("s", "t"));
var facts = (Dictionary<string, object?>)snapshot["facts"]!;
@@ -0,0 +1,107 @@
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Session eaf3d8c4 (2026-07-23): a previous enrollment's IME log surviving on disk made the
/// agent replay a week-old script history — 156 replayed <c>script_completed</c> events
/// inflated the session's RemediationScriptCount. Newer agents suppress the replay at the
/// source; this predicate covers the fleet of not-yet-updated agents by skipping the script
/// counter increment for events whose <c>rejectedSourceTimestamp</c> (the source log line's
/// own time, preserved by the agent's staleness clamp) is more than 24 h older than the
/// event's clock-clamped stamp.
/// </summary>
public class IngestHistoricImeReplayTests
{
private static readonly DateTime EventStamp = new(2026, 7, 23, 15, 42, 0, DateTimeKind.Utc);
private static EnrollmentEvent ScriptEvent(object? rejectedSourceTimestamp, bool includeKey = true)
{
var evt = new EnrollmentEvent
{
EventType = "script_completed",
Timestamp = EventStamp,
Data = new Dictionary<string, object> { ["scriptType"] = "remediation" },
};
if (includeKey)
evt.Data["rejectedSourceTimestamp"] = rejectedSourceTimestamp!;
return evt;
}
[Fact]
public void StaleRejectedSourceTimestamp_IsHistoricReplay()
{
var evt = ScriptEvent(EventStamp.AddDays(-7).ToString("o"));
Assert.True(EventIngestProcessor.IsHistoricImeReplay(evt));
}
[Fact]
public void StaleAppInstallEvent_IsHistoricReplay()
{
// The predicate is event-type-agnostic — it also guards AggregateAppInstallEvent
// (AppInstallSummaries would otherwise be overwritten with week-old runs).
var evt = new EnrollmentEvent
{
EventType = "app_install_completed",
Timestamp = EventStamp,
Data = new Dictionary<string, object>
{
["appName"] = "Contoso App",
["rejectedSourceTimestamp"] = EventStamp.AddDays(-7).ToString("o"),
},
};
Assert.True(EventIngestProcessor.IsHistoricImeReplay(evt));
}
[Fact]
public void JustOver24h_IsHistoricReplay()
{
var evt = ScriptEvent(EventStamp.AddHours(-25).ToString("o"));
Assert.True(EventIngestProcessor.IsHistoricImeReplay(evt));
}
[Fact]
public void Within24h_IsNotHistoricReplay()
{
var evt = ScriptEvent(EventStamp.AddHours(-23).ToString("o"));
Assert.False(EventIngestProcessor.IsHistoricImeReplay(evt));
}
[Fact]
public void FutureSkewRejection_IsNotHistoricReplay()
{
// A rejected source timestamp AHEAD of the event stamp is a mid-enrollment clock
// jump (WhiteGlove +1h CMTrace skew), not replayed history.
var evt = ScriptEvent(EventStamp.AddHours(2).ToString("o"));
Assert.False(EventIngestProcessor.IsHistoricImeReplay(evt));
}
[Fact]
public void MissingKey_IsNotHistoricReplay()
{
Assert.False(EventIngestProcessor.IsHistoricImeReplay(ScriptEvent(null, includeKey: false)));
}
[Fact]
public void NullData_IsNotHistoricReplay()
{
var evt = new EnrollmentEvent { EventType = "script_completed", Timestamp = EventStamp, Data = null! };
Assert.False(EventIngestProcessor.IsHistoricImeReplay(evt));
}
[Theory]
[InlineData("")]
[InlineData("not-a-date")]
public void UnparseableValue_IsNotHistoricReplay(string raw)
{
Assert.False(EventIngestProcessor.IsHistoricImeReplay(ScriptEvent(raw)));
}
[Fact]
public void NullEvent_IsNotHistoricReplay()
{
Assert.False(EventIngestProcessor.IsHistoricImeReplay(null));
}
}
@@ -235,6 +235,80 @@ public class RuleEnginePreconditionTests
Assert.Single(outcome.Results);
}
// ===== ANALYZE-SEC-001 v3: firmware-confirmation marker (uefiCa2023FirmwareConfirmed) =====
/// <summary>
/// v3 core fix: when the agent's direct UEFI db read confirms the 2023 certificate, the
/// one-sided marker is present and the not_exists precondition suppresses the rule — even
/// though the servicing status ("unknown") would still satisfy both required conditions.
/// </summary>
[Fact]
public async Task Sec001V3_FirmwareConfirmed_MarkerSuppressesRule()
{
var rule = MakeSecureBootRuleV3();
var events = new List<EnrollmentEvent>
{
HardwareSpecEvent(isVirtualMachine: "false"),
SecureBootStatusEvent(uefiCA2023Status: "unknown", secureBootEnabled: "true",
extraData: new Dictionary<string, object>
{
["uefiDbHasWindowsUefiCa2023"] = true,
["uefiCa2023FirmwareConfirmed"] = true
})
};
var outcome = await RunAsync(rule, events);
Assert.Empty(outcome.Results);
Assert.Single(outcome.EvaluatedRules);
}
/// <summary>
/// Backward compatibility: an old-agent payload (no firmware fields at all) must behave
/// exactly like v2 — marker absent → precondition passes → rule fires on the servicing
/// status, at base confidence (no firmware factors).
/// </summary>
[Fact]
public async Task Sec001V3_OldAgentPayload_FiresLikeV2()
{
var rule = MakeSecureBootRuleV3();
var events = new List<EnrollmentEvent>
{
HardwareSpecEvent(isVirtualMachine: "false"),
SecureBootStatusEvent(uefiCA2023Status: "unknown", secureBootEnabled: "true")
};
var outcome = await RunAsync(rule, events);
Assert.Single(outcome.Results);
Assert.Equal(70, outcome.Results[0].ConfidenceScore);
}
/// <summary>
/// New agent, firmware-verified missing: the marker is omitted (never emitted as false),
/// so the rule fires and the firmware evidence raises confidence (+20 db, +10 KEK).
/// </summary>
[Fact]
public async Task Sec001V3_FirmwareVerifiedMissing_RaisesConfidence()
{
var rule = MakeSecureBootRuleV3();
var events = new List<EnrollmentEvent>
{
HardwareSpecEvent(isVirtualMachine: "false"),
SecureBootStatusEvent(uefiCA2023Status: "unknown", secureBootEnabled: "true",
extraData: new Dictionary<string, object>
{
["uefiDbHasWindowsUefiCa2023"] = false,
["uefiKekHasMicrosoftKek2kCa2023"] = false
})
};
var outcome = await RunAsync(rule, events);
Assert.Single(outcome.Results);
Assert.Equal(100, outcome.Results[0].ConfidenceScore);
}
// ===== Helpers =====
private static AnalyzeRule MakeSecureBootRule(List<RulePrecondition>? preconditions)
@@ -258,6 +332,29 @@ public class RuleEnginePreconditionTests
};
}
/// <summary>
/// Mirrors the shipped ANALYZE-SEC-001 v3.0.0 (rules/analyze/ANALYZE-SEC-001.json):
/// VM-skip + firmware-confirmed-marker preconditions, the two required servicing
/// conditions, and the optional firmware conditions with their confidence weights.
/// Keep in sync when the rule JSON changes.
/// </summary>
private static AnalyzeRule MakeSecureBootRuleV3()
{
var rule = MakeSecureBootRule(preconditions: new List<RulePrecondition>
{
new() { Source = "event_data", EventType = "hardware_spec", DataField = "isVirtualMachine", Operator = "equals", Value = "false" },
new() { Source = "event_data", EventType = "secureboot_status", DataField = "uefiCa2023FirmwareConfirmed", Operator = "not_exists", Value = "" }
});
rule.Conditions.Add(new RuleCondition { Signal = "db_ca2023_firmware_missing", Source = "event_data", EventType = "secureboot_status", DataField = "uefiDbHasWindowsUefiCa2023", Operator = "equals", Value = "false", Required = false });
rule.Conditions.Add(new RuleCondition { Signal = "kek_2023_missing", Source = "event_data", EventType = "secureboot_status", DataField = "uefiKekHasMicrosoftKek2kCa2023", Operator = "equals", Value = "false", Required = false });
rule.ConfidenceFactors = new List<ConfidenceFactor>
{
new() { Signal = "db_ca2023_firmware_missing", Condition = "exists", Weight = 20 },
new() { Signal = "kek_2023_missing", Condition = "exists", Weight = 10 }
};
return rule;
}
private static EnrollmentEvent HardwareSpecEvent(string isVirtualMachine) => new()
{
EventId = Guid.NewGuid().ToString(),
@@ -280,20 +377,29 @@ public class RuleEnginePreconditionTests
Data = new Dictionary<string, object> { ["edition"] = edition }
};
private static EnrollmentEvent SecureBootStatusEvent(string uefiCA2023Status, string secureBootEnabled) => new()
private static EnrollmentEvent SecureBootStatusEvent(string uefiCA2023Status, string secureBootEnabled,
Dictionary<string, object>? extraData = null)
{
EventId = Guid.NewGuid().ToString(),
TenantId = TenantId,
SessionId = SessionId,
EventType = "secureboot_status",
Timestamp = DateTime.UtcNow,
Sequence = 3,
Data = new Dictionary<string, object>
var data = new Dictionary<string, object>
{
["uefiCA2023Status"] = uefiCA2023Status,
["uefiSecureBootEnabled"] = secureBootEnabled
}
};
};
if (extraData != null)
foreach (var kvp in extraData)
data[kvp.Key] = kvp.Value;
return new EnrollmentEvent
{
EventId = Guid.NewGuid().ToString(),
TenantId = TenantId,
SessionId = SessionId,
EventType = "secureboot_status",
Timestamp = DateTime.UtcNow,
Sequence = 3,
Data = data
};
}
private static async Task<AnalysisOutcome> RunAsync(AnalyzeRule rule, List<EnrollmentEvent> events)
{
@@ -22,6 +22,13 @@ namespace AutopilotMonitor.Functions.Services
if (!isRelevant) return;
// Legacy-agent stale-replay guard (session eaf3d8c4): app events replayed from a
// previous enrollment's IME log carry a rejectedSourceTimestamp > 24 h older than
// the event stamp. They would create/overwrite this session's AppInstallSummaries
// rows with week-old runs. The fixed agent suppresses them at the source; this
// covers agents not yet rolled out.
if (IsHistoricImeReplay(evt)) return;
var appName = evt.Data?.ContainsKey("appName") == true ? evt.Data["appName"]?.ToString()?.Trim() : null;
if (string.IsNullOrEmpty(appName)) return;
@@ -82,6 +82,13 @@ namespace AutopilotMonitor.Functions.Services
break;
case "script_completed":
case "script_failed":
// Legacy-agent stale-replay guard: events flagged with a
// rejectedSourceTimestamp > 24 h older than the event stamp are replayed
// history from a previous enrollment (session eaf3d8c4 inflated
// RemediationScriptCount to 156 in a 7-min session). The fixed agent
// suppresses them at the source; this covers agents not yet rolled out.
if (IsHistoricImeReplay(evt))
break;
var scriptType = evt.Data?.ContainsKey("scriptType") == true
? evt.Data["scriptType"]?.ToString() : null;
if (string.Equals(scriptType, "platform", StringComparison.OrdinalIgnoreCase))
@@ -119,6 +126,28 @@ namespace AutopilotMonitor.Functions.Services
return string.Equals(reason, "max_lifetime", StringComparison.OrdinalIgnoreCase);
}
/// <summary>
/// True for an agent event whose <c>Data.rejectedSourceTimestamp</c> is more than 24 h
/// older than the event's own (clock-clamped) timestamp: the agent replayed IME log
/// content from a previous enrollment that survived on disk. Such runs happened days ago
/// and must not count as this session's script executions. A rejected source timestamp
/// in the FUTURE (negative difference) is a mid-enrollment clock jump, not a replay.
/// The 24 h bound mirrors the agent's source-timestamp staleness clamp.
/// </summary>
internal static bool IsHistoricImeReplay(EnrollmentEvent? evt)
{
if (evt?.Data == null || !evt.Data.ContainsKey("rejectedSourceTimestamp"))
return false;
var raw = evt.Data["rejectedSourceTimestamp"]?.ToString();
if (string.IsNullOrEmpty(raw))
return false;
if (!DateTime.TryParse(raw, System.Globalization.CultureInfo.InvariantCulture,
System.Globalization.DateTimeStyles.AssumeUniversal | System.Globalization.DateTimeStyles.AdjustToUniversal,
out var rejected))
return false;
return evt.Timestamp - rejected > TimeSpan.FromHours(24);
}
/// <summary>
/// True for an <c>enrollment_failed</c> event whose <c>Data.failureType</c> is
/// <c>agent_timeout</c> — the V1-parity companion of the max-lifetime watchdog shutdown
@@ -0,0 +1,53 @@
/**
* Historic-replay predicate (session eaf3d8c4): legacy agents replay IME log content from a
* previous enrollment; the replayed events carry `data.rejectedSourceTimestamp` > 24 h older
* than the event stamp. `get_session_summary` filters them out so appInstalls counts,
* errorCount (replayed app_install_failed are Error-severity) and the keyEvents triage
* timeline reflect only this enrollment's activity.
*/
import { describe, it, expect } from 'vitest';
import { isHistoricImeReplay } from '../tools/sessions.js';
const BASE = Date.UTC(2026, 6, 23, 15, 42, 0);
const DAY_MS = 24 * 60 * 60 * 1000;
const evt = (rejected?: unknown, timestamp: string = new Date(BASE).toISOString()): Record<string, unknown> => ({
timestamp,
eventType: 'app_install_completed',
data: rejected === undefined ? {} : { rejectedSourceTimestamp: rejected },
});
describe('isHistoricImeReplay', () => {
it('detects a rejected source timestamp > 24h older than the event stamp', () => {
expect(isHistoricImeReplay(evt(new Date(BASE - 7 * DAY_MS).toISOString()))).toBe(true);
expect(isHistoricImeReplay(evt(new Date(BASE - 25 * 60 * 60 * 1000).toISOString()))).toBe(true);
});
it('keeps within-24h rejections', () => {
expect(isHistoricImeReplay(evt(new Date(BASE - 23 * 60 * 60 * 1000).toISOString()))).toBe(false);
});
it('keeps future-skew rejections (clock jump, not replay)', () => {
expect(isHistoricImeReplay(evt(new Date(BASE + 2 * 60 * 60 * 1000).toISOString()))).toBe(false);
});
it('keeps events without or with malformed rejectedSourceTimestamp', () => {
expect(isHistoricImeReplay(evt())).toBe(false);
expect(isHistoricImeReplay(evt(''))).toBe(false);
expect(isHistoricImeReplay(evt('not-a-date'))).toBe(false);
expect(isHistoricImeReplay(evt(12345))).toBe(false);
expect(isHistoricImeReplay({ timestamp: new Date(BASE).toISOString() })).toBe(false);
});
it('keeps events with an unparseable event timestamp', () => {
expect(isHistoricImeReplay(evt(new Date(BASE - 7 * DAY_MS).toISOString(), 'garbage'))).toBe(false);
});
it('reads the snake_case wire variant', () => {
const e = {
timestamp: new Date(BASE).toISOString(),
data: { rejected_source_timestamp: new Date(BASE - 7 * DAY_MS).toISOString() },
};
expect(isHistoricImeReplay(e)).toBe(true);
});
});
@@ -23,6 +23,9 @@ export const EVENT_TYPES_CATALOG = {
'esp_failure_advisory',
'esp_failure_settle_started',
'esp_failure_settle_recovered',
'esp_failure_retry_detected',
'esp_failure_recovered',
'esp_failure_advisory_resolved',
'esp_exiting',
'mdm_policy_reboot_required',
'esp_config_detected',
@@ -34,6 +37,7 @@ export const EVENT_TYPES_CATALOG = {
'desktop_detector_started',
'desktop_detector_first_poll',
'desktop_detector_no_candidate',
'oobe_state_completed',
],
whiteglove_events: [
'whiteglove_complete',
@@ -76,7 +80,7 @@ export const EVENT_TYPES_CATALOG = {
'office_install_failed',
'office_preinstalled_detected',
],
script_events: ['script_started', 'script_completed', 'script_failed', 'script_timeout_suspected'],
script_events: ['script_started', 'script_completed', 'script_failed', 'script_timeout_suspected', 'historic_ime_replay_detected'],
network_events: [
'network_state_change',
'network_connectivity_check',
@@ -166,6 +170,7 @@ export const EVENT_TYPES_CATALOG = {
'telemetry_upload_blocked',
'session_parked_without_deadline',
'disk_space_low',
'esp_policy_provider_stalled',
'agent_late_start',
],
metrics_events: ['performance_snapshot', 'agent_metrics_snapshot', 'ingress_backpressure'],
@@ -248,6 +253,12 @@ export const DEVICE_PROPERTIES_CATALOG = {
},
secureboot_status: {
'secureboot_status.uefiSecureBootEnabled': { type: 'boolean', description: 'Secure Boot enabled' },
'secureboot_status.uefiCA2023Status': { type: 'string', description: 'Windows servicing rollout status of the UEFI CA 2023 certificate (registry; "Updated" = rollout verified). Lags behind actual firmware state — prefer the firmware fields below' },
'secureboot_status.uefiFirmwareReadStatus': { type: 'string', description: 'Result of the direct UEFI variable read: ok, not_uefi (legacy BIOS), variable_not_found, privilege_denied, or error_<code>' },
'secureboot_status.uefiDbHasWindowsUefiCa2023': { type: 'boolean', description: 'Firmware db contains "Windows UEFI CA 2023" (authoritative 2023-cert deployment state; read directly from UEFI, only present when the read succeeded)' },
'secureboot_status.uefiDbHasWindowsProductionPca2011': { type: 'boolean', description: 'Firmware db still contains the expired "Microsoft Windows Production PCA 2011"' },
'secureboot_status.uefiKekHasMicrosoftKek2kCa2023': { type: 'boolean', description: 'Firmware KEK contains "Microsoft Corporation KEK 2K CA 2023" (required to keep receiving db/dbx updates after the 2011 KEK expiry)' },
'secureboot_status.uefiCa2023FirmwareConfirmed': { type: 'boolean', description: 'One-sided marker: present (true) only when the firmware db positively confirms the 2023 certificate — suppresses ANALYZE-SEC-001 via not_exists precondition. Never emitted as false' },
},
bitlocker_status: {
'bitlocker_status.systemDriveProtected': { type: 'boolean', description: 'System drive (C:) BitLocker protection' },
@@ -28,7 +28,8 @@ export const KEY_EVENT_TYPES = new Set([
'enrollment_complete', 'enrollment_failed', 'completion_check',
'desktop_arrived', 'hello_policy_detected', 'waiting_for_hello', 'hello_completion_timeout',
'agent_started', 'agent_shutdown', 'agent_shutting_down', 'agent_trace',
'script_started', 'script_completed', 'script_failed', 'vulnerability_report',
'script_started', 'script_completed', 'script_failed', 'historic_ime_replay_detected',
'vulnerability_report',
]);
// Phase-defining events promoted to the top of the triage timeline. Module-level
// (not handler-local) so the same drift test can validate it.
@@ -36,6 +37,27 @@ export const PHASE_EVENT_TYPES = new Set([
'phase_transition', 'esp_phase_changed', 'enrollment_type_detected',
'enrollment_complete', 'enrollment_failed', 'desktop_arrived',
]);
// Mirrors the agent's 24 h source-timestamp staleness clamp (session eaf3d8c4).
const HISTORIC_REPLAY_THRESHOLD_MS = 24 * 60 * 60 * 1000;
/**
* True when an event is replayed history from a previous enrollment: legacy agents replay
* IME log content surviving a re-enrollment, stamping the rejected source-line time as
* `data.rejectedSourceTimestamp` — more than 24 h older than the event stamp means the
* activity happened during a previous enrollment. Newer agents suppress these at the
* source; this filter covers sessions recorded by older agents. A rejected timestamp in
* the FUTURE (clock jump) is genuine current activity and passes through. Exported so the
* unit tests can pin the predicate.
*/
export function isHistoricImeReplay(e: Record<string, unknown>): boolean {
const data = e.data as Record<string, unknown> | undefined;
const rejected = data?.rejectedSourceTimestamp ?? data?.rejected_source_timestamp;
if (typeof rejected !== 'string' || rejected.length === 0) return false;
const rej = Date.parse(rejected);
const ts = Date.parse(String(e.timestamp ?? ''));
return Number.isFinite(rej) && Number.isFinite(ts) && ts - rej > HISTORIC_REPLAY_THRESHOLD_MS;
}
// Permissive ISO-8601 guard: rejects unparseable junk (which the backend would
// silently treat as no filter) while still accepting the date-only and
// timezone-offset forms the backend honors.
@@ -482,7 +504,12 @@ export function registerSessionTools(server: McpServer, ga: boolean, delegated:
: null,
};
const allEvents = (eventsData?.events ?? []) as Array<Record<string, unknown>>;
// Historic-replay filter: replayed app_install_failed events are Error-severity and
// would inflate errorCount; replayed app_install_* would inflate the appInstalls
// stats; replayed events would pollute the keyEvents triage timeline. One filter
// cleans all three. stats.totalEvents therefore counts non-replayed events.
const allEvents = ((eventsData?.events ?? []) as Array<Record<string, unknown>>)
.filter((e) => !isHistoricImeReplay(e));
let errorCount = 0;
let warningCount = 0;
@@ -73,6 +73,11 @@ namespace AutopilotMonitor.DecisionCore.Engine
/// The advisory variant (<see cref="DecisionState.EspAdvisoryFailureRecordedUtc"/>
/// set) is deliberately NOT touched: its anchor is a real ESP terminal failure and
/// a reboot does not un-happen that failure — the window must still un-defang it.
/// EXCEPTION (session 4910a5a5, 2026-07-23): when
/// <see cref="DecisionState.EspAdvisoryFailureResolvedUtc"/> is set the failed
/// category demonstrably recovered to success — the failure DID un-happen, and a
/// reboot mid-recovery (ESP continuing into AccountSetup) must re-base the window
/// exactly like the esp-exit variant instead of letting it fail a live enrollment.
/// </para>
/// Returns the effects (cancel + schedule + a fingerprint-bypassing
/// <c>completion_waiting</c> carrying the re-based due-time, mirroring the
@@ -84,7 +89,9 @@ namespace AutopilotMonitor.DecisionCore.Engine
DecisionStateBuilder builder,
DecisionSignal signal)
{
if (state.EspAdvisoryFailureRecordedUtc != null || !HasAdvisoryCompletionDeadline(state))
var unresolvedAdvisoryAnchored = state.EspAdvisoryFailureRecordedUtc != null
&& state.EspAdvisoryFailureResolvedUtc == null;
if (unresolvedAdvisoryAnchored || !HasAdvisoryCompletionDeadline(state))
{
return Array.Empty<DecisionEffect>();
}
@@ -139,6 +146,10 @@ namespace AutopilotMonitor.DecisionCore.Engine
"mayHaveContinuedAnyway",
"continueAnywayHint",
"advisoryReason",
// Session 4910a5a5 — recovery story context.
"failureRecovered",
"likelyCulpritApps",
"likelyCulpritAppCount",
};
/// <summary>
@@ -408,10 +419,17 @@ namespace AutopilotMonitor.DecisionCore.Engine
string reason,
Dictionary<string, string> parameters)
{
// Session 4910a5a5: remember WHICH category failed so the recovery hook
// (TryResolveAdvisoryOnCategoryRecovery) can later match a ProvisioningComplete
// signal of the same category against this advisory. Absent on event-log-derived
// failures — the hook then never matches.
string? failedCategory = null;
signal.Payload?.TryGetValue("category", out failedCategory);
var builder = state.ToBuilder()
.WithStepIndex(nextStep)
.WithLastAppliedSignalOrdinal(signal.SessionSignalOrdinal)
.WithEspAdvisoryFailureRecorded(signal.OccurredAtUtc, signal.SessionSignalOrdinal);
.WithEspAdvisoryFailureRecorded(signal.OccurredAtUtc, signal.SessionSignalOrdinal, failedCategory);
// Stage / Outcome deliberately untouched — monitoring continues. The session's
// normal completion paths (Hello/Desktop, IME pattern, AccountSetup provisioning
@@ -454,6 +472,63 @@ namespace AutopilotMonitor.DecisionCore.Engine
return new DecisionStep(newState, transition, effects);
}
/// <summary>
/// Session 4910a5a5 (2026-07-23) — recovery hook for a defanged ESP failure. When a
/// <c>DeviceSetupProvisioningComplete</c> / <c>AccountSetupProvisioningComplete</c>
/// signal reports the SAME category the advisory failure was recorded for, the failure
/// demonstrably un-happened (DeviceSetup/Apps failed → user pressed "Try again" ~23 min
/// later → apps re-ran to 10/10 → category resolved). Sets the set-once
/// <see cref="DecisionState.EspAdvisoryFailureResolvedUtc"/> fact on
/// <paramref name="builder"/> and returns an <c>esp_failure_advisory_resolved</c>
/// timeline effect telling the recovery story; returns null (builder untouched) when
/// there is no advisory, it is already resolved, or the category does not match.
/// <para>
/// The fact lifts the advisory variant's exemption from the AdvisoryCompletion
/// re-arm/rebase guards (see <see cref="HandleAdvisoryCompletionDeadlineFired"/> and
/// <see cref="RebaseEspExitVariantAdvisoryWindowOnReboot"/>) — without it the window
/// un-defangs the stale failure and fails a live enrollment mid-AccountSetup.
/// </para>
/// </summary>
private static DecisionEffect? TryResolveAdvisoryOnCategoryRecovery(
DecisionState state,
DecisionStateBuilder builder,
DecisionSignal signal,
string resolvedCategory)
{
if (state.EspAdvisoryFailureRecordedUtc == null) return null;
if (state.EspAdvisoryFailureResolvedUtc != null) return null;
var advisoryCategory = state.EspAdvisoryFailureCategory?.Value;
if (!string.Equals(advisoryCategory, resolvedCategory, StringComparison.OrdinalIgnoreCase))
{
return null;
}
builder.WithEspAdvisoryFailureResolved(signal.OccurredAtUtc, signal.SessionSignalOrdinal);
var minutesSinceAdvisory = Math.Max(0,
(int)Math.Round((signal.OccurredAtUtc - state.EspAdvisoryFailureRecordedUtc.Value).TotalMinutes));
var continueAnywayAllowed = state.ScenarioObservations?.EspAllowContinueAnyway?.Value == true;
var parameters = new Dictionary<string, string>
{
["eventType"] = SharedConstants.EventTypes.EspFailureAdvisoryResolved,
["source"] = "DecisionEngine",
["severity"] = "Info",
["immediateUpload"] = "false",
["message"] = $"ESP failure advisory resolved: {resolvedCategory} completed successfully "
+ $"{minutesSinceAdvisory} min after the advisory — the ESP re-ran the failed step "
+ "(consistent with a user-initiated retry on the ESP failure page"
+ (continueAnywayAllowed ? "; the profile allows Try again / Continue anyway" : "")
+ "). The earlier failure no longer blocks completion.",
["category"] = resolvedCategory,
["minutesSinceAdvisory"] = minutesSinceAdvisory.ToString(CultureInfo.InvariantCulture),
["espAllowContinueAnyway"] = continueAnywayAllowed ? "true" : "false",
};
return new DecisionEffect(DecisionEffectKind.EmitEventTimelineEntry, parameters: parameters);
}
/// <summary>
/// <see cref="DeadlineNames.AdvisoryCompletion"/> fired — the resolution window after a
/// completion dead-end expired without one of the normal completion paths terminating
@@ -511,6 +586,12 @@ namespace AutopilotMonitor.DecisionCore.Engine
var deadlineStillArmed = armedDeadline != null;
var hasAdvisoryAnchor = state.EspAdvisoryFailureRecordedUtc != null;
// Session 4910a5a5: once the failed category demonstrably recovered
// (TryResolveAdvisoryOnCategoryRecovery set the fact), the advisory variant's
// "a real ESP terminal failure does not un-happen" premise is disproven — the
// guards below that protect live enrollments (enforcement-progress re-arm,
// hello-never-observed promote) apply to it again.
var advisoryResolved = state.EspAdvisoryFailureResolvedUtc != null;
// Presence-only anchor (L5, delta review 2026-07-02). The esp-exit arming site only
// runs when AccountSetupEnteredUtc is already set, so both facts being present is
// the anchor; `deadlineStillArmed` below is what rejects stray timer fires. The old
@@ -635,8 +716,13 @@ namespace AutopilotMonitor.DecisionCore.Engine
// When user-phase enforcement demonstrably progressed since arming, the dead-end
// shape is disproven: re-arm a fresh window with updated baselines instead of
// failing a live enrollment mid-install. The advisory variant is exempt — its
// anchor is a real ESP terminal failure that a busy IME does not un-happen.
if (!hasAdvisoryAnchor && HasEnforcementProgressSinceArming(state, armedDeadline!))
// anchor is a real ESP terminal failure that a busy IME does not un-happen —
// UNLESS the failed category later resolved to success (session 4910a5a5:
// DeviceSetup/Apps failed, user retried, apps reached 10/10, DeviceSetup resolved
// — the failure DID un-happen, so a live AccountSetup must not be failed on the
// stale advisory). Convergent either way: each re-arm renews the baselines, so a
// session that stalls after the recovery still fails one window later.
if ((!hasAdvisoryAnchor || advisoryResolved) && HasEnforcementProgressSinceArming(state, armedDeadline!))
{
var rearmedDeadline = BuildAdvisoryCompletionDeadline(state, signal);
builder.AddDeadline(rearmedDeadline);
@@ -692,7 +778,7 @@ namespace AutopilotMonitor.DecisionCore.Engine
// through Finalizing. Deliberately does NOT require the IME user-session gate —
// the FP sessions never saw an IME user-session completion (that gate guards
// completion evidence, not user presence; presence is the DAD-validated desktop).
if (!hasAdvisoryAnchor
if ((!hasAdvisoryAnchor || advisoryResolved)
&& desktopArrived
&& state.HelloResolvedUtc == null
&& state.HelloPolicyEnabled == null
@@ -741,7 +827,30 @@ namespace AutopilotMonitor.DecisionCore.Engine
// stays the deadline, not EspTerminalFailure).
string reason;
Dictionary<string, string> parameters;
if (hasAdvisoryAnchor)
if (hasAdvisoryAnchor && advisoryResolved)
{
// Session 4910a5a5: the original ESP failure demonstrably recovered (failed
// category resolved to success after the advisory), so claiming
// `esp_terminal_failure` would blame a failure that un-happened. The device
// recovered its device phase but then produced no completion evidence for a
// whole (re-armed, progress-gated) window. LastFailureTrigger stays the
// deadline so the EnrollmentTerminationHandler's likely-stuck app promotion
// stays OFF — the apps are not stuck, they succeeded.
reason = "esp_recovered_without_completion_evidence";
parameters = BuildEspFailureParameters(
signal: signal,
reason: reason,
advisoryPath: false,
observations: state.ScenarioObservations);
parameters["advisoryReason"] = "advisory_recovered_window_expired_without_completion_evidence";
parameters["failureRecovered"] = "true";
if (state.EspAdvisoryFailureCategory != null)
{
parameters["category"] = state.EspAdvisoryFailureCategory.Value;
}
builder.WithLastFailureTrigger(nameof(DecisionSignalKind.DeadlineFired), signal.SessionSignalOrdinal);
}
else if (hasAdvisoryAnchor)
{
// Registry-derived failure context (failureType/errorCode/...) lived on the
// original signal and is already on the wire via the esp_failure_advisory
@@ -885,6 +994,12 @@ namespace AutopilotMonitor.DecisionCore.Engine
parameters["failedSubcategory"] = failedSubcategory;
if (signal.Payload.TryGetValue("category", out var category) && !string.IsNullOrEmpty(category))
parameters["category"] = category;
// Session 4910a5a5 — tracker-snapshotted culprit-app names for Apps failures,
// so the advisory/terminal event tells WHICH app ESP most likely failed on.
if (signal.Payload.TryGetValue("likelyCulpritApps", out var culpritApps) && !string.IsNullOrEmpty(culpritApps))
parameters["likelyCulpritApps"] = culpritApps;
if (signal.Payload.TryGetValue("likelyCulpritAppCount", out var culpritCount) && !string.IsNullOrEmpty(culpritCount))
parameters["likelyCulpritAppCount"] = culpritCount;
}
// FirstSync-derived ESP failure-handling settings — observable hint that
@@ -98,17 +98,28 @@ namespace AutopilotMonitor.DecisionCore.Engine
// was already armed (or already cancelled by AccountSetup). Pass-through.
if (state.DeviceSetupResolvedUtc != null)
{
var passthroughState = state.ToBuilder()
var passthroughBuilder = state.ToBuilder()
.WithStepIndex(nextStep)
.WithLastAppliedSignalOrdinal(signal.SessionSignalOrdinal)
.Build();
.WithLastAppliedSignalOrdinal(signal.SessionSignalOrdinal);
// Session 4910a5a5 recovery hook: a duplicate arrival (e.g. the post-reboot
// registry re-read) can still be the first evidence that the advisory's failed
// category recovered — set-once + category-match inside the helper keep this
// a no-op everywhere else.
var passthroughResolveEffect = TryResolveAdvisoryOnCategoryRecovery(
state, passthroughBuilder, signal, resolvedCategory: "DeviceSetup");
var passthroughState = passthroughBuilder.Build();
var passthroughTransition = BuildTakenTransition(
before: state,
signal: signal,
toStage: state.Stage,
nextStepIndex: nextStep,
trigger: nameof(DecisionSignalKind.DeviceSetupProvisioningComplete) + ":AnchorAlreadySet");
return new DecisionStep(passthroughState, passthroughTransition, Array.Empty<DecisionEffect>());
return new DecisionStep(
passthroughState,
passthroughTransition,
passthroughResolveEffect != null
? new[] { passthroughResolveEffect }
: Array.Empty<DecisionEffect>());
}
// Step 2 — Set the DeviceSetupResolvedUtc anchor UNCONDITIONALLY. Even when Classic
@@ -119,6 +130,13 @@ namespace AutopilotMonitor.DecisionCore.Engine
.WithLastAppliedSignalOrdinal(signal.SessionSignalOrdinal);
builder.DeviceSetupResolvedUtc = new SignalFact<DateTime>(signal.OccurredAtUtc, signal.SessionSignalOrdinal);
// Session 4910a5a5 recovery hook: DeviceSetup resolving to success while a
// DeviceSetup advisory failure is on record means the failure un-happened (user
// "Try again" retry). Sets the set-once resolved fact + emits the
// esp_failure_advisory_resolved story event; no-op otherwise.
var advisoryResolveEffect = TryResolveAdvisoryOnCategoryRecovery(
state, builder, signal, resolvedCategory: "DeviceSetup");
// Step 3 — Classic-path short-circuit: AccountSetup already started. The DeviceOnly
// deadline is moot (already cancelled by HandleEspPhaseChangedV1's AccountSetup branch,
// or never armed under the new code). No deadline arm here.
@@ -136,7 +154,12 @@ namespace AutopilotMonitor.DecisionCore.Engine
toStage: state.Stage,
nextStepIndex: nextStep,
trigger: nameof(DecisionSignalKind.DeviceSetupProvisioningComplete) + ":AccountSetupAlreadyEntered");
return new DecisionStep(classicState, classicTransition, Array.Empty<DecisionEffect>());
return new DecisionStep(
classicState,
classicTransition,
advisoryResolveEffect != null
? new[] { advisoryResolveEffect }
: Array.Empty<DecisionEffect>());
}
// Step 4 — Arm the deadline. Cancel any existing DeviceOnlyEspDetection first
@@ -156,13 +179,14 @@ namespace AutopilotMonitor.DecisionCore.Engine
nextStepIndex: nextStep,
trigger: nameof(DecisionSignalKind.DeviceSetupProvisioningComplete) + ":DeadlineArmed");
var effects = new[]
var effects = new List<DecisionEffect>(3)
{
new DecisionEffect(DecisionEffectKind.CancelDeadline, cancelDeadlineName: DeadlineNames.DeviceOnlyEspDetection),
new DecisionEffect(DecisionEffectKind.ScheduleDeadline, deadline: deadline),
};
if (advisoryResolveEffect != null) effects.Add(advisoryResolveEffect);
return new DecisionStep(newState, transition, effects);
return new DecisionStep(newState, transition, effects.ToArray());
}
/// <summary>
@@ -836,6 +836,13 @@ namespace AutopilotMonitor.DecisionCore.Engine
.WithStepIndex(nextStep)
.WithLastAppliedSignalOrdinal(signal.SessionSignalOrdinal);
// Session 4910a5a5 recovery hook (AccountSetup counterpart): the user-phase
// category resolving to success while an AccountSetup advisory failure is on
// record means that failure un-happened. Sets the set-once resolved fact + emits
// the esp_failure_advisory_resolved story event; no-op otherwise.
var advisoryResolveEffect = TryResolveAdvisoryOnCategoryRecovery(
state, builder, signal, resolvedCategory: "AccountSetup");
var alreadyRecorded = state.AccountSetupProvisioningSucceededUtc != null;
if (!alreadyRecorded)
{
@@ -876,10 +883,10 @@ namespace AutopilotMonitor.DecisionCore.Engine
toStage: state.Stage,
nextStepIndex: nextStep,
trigger: nameof(DecisionSignalKind.AccountSetupProvisioningComplete));
return new DecisionStep(
newState,
transition,
waitingEffect != null ? new[] { waitingEffect } : Array.Empty<DecisionEffect>());
var noPromoteEffects = new List<DecisionEffect>(2);
if (advisoryResolveEffect != null) noPromoteEffects.Add(advisoryResolveEffect);
if (waitingEffect != null) noPromoteEffects.Add(waitingEffect);
return new DecisionStep(newState, transition, noPromoteEffects.ToArray());
}
// Deferred-completion parity (session caa6cf50 fix, 2026-06-11): when the strong gate
@@ -910,14 +917,18 @@ namespace AutopilotMonitor.DecisionCore.Engine
builder.CancelDeadline(DeadlineNames.HelloSafety);
}
var deferredLeadingEffects = new List<DecisionEffect>(2);
if (advisoryResolveEffect != null) deferredLeadingEffects.Add(advisoryResolveEffect);
if (helloSafetyCancelEffect != null) deferredLeadingEffects.Add(helloSafetyCancelEffect);
return CompleteThroughFinalizingOrDefer(
state: state,
signal: signal,
preparedBuilder: builder,
nextStepIndex: nextStep,
trigger: nameof(DecisionSignalKind.AccountSetupProvisioningComplete) + ":DeferredCompletion",
leadingEffects: helloSafetyCancelEffect != null
? new[] { helloSafetyCancelEffect }
leadingEffects: deferredLeadingEffects.Count > 0
? deferredLeadingEffects.ToArray()
: null);
}
@@ -946,12 +957,13 @@ namespace AutopilotMonitor.DecisionCore.Engine
nextStepIndex: nextStep,
trigger: nameof(DecisionSignalKind.AccountSetupProvisioningComplete) + ":DeferredPromote");
var promotedEffects = new[]
var promotedEffects = new List<DecisionEffect>(2)
{
new DecisionEffect(DecisionEffectKind.ScheduleDeadline, deadline: helloSafety),
};
if (advisoryResolveEffect != null) promotedEffects.Add(advisoryResolveEffect);
return new DecisionStep(promotedState, promotedTransition, promotedEffects);
return new DecisionStep(promotedState, promotedTransition, promotedEffects.ToArray());
}
/// <summary>
@@ -110,6 +110,17 @@ namespace AutopilotMonitor.DecisionCore.Engine
timestamps["imeUserSessionCompleted"] = FormatUtc(state.ImeUserSessionCompletedUtc.Value);
evidence["imeUserSessionCompleted"] = TimestampedEvidence(state.ImeUserSessionCompletedUtc);
}
// Session 4910a5a5 — the ESP category behind a defanged advisory failure later
// resolved to success (user "Try again" recovery). Surfaced so terminal audit
// trails show that the original failure demonstrably un-happened — the
// discriminant between "failed and stayed failed" and "failed, recovered, but
// produced no completion evidence".
if (state.EspAdvisoryFailureResolvedUtc != null)
{
seen.Add("esp_advisory_failure_resolved");
timestamps["espAdvisoryFailureResolved"] = FormatUtc(state.EspAdvisoryFailureResolvedUtc.Value);
evidence["espAdvisoryFailureResolved"] = TimestampedEvidence(state.EspAdvisoryFailureResolvedUtc);
}
// Session 772fe502 — genuine Hello-wizard launch (Shell-Core 62404, CXID AADHello/
// NGC). Surfaced in the census so terminal audit trails show whether a wizard was
// observed — the discriminant between "Hello skipped by policy" and "Hello skipped
@@ -63,6 +63,8 @@ namespace AutopilotMonitor.DecisionCore.Engine
["helloPolicyEnabled"] = SerializeFact(state.HelloPolicyEnabled, v => (object)v),
["lastFailureTrigger"] = SerializeFact(state.LastFailureTrigger, v => v),
["espAdvisoryFailureRecordedUtc"] = SerializeFact(state.EspAdvisoryFailureRecordedUtc, FormatUtc),
["espAdvisoryFailureResolvedUtc"] = SerializeFact(state.EspAdvisoryFailureResolvedUtc, FormatUtc),
["espAdvisoryFailureCategory"] = SerializeFact(state.EspAdvisoryFailureCategory, v => v),
["imeUserSessionCompletedUtc"] = SerializeFact(state.ImeUserSessionCompletedUtc, FormatUtc),
["completionWaitingFingerprint"] = SerializeFact(state.CompletionWaitingFingerprint, v => v),
["helloWizardStartedUtc"] = SerializeFact(state.HelloWizardStartedUtc, FormatUtc),
@@ -64,7 +64,9 @@ namespace AutopilotMonitor.DecisionCore.State
SignalFact<DateTime>? espAdvisoryFailureRecordedUtc = null,
SignalFact<DateTime>? imeUserSessionCompletedUtc = null,
SignalFact<string>? completionWaitingFingerprint = null,
SignalFact<DateTime>? helloWizardStartedUtc = null)
SignalFact<DateTime>? helloWizardStartedUtc = null,
SignalFact<DateTime>? espAdvisoryFailureResolvedUtc = null,
SignalFact<string>? espAdvisoryFailureCategory = null)
{
if (string.IsNullOrEmpty(sessionId))
{
@@ -108,6 +110,8 @@ namespace AutopilotMonitor.DecisionCore.State
ImeUserSessionCompletedUtc = imeUserSessionCompletedUtc;
CompletionWaitingFingerprint = completionWaitingFingerprint;
HelloWizardStartedUtc = helloWizardStartedUtc;
EspAdvisoryFailureResolvedUtc = espAdvisoryFailureResolvedUtc;
EspAdvisoryFailureCategory = espAdvisoryFailureCategory;
}
public string SessionId { get; }
@@ -320,6 +324,36 @@ namespace AutopilotMonitor.DecisionCore.State
/// </summary>
public SignalFact<DateTime>? HelloWizardStartedUtc { get; }
/// <summary>
/// Set when the ESP category behind a recorded advisory failure
/// (<see cref="EspAdvisoryFailureRecordedUtc"/>) later resolves to success — i.e. the
/// failure demonstrably un-happened. Session 4910a5a5 (2026-07-23): DeviceSetup/Apps
/// failed, the user pressed "Try again" ~23 min later, the apps re-ran to 10/10 and
/// <c>DeviceSetupProvisioningComplete</c> arrived — yet the <c>AdvisoryCompletion</c>
/// window still un-defanged the stale failure and failed the live enrollment.
/// <para>
/// Set-once by the shared recovery hook in the ProvisioningComplete handlers when the
/// resolved category matches the <c>armFailedCategory</c> carried on the armed
/// <c>AdvisoryCompletion</c> deadline. Once set, the advisory variant loses its
/// exemption from the enforcement-progress re-arm, the reboot rebase and the Hello
/// promotes — those guards exist to protect live enrollments, and with this fact the
/// "a real ESP terminal failure does not un-happen" premise is disproven for this
/// session. Additive-nullable, no snapshot-schema bump.
/// </para>
/// </summary>
public SignalFact<DateTime>? EspAdvisoryFailureResolvedUtc { get; }
/// <summary>
/// The ESP category (<c>DeviceSetup</c> / <c>AccountSetup</c>) named by the advisory
/// failure's signal payload, recorded alongside <see cref="EspAdvisoryFailureRecordedUtc"/>.
/// Lives on state — not on the <c>AdvisoryCompletion</c> deadline payload — because
/// re-arm and reboot-rebase sites rebuild that deadline and would silently drop a
/// payload-carried category. Null when the arming signal carried no category (e.g.
/// event-log-derived failures); the recovery hook then never matches and behavior is
/// unchanged. Additive-nullable, no snapshot-schema bump.
/// </summary>
public SignalFact<string>? EspAdvisoryFailureCategory { get; }
public string SchemaVersion { get; }
/// <summary>
@@ -51,6 +51,8 @@ namespace AutopilotMonitor.DecisionCore.State
ImeUserSessionCompletedUtc = source.ImeUserSessionCompletedUtc;
CompletionWaitingFingerprint = source.CompletionWaitingFingerprint;
HelloWizardStartedUtc = source.HelloWizardStartedUtc;
EspAdvisoryFailureResolvedUtc = source.EspAdvisoryFailureResolvedUtc;
EspAdvisoryFailureCategory = source.EspAdvisoryFailureCategory;
}
public string SessionId { get; set; }
@@ -84,6 +86,8 @@ namespace AutopilotMonitor.DecisionCore.State
public SignalFact<DateTime>? ImeUserSessionCompletedUtc { get; set; }
public SignalFact<string>? CompletionWaitingFingerprint { get; set; }
public SignalFact<DateTime>? HelloWizardStartedUtc { get; set; }
public SignalFact<DateTime>? EspAdvisoryFailureResolvedUtc { get; set; }
public SignalFact<string>? EspAdvisoryFailureCategory { get; set; }
public string SchemaVersion { get; set; }
// ---------- fluent helpers for the most common reducer operations ----------
@@ -197,9 +201,25 @@ namespace AutopilotMonitor.DecisionCore.State
/// Fire-once gate: subsequent <c>EspTerminalFailure</c> signals see this fact set and
/// dead-end without further effect.
/// </summary>
public DecisionStateBuilder WithEspAdvisoryFailureRecorded(DateTime utc, long sourceSignalOrdinal)
public DecisionStateBuilder WithEspAdvisoryFailureRecorded(
DateTime utc, long sourceSignalOrdinal, string? failedCategory = null)
{
EspAdvisoryFailureRecordedUtc = new SignalFact<DateTime>(utc, sourceSignalOrdinal);
if (!string.IsNullOrEmpty(failedCategory))
{
EspAdvisoryFailureCategory = new SignalFact<string>(failedCategory!, sourceSignalOrdinal);
}
return this;
}
/// <summary>
/// Record that the ESP category behind the recorded advisory failure later resolved to
/// success (session 4910a5a5 — "Try again" recovery). Set-once; lifts the advisory
/// variant's exemption from the AdvisoryCompletion re-arm/rebase guards.
/// </summary>
public DecisionStateBuilder WithEspAdvisoryFailureResolved(DateTime utc, long sourceSignalOrdinal)
{
EspAdvisoryFailureResolvedUtc = new SignalFact<DateTime>(utc, sourceSignalOrdinal);
return this;
}
@@ -236,6 +256,8 @@ namespace AutopilotMonitor.DecisionCore.State
espAdvisoryFailureRecordedUtc: EspAdvisoryFailureRecordedUtc,
imeUserSessionCompletedUtc: ImeUserSessionCompletedUtc,
completionWaitingFingerprint: CompletionWaitingFingerprint,
helloWizardStartedUtc: HelloWizardStartedUtc);
helloWizardStartedUtc: HelloWizardStartedUtc,
espAdvisoryFailureResolvedUtc: EspAdvisoryFailureResolvedUtc,
espAdvisoryFailureCategory: EspAdvisoryFailureCategory);
}
}
@@ -317,6 +317,12 @@ namespace AutopilotMonitor.Shared
public const string DesktopDetectorStarted = "desktop_detector_started"; // 1x on Start() / ResetForRealUserSwitch()
public const string DesktopDetectorFirstPoll = "desktop_detector_first_poll"; // 1x after first PollForDesktop() completes
public const string DesktopDetectorNoCandidate = "desktop_detector_no_candidate"; // 1x after threshold polls without resolution (configurable)
// 1x when WinRT SystemSetupInfo.OutOfBoxExperienceState flips InProgress->Completed,
// sampled on the DAD 30s poll. Observational only (no decision-engine involvement) —
// an owner-independent desktop corroboration for sessions where WTS+WMI owner
// resolution starves and desktop_arrived never fires. Often absent in healthy
// sessions: the DAD timer is disposed at desktop arrival, which is when the flip lands.
public const string OobeStateCompleted = "oobe_state_completed";
public const string CompletionCheck = "completion_check";
// Liveness plan PR2 — emitted (state-change-only, deduped via the
// CompletionWaitingFingerprint state fact) whenever the DecisionEngine blocks or
@@ -334,6 +340,13 @@ namespace AutopilotMonitor.Shared
// start→completion duration (works on replay since start carries the source log
// timestamp). Warning, advisory ("suspected"); no state mutation.
public const string ScriptTimeoutSuspected = "script_timeout_suspected";
// One-shot per agent run: the IME adapter detected replayed log content from a
// previous enrollment (source lines > 24 h stale, e.g. IME logs surviving a
// re-enrollment) and suppressed the historic script (script_started/completed/
// failed) and app (app_install_*/download_progress/do_telemetry) events.
// Data.earliestRejectedSourceTimestamp dates the replayed content so operators
// know why those events are missing (session eaf3d8c4).
public const string HistoricImeReplayDetected = "historic_ime_replay_detected";
public const string EspProvisioningStatus = "esp_provisioning_status";
public const string SoftwareInventoryAnalysis = "software_inventory_analysis";
public const string VulnerabilityReport = "vulnerability_report";
@@ -362,6 +375,7 @@ namespace AutopilotMonitor.Shared
public const string TelemetryUploadBlocked = "telemetry_upload_blocked"; // Warning (one-shot) when the drain is blocked by a retained unknown permanent 4xx (400 contract bug / 404 route mismatch / tenant-validation). Data is NOT discarded. LOCAL diagnostic (agent log + diag ZIP): this marker is queued behind the blocking batch and only reaches the backend after the block clears — no out-of-band bypass by design.
public const string SessionParkedWithoutDeadline = "session_parked_without_deadline"; // One-shot Warning tripwire (liveness plan PR1): a non-terminal session entered the post-AccountSetup dead-end zone (ESP final exit or advisory failure) with NO resolution-capable deadline armed. After the 2026-06-12 dead-end fixes this must never fire in production — every occurrence is a bug report for an unknown parking variant.
public const string DiskSpaceLow = "disk_space_low"; // One-shot Warning (state-change-only, 1 GB hysteresis — NOT periodic) when the system drive drops below 2 GB free. Review MON-B10: disk-full was previously only visible inside Debug performance_snapshot payloads (filtered out of timelines), so a disk-starved enrollment carried no actionable threshold signal.
public const string EspPolicyProviderStalled = "esp_policy_provider_stalled"; // One-shot-per-provider Warning: a registered EnrollmentStatusTracking CSP policy provider stayed incomplete >= 15 min (Setup/Apps TrackingPoliciesCreated != 1, or DevicePreparation InstallationState not in {2,3}). The ESP waits on Setup/Apps providers WITHOUT any OS timeout — e.g. pre-provisioning + co-management leaves only "ConfigMgr" registered (no "Sidecar" = IME) and the user ESP sits at "Apps (Identifying)" for days. Observational only; payload carries the full provider list. See EspPolicyProviderProbe.
public const string KeepAwakeEngaged = "keep_awake_engaged"; // Info (state-change, NOT periodic) emitted by the opt-in UserEspKeepAwake host when it holds the device awake (system + display) on entering the User-ESP/AccountSetup phase, so idle standby cannot stall app installs / account provisioning. Reboots are unaffected. Per-tenant toggle (AnalyzerConfiguration.KeepAwakeDuringUserEsp, default off).
public const string KeepAwakeReleased = "keep_awake_released"; // Info (Warning when reason=safety_cap) emitted when the keep-awake hold is released — reason = account_setup_complete | host_stop | safety_cap. Brackets KeepAwakeEngaged; the OS also auto-clears the hold on process exit/reboot.
@@ -532,6 +546,17 @@ namespace AutopilotMonitor.Shared
public const string EspFailureSettleRecovered = "esp_failure_settle_recovered"; // The failed subcategory left the "failed" state during the settle window (e.g. ESP "Try again" retry, session c071e92b) — the terminal EspFailureDetected fire is suppressed and monitoring continues; a subsequent failure re-arms a fresh settle window.
public const string EspAppxFailureAnalysis = "esp_appx_failure_analysis"; // One-shot AppXDeploymentServer/Operational scan during the ESP failure settle window: candidate MSIX/Store packages behind an "Apps (0x…)" subcategory failure invisible to ImeLogTracker (session 2bc884b6). Assessment, not a confirmed root cause.
public const string EspProvisioningRaw = "esp_provisioning_raw";
// Recovery story AFTER a terminally fired ESP failure (session 4910a5a5, 2026-07-23 —
// DeviceSetup/Apps failed, user pressed "Try again" ~23 min later, apps re-ran to
// 10/10). Distinct from EspFailureSettleRecovered, which covers recovery INSIDE the
// 30s settle window (terminal fire suppressed); these fire when the failure already
// went terminal (advisory/enrollment_failed) and the registry later disproves it.
public const string EspFailureRetryDetected = "esp_failure_retry_detected"; // failed subcategory left "failed" post-terminal — consistent with the user pressing "Try again"
public const string EspFailureRecovered = "esp_failure_recovered"; // previously failed category reached all-subcategories-succeeded post-terminal
// DecisionEngine counterpart (survives reboots via snapshot, unlike the in-run tracker
// pair above): the category behind a defanged esp_failure_advisory later resolved to
// success — the advisory no longer blocks; re-arm/rebase guards apply again.
public const string EspFailureAdvisoryResolved = "esp_failure_advisory_resolved";
public const string EspFailure = "esp_failure";
public const string EspExiting = "esp_exiting";
@@ -32,6 +32,8 @@ export const KNOWN_EVENT_TYPES: EventTypeEntry[] = [
description: "Pre-Provisioning Part 2 resumed after the user powered on the device." },
{ value: "desktop_arrived", label: "desktop_arrived", category: "enrollment",
description: "User desktop detected (explorer.exe under real user)." },
{ value: "oobe_state_completed", label: "oobe_state_completed", category: "enrollment",
description: "Windows OOBE state flipped InProgress→Completed (WinRT SystemSetupInfo, sampled on the desktop-arrival poll). One-shot and observational only; valuable as an owner-independent desktop corroboration in sessions where desktop_arrived never fires." },
{ value: "completion_check", label: "completion_check", category: "enrollment",
description: "Throttled state-machine snapshot during completion evaluation." },
{ value: "completion_waiting", label: "completion_waiting", category: "enrollment",
@@ -56,6 +58,8 @@ export const KNOWN_EVENT_TYPES: EventTypeEntry[] = [
description: "Stall probe found an anomaly (ModernDeployment error, DeploymentErrorCode≠0, IME EnforcementState Failed)." },
{ value: "stall_probe_check", label: "stall_probe_check", category: "stall",
description: "Trace-level heartbeat from a stall probe — proves the logic ran (default: only the 15 min probe)." },
{ value: "esp_policy_provider_stalled", label: "esp_policy_provider_stalled", category: "stall",
description: "A registered ESP policy provider (EnrollmentStatusTracking CSP) stayed incomplete for 15+ minutes — e.g. co-management leaves only ConfigMgr registered without the Intune Sidecar provider, which parks the user ESP at 'Apps (Identifying)' with no OS timeout. One-shot Warning; payload carries the full provider table, per-provider state, and sidecarRegistered." },
{ value: "agent_late_start", label: "agent_late_start", category: "stall",
description: "Low observation coverage: the agent started long after device boot (≥10 min) AND lived only briefly (≤5 min) before a terminal outcome — it arrived after the enrollment had already decided, so its diagnosis is a post-mortem of the end-state, not a live observation. Carries bootToAgentStartSeconds, agentUptimeSeconds and outcome. Often paired with script_timeout_suspected (a script that hung ahead of the agent bootstrap)." },
@@ -94,6 +98,12 @@ export const KNOWN_EVENT_TYPES: EventTypeEntry[] = [
description: "ESP reported a subcategory failure but the device had already progressed to AccountSetup with ContinueAnyway enabled — non-terminal advisory, the agent continues monitoring." },
{ value: "esp_appx_failure_analysis", label: "esp_appx_failure_analysis", category: "esp",
description: "AppX deployment log scan during the ESP failure settle window: suspected MSIX/Store package candidates behind an Apps-subcategory failure (assessment, not a confirmed root cause)." },
{ value: "esp_failure_retry_detected", label: "esp_failure_retry_detected", category: "esp",
description: "After a terminally reported ESP failure, the failed subcategory left the failed state — consistent with the user pressing 'Try again' on the ESP failure page." },
{ value: "esp_failure_recovered", label: "esp_failure_recovered", category: "esp",
description: "The ESP category behind a terminally reported failure later completed successfully — the earlier failure recovered (e.g. after a user retry)." },
{ value: "esp_failure_advisory_resolved", label: "esp_failure_advisory_resolved", category: "esp",
description: "DecisionEngine: the category behind a defanged esp_failure_advisory resolved to success — the advisory no longer blocks completion." },
// -------- App installs --------
{ value: "app_install_started", label: "app_install_started", category: "app",
@@ -132,6 +142,8 @@ export const KNOWN_EVENT_TYPES: EventTypeEntry[] = [
description: "A platform script crashed (exit code != 0), or the remediation phase of a health script crashed. Health-script detection / post-detection phases use exit code as a compliance verdict (non-zero = non-compliant) and emit script_completed even on non-zero exit; only a true script crash in those phases would surface here." },
{ value: "script_timeout_suspected", label: "script_timeout_suspected", category: "script",
description: "A platform script ran to the IME script-execution timeout (~30 min; threshold 25 min) and was marked Failed while the enrollment was in progress. IME runs platform scripts serially, so a hung script starves app installs and the Autopilot-Monitor bootstrap — the prime suspect behind a late agent start / pre-failed ESP. Advisory (one per policyId); carries durationSeconds, exitCode and espPhase." },
{ value: "historic_ime_replay_detected", label: "historic_ime_replay_detected", category: "diagnostics",
description: "One-shot per agent run: the IME log contained replayed script and app activity from a previous enrollment (source lines more than 24 h stale, e.g. IME logs surviving a re-enrollment). The historic script_* and app_install_*/download_progress/do_telemetry events were suppressed for this session; carries earliestRejectedSourceTimestamp so the replayed window is datable." },
// -------- Diagnostics / misc --------
{ value: "error_detected", label: "error_detected", category: "diagnostics",
@@ -297,7 +297,25 @@ export default function DeviceDetailsCard({ events, latestAgentVersion }: { even
{(bitLockerStatus || secureBootStatus || tpmStatus) && (
<DetailSection title="Security">
{secureBootStatus && (
<DetailRow label="SecureBoot" value={secureBootStatus.uefiSecureBootEnabled ? "Enabled" : "Disabled"} />
<>
<DetailRow label="SecureBoot" value={secureBootStatus.uefiSecureBootEnabled ? "Enabled" : "Disabled"} />
{/* Firmware verdict (direct UEFI db read, new agents) beats the servicing
registry status, which only reflects the Windows Update rollout. */}
{secureBootStatus.uefiDbHasWindowsUefiCa2023 !== undefined ? (
<DetailRow
label="UEFI CA 2023 (firmware)"
value={secureBootStatus.uefiDbHasWindowsUefiCa2023 ? "Deployed" : "Missing"}
/>
) : secureBootStatus.uefiFirmwareReadStatus ? (
<DetailRow
label="UEFI CA 2023 (firmware)"
value={`Unknown (${secureBootStatus.uefiFirmwareReadStatus})`}
/>
) : null}
{secureBootStatus.uefiCA2023Status && (
<DetailRow label="UEFI CA 2023 (servicing)" value={`${secureBootStatus.uefiCA2023Status}`} />
)}
</>
)}
{tpmStatus && (
<>
@@ -521,6 +521,46 @@ function EventRow({ event, showScriptOutput }: { event: EnrollmentEvent; showScr
subcategory: <span className="font-mono">{String(sub)}</span>
</span>
)}
{event.data?.likelyCulpritApps && (
<span
className="px-1.5 py-0.5 rounded bg-orange-100 text-orange-800 font-medium"
title="Tracked app(s) ESP most likely failed on — never-started apps ranked first (snapshot at failure time)."
>
Likely app: {String(event.data.likelyCulpritApps)}
</span>
)}
</div>
);
})()}
{/* Recovery-story badges (session 4910a5a5): a terminally reported ESP failure that
later un-happened — the user retried ("Try again"), the failed step re-ran and
recovered. Green/teal palette so the timeline visually closes the earlier red/amber
failure arc. */}
{(event.eventType === "esp_failure_retry_detected" || event.eventType === "esp_failure_recovered" || event.eventType === "esp_failure_advisory_resolved") && (() => {
const isRetry = event.eventType === "esp_failure_retry_detected";
const label = isRetry ? "User retry" : "Recovered";
const badgeCls = isRetry ? "bg-sky-100 text-sky-800" : "bg-emerald-100 text-emerald-800";
const cat = event.data?.category;
const sub = event.data?.subcategory ?? event.data?.failedSubcategory;
const mins = event.data?.minutesSinceFailure ?? event.data?.minutesSinceAdvisory;
return (
<div className="mt-1 flex flex-wrap items-center gap-2 text-xs">
<span
className={`px-1.5 py-0.5 rounded ${badgeCls} font-medium`}
title={isRetry
? "The failed ESP step left the failed state — consistent with the user pressing 'Try again' on the ESP failure page."
: "The previously failed ESP category completed successfully — the earlier failure no longer applies."}
>
{label}
</span>
{cat && (
<span className="text-gray-500">
{String(cat)}{sub ? <> / <span className="font-mono">{String(sub)}</span></> : null}
</span>
)}
{mins !== undefined && mins !== null && (
<span className="text-gray-500">{String(mins)} min after the failure</span>
)}
</div>
);
})()}
@@ -3,7 +3,7 @@
import { useMemo } from "react";
import { EnrollmentEvent, Session } from "@/types";
import { V1_PHASE_NAMES, V2_PHASE_NAMES, V1_PHASE_ORDER, V2_PHASE_ORDER } from "../utils/phaseConstants";
import { computeWhiteGloveSplitSequence, groupEventsByPhase } from "../utils/eventHelpers";
import { computeWhiteGloveDurations, computeWhiteGloveSplitSequence, groupEventsByPhase } from "../utils/eventHelpers";
interface PhaseGrouping {
eventsByPhase: Record<string, EnrollmentEvent[]>;
@@ -62,6 +62,10 @@ export function useSessionDerivedData(
// Extract latest app_tracking_summary state-breakdown for progress headers.
// Schema is the flat V1 shape — see AppTrackingSummaryBuilder.cs for the contract.
// Known legacy limitation: on sessions recorded by agents without the tracker-level
// historic-replay guard, the snapshot may count apps replayed from a previous
// enrollment's IME log — accepted; fixed agents keep replayed apps out of the summary
// at the source (ImeLogTracker AppMutatingActions guard).
const appSummaryStats = useMemo(() => {
const summaryEvents = events.filter(e => e.eventType === "app_tracking_summary");
if (summaryEvents.length === 0) return null;
@@ -176,33 +180,8 @@ export function useSessionDerivedData(
if (!isWhiteGloveSession) {
return { preProvDuration: null as string | null, userEnrollDuration: null as string | null, combinedDuration: null as string | null };
}
const preProvEvts = whiteGloveSplitSequence < 0 ? events : events.filter(e => e.sequence <= whiteGloveSplitSequence);
const userEnrollEvts = whiteGloveSplitSequence < 0 ? [] : events.filter(e => e.sequence > whiteGloveSplitSequence);
const calcMs = (evts: EnrollmentEvent[]): number => {
if (evts.length === 0) return 0;
const ts = evts.map(e => new Date(e.timestamp).getTime());
return Math.max(...ts) - Math.min(...ts);
};
const fmt = (ms: number): string | null => {
const sec = Math.round(ms / 1000);
if (sec < 1) return null;
if (sec < 60) return `${sec}s`;
if (sec < 3600) return `${Math.floor(sec / 60)}m ${sec % 60}s`;
return `${Math.floor(sec / 3600)}h ${Math.floor((sec % 3600) / 60)}m`;
};
const preProvMs = calcMs(preProvEvts);
const userEnrollMs = calcMs(userEnrollEvts);
return {
preProvDuration: fmt(preProvMs),
userEnrollDuration: fmt(userEnrollMs),
combinedDuration: fmt(preProvMs + userEnrollMs),
};
}, [events, isWhiteGloveSession, whiteGloveSplitSequence]);
return computeWhiteGloveDurations(events, whiteGloveSplitSequence, session?.startedAt);
}, [events, isWhiteGloveSession, whiteGloveSplitSequence, session?.startedAt]);
// Group events by phase — single timeline for normal sessions, two groups for WhiteGlove
const { eventsByPhase, orderedPhases } = useMemo(() => {
@@ -66,6 +66,69 @@ export function computeWhiteGloveSplitSequence(events: EnrollmentEvent[]): numbe
return -1;
}
// Pure helper — per-block durations for WhiteGlove sessions.
// Duration 1 = pre-provisioning, Duration 2 = user enrollment, combined = D1 + D2
// (the sealed pause between reseal and resume is excluded).
//
// Block spans are anchored on the SEQUENCE-canonical edge events (first/last event by
// sequence), NOT on min/max over all timestamps in the block. Two real-world cases break
// the min/max approach:
// - Events re-emitted after the Part-2 resume can legitimately carry historical event
// times (e.g. an app_install_started the ImeLogTracker re-parses from the IME log,
// stamped with the Part-1 install start). min() would stretch the User Enrollment
// block back across the sealed pause.
// - A single clock-skewed source timestamp (e.g. an IME log line with a wrong timezone
// bias that slips under the agent's 1h future-skew clamp) would stretch the block
// the other way.
// Sequence is the canonical event order, so the block edges by sequence are the robust
// span anchors. Part 1 additionally starts at session.startedAt (registration) when
// provided — the same anchor the backend uses for DurationSeconds.
export interface WhiteGloveDurations {
preProvDuration: string | null;
userEnrollDuration: string | null;
combinedDuration: string | null;
}
export function computeWhiteGloveDurations(
events: EnrollmentEvent[],
splitSequence: number,
startedAt?: string,
): WhiteGloveDurations {
const preProvEvts = splitSequence < 0 ? events : events.filter(e => e.sequence <= splitSequence);
const userEnrollEvts = splitSequence < 0 ? [] : events.filter(e => e.sequence > splitSequence);
const spanMs = (evts: EnrollmentEvent[], startOverrideMs?: number): number => {
if (evts.length === 0) return 0;
let first = evts[0];
let last = evts[0];
for (const e of evts) {
if (e.sequence < first.sequence) first = e;
if (e.sequence > last.sequence) last = e;
}
const start = startOverrideMs ?? new Date(first.timestamp).getTime();
const end = new Date(last.timestamp).getTime();
return Math.max(0, end - start);
};
const fmt = (ms: number): string | null => {
const sec = Math.round(ms / 1000);
if (sec < 1) return null;
if (sec < 60) return `${sec}s`;
if (sec < 3600) return `${Math.floor(sec / 60)}m ${sec % 60}s`;
return `${Math.floor(sec / 3600)}h ${Math.floor((sec % 3600) / 60)}m`;
};
const startedAtMs = startedAt ? new Date(startedAt).getTime() : NaN;
const preProvMs = spanMs(preProvEvts, Number.isFinite(startedAtMs) ? startedAtMs : undefined);
const userEnrollMs = spanMs(userEnrollEvts);
return {
preProvDuration: fmt(preProvMs),
userEnrollDuration: fmt(userEnrollMs),
combinedDuration: fmt(preProvMs + userEnrollMs),
};
}
// Pure helper — groups a flat event list into phase buckets.
// Extracted from the useMemo so it can be called multiple times for WhiteGlove split timelines.
//
@@ -1,6 +1,7 @@
"use client";
import { useMemo, useState } from "react";
import { partitionHistoricReplayEvents } from "@/lib/historicReplay";
import { shouldSkipLowBytesTotal, shouldSkipNoActivity } from "@/lib/downloadProgressFilters";
import { formatBytes, formatThroughput, formatDuration } from "@/lib/formatting";
import DoBreakdownBar from "./DoBreakdownBar";
@@ -84,10 +85,18 @@ function formatDoDuration(duration: string): string {
}
export default function DownloadProgress({ events, summaryStats }: DownloadProgressProps) {
const downloads = useMemo(() => {
if (events.length === 0) return [];
// Legacy-agent guard: drop download events replayed from a previous enrollment's IME log.
// Silent (empty finals set) — the InstallProgress panel already reports the hidden count
// for the same apps; a second note here would double-report them.
const current = useMemo(
() => partitionHistoricReplayEvents(events, new Set<string>()).current,
[events]
);
const sortedEvents = [...events].sort(
const downloads = useMemo(() => {
if (current.length === 0) return [];
const sortedEvents = [...current].sort(
(a, b) => new Date(a.timestamp).getTime() - new Date(b.timestamp).getTime()
);
@@ -185,7 +194,7 @@ export default function DownloadProgress({ events, summaryStats }: DownloadProgr
// This avoids active items jumping when completion status changes.
return a.firstSeenIndex - b.firstSeenIndex;
});
}, [events]);
}, [current]);
const [expanded, setExpanded] = useState(true);
const [showSkipped, setShowSkipped] = useState(false);
@@ -2,6 +2,7 @@
import { useEffect, useMemo, useState } from "react";
import { getErrorCodeEntry, formatErrorCode } from "@/utils/errorCodeMap";
import { partitionHistoricReplayEvents } from "@/lib/historicReplay";
interface InstallEvent {
timestamp: string;
@@ -29,6 +30,10 @@ const ESP_APPS_TIMEOUT = "esp_apps_timeout";
const ESP_APPS_DETECTION_FAILURE = "esp_apps_detection_failure";
const ESP_APPS_INSTALL_FAILURE = "esp_apps_install_failure";
// Finals counted by the historic-replay partition — one per hidden install, so the note
// is not inflated by started/progress events of the same app.
const APP_FINAL_TYPES: ReadonlySet<string> = new Set(["app_install_completed", "app_install_failed"]);
interface InstallItem {
appName: string;
appId: string;
@@ -76,10 +81,18 @@ function formatDuration(ms: number): string {
}
export default function InstallProgress({ events, summaryStats }: InstallProgressProps) {
const installs = useMemo(() => {
if (events.length === 0) return [];
// Legacy-agent guard: split off app events replayed from a previous enrollment's IME log
// (newer agents suppress them at the source) so week-old installs never render as current.
// office_*/realmjoin_* events never carry rejectedSourceTimestamp and pass through untouched.
const { current, historicCount } = useMemo(
() => partitionHistoricReplayEvents(events, APP_FINAL_TYPES),
[events]
);
const sortedEvents = [...events].sort(
const installs = useMemo(() => {
if (current.length === 0) return [];
const sortedEvents = [...current].sort(
(a, b) => new Date(a.timestamp).getTime() - new Date(b.timestamp).getTime()
);
@@ -265,7 +278,7 @@ export default function InstallProgress({ events, summaryStats }: InstallProgres
}
return Array.from(installMap.values()).sort((a, b) => a.firstSeenIndex - b.firstSeenIndex);
}, [events]);
}, [current]);
const [expanded, setExpanded] = useState(true);
const [showSkipped, setShowSkipped] = useState(false);
@@ -288,7 +301,7 @@ export default function InstallProgress({ events, summaryStats }: InstallProgres
return showSkipped ? installs : installs.filter(d => d.state !== "Skipped");
}, [installs, showSkipped]);
if (installs.length === 0) return null;
if (installs.length === 0 && historicCount === 0) return null;
const activeCount = installs.filter(d => d.state === "Installing").length;
const completedCount = installs.filter(d => d.state === "Installed").length;
@@ -376,6 +389,14 @@ export default function InstallProgress({ events, summaryStats }: InstallProgres
{filteredInstalls.map((item) => (
<InstallItemRow key={item.appName} item={item} />
))}
{historicCount > 0 && (
<div
className="text-xs text-gray-400 italic"
title="These installs were replayed from IME log content that predates this enrollment by more than 24 hours — they ran during a previous enrollment on this device"
>
{historicCount} historic app install{historicCount === 1 ? "" : "s"} from a previous enrollment hidden
</div>
)}
</div>}
</div>
);
@@ -10,6 +10,7 @@ import {
isDetectOnlyRow,
isNonCompliantReport,
mapRemediationStatus,
partitionHistoricScriptEvents,
reduceScriptEvents,
scriptCardKey,
scriptItemKey,
@@ -38,11 +39,15 @@ interface ScriptExecutionsProps {
}
export default function ScriptExecutions({ events, showScriptOutput, latestBootstrapVersion, displayNamesByRefKey }: ScriptExecutionsProps) {
const cards = useMemo(() => groupScriptItems(reduceScriptEvents(events)), [events]);
// Legacy-agent guard: split off script events replayed from a previous enrollment's IME
// log (newer agents suppress them at the source) so week-old runs never render as current
// executions. The muted note below keeps the gap explainable.
const { current, historicCount } = useMemo(() => partitionHistoricScriptEvents(events), [events]);
const cards = useMemo(() => groupScriptItems(reduceScriptEvents(current)), [current]);
const [expanded, setExpanded] = useState(true);
if (cards.length === 0) return null;
if (cards.length === 0 && historicCount === 0) return null;
// Header counters work on cards (one per policy), so a remediated cycle reads as
// "1 succeeded" rather than "1 succeeded + 2 non-compliant" when its detection /
@@ -114,6 +119,14 @@ export default function ScriptExecutions({ events, showScriptOutput, latestBoots
displayNamesByRefKey={displayNamesByRefKey}
/>
))}
{historicCount > 0 && (
<div
className="text-xs text-gray-400 italic"
title="These executions were replayed from IME log content that predates this enrollment by more than 24 hours — they ran during a previous enrollment on this device"
>
{historicCount} historic script execution{historicCount === 1 ? "" : "s"} from a previous enrollment hidden
</div>
)}
</div>
)}
</div>
@@ -0,0 +1,77 @@
import { describe, it, expect } from "vitest";
import {
HISTORIC_REPLAY_THRESHOLD_MS,
partitionHistoricReplayEvents,
type ReplayInputEvent,
} from "../historicReplay";
const BASE = 1700000000000;
const DAY_MS = 24 * 60 * 60 * 1000;
const evt = (overrides: Partial<ReplayInputEvent> = {}): ReplayInputEvent => ({
timestamp: new Date(BASE).toISOString(),
eventType: "app_install_completed",
data: {},
...overrides,
});
const APP_FINALS = new Set(["app_install_completed", "app_install_failed"]);
describe("partitionHistoricReplayEvents", () => {
it("drops events whose rejectedSourceTimestamp is > 24h older than the event stamp", () => {
const historic = evt({ data: { rejectedSourceTimestamp: new Date(BASE - 7 * DAY_MS).toISOString() } });
const fresh = evt({ data: {} });
const { current, historicCount } = partitionHistoricReplayEvents([historic, fresh], APP_FINALS);
expect(current).toEqual([fresh]);
expect(historicCount).toBe(1);
});
it("counts only finals — dropped non-final events are removed uncounted", () => {
const staleTs = new Date(BASE - 7 * DAY_MS).toISOString();
const events = [
evt({ eventType: "app_download_started", data: { rejectedSourceTimestamp: staleTs } }),
evt({ eventType: "download_progress", data: { rejectedSourceTimestamp: staleTs } }),
evt({ eventType: "app_install_started", data: { rejectedSourceTimestamp: staleTs } }),
evt({ eventType: "app_install_completed", data: { rejectedSourceTimestamp: staleTs } }),
evt({ eventType: "app_install_failed", data: { rejectedSourceTimestamp: staleTs } }),
];
const { current, historicCount } = partitionHistoricReplayEvents(events, APP_FINALS);
expect(current).toEqual([]);
expect(historicCount).toBe(2);
});
it("keeps future-skew rejections (source timestamp ahead of the event stamp)", () => {
const skewed = evt({ data: { rejectedSourceTimestamp: new Date(BASE + 2 * 60 * 60 * 1000).toISOString() } });
const { current, historicCount } = partitionHistoricReplayEvents([skewed], APP_FINALS);
expect(current).toEqual([skewed]);
expect(historicCount).toBe(0);
});
it("keeps within-24h rejections, missing and malformed rejectedSourceTimestamp", () => {
const within = evt({ data: { rejectedSourceTimestamp: new Date(BASE - 23 * 60 * 60 * 1000).toISOString() } });
const none = evt({ data: {} });
const noData = evt({ data: undefined });
const malformed = evt({ data: { rejectedSourceTimestamp: "not-a-date" } });
const { current, historicCount } = partitionHistoricReplayEvents([within, none, noData, malformed], APP_FINALS);
expect(current).toHaveLength(4);
expect(historicCount).toBe(0);
});
it("reads the snake_case wire variant", () => {
const historic = evt({ data: { rejected_source_timestamp: new Date(BASE - 7 * DAY_MS).toISOString() } });
const { current, historicCount } = partitionHistoricReplayEvents([historic], APP_FINALS);
expect(current).toEqual([]);
expect(historicCount).toBe(1);
});
it("empty finals set filters silently (DownloadProgress mode)", () => {
const historic = evt({ data: { rejectedSourceTimestamp: new Date(BASE - 7 * DAY_MS).toISOString() } });
const { current, historicCount } = partitionHistoricReplayEvents([historic], new Set<string>());
expect(current).toEqual([]);
expect(historicCount).toBe(0);
});
it("exposes the 24h threshold constant", () => {
expect(HISTORIC_REPLAY_THRESHOLD_MS).toBe(DAY_MS);
});
});
@@ -7,6 +7,7 @@ import {
isDetectOnlyRow,
isNonCompliantReport,
mapRemediationStatus,
partitionHistoricScriptEvents,
reduceScriptEvents,
scriptItemKey,
toNumber,
@@ -905,3 +906,82 @@ describe("groupScriptItems — card durationSeconds is the max phase run time",
expect(cards[0].durationSeconds).toBeUndefined();
});
});
describe("partitionHistoricScriptEvents", () => {
// Agent-side clamp evidence: rejectedSourceTimestamp = the source log line's own time,
// timestamp = the clock-clamped event stamp. > 24 h older ⇒ replayed history from a
// previous enrollment (session eaf3d8c4).
const DAY_MS = 24 * 60 * 60 * 1000;
it("drops events whose rejectedSourceTimestamp is > 24 h older than the event stamp and counts finals", () => {
const historicFinal = finalEvent({
ts: 0,
data: { policyId: "p1", scriptType: "remediation", rejectedSourceTimestamp: new Date(1700000000000 - 7 * DAY_MS).toISOString() },
});
const historicStart = finalEvent({
eventType: "script_started",
ts: 1,
data: { policyId: "p1", scriptType: "remediation", rejectedSourceTimestamp: new Date(1700000000000 - 7 * DAY_MS).toISOString() },
});
const fresh = finalEvent({ ts: 2, data: { policyId: "p2", scriptType: "platform" } });
const { current, historicCount } = partitionHistoricScriptEvents([historicFinal, historicStart, fresh]);
expect(current).toEqual([fresh]);
// Only finals count toward the "hidden executions" note — dropped starts would
// double-count the same run.
expect(historicCount).toBe(1);
});
it("keeps future-skew rejections (source timestamp ahead of the event stamp)", () => {
const skewed = finalEvent({
ts: 0,
data: { policyId: "p1", scriptType: "platform", rejectedSourceTimestamp: new Date(1700000000000 + 2 * 60 * 60 * 1000).toISOString() },
});
const { current, historicCount } = partitionHistoricScriptEvents([skewed]);
expect(current).toEqual([skewed]);
expect(historicCount).toBe(0);
});
it("keeps within-24h rejections and events without or with malformed rejectedSourceTimestamp", () => {
const within = finalEvent({
ts: 0,
data: { policyId: "p1", scriptType: "platform", rejectedSourceTimestamp: new Date(1700000000000 - 23 * 60 * 60 * 1000).toISOString() },
});
const none = finalEvent({ ts: 1, data: { policyId: "p2", scriptType: "platform" } });
const malformed = finalEvent({ ts: 2, data: { policyId: "p3", scriptType: "platform", rejectedSourceTimestamp: "not-a-date" } });
const { current, historicCount } = partitionHistoricScriptEvents([within, none, malformed]);
expect(current).toHaveLength(3);
expect(historicCount).toBe(0);
});
it("reads the snake_case wire variant too", () => {
const historic = finalEvent({
ts: 0,
data: { policyId: "p1", scriptType: "platform", rejected_source_timestamp: new Date(1700000000000 - 7 * DAY_MS).toISOString() },
});
const { current, historicCount } = partitionHistoricScriptEvents([historic]);
expect(current).toEqual([]);
expect(historicCount).toBe(1);
});
});
describe("reduceScriptEvents — implausible duration blanking", () => {
it("blanks durations above 24 h (mixed-timeline artifacts from legacy agents)", () => {
// Session eaf3d8c4: clock-clamped completion minus raw week-old start = 612225 s ≈ 170 h.
const items = reduceScriptEvents([
finalEvent({ data: { policyId: "p1", scriptType: "remediation", scriptPart: "detection", exitCode: "0", complianceResult: "True", durationSeconds: "612225.00", durationBasis: "script_runtime" } }),
]);
expect(items[0].durationSeconds).toBeUndefined();
expect(items[0].reportedAfterSeconds).toBeUndefined();
});
it("keeps the 24 h boundary value and blanks cycle-basis values the same way", () => {
const items = reduceScriptEvents([
finalEvent({ data: { policyId: "p1", scriptType: "platform", exitCode: "0", result: "Success", durationSeconds: "86400", durationBasis: "script_runtime" } }),
finalEvent({ ts: 1, data: { policyId: "p2", scriptType: "remediation", scriptPart: "detection", exitCode: "0", complianceResult: "True", durationSeconds: "90000", durationBasis: "cycle_including_reporting_latency" } }),
]);
expect(items[0].durationSeconds).toBe(86400);
expect(items[1].reportedAfterSeconds).toBeUndefined();
});
});
@@ -0,0 +1,103 @@
import { describe, it, expect } from "vitest";
import { computeWhiteGloveDurations } from "../../app/sessions/[sessionId]/utils/eventHelpers";
import type { EnrollmentEvent } from "@/types";
// Fixture builder — sequence and timestamp are the only fields that matter here.
function ev(seq: number, eventType: string, isoTs: string): EnrollmentEvent {
return {
eventId: `e${seq}`,
sessionId: "s1",
timestamp: isoTs,
eventType,
severity: "Info",
source: "Agent",
phase: 0,
message: eventType,
sequence: seq,
};
}
describe("computeWhiteGloveDurations", () => {
it("computes both blocks from the sequence-canonical edge events", () => {
const events = [
ev(1, "agent_started", "2026-07-23T08:31:00Z"),
ev(2, "esp_phase_changed", "2026-07-23T08:35:00Z"),
ev(3, "agent_shutting_down", "2026-07-23T09:28:00Z"), // Part 1 ends (reseal)
ev(4, "agent_started", "2026-07-23T11:00:00Z"), // Part 2 boot after sealed pause
ev(5, "enrollment_complete", "2026-07-23T11:56:00Z"),
];
const d = computeWhiteGloveDurations(events, 3);
expect(d.preProvDuration).toBe("57m 0s");
expect(d.userEnrollDuration).toBe("56m 0s");
expect(d.combinedDuration).toBe("1h 53m"); // pause 09:28→11:00 excluded
});
it("ignores a historical event time re-emitted into the user block (post-resume IME re-parse)", () => {
// Real case (session fcb44595): after the Part-2 resume the ImeLogTracker re-emits an
// app_install_started carrying the ORIGINAL Part-1 install time. min/max over timestamps
// stretched the user block back across the sealed pause (56m became 3h18m).
const events = [
ev(1, "agent_started", "2026-07-23T08:31:00Z"),
ev(2, "agent_shutting_down", "2026-07-23T09:28:00Z"),
ev(3, "agent_started", "2026-07-23T11:00:00Z"),
ev(4, "app_install_started", "2026-07-23T08:37:57Z"), // historical event time, Part-2 sequence
ev(5, "enrollment_complete", "2026-07-23T11:56:00Z"),
];
const d = computeWhiteGloveDurations(events, 2);
expect(d.userEnrollDuration).toBe("56m 0s");
expect(d.combinedDuration).toBe("1h 53m");
});
it("ignores a clock-skewed outlier timestamp inside the pre-provisioning block", () => {
// Real case (session fcb44595): a do_telemetry event carried a +1h source timestamp
// (IME log line with a wrong timezone bias, just under the agent's future-skew clamp).
// min/max stretched Part 1 from 57m to 1h44m.
const events = [
ev(1, "agent_started", "2026-07-23T08:31:00Z"),
ev(2, "do_telemetry", "2026-07-23T10:13:42Z"), // +1h skewed, mid-block by sequence
ev(3, "agent_shutting_down", "2026-07-23T09:28:00Z"),
ev(4, "agent_started", "2026-07-23T11:00:00Z"),
ev(5, "enrollment_complete", "2026-07-23T11:56:00Z"),
];
const d = computeWhiteGloveDurations(events, 3);
expect(d.preProvDuration).toBe("57m 0s");
});
it("anchors Part 1 at session.startedAt when provided (backend DurationSeconds parity)", () => {
const events = [
ev(1, "agent_started", "2026-07-23T08:31:00Z"),
ev(2, "agent_shutting_down", "2026-07-23T09:28:00Z"),
ev(3, "agent_started", "2026-07-23T11:00:00Z"),
ev(4, "enrollment_complete", "2026-07-23T11:56:00Z"),
];
const d = computeWhiteGloveDurations(events, 2, "2026-07-23T08:29:41Z");
expect(d.preProvDuration).toBe("58m 19s"); // 08:29:41 → 09:28:00
});
it("treats everything as Part 1 while no split point exists yet", () => {
const events = [
ev(1, "agent_started", "2026-07-23T08:31:00Z"),
ev(2, "esp_phase_changed", "2026-07-23T08:45:00Z"),
];
const d = computeWhiteGloveDurations(events, -1);
expect(d.preProvDuration).toBe("14m 0s");
expect(d.userEnrollDuration).toBeNull();
expect(d.combinedDuration).toBe("14m 0s");
});
it("returns nulls for an empty event list", () => {
const d = computeWhiteGloveDurations([], -1);
expect(d.preProvDuration).toBeNull();
expect(d.userEnrollDuration).toBeNull();
expect(d.combinedDuration).toBeNull();
});
it("never returns a negative span when edge timestamps are disordered", () => {
const events = [
ev(1, "agent_started", "2026-07-23T09:00:00Z"),
ev(2, "agent_shutting_down", "2026-07-23T08:59:00Z"), // straggler stamped before the boot
];
const d = computeWhiteGloveDurations(events, 2, "2026-07-23T09:30:00Z");
expect(d.preProvDuration).toBeNull(); // clamped to 0 → formatted as null
});
});
@@ -0,0 +1,55 @@
/**
* Client-side guard against historic IME-log replay (session eaf3d8c4): when a previous
* enrollment's IME log survives a re-enrollment, legacy agents replay its content and emit
* week-old script runs and app installs as if they were current activity. The agent's
* timestamp clamp preserves the evidence as `rejectedSourceTimestamp` on the event data —
* a source timestamp more than 24 h OLDER than the event stamp marks the event as replayed
* history. Newer agents suppress those events at the source (historic_ime_replay_detected);
* this partition covers sessions recorded by older agents.
*/
/** Mirrors the agent's 24 h source-timestamp staleness clamp. */
export const HISTORIC_REPLAY_THRESHOLD_MS = 24 * 60 * 60 * 1000;
export interface ReplayInputEvent {
timestamp: string;
eventType?: string;
data?: Record<string, any>;
}
export interface HistoricPartition<T extends ReplayInputEvent> {
current: T[];
/**
* Count of dropped events whose eventType is in `countedFinalTypes` — the number the UI
* reports as "hidden". Dropped non-final events (starts, progress ticks) are removed but
* not counted, so the note is not inflated by multiple events of the same run.
*/
historicCount: number;
}
/**
* Splits off events that are replayed history from a previous enrollment. Future-skew
* rejections (source timestamp AHEAD of the event stamp — mid-enrollment clock jump) are
* genuine current activity and stay in `current`, as do events without (or with malformed)
* `rejectedSourceTimestamp`.
*/
export function partitionHistoricReplayEvents<T extends ReplayInputEvent>(
events: T[],
countedFinalTypes: ReadonlySet<string>,
): HistoricPartition<T> {
const current: T[] = [];
let historicCount = 0;
for (const evt of events) {
const rejected = evt.data?.rejectedSourceTimestamp ?? evt.data?.rejected_source_timestamp;
if (typeof rejected === "string" && rejected.length > 0) {
const rej = Date.parse(rejected);
const ts = Date.parse(evt.timestamp);
if (Number.isFinite(rej) && Number.isFinite(ts) && ts - rej > HISTORIC_REPLAY_THRESHOLD_MS) {
if (evt.eventType && countedFinalTypes.has(evt.eventType)) historicCount++;
continue;
}
}
current.push(evt);
}
return { current, historicCount };
}
@@ -5,6 +5,7 @@
* mapping can be unit-tested without React Testing Library.
*/
import { extractBootstrapVersion } from "@/utils/bootstrapVersion";
import { HISTORIC_REPLAY_THRESHOLD_MS, partitionHistoricReplayEvents } from "./historicReplay";
export interface ScriptInputEvent {
timestamp: string;
@@ -119,6 +120,30 @@ export function scriptCardKey(card: Pick<ScriptCard, "policyId" | "scriptType" |
/** Threshold above which a Running placeholder is rendered as "stuck?" rather than animated. */
export const STALE_RUNNING_THRESHOLD_SECONDS = 600;
export { HISTORIC_REPLAY_THRESHOLD_MS };
const SCRIPT_FINAL_TYPES: ReadonlySet<string> = new Set(["script_completed", "script_failed"]);
export interface HistoricPartition {
current: ScriptInputEvent[];
/**
* Count of dropped FINAL events (script_completed / script_failed) — what the user reads
* as "hidden executions". Dropped script_started events are removed but not counted, so
* the note is not inflated by start/end pairs of the same run.
*/
historicCount: number;
}
/**
* Script-specific wrapper over `partitionHistoricReplayEvents` (see lib/historicReplay.ts
* for the replay semantics): drops script events replayed from a previous enrollment
* (session eaf3d8c4: 156 replayed runs shown with ~170 h durations) and counts the finals
* for the "N hidden" note.
*/
export function partitionHistoricScriptEvents(events: ScriptInputEvent[]): HistoricPartition {
return partitionHistoricReplayEvents(events, SCRIPT_FINAL_TYPES);
}
/**
* Score how complete a script item's data is (higher = better). Used by the reducer to
* pick the best entry when re-emissions of the same script collapse into one row.
@@ -261,7 +286,13 @@ export function reduceScriptEvents(events: ScriptInputEvent[]): ScriptItem[] {
// the script's own result line (runtime), remediation durations to HS-NEW-RESULT
// (cycle) — infer accordingly so old data is never displayed as run time when it
// isn't one.
const rawDuration = toNumber(d.durationSeconds ?? d.duration_seconds);
// Scripts cannot run > 24 h inside an enrollment (IME's execution timeout is ~30 min);
// larger values are mixed-timeline artifacts from legacy agents (raw stale start paired
// with a clock-clamped completion — session eaf3d8c4 showed 170 h). Hide, don't lie.
const rawDurationUnchecked = toNumber(d.durationSeconds ?? d.duration_seconds);
const rawDuration = rawDurationUnchecked != null && rawDurationUnchecked <= HISTORIC_REPLAY_THRESHOLD_MS / 1000
? rawDurationUnchecked
: undefined;
const rawBasis = d.durationBasis ?? d.duration_basis;
const isCycleBasis = typeof rawBasis === "string"
? rawBasis === "cycle_including_reporting_latency"
@@ -57,6 +57,7 @@ export const REGISTRY_PREFIX_CATEGORIES: readonly GuardrailCategory[] = [
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Setup",
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\OOBE",
"SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Winlogon",
"SOFTWARE\\Microsoft\\Windows\\Autopilot",
],
},
{
@@ -242,6 +243,7 @@ export const ALLOWED_REGISTRY_PREFIXES: readonly string[] = [
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Setup",
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\OOBE",
"SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Winlogon",
"SOFTWARE\\Microsoft\\Windows\\Autopilot",
"SYSTEM\\CurrentControlSet\\Services\\TPM",
"SOFTWARE\\Microsoft\\Tpm",
"SYSTEM\\CurrentControlSet\\Control\\SecureBoot",