From 022cc0354bdcc29023d55182665d4d5dccce659e Mon Sep 17 00:00:00 2001 From: Oliver Kieselbach Date: Thu, 20 Aug 2026 00:40:23 +0200 Subject: [PATCH] fix(engine): let the observed user-ESP skip stand in for the exit evidence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The five stuck sits-d Cloud PCs carried SkipUserStatusPage=True — verified across three agent processes (10+ registry reads, True every time) and six hours of live Shell-Core watching with zero 62407: Windows never rendered a user ESP page, so no exit and no AccountSetup provisioning registry can ever exist on this flow. Arm B of ShouldTransitionToAwaitingHello was open from the bootstrap EspConfigDetected onward, but every evaluation site hangs off a signal the flow cannot produce, and the AdvisoryCompletion backstop arms exclusively from an EspExiting signal — an open gate nobody knocks on, six hours to the max-lifetime watchdog. The earlier lost-exit-through-reboot diagnosis is retired by the same evidence. HandleImeUserSessionCompletedV1's completion attempt now accepts the observed skip in place of IsPostAccountSetupFinalExit (anchor + genuine IME user-session completion + real-user desktop stay mandatory; Device Preparation excluded). The knock deliberately lives on the IME edge, not on desktop arrival: Fix 10's AccountSetup bounce-back would undo a desktop-side promotion two signals later, while the IME phase line always precedes the user-session-complete line in log order, so a bounced promotion is re-knocked by the very next re-emission — restart re-parse included. Docs and comments no longer claim an ESP-exit recovery that does not exist (stray duplicate summary in ShellCoreTracker, ImeLogHost restore-callback comment); the OKF doc is reframed on the evidence and names the residual SkipUser=false exit-in-downtime gap honestly. Co-Authored-By: Claude Fable 5 --- docs/agent/post-reboot-completion-recovery.md | 286 ++++++++++-------- docs/index.md | 2 +- docs/log.md | 4 + .../SystemSignals/ShellCoreTracker.cs | 6 - .../Orchestration/Hosts/ImeLogHost.cs | 13 +- ...icSkipUserEspUserSessionCompletionTests.cs | 286 ++++++++++++++++++ .../Engine/DecisionEngine.Classic.cs | 31 +- 7 files changed, 482 insertions(+), 146 deletions(-) create mode 100644 src/Agent/AutopilotMonitor.DecisionCore.Tests/ClassicSkipUserEspUserSessionCompletionTests.cs diff --git a/docs/agent/post-reboot-completion-recovery.md b/docs/agent/post-reboot-completion-recovery.md index 95ff8a99..0af1625d 100644 --- a/docs/agent/post-reboot-completion-recovery.md +++ b/docs/agent/post-reboot-completion-recovery.md @@ -1,178 +1,200 @@ --- type: Concept -title: Post-Reboot Completion Recovery — surviving the forced mid-ESP restart -description: Why an agent restarted by a mid-ESP reboot used to lose the AccountSetup completion for the rest of the session (orphaned Shell-Core backfill + edge-only user-apps-settled synthesis), and the two mechanisms that recover it — a downtime-sized ESP-exit replay and a level-triggered re-check of the settled user-ESP apps. +title: ESP Completion Starvation — the sits-d Cloud-PC complex +description: Why five Windows 365 Cloud PCs finished enrolling but never completed in the product — a SkipUser=true flow produces none of the signals every completion-gate evaluation site hangs off — plus the two adjacent defects fixed alongside (edge-only user-apps-settled synthesis, orphaned Hello-wizard restart replay) and what a Shell-Core replay deliberately does not do. resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals tags: - agent - esp - completion + - skip-user - reboot - recovery -timestamp: 2026-08-19T00:00:00+02:00 +timestamp: 2026-08-20T00:00:00+02:00 --- -# Post-Reboot Completion Recovery +# ESP Completion Starvation — the sits-d Cloud-PC complex -A device-assigned policy that Windows flags as reboot-required (DeviceGuard/VBS, -DmaGuard — see [MDM Reboot Coalescing](mdm-reboot-coalescing.md)) forces a restart in the -middle of the ESP. The agent is killed (`previousExit=reboot_kill`), the user signs in a -second time, and AccountSetup finishes. Until 2026-08-19 the session did **not** finish -with it: the agent waited for signals whose window had already closed, emitted -`session_stalled` after 60 minutes, and ran out at the max-lifetime watchdog — verdict -`Incomplete`, "No Device Setup completion or explicit failure signal observed before -timeout", on devices that were completely healthy. +Seven Windows 365 Cloud PCs, same tenant, first sign-in the same morning. Two sessions +succeeded within minutes; five sat at `Completion is waiting on: hello_resolution` from +second five onward and ran into the six-hour max-lifetime watchdog — verdict `Incomplete` +on devices whose users were working normally the whole time. + +The first diagnosis blamed the mid-ESP reboot (DeviceGuard/DmaGuard, see +[MDM Reboot Coalescing](mdm-reboot-coalescing.md)): the agent is killed in +`Stage=EspAccountSetup` and, so the theory went, loses the Shell-Core ESP exit to the +downtime. The evidence disproved it. `8110e262` ran **six hours uninterrupted** after its +last reboot with the Shell-Core watcher armed — and Windows never wrote a 62407 in all +that time, because on these machines **the user ESP page never existed**. # Schema -The AccountSetup gate opens on one of two evidence pairs: +## The actual defect — a satisfied gate nobody evaluates -| Path | Evidence | -|---|---| -| Registry | `AccountSetupCategory.Status.categorySucceeded` confirmed by Windows | -| Synthesis | Shell-Core normal ESP exit (62407) **and** every required user-ESP app terminal with zero failures | +The five stuck machines carried `SkipUserStatusPage=True` in the enrollment `FirstSync` +key (read 10+ times across three agent processes, `True` on every read — not a stale +single read). Windows honours it: no user ESP page, hence -The synthesis exists because a policy-skipped user-ESP app leaves the registry's Apps -subcategory permanently `inProgress`, starving the registry path (session `caa6cf50`). +* no Shell-Core 62407 (`esp_exiting`) — the page that would exit does not exist, +* no `AccountSetupCategory` provisioning registry — nothing writes categories for a page + that never renders, +* desktop within seconds of sign-in. -Two independent defects made the synthesis unreachable after a reboot. +The reducer is *built* for this flow: **arm B** of `ShouldTransitionToAwaitingHello` +(`SkipUserEsp == true`) declares the gate open, and the missing-prerequisites bookkeeping +already exempts the AccountSetup gate — which is why the sessions reported only +`hello_resolution` as missing. But the gate is a predicate, not a process. Something has +to evaluate it, and every evaluation site hangs off a signal a skip-user flow structurally +cannot produce: -## Defect 1 — the restart replay was never wired up +| Evaluation site | Carrier signal | On SkipUser=true | +|---|---|---| +| `HandleEspExitingV1` | Shell-Core 62407 | never fires | +| `HandleEspPhaseChangedV1(FinalizingSetup)` | Shell-Core 62407 via coordinator | never fires | +| `HandleDesktopArrivedV1` fast-path | requires Hello policy **disabled** | Hello was enabled | +| `HandleImeUserSessionCompletedV1` arm-C attempt | requires the recorded final exit | unsatisfiable | -`ShellCoreTracker.BackfillRecentEspExitEvents()` had existed since session `772fe502` but -**no caller ever invoked it**. The Shell-Core `EventLogWatcher` only delivers records -written after `Start()`, so every agent restart silently dropped a Hello-wizard start -(62404) that occurred while the agent was down. +No knock, no promotion, no `HelloSafety` — and the `AdvisoryCompletion` backstop is armed +exclusively from an `EspExiting` signal, so it never armed either. The session idles to +the watchdog with an open gate. -`EspAndHelloHost.Start()` now calls it — as `BackfillRecentHelloWizardStart`, which is what -it does. +## The fix — the observed skip is the exit evidence -### What the replay recovers, and what it deliberately does not +`HandleImeUserSessionCompletedV1`'s completion attempt now accepts the observed skip in +place of the final exit (`skipUserExitEquivalent`): on a flow where no page exists, no +exit can ever say more than the `FirstSync` read already did. Everything else stays +mandatory — AccountSetup anchor, genuine (at-or-after-anchor) IME user-session completion, +DAD-validated real-user desktop — which makes this knock **stricter** than the existing +Shell-Core-carried arm-B promotion (pinned in `ClassicAwaitingHelloGuardTests`, which +requires no IME or desktop evidence at all). Hello semantics ride the existing rails: +enabled-or-unknown promotes to `AwaitingHello` with `HelloSafety` armed (on Cloud PCs +reached via RDP the Hello wizard never appears, so the synthetic timeout resolves the +wait), disabled completes directly with the synthetic `Skipped` outcome. -Wiring up dead code activated three rails at once. Only one of them is safe to replay, and -the asymmetry is the whole design: +The knock deliberately lives on the IME user-session edge, not on desktop arrival: + +* **Ordering-robust against Fix 10.** An `EspPhaseChanged(AccountSetup)` arriving on + `AwaitingHello` bounces the stage back and cancels `HelloSafety` (the premature-promotion + guard). The IME AccountSetup phase line lands *after* the desktop on this flow, so a + desktop-side knock would be undone two signals later. The IME phase line always precedes + the user-session-complete line in log order, so a bounced promotion is re-knocked by the + very next re-emission — including the restart re-parse. +* **Right semantics.** The IME user session completes when the required-app processing is + done; a desktop-side knock would mark the session `Succeeded` mid-installation. +* Device Preparation is excluded — WDP keeps its Hello+Desktop conjunction and the + `DevicePrepCompletion` backstop. + +`ClassicSkipUserEspUserSessionCompletionTests` replays the real `8110e262` signal ordering +end to end (bootstrap `EspConfigDetected` → Hello enabled → desktop → AccountSetup anchor → +IME completion → HelloSafety → `enrollment_complete`), including the Fix-10 +bounce-then-re-knock cycle, and pins the negative space (no skip observed / no desktop / +ghost pre-anchor IME completion / WDP). + +Why five of seven identically administered Cloud PCs carried `SkipUserStatusPage=True` is +a tenant-side question (ESP profile assignment state at each machine's provisioning +moment); the value did not change during any observed session, and machine age does not +cleanly explain the split. The agent's job is to complete the flow Windows actually ran — +which it now does. + +## Adjacent defect — the synthesis was edge-triggered + +This one is real but belongs to the `SkipUser=false` flow (the two sessions that +succeeded). The user-apps-settled synthesis +(`MaybeSynthesizeAccountSetupCompleteFromSettledUserApps`, session `caa6cf50`) ran **only** +from `OnEspExited`. On a tenant with ~138 required apps the ESP page exits while apps are +still in flight: the single attempt misses and nothing ever re-checks, even when every +tracked app reaches a terminal state minutes later. + +The ESP exit is an **edge** (happens once); settled user-ESP apps are a **level** (reached +later). `EspAndHelloTracker` records the edge in `_espExitObserved` and exposes +`ReevaluateUserAppsSettledSynthesis()`, chained into `ImeLogTracker.OnAppStateChanged` and +into the post-restore `onStateRestored` callback (a live exit observed before the IME +state restore completes would otherwise never get a second look). Gate conditions are +unchanged — no new completion path, only a second opportunity for the existing one. Only a +**live, confirmed post-AccountSetup** exit may be remembered +(`IsConfirmedPostAccountSetupExit`, stricter than the `IsIntermediateDeviceEspExit` +forward guard: unknown counts as not confirmed). + +## Adjacent defect — the restart replay was never wired up + +`ShellCoreTracker`'s backfill had existed since session `772fe502` with **no caller**; the +Shell-Core `EventLogWatcher` only delivers records written after `Start()`. +`EspAndHelloHost.Start()` now calls it as `BackfillRecentHelloWizardStart` — which is +exactly what it does and all it does: | Record | Replayed | Why | |---|---|---| -| 62404 Hello-wizard start | **yes** | A *conservative* fact: it vetoes a premature "Hello is disabled" skip and can never by itself complete a session. Replaying it can only make the agent wait longer, never finish early. This is the observation `772fe502` was about. | +| 62404 Hello-wizard start | **yes** | Conservative: vetoes a premature "Hello is disabled" skip, can never complete a session by itself. The observation `772fe502` was about. | | 62407 ESP exit | **no** | Cannot be placed in time — see below. | | 62407 ESP failure | **no** | Re-injecting a historic failure as fresh can fail a session that recovered on retry (`ANALYZE-ESP-006`). | -A replayed exit is unusable, and every candidate ordering mechanism fails for a different -reason: +A replayed exit is unusable, and every candidate ordering mechanism fails independently: 1. Windows writes the **identical** description `CommercialOOBE_ESPProgress_Page_Exiting` - for the intermediate DeviceSetup→AccountSetup transition and for the final - post-AccountSetup exit. The record carries no evidence of its own position. -2. Everything that could order it after the fact — the AccountSetup registry probe, the - settled-apps probe — reads state as it is **now**, not as it was at the event's time. An - agent that was down across the Device→AccountSetup transition would confirm the stale - intermediate exit. -3. The reducer orders exits by **ingest ordinal**, not by timestamp - (`IsPostAccountSetupFinalExit`, deliberately — replayed CMTrace lines carry backdated - source times). A historic exit replayed today is assigned a fresher ordinal than reality, - so it reads as post-AccountSetup by construction. -4. `HandleEspExitingV1` passes `espFinalExitInFlight: true` for every arriving exit — the - signal carries no provenance at all. With restored state (AccountSetup entered, a genuine - IME user session, desktop arrived) **arm C** of `ShouldTransitionToAwaitingHello` then - opens on a historic intermediate exit. + for the intermediate DeviceSetup→AccountSetup transition and the final exit. +2. Every post-hoc ordering probe (AccountSetup registry, settled-apps) reads state as it + is **now**, not as it was at the event's time. +3. The reducer orders exits by **ingest ordinal**, not timestamp + (`IsPostAccountSetupFinalExit`) — a historic exit replayed today gets a fresher ordinal + than reality and reads as post-AccountSetup by construction. +4. `HandleEspExitingV1` passes `espFinalExitInFlight: true` for every arriving exit; with + restored state, arm C would open on a historic intermediate exit. -Point 4 is why the fix cannot live in the reducer: the reducer is itself a completion gate -and cannot tell the two apart. `ClassicEspExitingOnRestoredStateTests` pins that it *does* -open on any arriving exit; `ShellCoreTrackerReplayScopeTests` pins that the replay never -produces one. The pair is the contract. +`ClassicEspExitingOnRestoredStateTests` pins that the reducer *does* open on any arriving +exit; `ShellCoreTrackerReplayScopeTests` pins that the replay never produces one. Skipped +62407 records are counted and reported once as an `agent_trace` +(`reason: replayed_62407_not_orderable`) — silently dropping evidence is how this class of +bug survives. -Records that are read but not replayed are counted and reported once as an `agent_trace` -(`skippedEspExits`, `skippedEspFailures`, oldest/newest). Silently dropping evidence is how -this class of bug survives. +**Residual gap, named honestly:** a `SkipUser=false` session whose final exit falls +exactly into an agent downtime *and* whose AccountSetup category never resolves +(`1ec8f4c6` shape) still has no completion carrier after the restart. The normal case is +covered — `ProvisioningStatusTracker` reads the current registry at startup, so a +category that resolved during the downtime fires arm A. The residual shape is rare, +surfaced by `ANALYZE-ESP-005` via the `session_stalled` interim trigger, and deliberately +not patched with a replay that cannot be ordered. -### Sizing the window +### Sizing the replay window -A constant is provably wrong: the agent's scheduled task carries a `BootTrigger` **only**, -with no restart-on-failure (`Program.InstallMode.BuildScheduledTaskXml`). After an -`exception_crash` the agent does not come back until the next boot — possibly hours later. -`ResolveEspExitBackfillLookbackMinutes` therefore reaches back to the last moment the -previous run is known to have been alive, taking the wider of two independent inputs: - -| Input | Available for | Source | -|---|---|---| -| Snapshot mtime | every exit type | `snapshot.json` is rewritten on every decision step, so its last-write time is "when we last knew what was happening" | -| `LastBootUtc` | `hard_kill` / `reboot_kill` | covers a missing or never-written snapshot | - -On `first_run` the replay stays **off** — no earlier process, no gap — so the happy path is -untouched. The 5-minute default is the **floor**, `ShellCoreTracker.ClampLookbackMinutes` -(360) the **ceiling**, and a timestamp in the future (clock skew across a reboot) is ignored -rather than producing a negative window. - -This is a new caller for existing code, not a change to any other backfill. The -Hello/MDM-reboot/Windows-Update/ModernDeployment trackers each own a private backfill called -from their own `Start()`; none of them is touched. - -## Defect 2 — the synthesis was edge-triggered - -`MaybeSynthesizeAccountSetupCompleteFromSettledUserApps()` ran **only** from `OnEspExited`. -On a tenant with a large required-app set the ESP page exits while apps are still in -flight: the single attempt misses, emits `app_install_starved` for the apps that never -started, and nothing ever re-checks — not even when every tracked app reaches a terminal -state minutes later. - -The ESP exit is an **edge** (it happens once); settled user-ESP apps are a **level** (they -can be reached later). `EspAndHelloTracker` now records the edge in `_espExitObserved` and -exposes `ReevaluateUserAppsSettledSynthesis()`, chained into `ImeLogTracker.OnAppStateChanged` -in `DefaultComponentFactory` (same preserve-previous pattern `DeliveryOptimizationHost` -uses). Every terminal app transition gives the synthesis another look. - -This adds **no new completion path**. The gate conditions are unchanged — real ESP exit, -every required user-ESP app terminal, zero failures. The re-check only grants the existing, -deliberately conservative check a second opportunity. Three guards keep it cheap and safe: -the fire-once claim is an `Interlocked.CompareExchange` (the re-check runs on the IME log -thread, the edge on the Shell-Core watcher thread); the re-check path suppresses -`app_install_starved` emission so the one-shot warning does not become a per-transition -stream; and the edge itself is gated. - -**Only a confirmed post-AccountSetup exit may be remembered.** Every exit that reaches the -coordinator is one the agent observed live — the replay never re-raises 62407 (above). That -is what makes the AccountSetup read below a valid ordering fact: the agent was continuously -observing up to that instant. On top of it, `IsConfirmedPostAccountSetupExit()` demands -positive evidence and is deliberately stricter than the existing -`IsIntermediateDeviceEspExit()` forward guard: - -**The re-check must also run after the IME state restore.** `ImeLogTracker.Start()` restores -the persisted package states via `LoadState()` and raises no `OnAppStateChanged` for them, and -`ImeLogHost` starts *after* `EspAndHelloHost` (pinned by -`DefaultComponentFactoryOrderingTests`). A live exit observed before that restore completes -would therefore never get a second look. `ImeLogHost` takes an `onStateRestored` callback, -wired in the factory to `ReevaluateUserAppsSettledSynthesis()`. - -The replay itself stays in `EspAndHelloHost.Start()` and does not depend on this ordering — -it feeds the reducer only, so there is nothing for it to read from the IME state. Keeping it -there also keeps this host's startup independent of the IME host's. +A constant is provably wrong: the scheduled task has a `BootTrigger` only +(`Program.InstallMode.BuildScheduledTaskXml`) — after an `exception_crash` the agent does +not return until the next boot. `ResolveEspExitBackfillLookbackMinutes` reaches back to +the last moment the previous run is known to have been alive (max of `snapshot.json` +mtime and `LastBootUtc` for kill-type exits), floored at 5 minutes, capped at 360, off on +`first_run`, and immune to cross-reboot clock skew (a future timestamp is ignored). # Examples -Tenant `sits-d.cloud`, 2026-08-19, seven Windows 365 Cloud PCs, identical configuration -(~138 required apps at the user-ESP gate, DeviceGuard + DmaGuard device-assigned): +Tenant `sits-d.cloud`, 2026-08-19, seven Windows 365 Cloud PCs, first sign-ins the same +morning. The discriminator is a single registry value: -* `08fc6bda`, `3b9291aa` — `RebootCount=0`. AccountSetup finished before the coalesced - reboot landed. Succeeded. -* `cb4a485a`, `e7ba63c9`, `8110e262`, `a89aac2d`, `3d6278fb` — `RebootCount>=1`. The agent - was killed in `Stage=EspAccountSetup`, and no completion was ever derived again. All five - ended `Incomplete` on healthy devices. +| Session | SkipUser | `esp_exiting` | AccountSetup registry | Reboots | Outcome | +|---|---|---|---|---|---| +| `08fc6bda` | False | yes | yes | 0 | Succeeded (10 min) | +| `3b9291aa` | False | yes | yes | 0 | Succeeded (2.1 h) | +| `8110e262` | True | — | — | 2 | Incomplete (6 h watchdog) | +| `a89aac2d` | True | — | — | 1 | Incomplete | +| `e7ba63c9` | True | — | — | 1 | Incomplete | +| `cb4a485a` | True | — | — | 2 | Incomplete | +| `3d6278fb` | True | — | — | 1 | Incomplete | -`8110e262` and `a89aac2d` are the sharpest evidence for defect 2: both reached -`App summary: 138/138 completed, 0 failed` and still never completed, because the settled -level arrived after the exit edge had already been consumed. +The reboot correlation was a red herring: `8110e262` ran six hours live after its final +reboot and would have seen every signal. `8110e262` and `a89aac2d` both reached +`138/138 completed, 0 failed` and a genuine IME user-session completion — with the fix, +all five complete a few minutes after their IME user session settles. -`3b9291aa` shows the independent starvation problem the same tenant has: 99 -`app_install_starved` events (required apps that never started installing) on a session -that was nevertheless recorded as `Succeeded` — surfaced from now on by `ANALYZE-APP-016`. +`08fc6bda` exercises the edge-level fix on the same day: ESP page exited normally while +`categorySucceeded` never confirmed — completed via the user-apps-settled synthesis. +`3b9291aa` shows the independent starvation visibility gap: 99 `app_install_starved` +events on a session recorded as `Succeeded` — surfaced from now on by `ANALYZE-APP-016`. # Citations -* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/ShellCoreTracker.cs` — `BackfillRecentEspExitEvents(int)`, `ClampLookbackMinutes` -* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/EspAndHelloTracker.cs` — `_espExitObserved`, `ReevaluateUserAppsSettledSynthesis`, `MaybeSynthesizeAccountSetupCompleteFromSettledUserApps` -* `src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/Hosts/EspAndHelloHost.cs` — `Start()` ordering +* `src/Shared/AutopilotMonitor.DecisionCore/Engine/DecisionEngine.Classic.cs` — `HandleImeUserSessionCompletedV1`, `skipUserExitEquivalent` +* `src/Shared/AutopilotMonitor.DecisionCore/Engine/DecisionEngine.Shared.cs` — `ShouldTransitionToAwaitingHello` (arm B) +* `src/Agent/AutopilotMonitor.DecisionCore.Tests/ClassicSkipUserEspUserSessionCompletionTests.cs` — end-to-end replay of the 8110e262 ordering +* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/ShellCoreTracker.cs` — `BackfillRecentHelloWizardStart`, `ReplayBackfillRecords` +* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/EspAndHelloTracker.cs` — `_espExitObserved`, `ReevaluateUserAppsSettledSynthesis` * `src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/DefaultComponentFactory.cs` — lookback sizing, `OnAppStateChanged` chaining -* `src/Agent/AutopilotMonitor.Agent.V2/Runtime/AgentRuntimeHost.cs` — `previousBootUtc` hand-off -* [MDM Reboot Coalescing](mdm-reboot-coalescing.md) — the policy attribution for the reboot itself +* [MDM Reboot Coalescing](mdm-reboot-coalescing.md) — the policy attribution for the reboots themselves * [Decision Engine](decision-engine.md) — the AccountSetup gate and completion arms diff --git a/docs/index.md b/docs/index.md index dc2f9874..5c6d4d0d 100644 --- a/docs/index.md +++ b/docs/index.md @@ -24,7 +24,7 @@ https://docs.autopilotmonitor.com (separate repository). * [Agent Endpoint Migration](agent/endpoint-migration.md) - How the backend re-homes agents to a new API base URL via the config channel (backend move or per-tenant region move): live-fetch-only, allowlist-validated on both sides, one hop, kill wins. * [Gather Rule Debug Log](agent/gather-rule-debug-log.md) - EnableGatherRuleDebugLog (ConfigVersion 36): the per-tenant local trace file that explains every gather-rule evaluation decision — scope skips, on_change suppression, empty collector results, logparser position/parse-mode outcomes — for rules that produce no timeline events. * [MDM Reboot Coalescing](agent/mdm-reboot-coalescing.md) - Attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: aggregated DM-Enterprise 2800 observations (neutral, flush-time watermark) + the ANALYZE-ESP-005 advisory gated on an actually-observed reboot. -* [Post-Reboot Completion Recovery](agent/post-reboot-completion-recovery.md) - Why a forced mid-ESP reboot used to cost the session its AccountSetup completion for good (the Shell-Core ESP-exit backfill was never wired to a caller, and the user-apps-settled synthesis only ran on the exit edge), and the two recoveries: a replay window sized to the agent's downtime, plus a level-triggered re-check on every terminal IME app transition. +* [ESP Completion Starvation — the sits-d Cloud-PC complex](agent/post-reboot-completion-recovery.md) - Why a SkipUser=true flow (no user ESP page → no Shell-Core 62407, no AccountSetup registry) left the completion gate satisfied but never evaluated — five Cloud PCs idled six hours at "waiting on: hello_resolution" — and the fix: the observed skip stands in for the final-exit evidence in the IME user-session completion attempt. Plus the two adjacent defects fixed alongside: the edge-only user-apps-settled synthesis (now level-triggered) and the never-wired Shell-Core restart replay (now Hello-wizard-only, with the reasoning why a replayed 62407 must never reach the reducer). * [Secure Boot CA-2023 Detection](agent/secureboot-ca2023-detection.md) - Why ANALYZE-SEC-001 v3 reads the UEFI db/KEK variables directly instead of trusting the SecureBoot\Servicing registry, and how the one-sided uefiCa2023FirmwareConfirmed marker plus a not_exists precondition give absence-tolerant suppression across old and new agents. * [Continue-Anyway Observation Mode](agent/continue-anyway-observation.md) - EnableEspContinueAnywayObservation (ConfigVersion 37, operator-set): Device-phase ESP terminal failures on Continue-Anyway profiles get a bounded 60-min observation instead of an immediate hard fail; the DAD-validated real-user desktop + Hello gate detect the true end, the session completes Succeeded + EspSoftFailure (amber "with issues" badge), expiry un-defangs to the original esp_terminal_failure. * [Registry App-State Second Pillar](agent/registry-app-state-pillar.md) - IME's authoritative per-app state read straight from the registry (Win32Apps EnforcementStateMessage, EspTrackingWin32Apps, StatusServiceReports) via a snapshot-and-diff watcher; registry/log reconciliation doubles as the built-in IME log-pattern drift alarm. diff --git a/docs/log.md b/docs/log.md index 60e7b279..11c40186 100644 --- a/docs/log.md +++ b/docs/log.md @@ -1,5 +1,9 @@ # Log +## 2026-08-20 + +* **Update**: `agent/post-reboot-completion-recovery.md` — retitled to "ESP Completion Starvation — the sits-d Cloud-PC complex" and reframed on the evidence: the reboot was a red herring. The five stuck Cloud PCs carried `SkipUserStatusPage=True` (read 10+ times across three agent processes, `True` every time) and Windows honoured it — no user ESP page, hence no Shell-Core 62407 and no AccountSetup provisioning registry, verified over a six-hour uninterrupted post-reboot run with the watcher armed. Arm B of `ShouldTransitionToAwaitingHello` was open from the bootstrap onward, but every evaluation site hangs off a signal a skip-user flow cannot produce, and the `AdvisoryCompletion` backstop arms exclusively from an `EspExiting` signal — an open gate nobody knocks on. Fix: `HandleImeUserSessionCompletedV1` accepts the observed skip in place of the final-exit evidence (`skipUserExitEquivalent`; anchor + genuine IME completion + real-user desktop stay mandatory, WDP excluded). The knock deliberately lives on the IME edge, not desktop arrival — ordering-robust against Fix 10's AccountSetup bounce-back and semantically "user-phase work finished" rather than mid-install. `ClassicSkipUserEspUserSessionCompletionTests` replays the real 8110e262 ordering end to end. Docs and comments no longer claim an ESP-exit recovery that does not exist (stray duplicate summary in `ShellCoreTracker`, `ImeLogHost` restore-callback comment); the residual `SkipUser=false` exit-in-downtime gap is named honestly and surfaced by `ANALYZE-ESP-005` via `session_stalled` instead of being patched with an unorderable replay. + ## 2026-08-19 * **New**: `agent/post-reboot-completion-recovery.md` — a forced mid-ESP reboot (DeviceGuard/DmaGuard) used to cost the session its AccountSetup completion permanently. Two defects: `ShellCoreTracker.BackfillRecentEspExitEvents()` existed but had no caller, so every agent restart dropped an ESP exit (62407) written while it was down; and the user-apps-settled synthesis was edge-triggered on `EspExited` only, so apps settling afterwards never re-opened the gate. `EspAndHelloHost.Start()` now backfills with a lookback sized to the downtime (`previousExit=reboot_kill` → since `LastBootUtc`, clamped to 360 min), and `ReevaluateUserAppsSettledSynthesis()` is chained into `ImeLogTracker.OnAppStateChanged`. No new completion path — the gate conditions are unchanged. Evidence: sits-d Cloud PCs, five `Incomplete` sessions on healthy devices, two of them at `138/138 apps, 0 failed`. Hardened over two review rounds: the remembered edge must be a **live** exit that is **confirmed** post-AccountSetup (a replay carries no evidence of its own position — Shell-Core writes the same `CommercialOOBE_ESPProgress_Page_Exiting` for the intermediate and the final exit, and the registry check reads *now*, not the event's time); the replay hands over the **newest** exit rather than the oldest; the replay is narrowed to the Hello-wizard start (62404) alone — a conservative fact that can only make the agent wait longer. A replayed ESP exit is never re-raised at all: Windows writes the identical description for the intermediate and the final exit, the reducer orders exits by ingest ordinal (fresher than reality for a replay) and `HandleEspExitingV1` passes `espFinalExitInFlight: true` for every arriving exit, so arm C of `ShouldTransitionToAwaitingHello` would open on a historic handoff once the restored state carries AccountSetupEntered + IME user session + desktop. A replayed ESP failure is likewise dropped (it could fail a session that recovered). Skipped records are reported as an `agent_trace`. The window is **measured** from the previous run's last known liveness (snapshot mtime, widened by `LastBootUtc`) instead of a constant — the scheduled task has a BootTrigger only and no restart-on-failure, so a crashed agent returns hours later and five minutes would lose the exit every time. On `first_run` the replay stays off, leaving the happy path untouched. diff --git a/src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/ShellCoreTracker.cs b/src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/ShellCoreTracker.cs index b8ce70f6..c9e6adc5 100644 --- a/src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/ShellCoreTracker.cs +++ b/src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/ShellCoreTracker.cs @@ -358,12 +358,6 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals // Backfill (public — called by coordinator) // ===================================================================== - /// - /// Backfills recent ESP exit, ESP failure and Hello-wizard-start events from the - /// Shell-Core log on startup. Secondary recovery mechanism when state persistence is - /// unavailable — an agent restart while the user sits inside the Hello wizard would - /// otherwise lose the wizard-start observation (session 772fe502). - /// /// Clamps a caller-supplied lookback into [1, ]. internal static int ClampLookbackMinutes(int lookbackMinutes) { diff --git a/src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/Hosts/ImeLogHost.cs b/src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/Hosts/ImeLogHost.cs index 43882016..2398e5cf 100644 --- a/src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/Hosts/ImeLogHost.cs +++ b/src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/Hosts/ImeLogHost.cs @@ -134,12 +134,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration /// /// Codex review P2 (2026-08-19) — restores the persisted /// package states (LoadState) and does NOT raise OnAppStateChanged for them. This - /// host is started AFTER EspAndHelloHost, so on an agent restart the Shell-Core - /// ESP-exit replay runs against an empty app list and synthesises nothing — and without a - /// nudge here nothing would ever re-check, which is precisely the reboot case the - /// completion recovery exists for (apps already terminal before the reboot, only the final - /// exit lost to the downtime). fires once, after the - /// restore, so the deferred synthesis gets its look at the restored level. + /// host is started AFTER EspAndHelloHost, so a LIVE ESP exit observed before this + /// restore completes would evaluate the user-apps-settled synthesis against an empty app + /// list — and without a nudge here nothing would ever re-check the restored level. The + /// onStateRestored callback fires once, after the restore, so the deferred + /// synthesis gets its look. (It deliberately covers ONLY live exits: a 62407 that fired + /// while no agent was running is never replayed — see + /// ShellCoreTracker.ReplayBackfillRecords for why it cannot be placed in time.) /// public void Start() { diff --git a/src/Agent/AutopilotMonitor.DecisionCore.Tests/ClassicSkipUserEspUserSessionCompletionTests.cs b/src/Agent/AutopilotMonitor.DecisionCore.Tests/ClassicSkipUserEspUserSessionCompletionTests.cs new file mode 100644 index 00000000..fd08984f --- /dev/null +++ b/src/Agent/AutopilotMonitor.DecisionCore.Tests/ClassicSkipUserEspUserSessionCompletionTests.cs @@ -0,0 +1,286 @@ +using System; +using System.Collections.Generic; +using AutopilotMonitor.DecisionCore.Engine; +using AutopilotMonitor.DecisionCore.Signals; +using AutopilotMonitor.DecisionCore.State; +using Xunit; + +namespace AutopilotMonitor.DecisionCore.Tests +{ + /// + /// sits-d Cloud-PC fix (2026-08-20, sessions 8110e262 / a89aac2d / e7ba63c9 / cb4a485a / + /// 3d6278fb). On a SkipUser=true flow Windows never renders the user ESP page: Shell-Core + /// 62407 does not fire and the AccountSetup provisioning categories are never written — + /// verified live (Shell-Core watcher armed across three agent runs, zero 62407 in six + /// hours; SkipUserStatusPage read 10+ times across all runs, True every time). Arm B of + /// ShouldTransitionToAwaitingHello was therefore open from the bootstrap + /// EspConfigDetected onward, but every site that evaluates the gate hangs off a signal the + /// flow structurally cannot produce, so five sessions sat at "waiting on: hello_resolution" + /// until the max-lifetime watchdog. + /// + /// The fix lets the observed skip stand in for the final-exit evidence inside + /// HandleImeUserSessionCompletedV1's completion attempt. These tests replay the real + /// 8110e262 signal ordering end to end (including the Fix-10 AccountSetup bounce-back a + /// restart re-emission triggers) and pin the negative space: no skip observed, no desktop, + /// ghost IME completion, and Device Preparation flows are all unchanged. + /// + /// + public sealed class ClassicSkipUserEspUserSessionCompletionTests + { + // 8110e262 run 1: agent_started 08:30:08Z. + private static readonly DateTime T0 = new DateTime(2026, 8, 19, 8, 30, 8, DateTimeKind.Utc); + + [Fact] + public void Sits_d_signal_ordering_completes_via_hello_safety() + { + var engine = new DecisionEngine(); + var state = DecisionState.CreateInitial( + "8110e262-da57-4282-9290-bb0c96d05614", "5ca2b350", T0); + + state = engine.Reduce(state, MakeSignal(0, DecisionSignalKind.SessionStarted, T0, null)).NewState; + + // Bootstrap EspConfigDetected — SkipUserStatusPage=True, read from FirstSync 2 s in. + state = engine.Reduce(state, MakeEspConfigSignal(1, skipUser: "true")).NewState; + Assert.True(state.ScenarioObservations.SkipUserEsp!.Value); + + // Hello policy enabled (device-scoped CSP) — the reason the desktop fast-path + // could never take this flow. + state = engine.Reduce(state, MakeSignal(2, DecisionSignalKind.HelloPolicyDetected, T0.AddSeconds(1), + new Dictionary { [SignalPayloadKeys.HelloEnabled] = "true" })).NewState; + + // Desktop 5 s after agent start (the user ESP page does not exist — the user is + // simply logged on). Pins that desktop arrival alone does NOT promote: the knock + // deliberately lives on the IME user-session edge, because a desktop-side promotion + // would be undone by Fix 10's AccountSetup bounce-back two signals later (the IME + // AccountSetup phase line lands AFTER the desktop on this flow). + state = engine.Reduce(state, MakeSignal(3, DecisionSignalKind.DesktopArrived, T0.AddSeconds(5), null)).NewState; + Assert.NotEqual(SessionStage.AwaitingHello, state.Stage); + Assert.DoesNotContain(state.Deadlines, d => d.Name == DeadlineNames.HelloSafety); + + // IME's AccountSetup phase line (source ts 08:29:56 — before agent start; the IME + // logs the phase for its user-session app processing even though no page is shown). + state = engine.Reduce(state, MakeSignal(4, DecisionSignalKind.EspPhaseChanged, T0.AddSeconds(-12), + new Dictionary { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState; + Assert.Equal(SessionStage.EspAccountSetup, state.Stage); + + // 08:36:59 — IME user session completed (138/138 apps, 0 failed). THE knock: + // skip-user stands in for the exit that cannot exist; Hello enabled + unresolved + // promotes to AwaitingHello with HelloSafety armed. + var promoted = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.ImeUserSessionCompleted, + T0.AddMinutes(6).AddSeconds(51), null)); + Assert.Equal(SessionStage.AwaitingHello, promoted.NewState.Stage); + var helloSafety = Assert.Single(promoted.NewState.Deadlines, d => d.Name == DeadlineNames.HelloSafety); + Assert.Contains(promoted.Effects, + e => e.Kind == DecisionEffectKind.ScheduleDeadline && e.Deadline?.Name == DeadlineNames.HelloSafety); + + // The Hello wizard never appears over RDP — HelloSafety resolves synthetically. + var afterSafety = engine.Reduce(promoted.NewState, + MakeSignal(6, DecisionSignalKind.DeadlineFired, helloSafety.DueAtUtc, + new Dictionary { [SignalPayloadKeys.Deadline] = DeadlineNames.HelloSafety })); + Assert.Equal(SessionStage.Finalizing, afterSafety.NewState.Stage); + Assert.Equal("Timeout", afterSafety.NewState.HelloOutcome!.Value); + + var completed = engine.Reduce(afterSafety.NewState, + MakeSignal(7, DecisionSignalKind.DeadlineFired, helloSafety.DueAtUtc.AddSeconds(5), + new Dictionary { [SignalPayloadKeys.Deadline] = DeadlineNames.FinalizingGrace })); + Assert.Equal(SessionStage.Completed, completed.NewState.Stage); + Assert.Equal(SessionOutcome.EnrollmentComplete, completed.NewState.Outcome); + Assert.Contains(completed.Effects, + e => e.Kind == DecisionEffectKind.EmitEventTimelineEntry + && e.Parameters != null + && e.Parameters.TryGetValue("eventType", out var et) && et == "enrollment_complete"); + } + + [Fact] + public void Fix10_bounceback_is_healed_by_the_ime_reemission() + { + // Restart shape (8110e262 run 3): the promoted AwaitingHello gets bounced back by a + // re-emitted AccountSetup phase line (Fix 10 cancels HelloSafety, deliberately — it + // guards the premature-promotion case). The IME phase line always precedes the + // user-session-complete line in log order, so the re-emitted completion re-knocks. + var engine = new DecisionEngine(); + var state = ProgressToPromotedAwaitingHello(engine); + + var bounced = engine.Reduce(state, MakeSignal(10, DecisionSignalKind.EspPhaseChanged, + T0.AddMinutes(18).AddSeconds(10), + new Dictionary { [SignalPayloadKeys.EspPhase] = "AccountSetup" })); + Assert.Equal(SessionStage.EspAccountSetup, bounced.NewState.Stage); + Assert.DoesNotContain(bounced.NewState.Deadlines, d => d.Name == DeadlineNames.HelloSafety); + + var reknocked = engine.Reduce(bounced.NewState, + MakeSignal(11, DecisionSignalKind.ImeUserSessionCompleted, T0.AddMinutes(6).AddSeconds(51), null)); + Assert.Equal(SessionStage.AwaitingHello, reknocked.NewState.Stage); + Assert.Single(reknocked.NewState.Deadlines, d => d.Name == DeadlineNames.HelloSafety); + } + + [Theory] + // No EspConfigDetected at all — SkipUserEsp unknown. + [InlineData(null)] + // Explicit SkipUser=false (the two sits-d machines that succeeded had this shape and + // a REAL page: exit + registry evidence. Without the exit the gate must stay shut.) + [InlineData("false")] + public void No_promotion_without_an_observed_skip(string? skipUser) + { + var engine = new DecisionEngine(); + var state = DecisionState.CreateInitial("s", "t", T0); + state = engine.Reduce(state, MakeSignal(0, DecisionSignalKind.SessionStarted, T0, null)).NewState; + if (skipUser != null) + { + state = engine.Reduce(state, MakeEspConfigSignal(1, skipUser)).NewState; + } + state = engine.Reduce(state, MakeSignal(2, DecisionSignalKind.HelloPolicyDetected, T0.AddSeconds(1), + new Dictionary { [SignalPayloadKeys.HelloEnabled] = "true" })).NewState; + state = engine.Reduce(state, MakeSignal(3, DecisionSignalKind.DesktopArrived, T0.AddSeconds(5), null)).NewState; + state = engine.Reduce(state, MakeSignal(4, DecisionSignalKind.EspPhaseChanged, T0.AddSeconds(10), + new Dictionary { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState; + + var step = engine.Reduce(state, + MakeSignal(5, DecisionSignalKind.ImeUserSessionCompleted, T0.AddMinutes(7), null)); + + Assert.Equal(SessionStage.EspAccountSetup, step.NewState.Stage); + Assert.DoesNotContain(step.NewState.Deadlines, d => d.Name == DeadlineNames.HelloSafety); + } + + [Fact] + public void No_promotion_without_the_desktop() + { + var engine = new DecisionEngine(); + var state = DecisionState.CreateInitial("s", "t", T0); + state = engine.Reduce(state, MakeSignal(0, DecisionSignalKind.SessionStarted, T0, null)).NewState; + state = engine.Reduce(state, MakeEspConfigSignal(1, skipUser: "true")).NewState; + state = engine.Reduce(state, MakeSignal(2, DecisionSignalKind.EspPhaseChanged, T0.AddSeconds(10), + new Dictionary { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState; + + var step = engine.Reduce(state, + MakeSignal(3, DecisionSignalKind.ImeUserSessionCompleted, T0.AddMinutes(7), null)); + + Assert.Equal(SessionStage.EspAccountSetup, step.NewState.Stage); + Assert.DoesNotContain(step.NewState.Deadlines, d => d.Name == DeadlineNames.HelloSafety); + } + + [Fact] + public void A_ghost_pre_anchor_ime_completion_does_not_knock() + { + // defaultuser0-ghost guard stays load-bearing: an IME completion whose source time + // precedes the AccountSetup anchor is the OOBE/technician frame, not the user. + var engine = new DecisionEngine(); + var state = DecisionState.CreateInitial("s", "t", T0); + state = engine.Reduce(state, MakeSignal(0, DecisionSignalKind.SessionStarted, T0, null)).NewState; + state = engine.Reduce(state, MakeEspConfigSignal(1, skipUser: "true")).NewState; + state = engine.Reduce(state, MakeSignal(2, DecisionSignalKind.DesktopArrived, T0.AddSeconds(5), null)).NewState; + state = engine.Reduce(state, MakeSignal(3, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(2), + new Dictionary { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState; + + var step = engine.Reduce(state, + MakeSignal(4, DecisionSignalKind.ImeUserSessionCompleted, T0.AddMinutes(1), null)); + + Assert.Equal(SessionStage.EspAccountSetup, step.NewState.Stage); + Assert.DoesNotContain(step.NewState.Deadlines, d => d.Name == DeadlineNames.HelloSafety); + } + + [Fact] + public void Device_preparation_flows_keep_their_own_rails() + { + // WDP has no ESP at all — completion rides the Hello+Desktop conjunction and the + // DevicePrepCompletion backstop (afee7ae0). Even a (spurious) SkipUser=true read + // must not route WDP through the classic skip-user knock. + var engine = new DecisionEngine(); + var seed = DecisionState.CreateInitial("s", "t", T0) + .ToBuilder() + .WithStage(SessionStage.SessionStarted) + .WithStepIndex(5) + .WithLastAppliedSignalOrdinal(4) + .WithScenarioProfile(EnrollmentScenarioProfile.Empty.With( + mode: EnrollmentMode.DevicePreparation, + confidence: ProfileConfidence.High, + evidenceOrdinal: 1, + reason: "test")); + seed.ScenarioObservations = EnrollmentScenarioObservations.Empty + .WithSkipUserEsp(true, 1); + seed.AccountSetupEnteredUtc = new SignalFact(T0.AddMinutes(1), 2); + seed.DesktopArrivedUtc = new SignalFact(T0.AddMinutes(2), 3); + var state = seed.Build(); + + var step = engine.Reduce(state, + MakeSignal(5, DecisionSignalKind.ImeUserSessionCompleted, T0.AddMinutes(3), null)); + + Assert.DoesNotContain(step.NewState.Deadlines, d => d.Name == DeadlineNames.HelloSafety); + Assert.NotEqual(SessionStage.AwaitingHello, step.NewState.Stage); + } + + [Fact] + public void Hello_disabled_skip_user_completes_directly_from_the_knock() + { + // HelloSatisfiedForCompletion holds (policy disabled, no wizard observed) — the + // knock completes through Finalizing with the synthetic Skipped outcome instead of + // parking 5 minutes in AwaitingHello. + var engine = new DecisionEngine(); + var state = DecisionState.CreateInitial("s", "t", T0); + state = engine.Reduce(state, MakeSignal(0, DecisionSignalKind.SessionStarted, T0, null)).NewState; + state = engine.Reduce(state, MakeEspConfigSignal(1, skipUser: "true")).NewState; + state = engine.Reduce(state, MakeSignal(2, DecisionSignalKind.HelloPolicyDetected, T0.AddSeconds(1), + new Dictionary { [SignalPayloadKeys.HelloEnabled] = "false" })).NewState; + state = engine.Reduce(state, MakeSignal(3, DecisionSignalKind.EspPhaseChanged, T0.AddSeconds(10), + new Dictionary { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState; + // Desktop arrives; the hello-disabled fast-path already promotes/completes via the + // gate's arm B on this ordering, so the knock is not even needed here — assert the + // flow terminates either way (belt for orderings where desktop precedes the config). + state = engine.Reduce(state, MakeSignal(4, DecisionSignalKind.DesktopArrived, T0.AddMinutes(5), null)).NewState; + + if (state.Stage != SessionStage.Finalizing && !state.Stage.IsTerminal()) + { + var step = engine.Reduce(state, + MakeSignal(5, DecisionSignalKind.ImeUserSessionCompleted, T0.AddMinutes(7), null)); + Assert.Equal(SessionStage.Finalizing, step.NewState.Stage); + Assert.Equal("Skipped", step.NewState.HelloOutcome!.Value); + } + else + { + Assert.Equal(SessionStage.Finalizing, state.Stage); + } + } + + // ------------------------------------------------------------------------------ + + private static DecisionState ProgressToPromotedAwaitingHello(DecisionEngine engine) + { + var state = DecisionState.CreateInitial("s", "t", T0); + state = engine.Reduce(state, MakeSignal(0, DecisionSignalKind.SessionStarted, T0, null)).NewState; + state = engine.Reduce(state, MakeEspConfigSignal(1, skipUser: "true")).NewState; + state = engine.Reduce(state, MakeSignal(2, DecisionSignalKind.HelloPolicyDetected, T0.AddSeconds(1), + new Dictionary { [SignalPayloadKeys.HelloEnabled] = "true" })).NewState; + state = engine.Reduce(state, MakeSignal(3, DecisionSignalKind.DesktopArrived, T0.AddSeconds(5), null)).NewState; + state = engine.Reduce(state, MakeSignal(4, DecisionSignalKind.EspPhaseChanged, T0.AddSeconds(-12), + new Dictionary { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState; + state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.ImeUserSessionCompleted, + T0.AddMinutes(6).AddSeconds(51), null)).NewState; + Assert.Equal(SessionStage.AwaitingHello, state.Stage); + return state; + } + + private static DecisionSignal MakeEspConfigSignal(long ordinal, string skipUser) => + MakeSignal(ordinal, DecisionSignalKind.EspConfigDetected, T0.AddSeconds(ordinal), + new Dictionary + { + [SignalPayloadKeys.SkipUserEsp] = skipUser, + [SignalPayloadKeys.SkipDeviceEsp] = "false", + }); + + private static DecisionSignal MakeSignal( + long ordinal, + DecisionSignalKind kind, + DateTime occurredAtUtc, + IReadOnlyDictionary? payload) + { + return new DecisionSignal( + sessionSignalOrdinal: ordinal, + sessionTraceOrdinal: ordinal, + kind: kind, + kindSchemaVersion: 1, + occurredAtUtc: occurredAtUtc, + sourceOrigin: "test", + evidence: new Evidence(EvidenceKind.Synthetic, $"test-{kind}-{ordinal}", $"synthetic {kind}"), + payload: payload); + } + } +} diff --git a/src/Shared/AutopilotMonitor.DecisionCore/Engine/DecisionEngine.Classic.cs b/src/Shared/AutopilotMonitor.DecisionCore/Engine/DecisionEngine.Classic.cs index 21c89a80..4d0870b1 100644 --- a/src/Shared/AutopilotMonitor.DecisionCore/Engine/DecisionEngine.Classic.cs +++ b/src/Shared/AutopilotMonitor.DecisionCore/Engine/DecisionEngine.Classic.cs @@ -720,6 +720,13 @@ namespace AutopilotMonitor.DecisionCore.Engine /// with the HelloSafety window armed at this /// signal's instant. /// + /// + /// sits-d Cloud PCs (2026-08-20): on SkipUser=true flows the observed skip stands in + /// for the final-exit evidence — Windows never renders the user ESP page, so no + /// Shell-Core 62407 and no AccountSetup provisioning category can ever exist, and this + /// handler is the only remaining site that can knock on the (open) arm-B gate. See the + /// inline skipUserExitEquivalent comment for the full derivation. + /// /// private DecisionStep HandleImeUserSessionCompletedV1(DecisionState state, DecisionSignal signal) { @@ -768,11 +775,33 @@ namespace AutopilotMonitor.DecisionCore.Engine // session is still parked pre-Hello (same stage guard as the deferred-promote path // in HandleAccountSetupProvisioningCompleteV1) — AwaitingHello/AwaitingDesktop/ // Finalizing/terminal stages keep their existing rails untouched. + // + // sits-d Cloud PCs (2026-08-20): on a SkipUser=true flow Windows never renders the + // user ESP page — Shell-Core 62407 does not fire and the AccountSetup provisioning + // categories are never written, so IsPostAccountSetupFinalExit is unsatisfiable by + // construction (there is no exit to record). Arm B of ShouldTransitionToAwaitingHello + // is open from the EspConfigDetected bootstrap onward, but every site that evaluates + // the gate hangs off a signal this flow cannot produce (EspExiting and + // EspPhaseChanged(FinalizingSetup) ride Shell-Core; the desktop fast-path requires + // the Hello policy to be disabled) — five sessions sat at "waiting on: + // hello_resolution" for six hours with the gate open and nothing left to knock. + // On such a flow the observed skip IS the exit evidence: no page exists whose exit + // could ever say more. The other three facts (AccountSetup anchor, genuine IME + // user-session completion at-or-after it, real-user desktop) stay mandatory, which + // keeps this STRICTER than the existing Shell-Core-carried arm-B promotion (pinned + // in ClassicAwaitingHelloGuardTests, which needs no IME/desktop evidence at all). + // This knock is also self-healing against Fix 10's AccountSetup bounce-back: the + // IME phase line always precedes the user-session-complete line in log order, so a + // bounced promotion is re-attempted by the very next user-session re-emission. + // Device Preparation keeps its own rails (arm D + DevicePrepCompletion backstop). + var skipUserExitEquivalent = + state.ScenarioObservations.SkipUserEsp?.Value == true + && state.ScenarioProfile.Mode != EnrollmentMode.DevicePreparation; var armCSatisfied = builder.ImeUserSessionCompletedUtc != null && state.AccountSetupEnteredUtc != null && builder.ImeUserSessionCompletedUtc.Value >= state.AccountSetupEnteredUtc.Value - && IsPostAccountSetupFinalExit(state) + && (IsPostAccountSetupFinalExit(state) || skipUserExitEquivalent) && state.DesktopArrivedUtc != null; var parkedPreHello = state.Stage == SessionStage.SessionStarted || state.Stage == SessionStage.EspDeviceSetup