Compare commits

...

3 Commits

Author SHA1 Message Date
Oliver Kieselbach 54044c5d9c esp: continue-anyway observation mode — bounded 60-min defang for device-phase failures
Classic UD fleets held at the Device-ESP timeout wall by one slow blocking
app showed ~100% Failed (tenant c9787ba2: 948/1000 esp_terminal_failure)
although the profile allows "Continue anyway" and users demonstrably reach
the desktop — the classic advisory defang requires AccountSetupEnteredUtc
and never applied there.

New operator-set tenant opt-in EnableEspContinueAnywayObservation
(ConfigVersion 36 -> 37, default off):

- DecisionCore: a Device-phase EspTerminalFailure with ContinueAnyway
  allowed + opt-in observed (EspConfigDetected payload fact — kernel stays
  config-free) + Classic/Unknown non-pre-prov scenario defangs into the
  advisory rail with a 60-min AdvisoryCompletion window. Real end via the
  DAD-validated real-user desktop + Hello gate (IME user-session gate is
  structurally unavailable without the AccountSetup anchor): eager
  completion on DesktopArrived, category recovery via the 4910a5a5 hook,
  expiry un-defangs to the original esp_terminal_failure — every observed
  session still gets a terminal verdict.
- Honest outcome: enrollment_complete under an unresolved advisory carries
  espSoftFailure/completionSource (also on the classic post-AccountSetup
  advisory path); backend stamps EspSoftFailure/CompletionSource on the
  Sessions row (+ index mirror, terminal-tuple preservation); web renders
  an amber "with issues" badge plus a session-detail banner. No new
  terminal status — Succeeded + marker.
- Agent: remote-config field merged into AgentConfiguration; stamped into
  both esp_config_detected emitters (orchestrator bootstrap via live
  Func<bool> seam, DeviceInfoCollector via host/factory pass-through).
- Admin UI: GA-only toggle in the tenant management modal (next to
  Bootstrap Token) + stored/runtime rows in the tenant config report;
  GA-gated on both write paths (PUT silently reverts for tenant admins,
  field patch rejects with 400).

Tests: 13 new ContinueAnywayObservationTests; DecisionCore 513, Agent V2
2162, Functions 3948, SummaryDialog 44, vitest 632, tsc clean.
Docs: docs/agent/continue-anyway-observation.md (OKF) + index/log.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 12:28:45 +02:00
Oliver Kieselbach 3b3e8a7614 analyze: delay auto-analyze visibility 30s so rules stop racing the terminal flush
The enqueue fires in the very ingest batch that carries enrollment_failed /
enrollment_complete — by design the FIRST event of the agent's terminal
sequence. The worker's hot loop dequeued sub-second and evaluated rules
against a stream still missing the terminal-flush rows (stuck-promotion's
synthetic app_install_failed, app_tracking_summary, agent_shutting_down),
and the RuleResult dedup made the gap permanent. Session 53d1e9f6:
ANALYZE-ESP-004's optional app_install_failed condition never matched and
{{appName}} rendered as a literal placeholder (confidence 80, factor never
fired). A 30 s visibilityTimeout covers the flush including the agent's
late-event grace.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 12:28:25 +02:00
Oliver Kieselbach 115b4ec8c2 bootstrap: promote Windows 365 Cloud PC relax to v2.3 production script
Install-AutopilotMonitor.ps1 takes over the field-tested v2.3-dev.3 logic
unchanged (code-only diff vs dev: ScriptVersion + log file name); comments
condensed to customer-facing descriptions, dev changelog collapsed into one
v2.3 entry. Pester suite moved to Install-AutopilotMonitor.Tests.ps1 (neutral
test user names), dev script and dev suite removed. Test-ShouldBootstrapAgent
dry-run tester mirrors the Cloud PC trigger and the guard-4 uptime exemption.

Pester 29/29 green under PS 5.1 and pwsh 7; parse + pure-ASCII checks pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 09:18:13 +02:00
42 changed files with 1306 additions and 802 deletions
+114
View File
@@ -0,0 +1,114 @@
---
type: Concept
title: Continue-Anyway Observation Mode — Device-Phase ESP Failure Defang
description: Operator-set tenant opt-in that turns an immediate Device-phase ESP terminal failure on a Continue-Anyway-enabled profile into a bounded 60-minute observation; the DAD-validated real-user desktop plus the Hello gate detect the true end, and the session completes as Succeeded with an esp-soft-failure marker.
resource: /src/Shared/AutopilotMonitor.DecisionCore/Engine/DecisionEngine.Edge.cs
tags:
- agent
- decision-engine
- esp
- continue-anyway
- soft-failure
- tenant-config
timestamp: 2026-08-08T00:00:00+02:00
---
# Continue-Anyway Observation Mode
## Problem
A Classic user-driven enrollment whose Device ESP hits the configured terminal timeout
(`syncFailureTimeoutMinutes`, typically 30) fails **before AccountSetup is ever entered**.
The classic advisory defang (`HandleEspTerminalFailureV1`) requires
`AccountSetupEnteredUtc != null`, so these sessions hard-fail immediately — even when the
ESP profile allows "Continue anyway" and users demonstrably dismiss the failure screen and
reach a working desktop. A fleet held at the timeout wall by one slow blocking app then
shows ~100% Failed (tenant c9787ba2: 948/1000 `esp_terminal_failure`, culprit "Encompass
Hybrid Installer"), which tells the tenant nothing.
## Mechanism
Opt-in flow (default off — absence keeps the immediate hard fail):
1. **Tenant setting** `EnableEspContinueAnywayObservation` (`TenantConfiguration`,
operator-set only, NOT exposed in the tenant admin settings UI; the toggle lives in
the Global-Admin tenant management modal next to Bootstrap Token / Unrestricted Mode,
and the field is GA-gated on both write paths — the generic PUT silently reverts it
for tenant admins, the field patch rejects it with 400) → served via `agent/config`
(`AgentConfigResponse`, ConfigVersion 37).
2. **Agent** stamps `espContinueAnywayObservationEnabled=true` into the
`EspConfigDetected` signal payload (both emitters: the orchestrator FirstSync
bootstrap and `DeviceInfoCollector.PostEspConfigDetectedSignal`) — only when enabled,
so the kernel stays config-free and replay/recovery reconstruct the fact from the
signal log.
3. **Engine** (`HandleEspTerminalFailureV1`): a terminal ESP failure is defanged into an
`esp_failure_advisory` (`advisoryReason=esp_failure_defanged_continueanyway_observation`)
when ALL hold: profile allows Continue-Anyway, observation opt-in observed,
`AccountSetupEnteredUtc == null`, and the scenario is Classic/Unknown with
`PreProvisioningSide=None` and no registry self-deploying marker (WhiteGlove /
SelfDeploying / DevicePreparation have no interactive user to press the button).
4. **Observation window**: the shared `AdvisoryCompletion` deadline is armed with a
**60-minute** window (classic advisory: 30) — long enough for failure screen →
Continue anyway → sign-in → desktop, or a full device-phase re-run via "Try again".
## Detecting the real end
The active-advisory-without-AccountSetup shape is unique to this arming site, so
`IsContinueAnywayObservationActive(state)` (advisory recorded ∧ not resolved ∧
AccountSetup never entered) needs no separate opt-in re-check. Three outcomes:
* **Continue anyway** — the DAD-validated real-user desktop arrives (defaultuser0/SYSTEM
excluded at the detector) and the Hello gate is satisfied (resolved, or policy
explicitly disabled without an observed wizard): `HandleDesktopArrivedV1` completes
eagerly through Finalizing (trigger `DesktopArrived:ContinueAnywayObservation`) — the
session end reflects the real end, not the window edge. The IME user-session gate is
deliberately waived: it anchors on `AccountSetupEnteredUtc`, which is structurally
absent here. The terminal `enrollment_complete` carries `espSoftFailure=true`,
`completionSource=continue_anyway_observation` and the failed category; the backend
stamps `EspSoftFailure`/`CompletionSource` on the Sessions row (+ SessionsIndex
mirror) and the web renders the amber "with issues" badge plus a detail banner.
Sessions completing under the CLASSIC advisory (AccountSetup entered) carry the same
marker with `completionSource=continue_anyway_post_accountsetup`.
* **Try again** — the ESP re-runs the failed category; `DeviceSetupProvisioningComplete`
resolves the advisory via the 4910a5a5 recovery hook (`esp_failure_advisory_resolved`)
and the session continues on the normal rails, completing clean (no marker).
* **Nothing** — the window expires: un-defang to `enrollment_failed` with the original
`esp_terminal_failure` reason and
`advisoryReason=continue_anyway_observation_window_expired_without_completion_evidence`;
`LastFailureTrigger=EspTerminalFailure` keeps the likely-stuck app promotion on. Every
observed session therefore still gets a terminal verdict ≤ ~60 min after the failure.
Deadline-fire edge cases mirror the classic paths: desktop + unknown Hello promotes to
`AwaitingHello` with the HelloSafety window (never-observed promote), and enforcement
progress since arming re-arms the window (30-min re-arm, convergent) instead of failing a
device mid-"Try again".
## Status semantics
No new terminal status: the session is `Succeeded` + `EspSoftFailure=true` (user
decision 2026-08-08). Success-rate conventions stay untouched; honesty lives in the
badge, the detail banner, and ANALYZE-ESP-004 (which continues to fire on the un-defang
expiry path via `mayHaveContinuedAnyway`).
# Examples
Session `53d1e9f6-4eef-4f3b-a999-c63e96c5148d` (tenant c9787ba2, 2026-08-07) is the
motivating shape: DeviceSetup/Certificates failed at the 30-min wall while "Encompass
Hybrid Installer" was still installing, AccountSetup 0/5, Continue-Anyway allowed —
hard-failed under the default semantics.
Tests: `ContinueAnywayObservationTests` (DecisionCore.Tests) — defang, 60-min arming,
opt-out/scenario regressions, eager desktop completion with marker, clean-completion
marker absence, expiry un-defang, hello promotes, category recovery.
# Citations
* `src/Shared/AutopilotMonitor.DecisionCore/Engine/DecisionEngine.Edge.cs` — arming site,
observation branches in `HandleAdvisoryCompletionDeadlineFired`,
`IsContinueAnywayObservationActive`.
* `src/Shared/AutopilotMonitor.DecisionCore/Engine/DecisionEngine.Classic.cs` —
`HandleDesktopArrivedV1` observation completion, `BuildEnrollmentCompleteEffect`
soft-failure marker.
* `src/Backend/AutopilotMonitor.Functions/Services/EventIngestProcessor.Classification.cs`
/ `TableStorageService.Sessions.cs` — session stamping + index mirror.
* `src/Web/autopilot-monitor-web/components/SessionStatusBadge.tsx` — amber badge.
+1
View File
@@ -23,6 +23,7 @@ https://docs.autopilotmonitor.com (separate repository).
* [Gather Rule Debug Log](agent/gather-rule-debug-log.md) - EnableGatherRuleDebugLog (ConfigVersion 36): the per-tenant local trace file that explains every gather-rule evaluation decision — scope skips, on_change suppression, empty collector results, logparser position/parse-mode outcomes — for rules that produce no timeline events.
* [MDM Reboot Coalescing](agent/mdm-reboot-coalescing.md) - Attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: aggregated DM-Enterprise 2800 observations (neutral, flush-time watermark) + the ANALYZE-ESP-005 advisory gated on an actually-observed reboot.
* [Secure Boot CA-2023 Detection](agent/secureboot-ca2023-detection.md) - Why ANALYZE-SEC-001 v3 reads the UEFI db/KEK variables directly instead of trusting the SecureBoot\Servicing registry, and how the one-sided uefiCa2023FirmwareConfirmed marker plus a not_exists precondition give absence-tolerant suppression across old and new agents.
* [Continue-Anyway Observation Mode](agent/continue-anyway-observation.md) - EnableEspContinueAnywayObservation (ConfigVersion 37, operator-set): Device-phase ESP terminal failures on Continue-Anyway profiles get a bounded 60-min observation instead of an immediate hard fail; the DAD-validated real-user desktop + Hello gate detect the true end, the session completes Succeeded + EspSoftFailure (amber "with issues" badge), expiry un-defangs to the original esp_terminal_failure.
* [ESP Policy-Provider Stall Detection](agent/esp-policy-provider-stall.md) - The EnrollmentStatusTracking CSP's no-timeout Setup/Apps wait is keyed to the Intune-registered "Sidecar" provider by NAME: co-management's "ConfigMgr" parks the user ESP at "Apps (Identifying)" for days even with TrackingPoliciesCreated=1 (issue #106); the always-on two-arm esp_policy_provider_stalled tripwire (provider_incomplete | sidecar_provider_missing) and the SOFTWARE\Microsoft\Windows\Autopilot gather-rule allowlist entry that surface it.
# Rules
+4
View File
@@ -1,5 +1,9 @@
# Log
## 2026-08-08
* **New**: `agent/continue-anyway-observation.md` — Continue-Anyway observation mode (`EnableEspContinueAnywayObservation`, ConfigVersion 37, operator-set): Device-phase ESP terminal failures on Continue-Anyway profiles are defanged into a 60-min observation window (Classic/Unknown, non-pre-prov only); real-user desktop + Hello gate complete the session as Succeeded + `EspSoftFailure`/`CompletionSource` (amber "with issues" badge + detail banner), category recovery resolves clean, expiry un-defangs to the original `esp_terminal_failure`. Registered in the Agent index section. Related backend fix (no doc): the auto-analyze enqueue now carries a 30 s `visibilityTimeout` so rule evaluation no longer races the agent's terminal flush (ANALYZE-ESP-004's `{{appName}}` placeholder).
## 2026-08-07 (later)
* **Update**: `backend/cloudpc-device-validation.md` — third `IsCloudPc` transport rail: the pre-auth distress channel. `DistressReport.IsCloudPc` (CloudPcDetector verdict captured at `DistressReporter` construction) → `DistressReportEntry`/table column → "Devices Not Registered" aggregation (sticky-true per serial, same OR semantics as the Sessions merge) → portal Cloud PC badge + actionable hint ("enable Windows 365 Cloud PC Validation + W365CloudPcValidation add-on"). Motivated by the first W365 field run: the rejected Cloud PC showed up as anonymous "Microsoft Corporation / Virtual Machine" with no clue that the fix is the Cloud PC validator, not Autopilot registration. UNVERIFIED like all distress fields; ≤18 bytes, fits the 1536-byte cap.
@@ -1,523 +0,0 @@
<#
.SYNOPSIS
DEV bootstrap script (v2.3-dev) to deploy and start the Autopilot Monitor agent.
.DESCRIPTION
DEV VARIANT of Install-AutopilotMonitor.ps1 for validating the v2.3 Cloud-PC-relax
on Windows 365 before promoting it to the production script. Differences to v2.2:
- Cloud-PC-relax (see below) as a second trigger for the guard 2+3 exemption
- Raw OutOfBoxExperienceState value is logged (diagnosis)
- Logs to bootstrap_agent_dev.log so runs never interleave with the prod script
- Testable structure: guard/relax logic lives in functions; the script only
executes when run directly. Dot-sourcing it (as the Pester suite in
Install-AutopilotMonitor-dev.Tests.ps1 does) loads the functions without
side effects.
Designed to be deployed via Intune as a PowerShell Script during Autopilot.
Runs very early in the enrollment process (first Intune action) and:
1. Runs pre-flight guards to skip productive devices and ghost re-installs.
A relax exemption allows install when a single freshly-created profile
appears DURING OOBE (Windows Backup for Organizations restore) or on a
freshly provisioned Windows 365 Cloud PC.
2. Downloads the monitoring agent ZIP from Azure Blob Storage
3. Verifies integrity via SHA-256 hash from the version manifest (mandatory)
4. Extracts the agent into %ProgramData%\AutopilotMonitor\Agent
5. Runs the agent in --install mode (registers Scheduled Task, spawns the runtime)
6. Verifies the runtime process actually launched
Agent self-destructs when enrollment completes.
Relax flow (guards 2+3): a real user profile (or a real LastLoggedOnUser)
normally means "productive -> skip". Two legitimate flows break that assumption:
(1) Windows Backup for Organizations: one user signs in + restores DURING OOBE.
Trigger: the OS positively reports OOBE InProgress (Test via WinRT
SystemSetupInfo.OutOfBoxExperienceState).
(2) Windows 365 Cloud PC: W365 provisions headless -- OOBE and Intune enrollment
complete before any user ever connects, so OutOfBoxExperienceState is NEVER
InProgress when this script runs. The assigned user's profile is created at
first connect, typically seconds before IME pulls this script. Trigger: the
device positively identifies as a Cloud PC via TWO local markers (both
required; captured from a real Cloud PC -- local WMI reports only the
generic 'Virtual Machine' model): HKLM\SOFTWARE\Microsoft\Windows365
registry key AND the CloudManagedDesktopExtension service.
Either trigger relaxes guards 2+3 ONLY when additionally:
(a) there is exactly ONE real user profile, AND
(b) that profile was created within $OobeProfileMaxAgeMinutes minutes.
The Cloud PC trigger additionally exempts guard 4 (uptime window) -- see
.PARAMETER MaxBootstrapWindowHours for why boot uptime is meaningless there.
Any miss keeps the original skip (SKIP-safe: we never install on uncertainty).
On productive devices the profile is old, so (b) keeps skipping correctly.
Known accepted edge: a W365 Frontline shared-mode device can legitimately show a
single fresh profile while productive; a mis-install there is one-time (agent
finds enrollment complete and self-destructs; Deployed marker prevents repeats).
.PARAMETER AgentDownloadUrl
URL to download the agent ZIP from. Defaults to the production download endpoint;
override only for parallel lab/dev assignments that point at a separate pre-release ZIP.
.PARAMETER VersionJsonName
Filename of the integrity manifest in the same blob container as the agent ZIP.
Defaults to "version.json"; override only for parallel lab/dev manifests.
.PARAMETER MaxBootstrapWindowHours
Maximum device uptime (hours) within which the bootstrap is still considered
valid. Devices booted more than this many hours ago are skipped because we no
longer trust their OOBE state. Default: 12.
Cloud PC note: NOT applied when the Cloud-PC relax is active. A Cloud PC runs
headless for days between provisioning and the user's first connect, and the
phase worth monitoring (Account Setup) only starts at that first connect --
there the fresh-profile window (OobeProfileMaxAgeMinutes) is the time anchor,
not boot uptime.
.PARAMETER OobeProfileMaxAgeMinutes
Maximum age (minutes) of the single profile that may appear for the relax
exemption to apply. The restore/first-connect -> bootstrap gap is always
minutes; an older profile is treated as a productive device. Default: 15.
.NOTES
- Agent is temporary and auto-removes after enrollment.
- All files live under C:\ProgramData\AutopilotMonitor (easy cleanup).
- One registry key remains after removal: HKLM\SOFTWARE\AutopilotMonitor\Deployed
(prevents ghost re-installs on re-Autopilot of the same device).
- Scheduled Task survives reboots during enrollment.
- OOBE detection uses Windows.System.Profile.SystemSetupInfo.OutOfBoxExperienceState.
Requires Windows 10, version 1809 (introduced in 10.0.17763.0). Older builds lack
the class -> state reads as 'Unavailable' -> original (skip) behaviour. Docs:
https://learn.microsoft.com/en-us/uwp/api/windows.system.profile.systemsetupinfo
- This script MUST remain pure ASCII (no Unicode/UTF-8 special chars).
PowerShell 5.1 (IME) reads scripts without BOM as ANSI, corrupting multi-byte chars.
.CHANGELOG
2026-08-06 v2.3-dev.3 Guard 4 (uptime window) exempted while the Cloud-PC relax is
active: Cloud PCs run headless for days before the user's first
connect, so boot uptime says nothing about enrollment freshness --
the single-fresh-profile window (< OobeProfileMaxAgeMinutes) already
anchors the relax in time. OOBE-restore keeps the uptime guard.
2026-08-06 v2.3-dev.2 Cloud PC detection rebuilt on captured evidence: first field
test proved local WMI reports only Manufacturer='Microsoft
Corporation' + Model='Virtual Machine' (the 'Cloud PC ...' model
string exists only in Intune/Graph). Detection now requires the
HKLM\SOFTWARE\Microsoft\Windows365 registry key AND the
CloudManagedDesktopExtension service (both verified present on a
real Cloud PC); raw identity + marker states are always logged.
2026-08-05 v2.3-dev DEV variant (-dev.ps1, own log file bootstrap_agent_dev.log)
to validate on Windows 365 before promotion to v2.3:
Cloud-PC-relax -- a positively identified Cloud PC
(Win32_ComputerSystem Manufacturer 'Microsoft Corporation' +
Model 'Cloud PC*') now triggers the guard 2+3 relax alongside
OOBE InProgress; the single-fresh-profile conditions are
unchanged. Raw OutOfBoxExperienceState is now logged.
Refactored into functions with a dot-source entry guard so the
Pester suite (*.Tests.ps1) can load the logic without executing
the bootstrap; behaviour when run directly is unchanged.
2026-07-20 v2.2 Switched default download endpoint from the legacy blob
(autopilotmonitor.blob.core.windows.net) to the Front Door alias
https://download.autopilotmonitor.com/agent/ (route /agent/*).
version.json is still derived from the same container.
2026-06-25 v2.1 OOBE-relax: detect genuine OOBE via WinRT
SystemSetupInfo.OutOfBoxExperienceState and exempt guards 2+3 when a
single freshly-created profile appears in OOBE (Windows Backup for
Organizations). SKIP-safe when the API is absent (< 1809).
Removed legacy Content-MD5 fallback; SHA-256 manifest verification is
now mandatory (no hash -> bootstrap aborts, no unverified install).
2026-05-09 v2.0 Generic bootstrap: agent owns its own defaults (e.g. 600 s
TenantId-wait), so the script only calls `--install` plain.
Hardened post-install with a 10 s runtime-process verify.
(older entries: see Install-AutopilotMonitor.ps1)
#>
[CmdletBinding()]
param(
[Parameter(Mandatory = $false)]
[string]$AgentDownloadUrl = "https://download.autopilotmonitor.com/agent/AutopilotMonitor-Agent.zip",
[Parameter(Mandatory = $false)]
[string]$VersionJsonName = "version.json",
[Parameter(Mandatory = $false)]
[int]$MaxBootstrapWindowHours = 12,
[Parameter(Mandatory = $false)]
[int]$OobeProfileMaxAgeMinutes = 15
)
# Script version (bump on meaningful changes; see .CHANGELOG above)
$ScriptVersion = "2.3-dev.3"
# Configuration - Everything in ProgramData for easy cleanup
$AgentBasePath = "$env:ProgramData\AutopilotMonitor"
$AgentBinPath = "$AgentBasePath\Agent"
$AgentLogPath = "$AgentBasePath\Logs"
$LogFile = "$AgentLogPath\bootstrap_agent_dev.log"
# Write-Host (NOT Write-Output): log calls happen inside functions that return
# decision objects; Write-Output would leak every log line into those return
# values and turn them into arrays. Write-Host goes to the console host only.
function Write-Log {
param([string]$Message)
$timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
$logMessage = "[$timestamp] $Message"
Write-Host $logMessage
Add-Content -Path $LogFile -Value $logMessage -ErrorAction SilentlyContinue
}
# Raw OOBE state via the documented WinRT API
# Windows.System.Profile.SystemSetupInfo.OutOfBoxExperienceState.
# Returns the state string (InProgress/Completed/NotStarted). An older OS without the
# class (< Win10 1809 / 10.0.17763.0) or any error returns 'Unavailable'
# -> SKIP-safe (we never relax a guard on uncertainty).
# Docs: https://learn.microsoft.com/en-us/uwp/api/windows.system.profile.systemsetupinfo
function Get-OobeState {
try {
$null = [Windows.System.Profile.SystemSetupInfo, Windows.System.Profile, ContentType = WindowsRuntime]
return ([Windows.System.Profile.SystemSetupInfo]::OutOfBoxExperienceState).ToString()
}
catch {
return 'Unavailable'
}
}
# Positive Windows 365 Cloud PC detection.
# Evidence captured 2026-08-06 on a real, freshly provisioned W365 Enterprise Cloud PC:
# - Win32_ComputerSystem is GENERIC there: Manufacturer='Microsoft Corporation',
# Model='Virtual Machine'. The 'Cloud PC Enterprise ...' model string exists only
# in the Intune/Graph inventory, NOT in local WMI/SMBIOS -- it cannot be used here.
# - HKLM:\SOFTWARE\Microsoft\Windows365 exists.
# - Service 'CloudManagedDesktopExtension' (Microsoft Cloud Managed Desktop
# Extension, the W365 management agent that runs ON Cloud PCs) is installed.
# BOTH markers are required (AND): each alone has plausible look-alikes (W365-Boot
# physical clients may carry Windows365 policy state; the MCMD agent family also
# served other managed-desktop offerings). Raw identity and both marker states are
# always logged for field diagnosis; any error returns $false (SKIP-safe).
function Test-IsCloudPc {
try {
$cs = Get-CimInstance Win32_ComputerSystem -ErrorAction Stop
Write-Log "Computer identity: Manufacturer='$($cs.Manufacturer)' Model='$($cs.Model)'."
}
catch { Write-Log "INFO: computer identity query failed: $($_.Exception.Message)" }
try {
$regMarker = Test-Path 'HKLM:\SOFTWARE\Microsoft\Windows365'
$svcMarker = $null -ne (Get-Service -Name 'CloudManagedDesktopExtension' -ErrorAction SilentlyContinue)
Write-Log "Cloud PC markers: Windows365 registry key=$regMarker; CloudManagedDesktopExtension service=$svcMarker."
return ($regMarker -and $svcMarker)
}
catch {
Write-Log "INFO: Cloud PC marker check failed: $($_.Exception.Message)"
return $false
}
}
# Collects real (non-special) user profiles as full paths (WMI Special flag + filesystem).
# NOTE: callers MUST wrap the result in @(...). PowerShell unrolls function output, so an
# empty result returns $null and a single path returns a scalar string, whose [0] indexer
# would yield its first char ('C') instead of the full path.
function Get-RealUserProfilePaths {
$excludePattern = '^(defaultuser\d*|Public|Default( User)?|All Users|WDAGUtilityAccount)$'
@(
try {
Get-CimInstance Win32_UserProfile -ErrorAction Stop |
Where-Object { -not $_.Special -and $_.LocalPath -like 'C:\Users\*' } |
ForEach-Object { $_.LocalPath }
} catch { Write-Log "INFO: WMI profile query failed, continuing with filesystem check." }
(Get-ChildItem 'C:\Users' -Directory -ErrorAction SilentlyContinue).FullName
) | Where-Object { $_ -and (Split-Path $_ -Leaf) -notmatch $excludePattern } |
Select-Object -Unique
}
# Relax: the ONLY exemption to guards 2+3. Two triggers (see .DESCRIPTION):
# (1) OOBE InProgress -> Windows Backup for Organizations restore
# (2) Cloud PC identity -> Windows 365 (headless OOBE, profile at first connect)
# Either trigger additionally requires: exactly one real profile AND that profile
# created < $OobeProfileMaxAgeMinutes minutes ago. Any miss keeps the original skip.
# Returns an object with: Active (bool), OobeState (string), IsCloudPc (bool).
function Get-RelaxDecision {
param(
[string[]]$ProfilePaths,
[int]$OobeProfileMaxAgeMinutes
)
if ($null -eq $ProfilePaths) { $ProfilePaths = @() }
$relaxActive = $false
$oobeState = Get-OobeState
$oobeInProgress = ($oobeState -eq 'InProgress')
$isCloudPc = Test-IsCloudPc
if ($isCloudPc) {
Write-Log "Cloud PC detected (OOBE state: $oobeState)."
}
$relaxTrigger = $null
if ($oobeInProgress) {
$relaxTrigger = "OOBE InProgress (Windows Backup restore case)"
} elseif ($isCloudPc) {
$relaxTrigger = "Cloud PC (headless provisioning, OOBE state: $oobeState)"
}
if ($ProfilePaths.Count -eq 1 -and $relaxTrigger) {
try {
$created = (Get-Item -LiteralPath $ProfilePaths[0] -Force -ErrorAction Stop).CreationTimeUtc
$ageMin = ((Get-Date).ToUniversalTime() - $created).TotalMinutes
if ($ageMin -ge 0 -and $ageMin -lt $OobeProfileMaxAgeMinutes) {
$relaxActive = $true
Write-Log ("Relax active: {0} + single profile created {1:N1}min ago (< {2}min). Guards 2+3 relaxed." -f $relaxTrigger, $ageMin, $OobeProfileMaxAgeMinutes)
} else {
Write-Log ("Relax NOT applied: {0} + single profile '{1}', but profile created {2:N1}min ago (created {3:u}) is outside the [0, {4})min window." -f $relaxTrigger, (Split-Path $ProfilePaths[0] -Leaf), $ageMin, $created, $OobeProfileMaxAgeMinutes)
}
} catch { Write-Log "INFO: profile age check failed; relax not applied. $($_.Exception.Message)" }
} elseif ($ProfilePaths.Count -eq 1) {
Write-Log "Relax NOT applied: single profile present but neither OOBE InProgress nor Cloud PC (OOBE state: $oobeState)."
} elseif ($ProfilePaths.Count -gt 1) {
Write-Log ("Relax NOT applied: {0} real profiles present (relax requires exactly one). OOBE state: {1}; CloudPC: {2}." -f $ProfilePaths.Count, $oobeState, $isCloudPc)
}
return [pscustomobject]@{
Active = $relaxActive
OobeState = $oobeState
IsCloudPc = $isCloudPc
}
}
# Runs pre-flight guards 1-5 and returns the install decision WITHOUT side effects
# beyond logging. Returns an object with:
# Install (bool), ReasonCode (null | AlreadyDeployed | ProfilesFound |
# LastLoggedOnUser | UptimeExceeded | AgentPresent), RelaxActive (bool).
function Get-BootstrapDecision {
param(
[int]$MaxBootstrapWindowHours,
[int]$OobeProfileMaxAgeMinutes,
[string]$AgentBinPath
)
# Guard 1: Agent was already deployed on this device (registry marker survives self-destruct)
$deployed = (Get-ItemProperty -Path 'HKLM:\SOFTWARE\AutopilotMonitor' -Name 'Deployed' -ErrorAction SilentlyContinue).Deployed
if ($deployed) {
Write-Log "SKIP: Agent was previously deployed at $deployed."
return [pscustomobject]@{ Install = $false; ReasonCode = 'AlreadyDeployed'; RelaxActive = $false }
}
# The @() wrap is required, see NOTE on Get-RealUserProfilePaths.
$profilePaths = @(Get-RealUserProfilePaths)
$relax = Get-RelaxDecision -ProfilePaths $profilePaths -OobeProfileMaxAgeMinutes $OobeProfileMaxAgeMinutes
# Guard 2: No real user profile should exist yet (primary productive-device guard),
# unless the relax signature holds.
if ($profilePaths.Count -gt 0 -and -not $relax.Active) {
$details = (($profilePaths | ForEach-Object {
$leaf = Split-Path $_ -Leaf
try {
$c = (Get-Item -LiteralPath $_ -Force -ErrorAction Stop).CreationTimeUtc
"{0} (created {1:u}, {2:N1}min ago)" -f $leaf, $c, ((Get-Date).ToUniversalTime() - $c).TotalMinutes
} catch { "$leaf (age unknown)" }
}) | Select-Object -First 3) -join '; '
Write-Log "SKIP: Real user profile(s) found ($details). OOBE state: $($relax.OobeState). CloudPC: $($relax.IsCloudPc). Device appears productive."
return [pscustomobject]@{ Install = $false; ReasonCode = 'ProfilesFound'; RelaxActive = $false }
}
# Guard 3: LastLoggedOnUser - during Device ESP no real user has logged on yet
# (same relax exemption: the restoring/first-connect user signs in legitimately).
$lastLoggedOnUser = (Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Authentication\LogonUI' -Name 'LastLoggedOnUser' -EA SilentlyContinue).LastLoggedOnUser
if ($lastLoggedOnUser -and $lastLoggedOnUser -notmatch 'defaultuser\d*' -and -not $relax.Active) {
Write-Log "SKIP: LastLoggedOnUser found ($lastLoggedOnUser). Device appears productive."
return [pscustomobject]@{ Install = $false; ReasonCode = 'LastLoggedOnUser'; RelaxActive = $false }
}
# Guard 4: Bootstrap window check - device uptime must be within accepted OOBE window.
# Sleep/standby does not reset uptime, only real boot/restart does.
# Cloud PC exemption (v2.3-dev.3): a Cloud PC runs headless from provisioning until the
# user's first connect -- often days -- and the phase worth monitoring (Account Setup)
# only STARTS at that first connect. Boot uptime is therefore meaningless there; the
# single-fresh-profile window of the relax (< $OobeProfileMaxAgeMinutes min) is the time
# anchor instead. The OOBE-restore trigger keeps the original uptime guard (physical
# device, uptime is meaningful).
$lastBoot = (Get-CimInstance Win32_OperatingSystem).LastBootUpTime
$uptimeHours = ((Get-Date) - $lastBoot).TotalHours
Write-Log "Device uptime: $([int]$uptimeHours)h (last boot: $lastBoot)"
if ($uptimeHours -gt $MaxBootstrapWindowHours) {
if ($relax.Active -and $relax.IsCloudPc) {
Write-Log "Uptime $([int]$uptimeHours)h exceeds the ${MaxBootstrapWindowHours}h window, but the Cloud PC relax is active (fresh first-connect profile is the time anchor). Continuing."
} else {
Write-Log "SKIP: Device uptime is $([int]$uptimeHours)h. OOBE state is older than accepted bootstrap window of ${MaxBootstrapWindowHours}h."
return [pscustomobject]@{ Install = $false; ReasonCode = 'UptimeExceeded'; RelaxActive = $relax.Active }
}
}
# Guard 5: Is the agent already installed? (leftover from previous run)
if (Test-Path $AgentBinPath) {
$existingAgent = Get-ChildItem -Path $AgentBinPath -Filter "AutopilotMonitor.Agent.exe" -ErrorAction SilentlyContinue
if ($existingAgent) {
Write-Log "SKIP: Agent already installed at $($existingAgent.FullName)."
return [pscustomobject]@{ Install = $false; ReasonCode = 'AgentPresent'; RelaxActive = $relax.Active }
}
}
return [pscustomobject]@{ Install = $true; ReasonCode = $null; RelaxActive = $relax.Active }
}
# Main bootstrap flow. Only invoked when the script is run directly (see entry guard
# at the bottom); never on dot-source.
function Invoke-Bootstrap {
try {
New-Item -Path $AgentBasePath -ItemType Directory -Force -ErrorAction SilentlyContinue | Out-Null
New-Item -Path $AgentBinPath -ItemType Directory -Force -ErrorAction SilentlyContinue | Out-Null
New-Item -Path $AgentLogPath -ItemType Directory -Force -ErrorAction SilentlyContinue | Out-Null
Write-Log "===== Autopilot Monitor Bootstrap Started ====="
Write-Log "Bootstrap script version: v$ScriptVersion"
$decision = Get-BootstrapDecision -MaxBootstrapWindowHours $MaxBootstrapWindowHours -OobeProfileMaxAgeMinutes $OobeProfileMaxAgeMinutes -AgentBinPath $AgentBinPath
if (-not $decision.Install) {
exit 0
}
Write-Log "Pre-flight checks passed -- no prior deployment, no productive-device signals, within bootstrap window (relaxActive=$($decision.RelaxActive))"
# Download and extract agent binaries
$agentExePath = Join-Path $AgentBinPath "AutopilotMonitor.Agent.exe"
if (Test-Path $agentExePath) {
Write-Log "Agent already installed at $agentExePath"
}
else {
Write-Log "Downloading agent from $AgentDownloadUrl..."
try {
# Derive manifest URL from the agent download URL (same blob container)
$versionJsonUrl = $AgentDownloadUrl -replace '[^/]+$', $VersionJsonName
$expectedSha256 = $null
$manifestAttempts = 3
$manifestDelays = @(1, 2) # short backoff between attempts; do not block enrollment
for ($attempt = 1; $attempt -le $manifestAttempts; $attempt++) {
try {
Write-Log "Fetching $VersionJsonName (attempt ${attempt}/${manifestAttempts}) from $versionJsonUrl for integrity verification..."
$versionJsonResponse = Invoke-RestMethod -Uri $versionJsonUrl -UseBasicParsing -TimeoutSec 5 -ErrorAction Stop
if ($versionJsonResponse.sha256) {
$expectedSha256 = $versionJsonResponse.sha256.ToLowerInvariant()
Write-Log "SHA-256 hash from manifest: $expectedSha256 (version: $($versionJsonResponse.version))"
} else {
Write-Log "WARNING: Manifest has no sha256 field - integrity cannot be verified."
}
break # manifest reached (with or without hash); retrying would not help
}
catch {
Write-Log "WARNING: Could not fetch integrity manifest (attempt $attempt): $($_.Exception.Message)"
if ($attempt -lt $manifestAttempts) {
Start-Sleep -Seconds $manifestDelays[$attempt - 1]
}
}
}
$zipPath = Join-Path $env:TEMP "AutopilotMonitor-Agent.zip"
$maxDownloadAttempts = 3
$downloadAttempt = 0
do {
$downloadAttempt++
try {
Write-Log "Download attempt ${downloadAttempt}/${maxDownloadAttempts}"
$downloadStopwatch = [System.Diagnostics.Stopwatch]::StartNew()
Invoke-WebRequest `
-Uri $AgentDownloadUrl `
-OutFile $zipPath `
-UseBasicParsing `
-TimeoutSec 30 `
-ErrorAction Stop
$downloadStopwatch.Stop()
$downloadSeconds = [math]::Round($downloadStopwatch.Elapsed.TotalSeconds, 1)
Write-Log "Downloaded agent to $zipPath (took ${downloadSeconds}s)"
break
}
catch {
if ($downloadAttempt -ge $maxDownloadAttempts) {
throw
}
$retryDelaysInSeconds = @(2, 4, 8)
$retryDelaySeconds = $retryDelaysInSeconds[$downloadAttempt - 1]
Write-Log "Download failed (attempt $downloadAttempt): $($_.Exception.Message). Retrying in ${retryDelaySeconds}s..."
Start-Sleep -Seconds $retryDelaySeconds
}
} while ($downloadAttempt -lt $maxDownloadAttempts)
# SHA-256 from the version manifest is the only accepted integrity proof.
# No hash -> refuse to install an unverified ZIP (SKIP-safe security posture).
if (-not $expectedSha256) {
throw "No SHA-256 hash available from manifest - refusing to install unverified agent ZIP."
}
$actualSha256 = (Get-FileHash $zipPath -Algorithm SHA256).Hash.ToLowerInvariant()
Write-Log "Validating SHA-256 hash against manifest"
if ($actualSha256 -ne $expectedSha256) {
throw "SHA-256 integrity check FAILED. Expected='$expectedSha256', Actual='$actualSha256'. Download may be tampered or corrupted."
}
Write-Log "SHA-256 integrity check passed"
Expand-Archive -Path $zipPath -DestinationPath $AgentBinPath -Force
Write-Log "Extracted agent to $AgentBinPath"
Remove-Item $zipPath -Force -ErrorAction SilentlyContinue
Write-Log "Cleaned up temporary files"
if (-not (Test-Path $agentExePath)) {
throw "Agent executable not found after extraction at $agentExePath"
}
Write-Log "Agent installation completed successfully"
}
catch {
Write-Log "ERROR downloading/extracting agent: $($_.Exception.Message)"
throw
}
}
Write-Log "Calling agent install mode (--install)..."
& $agentExePath --install
$installExitCode = $LASTEXITCODE
if ($installExitCode -ne 0) {
throw "Agent install failed with exit code $installExitCode"
}
Write-Log "Agent install mode completed successfully"
$runtimeProcessName = 'AutopilotMonitor.Agent'
$verifyTimeoutSec = 10
$verifyDeadline = (Get-Date).AddSeconds($verifyTimeoutSec)
$runtimeProc = $null
while ((Get-Date) -lt $verifyDeadline) {
$runtimeProc = Get-Process -Name $runtimeProcessName -ErrorAction SilentlyContinue | Select-Object -First 1
if ($null -ne $runtimeProc) { break }
Start-Sleep -Milliseconds 500
}
if ($null -ne $runtimeProc) {
$startedUtc = 'unavailable'
try { $startedUtc = $runtimeProc.StartTime.ToUniversalTime().ToString('o') } catch { }
Write-Log ("Runtime process verified: name={0}.exe pid={1} startedUtc={2}" -f $runtimeProcessName, $runtimeProc.Id, $startedUtc)
} else {
Write-Log ("WARNING: Runtime process verification FAILED. Agent --install reported success but no '{0}.exe' process appeared within {1}s. Likely silent block (AV/EDR, AppLocker/WDAC) of the runtime launch. Agent should still come up at next boot via the BootTrigger task. Check Event Viewer > Microsoft > Windows > TaskScheduler/Operational and AV/EDR logs for '{0}.exe'." -f $runtimeProcessName, $verifyTimeoutSec)
}
Write-Log "===== Bootstrap Completed Successfully ====="
exit 0
}
catch {
Write-Log "===== Bootstrap FAILED ====="
Write-Log "ERROR: $($_.Exception.Message)"
Write-Log "Stack trace: $($_.ScriptStackTrace)"
Write-Log "Please check log file: $LogFile"
$errMsg = "AutopilotMonitor bootstrap failed: $($_.Exception.Message)"
if ($errMsg.Length -gt 2048) { $errMsg = $errMsg.Substring(0, 1045) + '...' }
[Console]::Error.WriteLine($errMsg)
exit 1
}
}
# Entry guard: execute only when run as a script (IME, console). Dot-sourcing
# (". .\Install-AutopilotMonitor-dev.ps1", as the Pester suite does) only loads
# the functions above and MUST stay side-effect free.
if ($MyInvocation.InvocationName -ne '.') {
Invoke-Bootstrap
}
@@ -1,5 +1,5 @@
# Pester tests (Pester 5+) for the bootstrap guard/relax logic in
# Install-AutopilotMonitor-dev.ps1 (v2.3-dev).
# Install-AutopilotMonitor.ps1.
#
# The script under test is dot-sourced; its entry guard prevents the bootstrap
# from executing, so loading it here is side-effect free. External state (WMI,
@@ -12,7 +12,7 @@
# This file MUST remain pure ASCII (PS 5.1 reads BOM-less files as ANSI).
BeforeAll {
. (Join-Path $PSScriptRoot 'Install-AutopilotMonitor-dev.ps1')
. (Join-Path $PSScriptRoot 'Install-AutopilotMonitor.ps1')
function New-FakeProfile {
param([string]$Name, [double]$AgeMinutes)
@@ -24,9 +24,9 @@ BeforeAll {
}
Describe 'Test-IsCloudPc' {
# Marker set captured 2026-08-06 from a real W365 Enterprise Cloud PC: local WMI
# is generic ('Virtual Machine'), so detection rests on the Windows365 registry
# key AND the CloudManagedDesktopExtension service (both required).
# Local WMI on a Cloud PC is generic ('Virtual Machine'), so detection rests on
# the Windows365 registry key AND the CloudManagedDesktopExtension service
# (both required).
BeforeEach {
Mock Write-Log { }
Mock Get-CimInstance { [pscustomobject]@{ Manufacturer = 'Microsoft Corporation'; Model = 'Virtual Machine' } }
@@ -78,21 +78,21 @@ Describe 'Get-RealUserProfilePaths' {
It 'merges WMI and filesystem views, drops special/system profiles, dedupes' {
Mock Get-CimInstance {
@(
[pscustomobject]@{ Special = $false; LocalPath = 'C:\Users\LeonGottschalk' }
[pscustomobject]@{ Special = $false; LocalPath = 'C:\Users\UserOne' }
[pscustomobject]@{ Special = $true; LocalPath = 'C:\Users\systemprofile' }
[pscustomobject]@{ Special = $false; LocalPath = 'D:\Profiles\Elsewhere' }
)
}
Mock Get-ChildItem {
@(
[pscustomobject]@{ FullName = 'C:\Users\LeonGottschalk' }
[pscustomobject]@{ FullName = 'C:\Users\UserOne' }
[pscustomobject]@{ FullName = 'C:\Users\Public' }
[pscustomobject]@{ FullName = 'C:\Users\defaultuser0' }
[pscustomobject]@{ FullName = 'C:\Users\WDAGUtilityAccount' }
)
}
$result = @(Get-RealUserProfilePaths)
$result | Should -Be @('C:\Users\LeonGottschalk')
$result | Should -Be @('C:\Users\UserOne')
}
It 'falls back to the filesystem view when the WMI query fails' {
@@ -132,15 +132,15 @@ Describe 'Get-RelaxDecision' {
Mock Test-IsCloudPc { $true }
}
It 'relaxes for a single fresh profile although OOBE is Completed (W365 field case)' {
$p = New-FakeProfile 'LeonGottschalk' 0.6
It 'relaxes for a single fresh profile although OOBE is Completed (first connect)' {
$p = New-FakeProfile 'CloudPcUser' 0.6
$d = Get-RelaxDecision -ProfilePaths @($p) -OobeProfileMaxAgeMinutes 15
$d.Active | Should -BeTrue
$d.IsCloudPc | Should -BeTrue
}
It 'does not relax for an old profile (productive Cloud PC)' {
$p = New-FakeProfile 'LeonGottschalk' 30
$p = New-FakeProfile 'CloudPcUser' 30
(Get-RelaxDecision -ProfilePaths @($p) -OobeProfileMaxAgeMinutes 15).Active | Should -BeFalse
}
@@ -162,7 +162,7 @@ Describe 'Get-RelaxDecision' {
}
}
Context 'OOBE InProgress (Windows Backup for Organizations, v2.2 behaviour)' {
Context 'OOBE InProgress (Windows Backup for Organizations)' {
It 'relaxes for a single fresh profile during OOBE' {
Mock Get-OobeState { 'InProgress' }
$p = New-FakeProfile 'RestoreUser' 3
@@ -218,12 +218,12 @@ Describe 'Get-BootstrapDecision' {
$d.ReasonCode | Should -Be 'ProfilesFound'
}
It 'installs on a freshly provisioned Cloud PC with one fresh profile (W365 field case)' {
$script:fakeProfile = New-FakeProfile 'LeonGottschalk' 0.6
It 'installs on a freshly provisioned Cloud PC with one fresh profile' {
$script:fakeProfile = New-FakeProfile 'CloudPcUser' 0.6
Mock Get-RealUserProfilePaths { @($script:fakeProfile) }
Mock Test-IsCloudPc { $true }
# First-connect user is already visible in LogonUI on W365 -- must not block either.
Mock Get-ItemProperty { [pscustomobject]@{ LastLoggedOnUser = 'AzureAD\LeonGottschalk' } } -ParameterFilter { $Path -like '*LogonUI*' }
Mock Get-ItemProperty { [pscustomobject]@{ LastLoggedOnUser = 'AzureAD\CloudPcUser' } } -ParameterFilter { $Path -like '*LogonUI*' }
$d = Get-BootstrapDecision -MaxBootstrapWindowHours 12 -OobeProfileMaxAgeMinutes 15 -AgentBinPath $script:cleanAgentBin
$d.Install | Should -BeTrue
$d.RelaxActive | Should -BeTrue
@@ -248,7 +248,7 @@ Describe 'Get-BootstrapDecision' {
$d.ReasonCode | Should -Be 'UptimeExceeded'
}
It 'installs on a Cloud PC despite uptime over the window when the relax is active (guard 4 exemption, v2.3-dev.3)' {
It 'installs on a Cloud PC despite uptime over the window when the relax is active (guard 4 exemption)' {
# W365 reality: provisioned days ago, running headless ever since; the user's
# first connect just created the profile. Uptime is meaningless here.
$script:fakeProfile = New-FakeProfile 'FirstConnectUser' 0.6
+335 -210
View File
@@ -6,8 +6,9 @@
Designed to be deployed via Intune as a PowerShell Script during Autopilot.
Runs very early in the enrollment process (first Intune action) and:
1. Runs pre-flight guards to skip productive devices and ghost re-installs.
An OOBE-relax exemption allows install when a single freshly-created profile
appears DURING OOBE (the "Windows Backup for Organizations" restore case).
A relax exemption allows install when a single freshly-created profile
appears DURING OOBE (Windows Backup for Organizations restore) or on a
freshly provisioned Windows 365 Cloud PC.
2. Downloads the monitoring agent ZIP from Azure Blob Storage
3. Verifies integrity via SHA-256 hash from the version manifest (mandatory)
4. Extracts the agent into %ProgramData%\AutopilotMonitor\Agent
@@ -15,14 +16,27 @@
6. Verifies the runtime process actually launched
Agent self-destructs when enrollment completes.
OOBE-relax flow (guards 2+3): a real user profile (or a real LastLoggedOnUser)
normally means "productive -> skip". Windows Backup for Organizations breaks that
by restoring settings and creating ONE profile while still in OOBE. We relax guards
2+3 ONLY when all three positives hold, and nothing else:
(a) the OS positively reports OOBE InProgress (Test-OobeInProgress), AND
(b) there is exactly ONE real user profile, AND
(c) that profile was created within $OobeProfileMaxAgeMinutes minutes.
Any miss keeps the original skip (SKIP-safe: we never install on uncertainty).
Relax flow (guards 2+3): a real user profile (or a real LastLoggedOnUser)
normally means "productive -> skip". Two legitimate flows break that assumption:
(1) Windows Backup for Organizations: one user signs in + restores DURING OOBE.
Trigger: the OS positively reports OOBE InProgress.
(2) Windows 365 Cloud PC: W365 provisions headless -- OOBE and Intune enrollment
complete before any user ever connects, so OOBE is never InProgress when
this script runs. The assigned user's profile is created at first connect,
typically seconds before Intune pulls this script. Trigger: the device
positively identifies as a Cloud PC via TWO local markers (both required):
the HKLM\SOFTWARE\Microsoft\Windows365 registry key AND the
CloudManagedDesktopExtension service.
Either trigger relaxes guards 2+3 ONLY when additionally:
(a) there is exactly ONE real user profile, AND
(b) that profile was created within $OobeProfileMaxAgeMinutes minutes.
The Cloud PC trigger additionally exempts guard 4 (uptime window) -- see
.PARAMETER MaxBootstrapWindowHours. Any miss keeps the original skip
(SKIP-safe: we never install on uncertainty). On productive devices the
profile is old, so (b) keeps skipping correctly.
.PARAMETER AgentDownloadUrl
URL to download the agent ZIP from. Defaults to the production download endpoint;
@@ -36,11 +50,14 @@
Maximum device uptime (hours) within which the bootstrap is still considered
valid. Devices booted more than this many hours ago are skipped because we no
longer trust their OOBE state. Default: 12.
Not applied while the Cloud PC relax is active: a Cloud PC runs headless for
days between provisioning and the user's first connect, so the fresh-profile
window (OobeProfileMaxAgeMinutes) is the time anchor there, not boot uptime.
.PARAMETER OobeProfileMaxAgeMinutes
Maximum age (minutes) of the single profile that may appear during OOBE for the
OOBE-relax exemption to apply. The restore->bootstrap gap is always minutes; an
older profile is treated as a productive device. Default: 15.
Maximum age (minutes) of the single profile that may appear for the relax
exemption to apply. The restore/first-connect -> bootstrap gap is always
minutes; an older profile is treated as a productive device. Default: 15.
.NOTES
- Agent is temporary and auto-removes after enrollment.
@@ -50,12 +67,24 @@
- Scheduled Task survives reboots during enrollment.
- OOBE detection uses Windows.System.Profile.SystemSetupInfo.OutOfBoxExperienceState.
Requires Windows 10, version 1809 (introduced in 10.0.17763.0). Older builds lack
the class -> Test-OobeInProgress returns $false -> original (skip) behaviour. Docs:
the class -> state reads as 'Unavailable' -> original (skip) behaviour. Docs:
https://learn.microsoft.com/en-us/uwp/api/windows.system.profile.systemsetupinfo
- Guard/relax logic lives in functions; the script only executes when run
directly. Dot-sourcing it (as the Pester suite in
Install-AutopilotMonitor.Tests.ps1 does) loads the functions without side effects.
- This script MUST remain pure ASCII (no Unicode/UTF-8 special chars).
PowerShell 5.1 (IME) reads scripts without BOM as ANSI, corrupting multi-byte chars.
.CHANGELOG
2026-08-08 v2.3 Windows 365 Cloud PC support: a positively identified Cloud PC
(Windows365 registry key AND CloudManagedDesktopExtension
service) now triggers the guard 2+3 relax alongside OOBE
InProgress; the single-fresh-profile conditions are unchanged
and guard 4 (uptime window) is exempted for Cloud PCs.
Raw OOBE state and Cloud PC marker states are now logged.
Guard/relax logic refactored into functions with a dot-source
entry guard so the Pester suite can test it without executing
the bootstrap; behaviour when run directly is unchanged.
2026-07-20 v2.2 Switched default download endpoint from the legacy blob
(autopilotmonitor.blob.core.windows.net) to the Front Door alias
https://download.autopilotmonitor.com/agent/ (route /agent/*).
@@ -97,7 +126,7 @@ param(
)
# Script version (bump on meaningful changes; see .CHANGELOG above)
$ScriptVersion = "2.2"
$ScriptVersion = "2.3"
# Configuration - Everything in ProgramData for easy cleanup
$AgentBasePath = "$env:ProgramData\AutopilotMonitor"
@@ -105,89 +134,151 @@ $AgentBinPath = "$AgentBasePath\Agent"
$AgentLogPath = "$AgentBasePath\Logs"
$LogFile = "$AgentLogPath\bootstrap_agent.log"
New-Item -Path $AgentBasePath -ItemType Directory -Force -ErrorAction SilentlyContinue | Out-Null
New-Item -Path $AgentBinPath -ItemType Directory -Force -ErrorAction SilentlyContinue | Out-Null
New-Item -Path $AgentLogPath -ItemType Directory -Force -ErrorAction SilentlyContinue | Out-Null
# Write-Host (NOT Write-Output): log calls happen inside functions that return
# decision objects; Write-Output would leak every log line into those return values.
function Write-Log {
param([string]$Message)
$timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
$logMessage = "[$timestamp] $Message"
Write-Output $logMessage
Write-Host $logMessage
Add-Content -Path $LogFile -Value $logMessage -ErrorAction SilentlyContinue
}
# Positive OOBE check via the documented WinRT API
# Raw OOBE state via the documented WinRT API
# Windows.System.Profile.SystemSetupInfo.OutOfBoxExperienceState.
# Returns $true ONLY when the OS reports InProgress. Any other state (Completed/NotStarted),
# an older OS without the class (< Win10 1809 / 10.0.17763.0), or any error returns $false
# Returns the state string (InProgress/Completed/NotStarted). An older OS without the
# class (< Win10 1809 / 10.0.17763.0) or any error returns 'Unavailable'
# -> SKIP-safe (we never relax a guard on uncertainty).
# Docs: https://learn.microsoft.com/en-us/uwp/api/windows.system.profile.systemsetupinfo
function Test-OobeInProgress {
function Get-OobeState {
try {
$null = [Windows.System.Profile.SystemSetupInfo, Windows.System.Profile, ContentType = WindowsRuntime]
return ([Windows.System.Profile.SystemSetupInfo]::OutOfBoxExperienceState).ToString() -eq 'InProgress'
return ([Windows.System.Profile.SystemSetupInfo]::OutOfBoxExperienceState).ToString()
}
catch {
return 'Unavailable'
}
}
# Positive Windows 365 Cloud PC detection. Local WMI/SMBIOS on a Cloud PC only
# reports the generic Manufacturer='Microsoft Corporation' / Model='Virtual Machine',
# so identification uses two W365-specific markers instead. BOTH are required (AND):
# each alone has look-alikes (e.g. W365 Boot physical clients, other managed-desktop
# agents). Identity and marker states are logged for diagnosis; any error returns
# $false (SKIP-safe).
function Test-IsCloudPc {
try {
$cs = Get-CimInstance Win32_ComputerSystem -ErrorAction Stop
Write-Log "Computer identity: Manufacturer='$($cs.Manufacturer)' Model='$($cs.Model)'."
}
catch { Write-Log "INFO: computer identity query failed: $($_.Exception.Message)" }
try {
$regMarker = Test-Path 'HKLM:\SOFTWARE\Microsoft\Windows365'
$svcMarker = $null -ne (Get-Service -Name 'CloudManagedDesktopExtension' -ErrorAction SilentlyContinue)
Write-Log "Cloud PC markers: Windows365 registry key=$regMarker; CloudManagedDesktopExtension service=$svcMarker."
return ($regMarker -and $svcMarker)
}
catch {
Write-Log "INFO: Cloud PC marker check failed: $($_.Exception.Message)"
return $false
}
}
try {
Write-Log "===== Autopilot Monitor Bootstrap Started ====="
Write-Log "Bootstrap script version: v$ScriptVersion"
# Collects real (non-special) user profiles as full paths (WMI Special flag + filesystem).
# NOTE: callers MUST wrap the result in @(...). PowerShell unrolls function output, so an
# empty result returns $null and a single path returns a scalar string, whose [0] indexer
# would yield its first char ('C') instead of the full path.
function Get-RealUserProfilePaths {
$excludePattern = '^(defaultuser\d*|Public|Default( User)?|All Users|WDAGUtilityAccount)$'
@(
try {
Get-CimInstance Win32_UserProfile -ErrorAction Stop |
Where-Object { -not $_.Special -and $_.LocalPath -like 'C:\Users\*' } |
ForEach-Object { $_.LocalPath }
} catch { Write-Log "INFO: WMI profile query failed, continuing with filesystem check." }
(Get-ChildItem 'C:\Users' -Directory -ErrorAction SilentlyContinue).FullName
) | Where-Object { $_ -and (Split-Path $_ -Leaf) -notmatch $excludePattern } |
Select-Object -Unique
}
# Relax: the ONLY exemption to guards 2+3. Two triggers (see .DESCRIPTION):
# (1) OOBE InProgress -> Windows Backup for Organizations restore
# (2) Cloud PC identity -> Windows 365 (headless OOBE, profile at first connect)
# Either trigger additionally requires: exactly one real profile AND that profile
# created < $OobeProfileMaxAgeMinutes minutes ago. Any miss keeps the original skip.
# Returns an object with: Active (bool), OobeState (string), IsCloudPc (bool).
function Get-RelaxDecision {
param(
[string[]]$ProfilePaths,
[int]$OobeProfileMaxAgeMinutes
)
if ($null -eq $ProfilePaths) { $ProfilePaths = @() }
$relaxActive = $false
$oobeState = Get-OobeState
$oobeInProgress = ($oobeState -eq 'InProgress')
$isCloudPc = Test-IsCloudPc
if ($isCloudPc) {
Write-Log "Cloud PC detected (OOBE state: $oobeState)."
}
$relaxTrigger = $null
if ($oobeInProgress) {
$relaxTrigger = "OOBE InProgress (Windows Backup restore case)"
} elseif ($isCloudPc) {
$relaxTrigger = "Cloud PC (headless provisioning, OOBE state: $oobeState)"
}
if ($ProfilePaths.Count -eq 1 -and $relaxTrigger) {
try {
$created = (Get-Item -LiteralPath $ProfilePaths[0] -Force -ErrorAction Stop).CreationTimeUtc
$ageMin = ((Get-Date).ToUniversalTime() - $created).TotalMinutes
if ($ageMin -ge 0 -and $ageMin -lt $OobeProfileMaxAgeMinutes) {
$relaxActive = $true
Write-Log ("Relax active: {0} + single profile created {1:N1}min ago (< {2}min). Guards 2+3 relaxed." -f $relaxTrigger, $ageMin, $OobeProfileMaxAgeMinutes)
} else {
Write-Log ("Relax NOT applied: {0} + single profile '{1}', but profile created {2:N1}min ago (created {3:u}) is outside the [0, {4})min window." -f $relaxTrigger, (Split-Path $ProfilePaths[0] -Leaf), $ageMin, $created, $OobeProfileMaxAgeMinutes)
}
} catch { Write-Log "INFO: profile age check failed; relax not applied. $($_.Exception.Message)" }
} elseif ($ProfilePaths.Count -eq 1) {
Write-Log "Relax NOT applied: single profile present but neither OOBE InProgress nor Cloud PC (OOBE state: $oobeState)."
} elseif ($ProfilePaths.Count -gt 1) {
Write-Log ("Relax NOT applied: {0} real profiles present (relax requires exactly one). OOBE state: {1}; CloudPC: {2}." -f $ProfilePaths.Count, $oobeState, $isCloudPc)
}
return [pscustomobject]@{
Active = $relaxActive
OobeState = $oobeState
IsCloudPc = $isCloudPc
}
}
# Runs pre-flight guards 1-5 and returns the install decision WITHOUT side effects
# beyond logging. Returns an object with:
# Install (bool), ReasonCode (null | AlreadyDeployed | ProfilesFound |
# LastLoggedOnUser | UptimeExceeded | AgentPresent), RelaxActive (bool).
function Get-BootstrapDecision {
param(
[int]$MaxBootstrapWindowHours,
[int]$OobeProfileMaxAgeMinutes,
[string]$AgentBinPath
)
# Guard 1: Agent was already deployed on this device (registry marker survives self-destruct)
$deployed = (Get-ItemProperty -Path 'HKLM:\SOFTWARE\AutopilotMonitor' -Name 'Deployed' -ErrorAction SilentlyContinue).Deployed
if ($deployed) {
Write-Log "SKIP: Agent was previously deployed at $deployed."
exit 0
return [pscustomobject]@{ Install = $false; ReasonCode = 'AlreadyDeployed'; RelaxActive = $false }
}
# Collect real (non-special) user profiles as full paths (WMI Special flag + filesystem).
$excludePattern = '^(defaultuser\d*|Public|Default( User)?|All Users|WDAGUtilityAccount)$'
# NOTE: the outer @() is required. Without it the trailing Where/Select pipeline
# unwraps a single result back to a scalar string, so $profilePaths[0] would index
# into the string and return its first char ('C') instead of the full path.
$profilePaths = @(
@(
try {
Get-CimInstance Win32_UserProfile -ErrorAction Stop |
Where-Object { -not $_.Special -and $_.LocalPath -like 'C:\Users\*' } |
ForEach-Object { $_.LocalPath }
} catch { Write-Log "INFO: WMI profile query failed, continuing with filesystem check." }
(Get-ChildItem 'C:\Users' -Directory -ErrorAction SilentlyContinue).FullName
) | Where-Object { $_ -and (Split-Path $_ -Leaf) -notmatch $excludePattern } |
Select-Object -Unique
)
# OOBE-relax: the ONLY exemption to guards 2+3. Matches exactly the Windows Backup for
# Organizations restore (one user signs in + restores DURING OOBE). All three must hold:
# OOBE InProgress AND exactly one profile AND that profile created < N minutes ago.
$oobeRelax = $false
$oobeInProgress = Test-OobeInProgress
if ($profilePaths.Count -eq 1 -and $oobeInProgress) {
try {
$created = (Get-Item -LiteralPath $profilePaths[0] -Force -ErrorAction Stop).CreationTimeUtc
$ageMin = ((Get-Date).ToUniversalTime() - $created).TotalMinutes
if ($ageMin -ge 0 -and $ageMin -lt $OobeProfileMaxAgeMinutes) {
$oobeRelax = $true
Write-Log ("OOBE-relax active: OOBE InProgress + single profile created {0:N1}min ago (< {1}min). Guards 2+3 relaxed (Windows Backup restore case)." -f $ageMin, $OobeProfileMaxAgeMinutes)
} else {
Write-Log ("OOBE-relax NOT applied: OOBE InProgress + single profile '{0}', but profile created {1:N1}min ago (created {2:u}) is outside the [0, {3})min window." -f (Split-Path $profilePaths[0] -Leaf), $ageMin, $created, $OobeProfileMaxAgeMinutes)
}
} catch { Write-Log "INFO: profile age check failed; OOBE-relax not applied. $($_.Exception.Message)" }
} elseif ($profilePaths.Count -eq 1 -and -not $oobeInProgress) {
Write-Log "OOBE-relax NOT applied: single profile present but OutOfBoxExperienceState != InProgress (OOBE already finished / not detectable)."
} elseif ($profilePaths.Count -gt 1) {
Write-Log ("OOBE-relax NOT applied: {0} real profiles present (relax requires exactly one)." -f $profilePaths.Count)
}
# The @() wrap is required, see NOTE on Get-RealUserProfilePaths.
$profilePaths = @(Get-RealUserProfilePaths)
$relax = Get-RelaxDecision -ProfilePaths $profilePaths -OobeProfileMaxAgeMinutes $OobeProfileMaxAgeMinutes
# Guard 2: No real user profile should exist yet (primary productive-device guard),
# unless the OOBE-relax signature holds.
if ($profilePaths.Count -gt 0 -and -not $oobeRelax) {
# unless the relax signature holds.
if ($profilePaths.Count -gt 0 -and -not $relax.Active) {
$details = (($profilePaths | ForEach-Object {
$leaf = Split-Path $_ -Leaf
try {
@@ -195,26 +286,33 @@ try {
"{0} (created {1:u}, {2:N1}min ago)" -f $leaf, $c, ((Get-Date).ToUniversalTime() - $c).TotalMinutes
} catch { "$leaf (age unknown)" }
}) | Select-Object -First 3) -join '; '
Write-Log "SKIP: Real user profile(s) found ($details). OOBE InProgress=$oobeInProgress. Device appears productive."
exit 0
Write-Log "SKIP: Real user profile(s) found ($details). OOBE state: $($relax.OobeState). CloudPC: $($relax.IsCloudPc). Device appears productive."
return [pscustomobject]@{ Install = $false; ReasonCode = 'ProfilesFound'; RelaxActive = $false }
}
# Guard 3: LastLoggedOnUser - during Device ESP no real user has logged on yet
# (same OOBE-relax exemption: the restoring user signs in during OOBE).
# (same relax exemption: the restoring/first-connect user signs in legitimately).
$lastLoggedOnUser = (Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Authentication\LogonUI' -Name 'LastLoggedOnUser' -EA SilentlyContinue).LastLoggedOnUser
if ($lastLoggedOnUser -and $lastLoggedOnUser -notmatch 'defaultuser\d*' -and -not $oobeRelax) {
if ($lastLoggedOnUser -and $lastLoggedOnUser -notmatch 'defaultuser\d*' -and -not $relax.Active) {
Write-Log "SKIP: LastLoggedOnUser found ($lastLoggedOnUser). Device appears productive."
exit 0
return [pscustomobject]@{ Install = $false; ReasonCode = 'LastLoggedOnUser'; RelaxActive = $false }
}
# Guard 4: Bootstrap window check - device uptime must be within accepted OOBE window.
# Sleep/standby does not reset uptime, only real boot/restart does.
# Exempted while the Cloud PC relax is active: a Cloud PC runs headless from
# provisioning until the user's first connect (often days), so boot uptime says
# nothing there -- the fresh-profile window of the relax is the time anchor.
$lastBoot = (Get-CimInstance Win32_OperatingSystem).LastBootUpTime
$uptimeHours = ((Get-Date) - $lastBoot).TotalHours
Write-Log "Device uptime: $([int]$uptimeHours)h (last boot: $lastBoot)"
if ($uptimeHours -gt $MaxBootstrapWindowHours) {
Write-Log "SKIP: Device uptime is $([int]$uptimeHours)h. OOBE state is older than accepted bootstrap window of ${MaxBootstrapWindowHours}h."
exit 0
if ($relax.Active -and $relax.IsCloudPc) {
Write-Log "Uptime $([int]$uptimeHours)h exceeds the ${MaxBootstrapWindowHours}h window, but the Cloud PC relax is active (fresh first-connect profile is the time anchor). Continuing."
} else {
Write-Log "SKIP: Device uptime is $([int]$uptimeHours)h. OOBE state is older than accepted bootstrap window of ${MaxBootstrapWindowHours}h."
return [pscustomobject]@{ Install = $false; ReasonCode = 'UptimeExceeded'; RelaxActive = $relax.Active }
}
}
# Guard 5: Is the agent already installed? (leftover from previous run)
@@ -222,148 +320,175 @@ try {
$existingAgent = Get-ChildItem -Path $AgentBinPath -Filter "AutopilotMonitor.Agent.exe" -ErrorAction SilentlyContinue
if ($existingAgent) {
Write-Log "SKIP: Agent already installed at $($existingAgent.FullName)."
return [pscustomobject]@{ Install = $false; ReasonCode = 'AgentPresent'; RelaxActive = $relax.Active }
}
}
return [pscustomobject]@{ Install = $true; ReasonCode = $null; RelaxActive = $relax.Active }
}
# Main bootstrap flow. Only invoked when the script is run directly (see entry guard
# at the bottom); never on dot-source.
function Invoke-Bootstrap {
try {
New-Item -Path $AgentBasePath -ItemType Directory -Force -ErrorAction SilentlyContinue | Out-Null
New-Item -Path $AgentBinPath -ItemType Directory -Force -ErrorAction SilentlyContinue | Out-Null
New-Item -Path $AgentLogPath -ItemType Directory -Force -ErrorAction SilentlyContinue | Out-Null
Write-Log "===== Autopilot Monitor Bootstrap Started ====="
Write-Log "Bootstrap script version: v$ScriptVersion"
$decision = Get-BootstrapDecision -MaxBootstrapWindowHours $MaxBootstrapWindowHours -OobeProfileMaxAgeMinutes $OobeProfileMaxAgeMinutes -AgentBinPath $AgentBinPath
if (-not $decision.Install) {
exit 0
}
}
Write-Log "Pre-flight checks passed -- no prior deployment, no productive-device signals, within bootstrap window (oobeRelax=$oobeRelax)"
Write-Log "Pre-flight checks passed -- no prior deployment, no productive-device signals, within bootstrap window (relaxActive=$($decision.RelaxActive))"
# Download and extract agent binaries
$agentExePath = Join-Path $AgentBinPath "AutopilotMonitor.Agent.exe"
# Download and extract agent binaries
$agentExePath = Join-Path $AgentBinPath "AutopilotMonitor.Agent.exe"
if (Test-Path $agentExePath) {
Write-Log "Agent already installed at $agentExePath"
}
else {
Write-Log "Downloading agent from $AgentDownloadUrl..."
try {
# Derive manifest URL from the agent download URL (same blob container)
$versionJsonUrl = $AgentDownloadUrl -replace '[^/]+$', $VersionJsonName
$expectedSha256 = $null
$manifestAttempts = 3
$manifestDelays = @(1, 2) # short backoff between attempts; do not block enrollment
for ($attempt = 1; $attempt -le $manifestAttempts; $attempt++) {
try {
Write-Log "Fetching $VersionJsonName (attempt ${attempt}/${manifestAttempts}) from $versionJsonUrl for integrity verification..."
$versionJsonResponse = Invoke-RestMethod -Uri $versionJsonUrl -UseBasicParsing -TimeoutSec 5 -ErrorAction Stop
if ($versionJsonResponse.sha256) {
$expectedSha256 = $versionJsonResponse.sha256.ToLowerInvariant()
Write-Log "SHA-256 hash from manifest: $expectedSha256 (version: $($versionJsonResponse.version))"
} else {
Write-Log "WARNING: Manifest has no sha256 field - integrity cannot be verified."
}
break # manifest reached (with or without hash); retrying would not help
}
catch {
Write-Log "WARNING: Could not fetch integrity manifest (attempt $attempt): $($_.Exception.Message)"
if ($attempt -lt $manifestAttempts) {
Start-Sleep -Seconds $manifestDelays[$attempt - 1]
}
}
}
$zipPath = Join-Path $env:TEMP "AutopilotMonitor-Agent.zip"
$maxDownloadAttempts = 3
$downloadAttempt = 0
do {
$downloadAttempt++
try {
Write-Log "Download attempt ${downloadAttempt}/${maxDownloadAttempts}"
$downloadStopwatch = [System.Diagnostics.Stopwatch]::StartNew()
Invoke-WebRequest `
-Uri $AgentDownloadUrl `
-OutFile $zipPath `
-UseBasicParsing `
-TimeoutSec 30 `
-ErrorAction Stop
$downloadStopwatch.Stop()
$downloadSeconds = [math]::Round($downloadStopwatch.Elapsed.TotalSeconds, 1)
Write-Log "Downloaded agent to $zipPath (took ${downloadSeconds}s)"
break
}
catch {
if ($downloadAttempt -ge $maxDownloadAttempts) {
throw
}
$retryDelaysInSeconds = @(2, 4, 8)
$retryDelaySeconds = $retryDelaysInSeconds[$downloadAttempt - 1]
Write-Log "Download failed (attempt $downloadAttempt): $($_.Exception.Message). Retrying in ${retryDelaySeconds}s..."
Start-Sleep -Seconds $retryDelaySeconds
}
} while ($downloadAttempt -lt $maxDownloadAttempts)
# SHA-256 from the version manifest is the only accepted integrity proof.
# No hash -> refuse to install an unverified ZIP (SKIP-safe security posture).
if (-not $expectedSha256) {
throw "No SHA-256 hash available from manifest - refusing to install unverified agent ZIP."
}
$actualSha256 = (Get-FileHash $zipPath -Algorithm SHA256).Hash.ToLowerInvariant()
Write-Log "Validating SHA-256 hash against manifest"
if ($actualSha256 -ne $expectedSha256) {
throw "SHA-256 integrity check FAILED. Expected='$expectedSha256', Actual='$actualSha256'. Download may be tampered or corrupted."
}
Write-Log "SHA-256 integrity check passed"
Expand-Archive -Path $zipPath -DestinationPath $AgentBinPath -Force
Write-Log "Extracted agent to $AgentBinPath"
Remove-Item $zipPath -Force -ErrorAction SilentlyContinue
Write-Log "Cleaned up temporary files"
if (-not (Test-Path $agentExePath)) {
throw "Agent executable not found after extraction at $agentExePath"
}
Write-Log "Agent installation completed successfully"
if (Test-Path $agentExePath) {
Write-Log "Agent already installed at $agentExePath"
}
catch {
Write-Log "ERROR downloading/extracting agent: $($_.Exception.Message)"
throw
else {
Write-Log "Downloading agent from $AgentDownloadUrl..."
try {
# Derive manifest URL from the agent download URL (same blob container)
$versionJsonUrl = $AgentDownloadUrl -replace '[^/]+$', $VersionJsonName
$expectedSha256 = $null
$manifestAttempts = 3
$manifestDelays = @(1, 2) # short backoff between attempts; do not block enrollment
for ($attempt = 1; $attempt -le $manifestAttempts; $attempt++) {
try {
Write-Log "Fetching $VersionJsonName (attempt ${attempt}/${manifestAttempts}) from $versionJsonUrl for integrity verification..."
$versionJsonResponse = Invoke-RestMethod -Uri $versionJsonUrl -UseBasicParsing -TimeoutSec 5 -ErrorAction Stop
if ($versionJsonResponse.sha256) {
$expectedSha256 = $versionJsonResponse.sha256.ToLowerInvariant()
Write-Log "SHA-256 hash from manifest: $expectedSha256 (version: $($versionJsonResponse.version))"
} else {
Write-Log "WARNING: Manifest has no sha256 field - integrity cannot be verified."
}
break # manifest reached (with or without hash); retrying would not help
}
catch {
Write-Log "WARNING: Could not fetch integrity manifest (attempt $attempt): $($_.Exception.Message)"
if ($attempt -lt $manifestAttempts) {
Start-Sleep -Seconds $manifestDelays[$attempt - 1]
}
}
}
$zipPath = Join-Path $env:TEMP "AutopilotMonitor-Agent.zip"
$maxDownloadAttempts = 3
$downloadAttempt = 0
do {
$downloadAttempt++
try {
Write-Log "Download attempt ${downloadAttempt}/${maxDownloadAttempts}"
$downloadStopwatch = [System.Diagnostics.Stopwatch]::StartNew()
Invoke-WebRequest `
-Uri $AgentDownloadUrl `
-OutFile $zipPath `
-UseBasicParsing `
-TimeoutSec 30 `
-ErrorAction Stop
$downloadStopwatch.Stop()
$downloadSeconds = [math]::Round($downloadStopwatch.Elapsed.TotalSeconds, 1)
Write-Log "Downloaded agent to $zipPath (took ${downloadSeconds}s)"
break
}
catch {
if ($downloadAttempt -ge $maxDownloadAttempts) {
throw
}
$retryDelaysInSeconds = @(2, 4, 8)
$retryDelaySeconds = $retryDelaysInSeconds[$downloadAttempt - 1]
Write-Log "Download failed (attempt $downloadAttempt): $($_.Exception.Message). Retrying in ${retryDelaySeconds}s..."
Start-Sleep -Seconds $retryDelaySeconds
}
} while ($downloadAttempt -lt $maxDownloadAttempts)
# SHA-256 from the version manifest is the only accepted integrity proof.
# No hash -> refuse to install an unverified ZIP (SKIP-safe security posture).
if (-not $expectedSha256) {
throw "No SHA-256 hash available from manifest - refusing to install unverified agent ZIP."
}
$actualSha256 = (Get-FileHash $zipPath -Algorithm SHA256).Hash.ToLowerInvariant()
Write-Log "Validating SHA-256 hash against manifest"
if ($actualSha256 -ne $expectedSha256) {
throw "SHA-256 integrity check FAILED. Expected='$expectedSha256', Actual='$actualSha256'. Download may be tampered or corrupted."
}
Write-Log "SHA-256 integrity check passed"
Expand-Archive -Path $zipPath -DestinationPath $AgentBinPath -Force
Write-Log "Extracted agent to $AgentBinPath"
Remove-Item $zipPath -Force -ErrorAction SilentlyContinue
Write-Log "Cleaned up temporary files"
if (-not (Test-Path $agentExePath)) {
throw "Agent executable not found after extraction at $agentExePath"
}
Write-Log "Agent installation completed successfully"
}
catch {
Write-Log "ERROR downloading/extracting agent: $($_.Exception.Message)"
throw
}
}
}
Write-Log "Calling agent install mode (--install)..."
& $agentExePath --install
$installExitCode = $LASTEXITCODE
if ($installExitCode -ne 0) {
throw "Agent install failed with exit code $installExitCode"
}
Write-Log "Agent install mode completed successfully"
Write-Log "Calling agent install mode (--install)..."
& $agentExePath --install
$installExitCode = $LASTEXITCODE
if ($installExitCode -ne 0) {
throw "Agent install failed with exit code $installExitCode"
}
Write-Log "Agent install mode completed successfully"
$runtimeProcessName = 'AutopilotMonitor.Agent'
$verifyTimeoutSec = 10
$verifyDeadline = (Get-Date).AddSeconds($verifyTimeoutSec)
$runtimeProc = $null
while ((Get-Date) -lt $verifyDeadline) {
$runtimeProc = Get-Process -Name $runtimeProcessName -ErrorAction SilentlyContinue | Select-Object -First 1
if ($null -ne $runtimeProc) { break }
Start-Sleep -Milliseconds 500
}
if ($null -ne $runtimeProc) {
$startedUtc = 'unavailable'
try { $startedUtc = $runtimeProc.StartTime.ToUniversalTime().ToString('o') } catch { }
Write-Log ("Runtime process verified: name={0}.exe pid={1} startedUtc={2}" -f $runtimeProcessName, $runtimeProc.Id, $startedUtc)
} else {
Write-Log ("WARNING: Runtime process verification FAILED. Agent --install reported success but no '{0}.exe' process appeared within {1}s. Likely silent block (AV/EDR, AppLocker/WDAC) of the runtime launch. Agent should still come up at next boot via the BootTrigger task. Check Event Viewer > Microsoft > Windows > TaskScheduler/Operational and AV/EDR logs for '{0}.exe'." -f $runtimeProcessName, $verifyTimeoutSec)
}
$runtimeProcessName = 'AutopilotMonitor.Agent'
$verifyTimeoutSec = 10
$verifyDeadline = (Get-Date).AddSeconds($verifyTimeoutSec)
$runtimeProc = $null
while ((Get-Date) -lt $verifyDeadline) {
$runtimeProc = Get-Process -Name $runtimeProcessName -ErrorAction SilentlyContinue | Select-Object -First 1
if ($null -ne $runtimeProc) { break }
Start-Sleep -Milliseconds 500
}
if ($null -ne $runtimeProc) {
$startedUtc = 'unavailable'
try { $startedUtc = $runtimeProc.StartTime.ToUniversalTime().ToString('o') } catch { }
Write-Log ("Runtime process verified: name={0}.exe pid={1} startedUtc={2}" -f $runtimeProcessName, $runtimeProc.Id, $startedUtc)
} else {
Write-Log ("WARNING: Runtime process verification FAILED. Agent --install reported success but no '{0}.exe' process appeared within {1}s. Likely silent block (AV/EDR, AppLocker/WDAC) of the runtime launch. Agent should still come up at next boot via the BootTrigger task. Check Event Viewer > Microsoft > Windows > TaskScheduler/Operational and AV/EDR logs for '{0}.exe'." -f $runtimeProcessName, $verifyTimeoutSec)
}
Write-Log "===== Bootstrap Completed Successfully ====="
Write-Log "===== Bootstrap Completed Successfully ====="
exit 0
exit 0
}
catch {
Write-Log "===== Bootstrap FAILED ====="
Write-Log "ERROR: $($_.Exception.Message)"
Write-Log "Stack trace: $($_.ScriptStackTrace)"
Write-Log "Please check log file: $LogFile"
$errMsg = "AutopilotMonitor bootstrap failed: $($_.Exception.Message)"
if ($errMsg.Length -gt 2048) { $errMsg = $errMsg.Substring(0, 1045) + '...' }
[Console]::Error.WriteLine($errMsg)
exit 1
}
}
catch {
Write-Log "===== Bootstrap FAILED ====="
Write-Log "ERROR: $($_.Exception.Message)"
Write-Log "Stack trace: $($_.ScriptStackTrace)"
Write-Log "Please check log file: $LogFile"
$errMsg = "AutopilotMonitor bootstrap failed: $($_.Exception.Message)"
if ($errMsg.Length -gt 2048) { $errMsg = $errMsg.Substring(0, 1045) + '...' }
[Console]::Error.WriteLine($errMsg)
exit 1
# Entry guard: execute only when run as a script (IME, console). Dot-sourcing
# (". .\Install-AutopilotMonitor.ps1", as the Pester suite does) only loads
# the functions above and MUST stay side-effect free.
if ($MyInvocation.InvocationName -ne '.') {
Invoke-Bootstrap
}
+32 -12
View File
@@ -1,7 +1,9 @@
# Autopilot Monitor Bootstrap Guard Test
# Dry-run only: does not exit, does not install, only reports what WOULD happen.
# Mirrors Install-AutopilotMonitor.ps1 incl. the OOBE-relax exemption for the
# "Windows Backup for Organizations" restore case (guards 2+3).
# Mirrors Install-AutopilotMonitor.ps1 (v2.3) incl. the relax exemption for
# guards 2+3, triggered by "Windows Backup for Organizations" (OOBE InProgress)
# or a positively identified Windows 365 Cloud PC, and the guard 4 (uptime)
# exemption while the Cloud PC relax is active.
param(
[string]$AgentBinPath = 'C:\ProgramData\AutopilotMonitor\Agent',
@@ -54,6 +56,12 @@ function Test-OobeInProgress {
}
}
# Positive Windows 365 Cloud PC detection: two local markers, BOTH required (AND),
# because each alone has look-alikes. Mirrors Test-IsCloudPc in the bootstrapper.
$cloudPcRegMarker = Test-Path 'HKLM:\SOFTWARE\Microsoft\Windows365'
$cloudPcSvcMarker = $null -ne (Get-Service -Name 'CloudManagedDesktopExtension' -ErrorAction SilentlyContinue)
$isCloudPc = ($cloudPcRegMarker -and $cloudPcSvcMarker)
$AgentExePath = Join-Path $AgentBinPath 'AutopilotMonitor.Agent.exe'
$MaxBootstrapWindowHours = 12
@@ -104,8 +112,9 @@ if ($wmiProfileQueryFailed) {
$profileNames = $profilePaths | ForEach-Object { Split-Path $_ -Leaf }
# OOBE-relax: the ONLY exemption to guards 2+3. Matches exactly the Windows Backup for
# Organizations restore: OOBE InProgress AND exactly one profile AND that profile fresh.
# Relax: the ONLY exemption to guards 2+3. Two triggers -- OOBE InProgress (Windows
# Backup for Organizations restore) or Cloud PC identity (headless provisioning,
# profile at first connect). Either additionally requires exactly one fresh profile.
$oobeInProgress = Test-OobeInProgress
$profileAgeMin = $null
$oobeRelax = $false
@@ -114,12 +123,13 @@ if ($profilePaths.Count -eq 1) {
$created = (Get-Item -LiteralPath $profilePaths[0] -Force -ErrorAction Stop).CreationTimeUtc
$profileAgeMin = [math]::Round(((Get-Date).ToUniversalTime() - $created).TotalMinutes, 1)
} catch { }
if ($oobeInProgress -and $null -ne $profileAgeMin -and $profileAgeMin -ge 0 -and $profileAgeMin -lt $OobeProfileMaxAgeMinutes) {
if (($oobeInProgress -or $isCloudPc) -and $null -ne $profileAgeMin -and $profileAgeMin -ge 0 -and $profileAgeMin -lt $OobeProfileMaxAgeMinutes) {
$oobeRelax = $true
}
}
Write-Step -Status 'INFO' -Message "OOBE: OutOfBoxExperienceState InProgress = $oobeInProgress ; profileCount = $($profilePaths.Count) ; singleProfileAgeMin = $profileAgeMin ; OOBE-relax = $oobeRelax"
Write-Step -Status 'INFO' -Message "Cloud PC markers: Windows365 registry key = $cloudPcRegMarker ; CloudManagedDesktopExtension service = $cloudPcSvcMarker ; IsCloudPc = $isCloudPc"
Write-Step -Status 'INFO' -Message "OOBE: OutOfBoxExperienceState InProgress = $oobeInProgress ; profileCount = $($profilePaths.Count) ; singleProfileAgeMin = $profileAgeMin ; relax = $oobeRelax"
# Guard 2: No real user profile should exist yet (unless OOBE-relax holds)
if ($profilePaths.Count -gt 0 -and -not $oobeRelax) {
@@ -129,7 +139,8 @@ if ($profilePaths.Count -gt 0 -and -not $oobeRelax) {
$reasons.Add("Real user profile(s) found: $names")
} elseif ($profilePaths.Count -gt 0 -and $oobeRelax) {
$names = ($profileNames | Select-Object -First 3) -join ', '
Write-Step -Status 'PASS' -Message "Guard 2: Profile ($names) present but OOBE-relax active (single fresh profile in OOBE) -- not blocking."
$trigger = if ($oobeInProgress) { 'OOBE InProgress' } else { 'Cloud PC' }
Write-Step -Status 'PASS' -Message "Guard 2: Profile ($names) present but relax active ($trigger + single fresh profile) -- not blocking."
} else {
Write-Step -Status 'PASS' -Message 'Guard 2: No real user profiles found (combined WMI/filesystem view).'
}
@@ -141,7 +152,7 @@ if ($lastLoggedOnUser -and $lastLoggedOnUser -notmatch 'defaultuser\d*' -and -no
$wouldInstall = $false
$reasons.Add("LastLoggedOnUser found: $lastLoggedOnUser")
} elseif ($lastLoggedOnUser -and $lastLoggedOnUser -notmatch 'defaultuser\d*' -and $oobeRelax) {
Write-Step -Status 'PASS' -Message "Guard 3: LastLoggedOnUser ($lastLoggedOnUser) present but OOBE-relax active -- not blocking."
Write-Step -Status 'PASS' -Message "Guard 3: LastLoggedOnUser ($lastLoggedOnUser) present but relax active -- not blocking."
} else {
Write-Step -Status 'PASS' -Message "Guard 3: No real LastLoggedOnUser (value: '$lastLoggedOnUser')."
}
@@ -155,9 +166,15 @@ try {
$uptimeHours = ((Get-Date) - $lastBoot).TotalHours
if ($uptimeHours -gt $MaxBootstrapWindowHours) {
Write-Step -Status 'SKIP' -Message "Guard 4: Device uptime is $([int]$uptimeHours)h. Older than accepted bootstrap window of ${MaxBootstrapWindowHours}h."
$wouldInstall = $false
$reasons.Add("Uptime exceeds bootstrap window: $([int]$uptimeHours)h > ${MaxBootstrapWindowHours}h")
if ($oobeRelax -and $isCloudPc) {
# Cloud PCs run headless for days before the user's first connect; the
# fresh-profile window is the time anchor there, not boot uptime.
Write-Step -Status 'PASS' -Message "Guard 4: Device uptime is $([int]$uptimeHours)h (over ${MaxBootstrapWindowHours}h), but the Cloud PC relax is active -- not blocking."
} else {
Write-Step -Status 'SKIP' -Message "Guard 4: Device uptime is $([int]$uptimeHours)h. Older than accepted bootstrap window of ${MaxBootstrapWindowHours}h."
$wouldInstall = $false
$reasons.Add("Uptime exceeds bootstrap window: $([int]$uptimeHours)h > ${MaxBootstrapWindowHours}h")
}
} else {
Write-Step -Status 'PASS' -Message "Guard 4: Device uptime is $([int]$uptimeHours)h and within bootstrap window of ${MaxBootstrapWindowHours}h."
}
@@ -202,9 +219,12 @@ Write-Host "Deployed marker : $deployed"
Write-Host "DetectedProfileNames : $($profileNames -join ', ')"
Write-Host "ProfileCount : $($profilePaths.Count)"
Write-Host "OobeInProgress (WinRT) : $oobeInProgress"
Write-Host "CloudPcRegistryMarker : $cloudPcRegMarker"
Write-Host "CloudPcServiceMarker : $cloudPcSvcMarker"
Write-Host "IsCloudPc : $isCloudPc"
Write-Host "SingleProfileAgeMinutes : $profileAgeMin"
Write-Host "OobeProfileMaxAgeMinutes : $OobeProfileMaxAgeMinutes"
Write-Host "OobeRelaxActive : $oobeRelax"
Write-Host "RelaxActive : $oobeRelax"
Write-Host "LastLoggedOnUser : $lastLoggedOnUser"
Write-Host "MaxBootstrapWindowHours : $MaxBootstrapWindowHours"
Write-Host "LastBootUpTime : $lastBoot"
@@ -109,6 +109,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
/// </summary>
public bool EnableTimezoneAutoSet { get; set; } = false;
/// <summary>
/// Continue-Anyway observation mode (operator-set tenant opt-in, delivered via
/// remote config): stamped into the esp_config_detected decision-signal payload so
/// the DecisionEngine defangs a Device-phase ESP terminal failure into a bounded
/// 60-min observation instead of failing immediately. Default: false.
/// </summary>
public bool EnableEspContinueAnywayObservation { get; set; } = false;
/// <summary>
/// Optional custom path to IME logs directory for testing.
/// If set, overrides the default %ProgramData%\Microsoft\IntuneManagementExtension\Logs path.
@@ -103,6 +103,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
agentConfig.SendTraceEvents = remoteConfig.SendTraceEvents;
// ---------------- Continue-Anyway observation — remote wins (operator-set tenant
// opt-in; false must clear a stale cached true).
agentConfig.EnableEspContinueAnywayObservation = remoteConfig.EnableEspContinueAnywayObservation;
// ---------------- UnrestrictedMode audit — track old/new for the caller so the
// agent_unrestricted_mode_changed event can fire AFTER the
// orchestrator emitter is alive (V1 parity with
@@ -328,6 +328,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
EnableGeoLocation = true,
EnableImeMatchLog = false,
EnableGatherRuleDebugLog = false,
EnableEspContinueAnywayObservation = false,
Collectors = CollectorConfiguration.CreateDefault(),
GatherRules = new System.Collections.Generic.List<GatherRule>(),
ImeLogPatterns = new System.Collections.Generic.List<ImeLogPattern>()
@@ -1021,6 +1021,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo
.ToString(System.Globalization.CultureInfo.InvariantCulture);
if (snapshot.AllowContinueAnyway.HasValue)
payload[SignalPayloadKeys.EspAllowContinueAnyway] = snapshot.AllowContinueAnyway.Value ? "true" : "false";
// Tenant opt-in (remote config, not a registry fact) — only stamped when
// enabled; absent = default hard-fail semantics in the engine.
if (_espContinueAnywayObservationEnabled?.Invoke() == true)
payload[SignalPayloadKeys.EspContinueAnywayObservationEnabled] = "true";
var derivationInputs = new Dictionary<string, string>(payload, StringComparer.Ordinal)
{
@@ -35,6 +35,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo
private readonly IClock _clock;
private readonly Persistence.StartupEventGate _startupGate;
// Live accessor for AgentConfiguration.EnableEspContinueAnywayObservation — read at
// esp_config_detected stamping time so a late remote-config merge takes effect on
// the phase-triggered re-emissions. Null (tests / legacy callers) means disabled.
private readonly Func<bool> _espContinueAnywayObservationEnabled;
private const string RegKeyWindowsCurrentVersion = @"HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion";
// Restart dedup (StartupEventGate) — exemptions. boot_time is genuinely new per boot
@@ -65,7 +70,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo
AgentLogger logger,
ISignalIngressSink signalIngress = null,
IClock clock = null,
Persistence.StartupEventGate startupGate = null)
Persistence.StartupEventGate startupGate = null,
Func<bool> espContinueAnywayObservationEnabled = null)
{
_sessionId = sessionId ?? throw new ArgumentNullException(nameof(sessionId));
_tenantId = tenantId ?? throw new ArgumentNullException(nameof(tenantId));
@@ -74,6 +80,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo
_signalIngress = signalIngress;
_clock = clock;
_startupGate = startupGate;
_espContinueAnywayObservationEnabled = espContinueAnywayObservationEnabled;
}
/// <summary>
@@ -127,7 +127,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
ingress: ingress,
clock: clock,
logger: logger,
startupGate: _startupEventGate));
startupGate: _startupEventGate,
// Live accessor (not the current value): a remote-config merge landing after
// factory construction must still reach the phase-triggered re-emissions.
espContinueAnywayObservationEnabled: () => _agentConfig.EnableEspContinueAnywayObservation));
// Session caa6cf50 gate-starvation fix: the EspAndHelloTracker's user-ESP-apps-settled
// probe reads the IME log host, which is constructed further down. Same closure-over-
@@ -91,6 +91,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
// (tests / legacy callers) means trace events always flow.
private readonly Func<bool>? _traceEventsEnabled;
// Live accessor for AgentConfiguration.EnableEspContinueAnywayObservation — read at
// esp_config_detected stamping time so a remote-config merge that lands after
// construction still takes effect. Null (tests / legacy callers) means disabled.
private readonly Func<bool>? _espContinueAnywayObservationEnabled;
// Max-lifetime watchdog (M4.6.α). Timer is armed in Start() when _agentMaxLifetime != null.
private System.Threading.Timer? _maxLifetimeTimer;
private int _terminatedFired;
@@ -173,7 +178,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
TimeSpan? agentMaxLifetime = null,
int uploadBatchSize = 100,
IDeadlineScheduler? schedulerOverride = null,
Func<bool>? traceEventsEnabled = null)
Func<bool>? traceEventsEnabled = null,
Func<bool>? espContinueAnywayObservationEnabled = null)
{
if (string.IsNullOrEmpty(sessionId)) throw new ArgumentException("SessionId is mandatory.", nameof(sessionId));
if (string.IsNullOrEmpty(tenantId)) throw new ArgumentException("TenantId is mandatory.", nameof(tenantId));
@@ -214,6 +220,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
_schedulerOverride = schedulerOverride;
_traceEventsEnabled = traceEventsEnabled;
_espContinueAnywayObservationEnabled = espContinueAnywayObservationEnabled;
}
/// <summary>
@@ -773,6 +780,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
.ToString(System.Globalization.CultureInfo.InvariantCulture);
if (snapshot.AllowContinueAnyway.HasValue)
payload[SignalPayloadKeys.EspAllowContinueAnyway] = snapshot.AllowContinueAnyway.Value ? "true" : "false";
// Tenant opt-in (remote config, not a registry fact) — only stamped when
// enabled; absent = default hard-fail semantics in the engine.
if (_espContinueAnywayObservationEnabled?.Invoke() == true)
payload[SignalPayloadKeys.EspContinueAnywayObservationEnabled] = "true";
_ingress!.Post(
kind: DecisionSignalKind.EspConfigDetected,
@@ -82,7 +82,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
ISignalIngressSink ingress,
IClock clock,
AgentLogger logger,
Persistence.StartupEventGate? startupGate = null)
Persistence.StartupEventGate? startupGate = null,
Func<bool>? espContinueAnywayObservationEnabled = null)
{
if (ingress == null) throw new ArgumentNullException(nameof(ingress));
if (clock == null) throw new ArgumentNullException(nameof(clock));
@@ -92,7 +93,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
// when it reads the FirstSync SkipUser/SkipDevice registry values, so that Fix 8's
// reducer guards have the SkipUserEsp/SkipDeviceEsp state facts to read.
_collector = new Monitoring.Telemetry.DeviceInfo.DeviceInfoCollector(
sessionId, tenantId, post, logger, ingress, clock, startupGate);
sessionId, tenantId, post, logger, ingress, clock, startupGate,
espContinueAnywayObservationEnabled);
_observableIngress = ingress as SignalIngress;
}
@@ -135,7 +135,8 @@ namespace AutopilotMonitor.Agent.V2.Runtime
uploadBatchSize: uploadBatchSize,
// Live read: remote config is merged into agentConfig mid-session, so the
// trace-event gate must follow it rather than snapshot at construction.
traceEventsEnabled: () => agentConfig.SendTraceEvents))
traceEventsEnabled: () => agentConfig.SendTraceEvents,
espContinueAnywayObservationEnabled: () => agentConfig.EnableEspContinueAnywayObservation))
{
using (var shutdown = new ManualResetEventSlim(false))
using (var shutdownComplete = new ManualResetEventSlim(false))
@@ -0,0 +1,383 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.DecisionCore.State;
using Xunit;
namespace AutopilotMonitor.DecisionCore.Tests
{
/// <summary>
/// Continue-Anyway observation mode (tenant c9787ba2, session 53d1e9f6, 2026-08-08).
/// A Device-phase ESP terminal failure (AccountSetup never entered) on a
/// Continue-Anyway-enabled profile is a hard fail by default; with the tenant opt-in
/// (<c>EspContinueAnywayObservationEnabled</c> observation) it is defanged into a 60-min
/// observation advisory instead. Coverage:
/// <list type="bullet">
/// <item>Defang + 60-min AdvisoryCompletion arming (vs the classic 30-min window).</item>
/// <item>Opt-out / ContinueAnyway-off / self-deploying-scenario regressions still hard-fail.</item>
/// <item>Real-user desktop + Hello-disabled completes eagerly through Finalizing; the
/// terminal enrollment_complete carries the espSoftFailure marker.</item>
/// <item>Window expiry without desktop un-defangs to esp_terminal_failure.</item>
/// <item>Deadline fire with desktop but unknown Hello promotes to AwaitingHello.</item>
/// <item>Category recovery ("Try again") resolves the advisory (classic 4910a5a5 hook).</item>
/// </list>
/// </summary>
public sealed class ContinueAnywayObservationTests
{
private static readonly DateTime T0 = new DateTime(2026, 8, 7, 15, 16, 0, DateTimeKind.Utc);
private static DecisionSignal MakeSignal(
long ordinal,
DecisionSignalKind kind,
DateTime occurredAtUtc,
IReadOnlyDictionary<string, string>? payload = null,
string sourceOrigin = "test")
{
return new DecisionSignal(
sessionSignalOrdinal: ordinal,
sessionTraceOrdinal: ordinal,
kind: kind,
kindSchemaVersion: 1,
occurredAtUtc: occurredAtUtc,
sourceOrigin: sourceOrigin,
evidence: new Evidence(EvidenceKind.Synthetic, $"{kind}-{ordinal}", "test"),
payload: payload);
}
private static DecisionSignal DeadlineFired(long ordinal, DateTime occurredAtUtc, string deadlineName) =>
MakeSignal(ordinal, DecisionSignalKind.DeadlineFired, occurredAtUtc,
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = deadlineName });
private static DecisionEffect SingleTimelineEffect(DecisionStep step, string eventType) =>
step.Effects.Single(e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry
&& e.Parameters != null
&& e.Parameters.TryGetValue("eventType", out var et)
&& et == eventType);
private static ActiveDeadline? FindDeadline(DecisionState state, string name) =>
state.Deadlines.FirstOrDefault(d => d.Name == name);
/// <summary>
/// Replays the session-53d1e9f6 shape up to (but excluding) the ESP terminal failure:
/// Continue-Anyway profile with the tenant observation opt-in stamped into the
/// EspConfigDetected payload, Hello policy disabled, DeviceSetup entered —
/// AccountSetup never reached (the 30-min Device-ESP wall kills it first).
/// </summary>
private static DecisionState SetupObservationEligibleSession(
DecisionEngine engine,
bool observationOptIn = true,
bool continueAnyway = true,
bool helloPolicyDisabled = true)
{
var state = DecisionState.CreateInitial("sess-53d1e9f6", "tenant-c9787ba2", T0);
state = engine.Reduce(state, MakeSignal(0, DecisionSignalKind.SessionStarted, T0)).NewState;
var espConfig = new Dictionary<string, string>
{
[SignalPayloadKeys.SkipUserEsp] = "false",
[SignalPayloadKeys.SkipDeviceEsp] = "false",
[SignalPayloadKeys.EspSyncFailureTimeoutMinutes] = "30",
[SignalPayloadKeys.EspAllowContinueAnyway] = continueAnyway ? "true" : "false",
};
if (observationOptIn)
{
espConfig[SignalPayloadKeys.EspContinueAnywayObservationEnabled] = "true";
}
state = engine.Reduce(state, MakeSignal(
5, DecisionSignalKind.EspConfigDetected, T0.AddMinutes(1), espConfig)).NewState;
if (helloPolicyDisabled)
{
state = engine.Reduce(state, MakeSignal(
8, DecisionSignalKind.HelloPolicyDetected, T0.AddMinutes(1),
new Dictionary<string, string> { [SignalPayloadKeys.HelloEnabled] = "false" })).NewState;
}
state = engine.Reduce(state, MakeSignal(
10, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(2),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "DeviceSetup" })).NewState;
return state;
}
private static DecisionStep ApplyDeviceSetupTerminalFailure(DecisionEngine engine, DecisionState state, long ordinal = 50) =>
engine.Reduce(state, MakeSignal(
ordinal, DecisionSignalKind.EspTerminalFailure, T0.AddMinutes(38),
new Dictionary<string, string>
{
["failureType"] = "Provisioning_DeviceSetup_Certificates_Failed",
["failedSubcategory"] = "Certificates",
["category"] = "DeviceSetup",
}));
// ============================================================ defang path ====
[Fact]
public void DeviceSetupFailure_WithOptIn_DefangsToObservationAdvisory()
{
var engine = new DecisionEngine();
var state = SetupObservationEligibleSession(engine);
Assert.Null(state.AccountSetupEnteredUtc);
var preStage = state.Stage;
var step = ApplyDeviceSetupTerminalFailure(engine, state);
// No terminal — stage unchanged, advisory anchored, no LastFailureTrigger.
Assert.Equal(preStage, step.NewState.Stage);
Assert.Null(step.NewState.Outcome);
Assert.NotNull(step.NewState.EspAdvisoryFailureRecordedUtc);
Assert.Equal("DeviceSetup", step.NewState.EspAdvisoryFailureCategory!.Value);
Assert.Null(step.NewState.LastFailureTrigger);
var advisory = SingleTimelineEffect(step, "esp_failure_advisory");
Assert.Equal("esp_failure_defanged_continueanyway_observation", advisory.Parameters!["advisoryReason"]);
Assert.Equal("Warning", advisory.Parameters!["severity"]);
Assert.Equal("true", advisory.Parameters!["mayHaveContinuedAnyway"]);
Assert.DoesNotContain(step.Effects, e =>
e.Parameters != null
&& e.Parameters.TryGetValue("eventType", out var et)
&& et == "enrollment_failed");
}
[Fact]
public void DeviceSetupFailure_WithOptIn_Arms60MinObservationWindow()
{
var engine = new DecisionEngine();
var state = SetupObservationEligibleSession(engine);
var step = ApplyDeviceSetupTerminalFailure(engine, state);
var deadline = FindDeadline(step.NewState, DeadlineNames.AdvisoryCompletion);
Assert.NotNull(deadline);
// 60-min observation window (user decision 2026-08-08) — NOT the classic 30-min
// advisory-completion window.
Assert.Equal(T0.AddMinutes(38).AddMinutes(60), deadline!.DueAtUtc);
}
// ==================================================== negative / regression ====
[Fact]
public void DeviceSetupFailure_WithoutOptIn_StillHardFails()
{
var engine = new DecisionEngine();
var state = SetupObservationEligibleSession(engine, observationOptIn: false);
var step = ApplyDeviceSetupTerminalFailure(engine, state);
Assert.Equal(SessionStage.Failed, step.NewState.Stage);
Assert.Equal(SessionOutcome.EnrollmentFailed, step.NewState.Outcome);
Assert.Null(step.NewState.EspAdvisoryFailureRecordedUtc);
SingleTimelineEffect(step, "enrollment_failed");
}
[Fact]
public void DeviceSetupFailure_WithOptIn_ButContinueAnywayDisabled_StillHardFails()
{
// The observation only makes sense when the ESP profile actually offers the
// "Continue anyway" button — without it the user cannot reach the desktop.
var engine = new DecisionEngine();
var state = SetupObservationEligibleSession(engine, continueAnyway: false);
var step = ApplyDeviceSetupTerminalFailure(engine, state);
Assert.Equal(SessionStage.Failed, step.NewState.Stage);
Assert.Null(step.NewState.EspAdvisoryFailureRecordedUtc);
SingleTimelineEffect(step, "enrollment_failed");
}
[Fact]
public void DeviceSetupFailure_WithOptIn_OnSelfDeployingProfile_StillHardFails()
{
// Scope gate: self-deploying/kiosk flows have no interactive user who could press
// "Continue anyway" — observation would only ever expire. Immediate fail is truthful.
var engine = new DecisionEngine();
var state = SetupObservationEligibleSession(engine);
state = engine.Reduce(state, MakeSignal(
20, DecisionSignalKind.EnrollmentFactsObserved, T0.AddMinutes(3),
new Dictionary<string, string> { [SignalPayloadKeys.IsSelfDeployingProfile] = "true" })).NewState;
var step = ApplyDeviceSetupTerminalFailure(engine, state);
Assert.Equal(SessionStage.Failed, step.NewState.Stage);
Assert.Null(step.NewState.EspAdvisoryFailureRecordedUtc);
}
[Fact]
public void ClassicAdvisory_WithAccountSetupEntered_KeepsClassicReasonAndWindow()
{
// Regression: the opt-in must not change the classic post-AccountSetup advisory —
// same advisoryReason literal, same 30-min window.
var engine = new DecisionEngine();
var state = SetupObservationEligibleSession(engine);
state = engine.Reduce(state, MakeSignal(
20, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(17),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState;
Assert.NotNull(state.AccountSetupEnteredUtc);
var step = ApplyDeviceSetupTerminalFailure(engine, state);
var advisory = SingleTimelineEffect(step, "esp_failure_advisory");
Assert.Equal("esp_failure_defanged_continueanyway_with_accountsetup", advisory.Parameters!["advisoryReason"]);
var deadline = FindDeadline(step.NewState, DeadlineNames.AdvisoryCompletion);
Assert.Equal(T0.AddMinutes(38).AddMinutes(30), deadline!.DueAtUtc);
}
// ========================================== real end: desktop completion ====
[Fact]
public void ObservationAdvisory_ThenRealUserDesktop_CompletesThroughFinalizing()
{
var engine = new DecisionEngine();
var state = SetupObservationEligibleSession(engine);
state = ApplyDeviceSetupTerminalFailure(engine, state).NewState;
// User pressed "Continue anyway" → signed in → DAD-validated real-user desktop.
var step = engine.Reduce(state, MakeSignal(
60, DecisionSignalKind.DesktopArrived, T0.AddMinutes(42)));
Assert.Equal(SessionStage.Finalizing, step.NewState.Stage);
Assert.Contains(":ContinueAnywayObservation", step.Transition.Trigger);
// Hello disabled → synthetic skip recorded by the fast-path.
Assert.NotNull(step.NewState.HelloResolvedUtc);
Assert.NotNull(FindDeadline(step.NewState, DeadlineNames.FinalizingGrace));
}
[Fact]
public void ObservationCompletion_EnrollmentComplete_CarriesEspSoftFailureMarker()
{
var engine = new DecisionEngine();
var state = SetupObservationEligibleSession(engine);
state = ApplyDeviceSetupTerminalFailure(engine, state).NewState;
state = engine.Reduce(state, MakeSignal(
60, DecisionSignalKind.DesktopArrived, T0.AddMinutes(42))).NewState;
Assert.Equal(SessionStage.Finalizing, state.Stage);
var step = engine.Reduce(state, DeadlineFired(70, T0.AddMinutes(42.1), DeadlineNames.FinalizingGrace));
Assert.Equal(SessionStage.Completed, step.NewState.Stage);
Assert.Equal(SessionOutcome.EnrollmentComplete, step.NewState.Outcome);
var complete = SingleTimelineEffect(step, "enrollment_complete");
var payload = Assert.IsType<Dictionary<string, object>>(complete.TypedPayload);
Assert.Equal("true", payload["espSoftFailure"]);
Assert.Equal("continue_anyway_observation", payload["completionSource"]);
Assert.Equal("DeviceSetup", payload["espSoftFailureCategory"]);
}
[Fact]
public void CleanCompletion_EnrollmentComplete_OmitsEspSoftFailureMarker()
{
// A session without any advisory must not carry the marker — regression guard for
// the BuildEnrollmentCompleteEffect change.
var engine = new DecisionEngine();
var state = SetupObservationEligibleSession(engine);
state = engine.Reduce(state, MakeSignal(
20, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(17),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState;
state = engine.Reduce(state, MakeSignal(
30, DecisionSignalKind.AccountSetupProvisioningComplete, T0.AddMinutes(20))).NewState;
state = engine.Reduce(state, MakeSignal(
40, DecisionSignalKind.DesktopArrived, T0.AddMinutes(21))).NewState;
Assert.Equal(SessionStage.Finalizing, state.Stage);
var step = engine.Reduce(state, DeadlineFired(50, T0.AddMinutes(21.1), DeadlineNames.FinalizingGrace));
Assert.Equal(SessionStage.Completed, step.NewState.Stage);
var complete = SingleTimelineEffect(step, "enrollment_complete");
var payload = Assert.IsType<Dictionary<string, object>>(complete.TypedPayload);
Assert.False(payload.ContainsKey("espSoftFailure"));
Assert.False(payload.ContainsKey("completionSource"));
}
// =============================================== window expiry / deadline ====
[Fact]
public void ObservationWindow_ExpiresWithoutDesktop_UnDefangsToEspTerminalFailure()
{
var engine = new DecisionEngine();
var state = SetupObservationEligibleSession(engine);
state = ApplyDeviceSetupTerminalFailure(engine, state).NewState;
Assert.Null(state.DesktopArrivedUtc);
var step = engine.Reduce(state, DeadlineFired(60, T0.AddMinutes(98), DeadlineNames.AdvisoryCompletion));
Assert.Equal(SessionStage.Failed, step.NewState.Stage);
Assert.Equal(SessionOutcome.EnrollmentFailed, step.NewState.Outcome);
// The failure cause IS the original ESP failure — same reason literal, likely-stuck
// promotion stays armed via LastFailureTrigger=EspTerminalFailure.
Assert.Equal(nameof(DecisionSignalKind.EspTerminalFailure), step.NewState.LastFailureTrigger!.Value);
var failed = SingleTimelineEffect(step, "enrollment_failed");
Assert.Equal("esp_terminal_failure", failed.Parameters!["reason"]);
Assert.Equal(
"continue_anyway_observation_window_expired_without_completion_evidence",
failed.Parameters!["advisoryReason"]);
}
[Fact]
public void ObservationWindow_FiresWithDesktopAndDisabledHello_CompletesInsteadOfFailing()
{
// Ordering variant: desktop arrived but (hypothetically) no eager completion ran —
// the deadline-fire conjunction must accept desktop + hello for the observation
// variant without the (structurally impossible) IME user-session gate.
var engine = new DecisionEngine();
var state = SetupObservationEligibleSession(engine, helloPolicyDisabled: false);
state = ApplyDeviceSetupTerminalFailure(engine, state).NewState;
// Hello policy lands AFTER the failure; desktop arrives; no eager completion since
// the fast-path needs the policy fact at desktop-arrival time.
state = engine.Reduce(state, MakeSignal(
60, DecisionSignalKind.DesktopArrived, T0.AddMinutes(42))).NewState;
Assert.NotEqual(SessionStage.Finalizing, state.Stage);
state = engine.Reduce(state, MakeSignal(
65, DecisionSignalKind.HelloPolicyDetected, T0.AddMinutes(43),
new Dictionary<string, string> { [SignalPayloadKeys.HelloEnabled] = "false" })).NewState;
var step = engine.Reduce(state, DeadlineFired(70, T0.AddMinutes(98), DeadlineNames.AdvisoryCompletion));
Assert.Equal(SessionStage.Finalizing, step.NewState.Stage);
Assert.Null(step.NewState.Outcome);
Assert.NotNull(FindDeadline(step.NewState, DeadlineNames.FinalizingGrace));
}
[Fact]
public void ObservationWindow_FiresWithDesktopButUnknownHello_PromotesToAwaitingHello()
{
// Hello policy never observed (no PassportForWork value) — the hello prerequisite is
// structurally unresolvable. The never-observed promote must apply to the observation
// variant instead of failing a desktop-proven session.
var engine = new DecisionEngine();
var state = SetupObservationEligibleSession(engine, helloPolicyDisabled: false);
state = ApplyDeviceSetupTerminalFailure(engine, state).NewState;
state = engine.Reduce(state, MakeSignal(
60, DecisionSignalKind.DesktopArrived, T0.AddMinutes(42))).NewState;
Assert.Null(state.HelloPolicyEnabled);
var step = engine.Reduce(state, DeadlineFired(70, T0.AddMinutes(98), DeadlineNames.AdvisoryCompletion));
Assert.Equal(SessionStage.AwaitingHello, step.NewState.Stage);
Assert.Null(step.NewState.Outcome);
Assert.NotNull(FindDeadline(step.NewState, DeadlineNames.HelloSafety));
}
// ======================================================= category recovery ====
[Fact]
public void ObservationAdvisory_DeviceSetupRecovery_ResolvesAdvisory()
{
// "Try again" path: the ESP re-runs the device phase and DeviceSetup resolves to
// success — the 4910a5a5 recovery hook must match the observation advisory too.
var engine = new DecisionEngine();
var state = SetupObservationEligibleSession(engine);
state = ApplyDeviceSetupTerminalFailure(engine, state).NewState;
var step = engine.Reduce(state, MakeSignal(
60, DecisionSignalKind.DeviceSetupProvisioningComplete, T0.AddMinutes(55)));
Assert.NotNull(step.NewState.EspAdvisoryFailureResolvedUtc);
Assert.Contains(step.Effects, e =>
e.Parameters != null
&& e.Parameters.TryGetValue("eventType", out var et)
&& et == "esp_failure_advisory_resolved");
}
}
}
@@ -623,6 +623,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
{ "NtpServer", config.NtpServer },
{ "EnableImeMatchLog", config.EnableImeMatchLog },
{ "EnableGatherRuleDebugLog", config.EnableGatherRuleDebugLog },
{ "EnableEspContinueAnywayObservation", config.EnableEspContinueAnywayObservation },
{ "LogLevel", config.LogLevel },
{ "MaxBatchSize", config.MaxBatchSize },
{ "DiagnosticsBlobSasUrl", config.DiagnosticsBlobSasUrl },
@@ -729,6 +730,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
NtpServer = string.IsNullOrWhiteSpace(entity.GetString("NtpServer")) ? "time.windows.com" : entity.GetString("NtpServer"),
EnableImeMatchLog = entity.GetBoolean("EnableImeMatchLog"),
EnableGatherRuleDebugLog = entity.GetBoolean("EnableGatherRuleDebugLog"),
EnableEspContinueAnywayObservation = entity.GetBoolean("EnableEspContinueAnywayObservation"),
LogLevel = entity.GetString("LogLevel"),
MaxBatchSize = entity.GetInt32("MaxBatchSize"),
DiagnosticsBlobSasUrl = entity.GetString("DiagnosticsBlobSasUrl"),
@@ -65,13 +65,14 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
bool? isUserDriven = null, DateTime? resumedAt = null,
DateTime? stalledAt = null, bool clearStalledAt = false, bool clearFailureReason = false,
string? failureSource = null, string? adminMarkedAction = null,
string? failureSnapshotJson = null, bool allowTerminalReclassification = false)
string? failureSnapshotJson = null, bool allowTerminalReclassification = false,
bool espSoftFailure = false, string? completionSource = null)
{
var transitioned = await _storage.UpdateSessionStatusAsync(tenantId, sessionId, status,
currentPhase, failureReason, completedAt, earliestEventTimestamp,
latestEventTimestamp, isPreProvisioned, isUserDriven, resumedAt,
stalledAt, clearStalledAt, clearFailureReason, failureSource, adminMarkedAction,
failureSnapshotJson, allowTerminalReclassification);
failureSnapshotJson, allowTerminalReclassification, espSoftFailure, completionSource);
// Reconcile the authoritative EventCount + RebootCount whenever a session actually
// transitions to a terminal state through ANY caller — ingest, admin mark
@@ -274,13 +274,14 @@ namespace AutopilotMonitor.Functions.Functions.Config
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new AgentConfigResponse
{
ConfigVersion = 36, // EnableGatherRuleDebugLog (local gather-rule evaluation trace)
ConfigVersion = 37, // EnableEspContinueAnywayObservation (Continue-Anyway observation mode)
UploadIntervalSeconds = 10,
SelfDestructOnComplete = tenantConfig.SelfDestructOnComplete ?? true,
KeepLogFile = tenantConfig.KeepLogFile ?? false,
EnableGeoLocation = tenantConfig.EnableGeoLocation ?? true,
EnableImeMatchLog = tenantConfig.EnableImeMatchLog ?? false,
EnableGatherRuleDebugLog = tenantConfig.EnableGatherRuleDebugLog ?? false,
EnableEspContinueAnywayObservation = tenantConfig.EnableEspContinueAnywayObservation ?? false,
MaxAuthFailures = tenantConfig.MaxAuthFailures ?? 5,
AuthFailureTimeoutMinutes = tenantConfig.AuthFailureTimeoutMinutes ?? 0,
LogLevel = tenantConfig.LogLevel ?? "Info",
@@ -119,6 +119,7 @@ namespace AutopilotMonitor.Functions.Functions.Config
config.BootstrapTokenEnabled != existingConfig.BootstrapTokenEnabled ||
config.UnrestrictedModeEnabled != existingConfig.UnrestrictedModeEnabled ||
config.EntraAppRolesEnabled != existingConfig.EntraAppRolesEnabled ||
config.EnableEspContinueAnywayObservation != existingConfig.EnableEspContinueAnywayObservation ||
config.CustomRateLimitRequestsPerMinute != existingConfig.CustomRateLimitRequestsPerMinute ||
config.CustomUserRateLimitRequestsPerMinute != existingConfig.CustomUserRateLimitRequestsPerMinute ||
config.Disabled != existingConfig.Disabled ||
@@ -133,6 +134,9 @@ namespace AutopilotMonitor.Functions.Functions.Config
config.BootstrapTokenEnabled = existingConfig.BootstrapTokenEnabled;
config.UnrestrictedModeEnabled = existingConfig.UnrestrictedModeEnabled;
config.EntraAppRolesEnabled = existingConfig.EntraAppRolesEnabled;
// Continue-Anyway observation is an operator-set behavioral override —
// a tenant admin must not be able to relax their own failure semantics.
config.EnableEspContinueAnywayObservation = existingConfig.EnableEspContinueAnywayObservation;
config.CustomRateLimitRequestsPerMinute = existingConfig.CustomRateLimitRequestsPerMinute;
config.CustomUserRateLimitRequestsPerMinute = existingConfig.CustomUserRateLimitRequestsPerMinute;
config.Disabled = existingConfig.Disabled;
@@ -28,6 +28,19 @@ namespace AutopilotMonitor.Functions.Services.Analyze
private int _queueEnsured; // 0 = not yet ensured, 1 = CreateIfNotExistsAsync has run
/// <summary>
/// Delay before the enqueued message becomes visible to the worker. The trigger event
/// (enrollment_complete / enrollment_failed) is by design the FIRST event of the
/// agent's terminal sequence — the terminal flush (stuck-app promotion's synthetic
/// app_install_failed, app_tracking_summary, esp_apps_failure_correlation,
/// agent_shutting_down) lands in separate uploads seconds later. An immediately
/// visible message let the worker evaluate rules against a stream missing those rows
/// (session 53d1e9f6: ANALYZE-ESP-004's optional app_install_failed condition never
/// matched, {{appName}} stayed a literal placeholder — and the RuleResult dedup makes
/// that permanent). 30 s covers the flush including the agent's late-event grace.
/// </summary>
private static readonly TimeSpan s_visibilityDelay = TimeSpan.FromSeconds(30);
public AzureQueueAnalyzeOnEnrollmentEndProducer(
QueueClientFactory queueFactory,
ILogger<AzureQueueAnalyzeOnEnrollmentEndProducer> logger)
@@ -53,7 +66,10 @@ namespace AutopilotMonitor.Functions.Services.Analyze
try
{
var body = JsonConvert.SerializeObject(envelope);
await _queueClient.SendMessageAsync(body, cancellationToken).ConfigureAwait(false);
await _queueClient.SendMessageAsync(
body,
visibilityTimeout: s_visibilityDelay,
cancellationToken: cancellationToken).ConfigureAwait(false);
_logger.LogInformation(
"Analyze enqueued (tenant={Tenant} session={Session} reason={Reason})",
envelope.TenantId, envelope.SessionId, envelope.Reason);
@@ -391,11 +391,24 @@ namespace AutopilotMonitor.Functions.Services
if (c.CompletionEvent != null)
{
// Soft-failure marker from the V2 engine's enrollment_complete audit payload:
// the session completed while an ESP-failure advisory was still unresolved
// (Continue-Anyway observation / classic advisory defang). Stamped on the
// Sessions row so the UI can render the amber "completed with issues" badge.
var espSoftFailure = string.Equals(
c.CompletionEvent.Data?.ContainsKey("espSoftFailure") == true
? c.CompletionEvent.Data["espSoftFailure"]?.ToString() : null,
"true", StringComparison.OrdinalIgnoreCase);
var completionSource = c.CompletionEvent.Data?.ContainsKey("completionSource") == true
? c.CompletionEvent.Data["completionSource"]?.ToString() : null;
statusTransitioned = await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, SessionStatus.Succeeded, c.CompletionEvent.Phase,
completedAt: c.CompletionEvent.Timestamp,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp);
_logger.LogInformation("{SessionPrefix} Status: Succeeded (transitioned={Transitioned})", sessionPrefix, statusTransitioned);
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
espSoftFailure: espSoftFailure, completionSource: completionSource);
_logger.LogInformation("{SessionPrefix} Status: Succeeded (transitioned={Transitioned}, espSoftFailure={EspSoftFailure})",
sessionPrefix, statusTransitioned, espSoftFailure);
}
else if (c.FailureEvent != null)
{
@@ -175,6 +175,15 @@ namespace AutopilotMonitor.Functions.Services
if (!string.IsNullOrEmpty(reconcileReason))
indexEntity["ReconcileReason"] = reconcileReason;
// Soft-failure marker — rendered as the amber "completed with issues" badge in the
// index-served session list, so it must survive a StartedAt-shift full upsert too.
var espSoftFailure = sessionEntity.GetBoolean("EspSoftFailure");
if (espSoftFailure == true)
indexEntity["EspSoftFailure"] = true;
var completionSource = sessionEntity.GetString("CompletionSource");
if (!string.IsNullOrEmpty(completionSource))
indexEntity["CompletionSource"] = completionSource;
// Device-validation path recorded at registration (AutopilotV1 / CorporateIdentifier /
// DeviceAssociation / Bootstrap) — index-served session reads must carry it too.
var validatedBy = sessionEntity.GetString("ValidatedBy");
@@ -647,6 +656,7 @@ namespace AutopilotMonitor.Functions.Services
// fresh-read window isn't reverted by the Replace below (see guard).
"Status", "CurrentPhase", "CompletedAt", "FailureReason", "ReconcileReason",
"FailureSnapshotJson", "FailureSource", "AdminMarkedAction", "DurationSeconds",
"EspSoftFailure", "CompletionSource",
});
freshEtag = freshResponse.Value.ETag;
freshEntity = freshResponse.Value;
@@ -686,7 +696,7 @@ namespace AutopilotMonitor.Functions.Services
|| freshStatus == SessionStatus.Incomplete.ToString())
{
entity["Status"] = freshStatus;
foreach (var col in new[] { "CurrentPhase", "CompletedAt", "FailureReason", "ReconcileReason", "FailureSnapshotJson", "FailureSource", "AdminMarkedAction", "DurationSeconds" })
foreach (var col in new[] { "CurrentPhase", "CompletedAt", "FailureReason", "ReconcileReason", "FailureSnapshotJson", "FailureSource", "AdminMarkedAction", "DurationSeconds", "EspSoftFailure", "CompletionSource" })
{
if (freshEntity!.TryGetValue(col, out var val) && val is not null)
entity[col] = val;
@@ -1590,7 +1600,7 @@ namespace AutopilotMonitor.Functions.Services
: null;
}
public async Task<bool> UpdateSessionStatusAsync(string tenantId, string sessionId, SessionStatus status, EnrollmentPhase? currentPhase = null, string? failureReason = null, DateTime? completedAt = null, DateTime? earliestEventTimestamp = null, DateTime? latestEventTimestamp = null, bool? isPreProvisioned = null, bool? isUserDriven = null, DateTime? resumedAt = null, DateTime? stalledAt = null, bool clearStalledAt = false, bool clearFailureReason = false, string? failureSource = null, string? adminMarkedAction = null, string? failureSnapshotJson = null, bool allowTerminalReclassification = false)
public async Task<bool> UpdateSessionStatusAsync(string tenantId, string sessionId, SessionStatus status, EnrollmentPhase? currentPhase = null, string? failureReason = null, DateTime? completedAt = null, DateTime? earliestEventTimestamp = null, DateTime? latestEventTimestamp = null, bool? isPreProvisioned = null, bool? isUserDriven = null, DateTime? resumedAt = null, DateTime? stalledAt = null, bool clearStalledAt = false, bool clearFailureReason = false, string? failureSource = null, string? adminMarkedAction = null, string? failureSnapshotJson = null, bool allowTerminalReclassification = false, bool espSoftFailure = false, string? completionSource = null)
{
SecurityValidator.EnsureValidGuid(tenantId, nameof(tenantId));
SecurityValidator.EnsureValidGuid(sessionId, nameof(sessionId));
@@ -1701,6 +1711,18 @@ namespace AutopilotMonitor.Functions.Services
var reconcileReason = ComputeReconcileReason(existingStatusStr, failureReason, adminMarkedAction);
if (reconcileReason != null)
update["ReconcileReason"] = reconcileReason;
// Soft-failure marker (Continue-Anyway observation / advisory-defang):
// the session succeeded, but the ESP demonstrably gave up on at least
// one blocking item first. Rendered as the amber "completed with
// issues" badge. Only ever stamped alongside a Succeeded transition —
// absence means a clean success.
if (espSoftFailure)
{
update["EspSoftFailure"] = true;
if (!string.IsNullOrEmpty(completionSource))
update["CompletionSource"] = completionSource;
}
}
else if (status == SessionStatus.Failed)
{
@@ -1973,6 +1995,13 @@ namespace AutopilotMonitor.Functions.Services
var forceReconcileReason = ComputeReconcileReason(freshStatusStr, failureReason, adminMarkedAction);
if (forceReconcileReason != null)
forceUpdate["ReconcileReason"] = forceReconcileReason;
// Mirror the normal path's soft-failure marker.
if (espSoftFailure)
{
forceUpdate["EspSoftFailure"] = true;
if (!string.IsNullOrEmpty(completionSource))
forceUpdate["CompletionSource"] = completionSource;
}
}
else if (status == SessionStatus.Failed)
{
@@ -2928,6 +2957,8 @@ namespace AutopilotMonitor.Functions.Services
FailureReason = entity.GetString("FailureReason") ?? string.Empty,
FailureSource = entity.GetString("FailureSource") ?? string.Empty,
ReconcileReason = entity.GetString("ReconcileReason") ?? string.Empty,
EspSoftFailure = entity.GetBoolean("EspSoftFailure") ?? false,
CompletionSource = entity.GetString("CompletionSource") ?? string.Empty,
AdminMarkedAction = entity.GetString("AdminMarkedAction"),
ValidatedBy = entity.GetString("ValidatedBy") ?? string.Empty,
PendingActionsJson = entity.GetString("PendingActionsJson") ?? string.Empty,
@@ -132,6 +132,7 @@ namespace AutopilotMonitor.Functions.Services
"Disabled", "DisabledReason", "DisabledUntil",
"AllowInsecureAgentRequests", "BootstrapTokenEnabled",
"UnrestrictedModeEnabled", "EntraAppRolesEnabled",
"EnableEspContinueAnywayObservation",
"CustomRateLimitRequestsPerMinute", "CustomUserRateLimitRequestsPerMinute",
"ValidateDeviceAssociation", "MaxNdjsonPayloadSizeMB",
};
@@ -111,6 +111,16 @@ namespace AutopilotMonitor.DecisionCore.Engine
/// </summary>
public const string EspAllowContinueAnyway = "espAllowContinueAnyway";
/// <summary>
/// On <c>EspConfigDetected</c>: "true" when the tenant opted into the
/// Continue-Anyway observation mode (operator-set
/// <c>EnableEspContinueAnywayObservation</c> tenant setting, stamped into the signal
/// payload by the agent from RemoteConfig — NOT an ESP registry fact). Only stamped
/// when enabled; missing → fact stays unset and a Device-phase ESP terminal failure
/// keeps the immediate hard-fail semantics.
/// </summary>
public const string EspContinueAnywayObservationEnabled = "espContinueAnywayObservationEnabled";
/// <summary>On <c>HelloPolicyDetected</c>: "true" / "false". PR4 (882fef64 debrief).</summary>
public const string HelloEnabled = "helloEnabled";
@@ -565,13 +565,23 @@ namespace AutopilotMonitor.DecisionCore.Engine
// and surfaces hello_resolution via completion_waiting; the wait is bounded by an
// already-armed HelloSafety, the tracker's Hello completion timer, or the
// AdvisoryCompletion backstop.
// Continue-Anyway observation completion (2026-08-08): while a Device-phase
// observation advisory is live (see IsContinueAnywayObservationActive — ESP
// failed in DeviceSetup, tenant opted into observation, AccountSetup never
// entered), the strong post-AccountSetup gate is unsatisfiable by construction.
// The DAD-validated real-user desktop IS the observation's completion evidence:
// the user pressed "Continue anyway" and reached a working desktop. Complete
// eagerly here instead of waiting out the 60-min window so the session end
// reflects the real end.
var observationCompletion = IsContinueAnywayObservationActive(state);
if (!helloAlreadyResolved
&& HelloPolicyDisabledWithoutWizard(state.HelloPolicyEnabled, state.HelloWizardStartedUtc)
// desktopArrivedInFlight: the desktop fact was just recorded on the builder
// (line above) and is invisible to the pre-mutation state — arm C's "desktop
// arrives last" ordering (session a4537c36) must count it. Arms A/B ignore
// the flag entirely.
&& ShouldTransitionToAwaitingHello(state, desktopArrivedInFlight: true))
&& (ShouldTransitionToAwaitingHello(state, desktopArrivedInFlight: true) || observationCompletion))
{
SynthesizeHelloSkipped(builder, signal);
@@ -600,12 +610,16 @@ namespace AutopilotMonitor.DecisionCore.Engine
// Completion gates (ARCH-F1): synthetic Hello + Desktop are recorded; complete
// through Finalizing when all gates are open, else defer until a gate releases.
// The observation trigger label keeps the two doors apart on the audit trail —
// the observation path deliberately bypassed the post-AccountSetup gate.
return CompleteThroughFinalizingOrDefer(
state: state,
signal: signal,
preparedBuilder: builder,
nextStepIndex: nextStep,
trigger: nameof(DecisionSignalKind.DesktopArrived) + ":HelloDisabledFastPath",
trigger: ShouldTransitionToAwaitingHello(state, desktopArrivedInFlight: true)
? nameof(DecisionSignalKind.DesktopArrived) + ":HelloDisabledFastPath"
: nameof(DecisionSignalKind.DesktopArrived) + ":ContinueAnywayObservation",
leadingEffects: extraEffects);
}
@@ -1029,16 +1043,37 @@ namespace AutopilotMonitor.DecisionCore.Engine
/// </param>
internal static DecisionEffect BuildEnrollmentCompleteEffect(DecisionState state, string completionTrigger)
{
var trail = DecisionAuditTrailBuilder.Build(
postState: state,
decidedStage: state.Stage,
trigger: completionTrigger);
// Soft-failure honesty (2026-08-08): a session completing while an ESP-failure
// advisory is still unresolved reached the desktop, but the ESP demonstrably gave
// up on at least one blocking item first (Continue-Anyway observation, or the
// classic post-AccountSetup defang). Stamp that on the terminal event so the
// backend can mark the session "completed with issues" instead of presenting a
// clean success. A RESOLVED advisory (category re-ran to success after
// "Try again") is a genuine recovery and stays unstamped.
if (state.EspAdvisoryFailureRecordedUtc != null && state.EspAdvisoryFailureResolvedUtc == null)
{
trail["espSoftFailure"] = "true";
trail["completionSource"] = state.AccountSetupEnteredUtc == null
? "continue_anyway_observation"
: "continue_anyway_post_accountsetup";
if (state.EspAdvisoryFailureCategory != null)
{
trail["espSoftFailureCategory"] = state.EspAdvisoryFailureCategory.Value;
}
}
return new DecisionEffect(
kind: DecisionEffectKind.EmitEventTimelineEntry,
parameters: new Dictionary<string, string>
{
["eventType"] = SharedConstants.EventTypes.EnrollmentComplete,
},
typedPayload: DecisionAuditTrailBuilder.Build(
postState: state,
decidedStage: state.Stage,
trigger: completionTrigger));
typedPayload: trail);
}
}
}
@@ -237,20 +237,61 @@ namespace AutopilotMonitor.DecisionCore.Engine
var observations = state.ScenarioObservations;
var continueAnywayEnabled = observations?.EspAllowContinueAnyway?.Value == true;
var advisoryEligible = continueAnywayEnabled && state.AccountSetupEnteredUtc != null;
// Continue-Anyway observation mode (tenant c9787ba2, session 53d1e9f6, 2026-08-08):
// a Device-phase ESP terminal failure (AccountSetup never entered) is a hard fail
// by default — the classic advisory gate above requires AccountSetupEnteredUtc.
// Fleets held at the ESP timeout wall by one slow blocking app then show 100%
// Failed although the profile allows "Continue anyway" and users demonstrably
// reach the desktop. When the tenant explicitly opted in
// (EnableEspContinueAnywayObservation → EspContinueAnywayObservationEnabled
// observation), defang this case too: arm a 60-min observation window instead of
// failing immediately. The DAD-validated real-user desktop + the Hello gate then
// detect the true end (see HandleDesktopArrivedV1's observation completion and
// the observation branch in HandleAdvisoryCompletionDeadlineFired); an expired
// window un-defangs to the original esp_terminal_failure. Scope: Classic /
// Unknown user-driven flows only — WhiteGlove / SelfDeploying / DevicePreparation
// have no Continue-Anyway user desktop to observe.
var observationEligible = !advisoryEligible
&& continueAnywayEnabled
&& observations?.EspContinueAnywayObservationEnabled?.Value == true
&& state.AccountSetupEnteredUtc == null
&& IsObservationEligibleScenario(state);
var parameters = BuildEspFailureParameters(
signal: signal,
reason: reason,
advisoryPath: advisoryEligible,
advisoryPath: advisoryEligible || observationEligible,
observations: observations);
if (advisoryEligible)
if (advisoryEligible || observationEligible)
{
return BuildAdvisoryStep(state, signal, nextStep, reason, parameters);
if (observationEligible)
{
parameters["advisoryReason"] = "esp_failure_defanged_continueanyway_observation";
}
return BuildAdvisoryStep(state, signal, nextStep, reason, parameters, observationVariant: observationEligible);
}
return BuildFailedStep(state, signal, nextStep, reason, parameters);
}
/// <summary>
/// Scenario gate for the Continue-Anyway observation defang: only Classic (or
/// still-unclassified) non-pre-provisioning flows qualify. WhiteGlove technician
/// sessions, SelfDeploying/kiosk profiles and DevicePreparation have no interactive
/// user who could press "Continue anyway" and produce a real-user desktop, so the
/// observation window could only ever expire — the immediate hard fail is more
/// truthful there.
/// </summary>
private static bool IsObservationEligibleScenario(DecisionState state)
{
var profile = state.ScenarioProfile;
return (profile.Mode == EnrollmentMode.Unknown || profile.Mode == EnrollmentMode.Classic)
&& profile.PreProvisioningSide == PreProvisioningSide.None
&& state.ScenarioObservations?.RegistrySelfDeployingProfile?.Value != true;
}
/// <summary>
/// Build the advisory-enrichment step that runs when an EspTerminalFailure duplicate
/// arrives AFTER an advisory was already recorded AND the duplicate carries new
@@ -322,6 +363,17 @@ namespace AutopilotMonitor.DecisionCore.Engine
/// </summary>
private static readonly TimeSpan s_advisoryCompletionWindow = TimeSpan.FromMinutes(30);
/// <summary>
/// Observation window for the Continue-Anyway observation defang (Device-phase ESP
/// terminal failure, tenant opt-in — see <c>HandleEspTerminalFailureV1</c>). Longer
/// than <see cref="s_advisoryCompletionWindow"/> because the user must first notice
/// the failure screen, press "Continue anyway", sign in and reach the desktop —
/// or press "Try again" and let the ESP re-run the whole device phase. Fixed 60 min
/// (user decision 2026-08-08): well under the agent max lifetime, and every observed
/// session still gets a terminal verdict when the window expires.
/// </summary>
private static readonly TimeSpan s_observationCompletionWindow = TimeSpan.FromMinutes(60);
/// <summary>
/// Arming-time baseline keys carried on the AdvisoryCompletion deadline's
/// <see cref="ActiveDeadline.FiresPayload"/> (session 1924092e enforcement-progress
@@ -343,10 +395,10 @@ namespace AutopilotMonitor.DecisionCore.Engine
/// <see cref="HasEnforcementProgressSinceArming"/> can tell an idle dead-end from a
/// still-progressing enrollment at fire time.
/// </summary>
internal static ActiveDeadline BuildAdvisoryCompletionDeadline(DecisionState state, DecisionSignal signal) =>
internal static ActiveDeadline BuildAdvisoryCompletionDeadline(DecisionState state, DecisionSignal signal, TimeSpan? window = null) =>
new ActiveDeadline(
name: DeadlineNames.AdvisoryCompletion,
dueAtUtc: EffectiveDeadlineBase(state, signal).Add(s_advisoryCompletionWindow),
dueAtUtc: EffectiveDeadlineBase(state, signal).Add(window ?? s_advisoryCompletionWindow),
firesSignalKind: DecisionSignalKind.DeadlineFired,
firesPayload: new Dictionary<string, string>
{
@@ -358,6 +410,23 @@ namespace AutopilotMonitor.DecisionCore.Engine
.ToString(CultureInfo.InvariantCulture),
});
/// <summary>
/// True while a Continue-Anyway observation advisory is live: an ESP-failure advisory
/// is recorded, not resolved, and AccountSetup was never entered. Only the
/// observation arming site in <see cref="HandleEspTerminalFailureV1"/> can produce
/// this shape — the classic advisory requires <see cref="DecisionState.AccountSetupEnteredUtc"/>
/// at arm time and neither fact is ever retracted — so no separate opt-in re-check is
/// needed. Consumed by the observation completion path in
/// <c>HandleDesktopArrivedV1</c> and the observation branches in
/// <see cref="HandleAdvisoryCompletionDeadlineFired"/>. Deliberately flips false when
/// AccountSetup IS entered later (user pressed "Try again" and the ESP re-ran into
/// the user phase) — from that point the classic advisory semantics apply.
/// </summary>
internal static bool IsContinueAnywayObservationActive(DecisionState state) =>
state.EspAdvisoryFailureRecordedUtc != null
&& state.EspAdvisoryFailureResolvedUtc == null
&& state.AccountSetupEnteredUtc == null;
/// <summary>True when <see cref="DeadlineNames.AdvisoryCompletion"/> is armed in <paramref name="state"/>.</summary>
internal static bool HasAdvisoryCompletionDeadline(DecisionState state)
=> FindAdvisoryCompletionDeadline(state) != null;
@@ -417,7 +486,8 @@ namespace AutopilotMonitor.DecisionCore.Engine
DecisionSignal signal,
int nextStep,
string reason,
Dictionary<string, string> parameters)
Dictionary<string, string> parameters,
bool observationVariant = false)
{
// Session 4910a5a5: remember WHICH category failed so the recovery hook
// (TryResolveAdvisoryOnCategoryRecovery) can later match a ProvisioningComplete
@@ -443,7 +513,13 @@ namespace AutopilotMonitor.DecisionCore.Engine
// session either way (see HandleAdvisoryCompletionDeadlineFired). AddDeadline
// replaces by name, so an earlier esp_exiting arm is intentionally re-based on the
// (later) failure instant — the freshest dead-end signal owns the window.
var advisoryCompletion = BuildAdvisoryCompletionDeadline(state, signal);
//
// Observation variant (2026-08-08): the Device-phase Continue-Anyway observation
// uses the longer 60-min window — the user must traverse failure screen →
// Continue anyway → sign-in → desktop (or a full device-phase re-run via
// "Try again") before completion evidence can exist.
var advisoryCompletion = BuildAdvisoryCompletionDeadline(
state, signal, observationVariant ? s_observationCompletionWindow : (TimeSpan?)null);
builder.AddDeadline(advisoryCompletion);
var newState = builder.Build();
@@ -636,7 +712,17 @@ namespace AutopilotMonitor.DecisionCore.Engine
// Shared with ShouldTransitionToAwaitingHello's proactive arm C (session a4537c36).
var imeUserSessionGenuine = IsImeUserSessionGenuine(state);
if (desktopArrived && helloSatisfied && imeUserSessionGenuine)
// Continue-Anyway observation variant (2026-08-08): a Device-phase advisory —
// AccountSetup never entered — cannot satisfy the IME user-session gate by
// construction (the gate anchors on AccountSetupEnteredUtc). The DAD-validated
// real-user desktop plus the Hello gate ARE the completion evidence there: the
// user pressed "Continue anyway" and reached a working desktop. Only the
// observation arming site can produce this shape (the classic advisory requires
// AccountSetupEnteredUtc at arm time and the fact is never retracted), so the
// predicate needs no separate opt-in re-check.
var isObservationAdvisory = IsContinueAnywayObservationActive(state);
if (desktopArrived && helloSatisfied && (imeUserSessionGenuine || isObservationAdvisory))
{
if (state.HelloResolvedUtc == null)
{
@@ -669,7 +755,7 @@ namespace AutopilotMonitor.DecisionCore.Engine
// the Hello wizard right now. Promote to AwaitingHello and let a real
// HelloResolved — or the HelloSafety timeout — decide instead.
if (desktopArrived
&& imeUserSessionGenuine
&& (imeUserSessionGenuine || isObservationAdvisory)
&& state.HelloResolvedUtc == null
&& state.HelloPolicyEnabled?.Value == false
&& state.HelloWizardStartedUtc != null)
@@ -722,7 +808,13 @@ namespace AutopilotMonitor.DecisionCore.Engine
// — the failure DID un-happen, so a live AccountSetup must not be failed on the
// stale advisory). Convergent either way: each re-arm renews the baselines, so a
// session that stalls after the recovery still fails one window later.
if ((!hasAdvisoryAnchor || advisoryResolved) && HasEnforcementProgressSinceArming(state, armedDeadline!))
// The observation variant participates too (2026-08-08): its whole point is to
// keep watching while there is evidence of life — a user who pressed "Try again"
// late in the window keeps apps reaching terminal states, and failing mid-retry
// would be exactly the false positive the guard exists to prevent. Convergent:
// each re-arm (30 min) demands NEW progress.
if ((!hasAdvisoryAnchor || advisoryResolved || isObservationAdvisory)
&& HasEnforcementProgressSinceArming(state, armedDeadline!))
{
var rearmedDeadline = BuildAdvisoryCompletionDeadline(state, signal);
builder.AddDeadline(rearmedDeadline);
@@ -778,7 +870,7 @@ namespace AutopilotMonitor.DecisionCore.Engine
// through Finalizing. Deliberately does NOT require the IME user-session gate —
// the FP sessions never saw an IME user-session completion (that gate guards
// completion evidence, not user presence; presence is the DAD-validated desktop).
if ((!hasAdvisoryAnchor || advisoryResolved)
if ((!hasAdvisoryAnchor || advisoryResolved || isObservationAdvisory)
&& desktopArrived
&& state.HelloResolvedUtc == null
&& state.HelloPolicyEnabled == null
@@ -861,7 +953,14 @@ namespace AutopilotMonitor.DecisionCore.Engine
reason: reason,
advisoryPath: false,
observations: state.ScenarioObservations);
parameters["advisoryReason"] = "advisory_completion_window_expired_without_completion_evidence";
// Observation variant: the 60-min Continue-Anyway observation produced no
// real-user desktop — either nobody pressed the button or the device truly
// is stuck at the failure screen. The un-defang is identical (the failure
// cause IS the ESP failure), only the advisoryReason tells the two windows
// apart on the timeline.
parameters["advisoryReason"] = isObservationAdvisory
? "continue_anyway_observation_window_expired_without_completion_evidence"
: "advisory_completion_window_expired_without_completion_evidence";
builder.WithLastFailureTrigger(nameof(DecisionSignalKind.EspTerminalFailure), signal.SessionSignalOrdinal);
}
else
@@ -68,6 +68,18 @@ namespace AutopilotMonitor.DecisionCore.State
/// </summary>
public SignalFact<bool>? EspAllowContinueAnyway { get; init; }
/// <summary>
/// Tenant-config opt-in from <see cref="Signals.DecisionSignalKind.EspConfigDetected"/>
/// (payload <c>espContinueAnywayObservationEnabled</c>, stamped by the agent from
/// <c>RemoteConfig.EnableEspContinueAnywayObservation</c> — an operator-set tenant
/// setting, not an ESP registry fact). When <c>true</c> AND the ESP profile allows
/// "Continue anyway", a Device-phase ESP terminal failure (AccountSetup never entered)
/// is defanged into an observation advisory instead of failing the session immediately
/// — see <c>HandleEspTerminalFailureV1</c>. Default (absent/false) keeps the immediate
/// hard-fail semantics. Set-once.
/// </summary>
public SignalFact<bool>? EspContinueAnywayObservationEnabled { get; init; }
/// <summary>
/// Raw registry fact from <see cref="Signals.DecisionSignalKind.EnrollmentFactsObserved"/>
/// (payload <c>isSelfDeployingProfile</c>, from <c>CloudAssignedOobeConfig</c> bits
@@ -126,6 +138,11 @@ namespace AutopilotMonitor.DecisionCore.State
? this
: this with { EspAllowContinueAnyway = new SignalFact<bool>(value, sourceSignalOrdinal) };
public EnrollmentScenarioObservations WithEspContinueAnywayObservationEnabled(bool value, long sourceSignalOrdinal) =>
EspContinueAnywayObservationEnabled != null
? this
: this with { EspContinueAnywayObservationEnabled = new SignalFact<bool>(value, sourceSignalOrdinal) };
public EnrollmentScenarioObservations WithRegistrySelfDeployingProfile(bool value, long sourceSignalOrdinal) =>
RegistrySelfDeployingProfile != null
? this
@@ -226,6 +226,11 @@ namespace AutopilotMonitor.DecisionCore.State
{
newObservations = newObservations.WithEspAllowContinueAnyway(allowContinue, signal.SessionSignalOrdinal);
}
if (signal.Payload.TryGetValue(SignalPayloadKeys.EspContinueAnywayObservationEnabled, out var rawObservation)
&& bool.TryParse(rawObservation, out var observationEnabled))
{
newObservations = newObservations.WithEspContinueAnywayObservationEnabled(observationEnabled, signal.SessionSignalOrdinal);
}
}
var newProfile = currentProfile;
@@ -81,7 +81,8 @@ namespace AutopilotMonitor.Shared.DataAccess
bool? isUserDriven = null, DateTime? resumedAt = null,
DateTime? stalledAt = null, bool clearStalledAt = false, bool clearFailureReason = false,
string? failureSource = null, string? adminMarkedAction = null,
string? failureSnapshotJson = null, bool allowTerminalReclassification = false);
string? failureSnapshotJson = null, bool allowTerminalReclassification = false,
bool espSoftFailure = false, string? completionSource = null);
/// <summary>
/// Increments per-session counters via read-modify-write. Returns the post-merge session
/// snapshot (the RMW read with the applied increments) so hot-path callers can skip a
@@ -46,6 +46,14 @@ namespace AutopilotMonitor.Shared.Models
/// </summary>
public bool EnableGatherRuleDebugLog { get; set; } = false;
/// <summary>
/// Continue-Anyway observation mode (operator-set tenant opt-in): a Device-phase ESP
/// terminal failure on a Continue-Anyway-enabled profile is observed for up to 60
/// minutes instead of failing immediately — the real-user desktop then completes the
/// session with an esp-soft-failure marker. Default false = immediate hard fail.
/// </summary>
public bool EnableEspContinueAnywayObservation { get; set; } = false;
public CollectorConfiguration Collectors { get; set; } = default!;
/// <summary>
@@ -382,6 +382,17 @@ namespace AutopilotMonitor.Shared.Models
/// </summary>
public bool? EnableGatherRuleDebugLog { get; set; }
/// <summary>
/// Continue-Anyway observation mode: when true AND the ESP profile allows
/// "Continue anyway", a Device-phase ESP terminal failure (AccountSetup never
/// entered) does not fail the session immediately — the agent keeps monitoring for
/// up to 60 minutes and completes with an esp-soft-failure marker once the
/// DAD-validated real-user desktop (plus the Hello gate) proves the user continued;
/// an expired window fails with the original esp_terminal_failure. Operator-set only
/// (not exposed in the tenant admin UI). null = use agent default (false).
/// </summary>
public bool? EnableEspContinueAnywayObservation { get; set; }
/// <summary>
/// Log verbosity level override for this tenant's agents.
/// null = use agent default ("Info"). Values: "Info", "Debug", "Verbose", "Trace".
@@ -131,6 +131,8 @@ namespace AutopilotMonitor.Shared.Models
Status = s.Status,
FailureReason = s.FailureReason,
FailureSource = s.FailureSource,
EspSoftFailure = s.EspSoftFailure,
CompletionSource = s.CompletionSource,
EventCount = s.EventCount,
DurationSeconds = s.DurationSeconds,
EnrollmentType = s.EnrollmentType,
@@ -231,6 +231,26 @@ namespace AutopilotMonitor.Shared.Models
/// </summary>
public string ReconcileReason { get; set; } = string.Empty;
/// <summary>
/// True only on a Succeeded session that completed while an ESP-failure advisory was
/// still unresolved: the ESP demonstrably gave up on at least one blocking item
/// (typically the 30-min timeout wall) and the user most likely pressed
/// "Continue anyway" — the device reached a working desktop, but not everything the
/// ESP tracked finished inside its window. Rendered as the amber "completed with
/// issues" badge. False/absent = clean success.
/// </summary>
public bool EspSoftFailure { get; set; }
/// <summary>
/// How a soft-failure completion was detected. Values:
/// - "continue_anyway_observation": Device-phase ESP failure, tenant opted into the
/// Continue-Anyway observation window, real-user desktop arrived inside it
/// - "continue_anyway_post_accountsetup": classic advisory defang (AccountSetup had
/// already been entered when the ESP failure fired)
/// Empty on clean successes.
/// </summary>
public string CompletionSource { get; set; } = string.Empty;
/// <summary>
/// Non-null only when an administrator explicitly flipped the session via the portal
/// (MarkSessionSucceeded / MarkSessionFailed). Values: <c>null</c> (default, agent-driven),
@@ -33,6 +33,8 @@ export interface TenantConfiguration {
bootstrapTokenEnabled?: boolean;
unrestrictedModeEnabled?: boolean;
entraAppRolesEnabled?: boolean;
/** Operator-set (GA-only): Device-phase ESP failures on Continue-Anyway profiles get a 60-min observation instead of an immediate hard fail. */
enableEspContinueAnywayObservation?: boolean;
dataRetentionDays: number;
sessionTimeoutHours: number;
planTier?: string;
@@ -1005,6 +1007,24 @@ function TenantManagementSectionInner({
<span className="text-sm font-medium text-gray-700">Enable Unrestricted Mode</span>
</label>
<div>
<label className="flex items-center space-x-2 cursor-pointer">
<input
type="checkbox"
checked={editingTenant.enableEspContinueAnywayObservation ?? false}
onChange={(e) => setEditingTenant({ ...editingTenant, enableEspContinueAnywayObservation: e.target.checked })}
className="w-4 h-4 text-teal-600 border-gray-300 rounded focus:ring-teal-500"
/>
<span className="text-sm font-medium text-gray-700">Enable Continue-Anyway Observation</span>
</label>
<p className="text-xs text-gray-400 mt-1 ml-6">
Device-phase ESP terminal failures on profiles that allow &quot;Continue anyway&quot; are
observed for up to 60 min instead of failing immediately: a real-user desktop completes
the session as Succeeded with an amber &quot;with issues&quot; badge; otherwise it fails with
the original reason. Needs agent with ConfigVersion 37+; applies to new sessions only.
</p>
</div>
<div>
<label className="flex items-center space-x-2 cursor-pointer">
<input
@@ -59,6 +59,7 @@ interface TenantConfig {
ntpServer?: string | null;
enableImeMatchLog?: boolean | null;
enableGatherRuleDebugLog?: boolean | null;
enableEspContinueAnywayObservation?: boolean | null;
logLevel?: string | null;
maxBatchSize?: number | null;
showEnrollmentSummary?: boolean | null;
@@ -132,6 +133,7 @@ const DEFAULTS: Record<string, unknown> = {
ntpServer: 'time.windows.com',
enableImeMatchLog: null,
enableGatherRuleDebugLog: null,
enableEspContinueAnywayObservation: null,
logLevel: null,
maxBatchSize: null,
showEnrollmentSummary: null,
@@ -171,6 +173,7 @@ const RUNTIME_DEFAULTS: Record<string, unknown> = {
ntpServer: 'time.windows.com',
enableImeMatchLog: false,
enableGatherRuleDebugLog: false,
enableEspContinueAnywayObservation: false,
maxAuthFailures: 5,
authFailureTimeoutMinutes: 0,
logLevel: 'Info',
@@ -300,6 +303,7 @@ function computeRuntime(c: TenantConfig): Record<string, unknown> {
ntpServer: c.ntpServer ?? 'time.windows.com',
enableImeMatchLog: c.enableImeMatchLog ?? false,
enableGatherRuleDebugLog: c.enableGatherRuleDebugLog ?? false,
enableEspContinueAnywayObservation: c.enableEspContinueAnywayObservation ?? false,
maxAuthFailures: c.maxAuthFailures ?? 5,
authFailureTimeoutMinutes: c.authFailureTimeoutMinutes ?? 0,
logLevel: c.logLevel ?? 'Info',
@@ -581,6 +585,7 @@ export function SectionTenantConfigReport() {
<ConfigRow label="NTP Server" value={config.ntpServer} configKey="ntpServer" defaults={DEFAULTS} />
<ConfigRow label="IME Match Log" value={config.enableImeMatchLog} configKey="enableImeMatchLog" defaults={DEFAULTS} />
<ConfigRow label="Gather Rule Debug Log" value={config.enableGatherRuleDebugLog} configKey="enableGatherRuleDebugLog" defaults={DEFAULTS} />
<ConfigRow label="Continue-Anyway Observation" value={config.enableEspContinueAnywayObservation} configKey="enableEspContinueAnywayObservation" defaults={DEFAULTS} />
<ConfigRow label="Log Level" value={config.logLevel} configKey="logLevel" defaults={DEFAULTS} />
<ConfigRow label="Max Batch Size" value={config.maxBatchSize} configKey="maxBatchSize" defaults={DEFAULTS} />
<ConfigRow label="Show Script Output" value={config.showScriptOutput} configKey="showScriptOutput" defaults={DEFAULTS} />
@@ -654,6 +659,7 @@ export function SectionTenantConfigReport() {
<ConfigRow label="NTP Server" value={runtime.ntpServer} configKey="ntpServer" defaults={RUNTIME_DEFAULTS} />
<ConfigRow label="IME Match Log" value={runtime.enableImeMatchLog} configKey="enableImeMatchLog" defaults={RUNTIME_DEFAULTS} />
<ConfigRow label="Gather Rule Debug Log" value={runtime.enableGatherRuleDebugLog} configKey="enableGatherRuleDebugLog" defaults={RUNTIME_DEFAULTS} />
<ConfigRow label="Continue-Anyway Observation" value={runtime.enableEspContinueAnywayObservation} configKey="enableEspContinueAnywayObservation" defaults={RUNTIME_DEFAULTS} />
<ConfigRow label="Log Level" value={runtime.logLevel} configKey="logLevel" defaults={RUNTIME_DEFAULTS} />
<ConfigRow label="Send Trace Events" value={runtime.sendTraceEvents} configKey="sendTraceEvents" defaults={RUNTIME_DEFAULTS} />
<ConfigRow label="Unrestricted Mode" value={runtime.unrestrictedMode} configKey="unrestrictedMode" defaults={RUNTIME_DEFAULTS} />
@@ -862,7 +862,7 @@ function SessionCell({
{/* Qualifier pills stack BELOW the status badge (flex-col) so extra badges
(Hybrid / Self-Deploying / Blocked) never widen the status column. */}
<div className="flex flex-col items-start gap-1">
<SessionStatusBadge status={session.status} failureReason={session.failureReason} adminMarkedAction={session.adminMarkedAction} reconcileReason={session.reconcileReason} />
<SessionStatusBadge status={session.status} failureReason={session.failureReason} adminMarkedAction={session.adminMarkedAction} reconcileReason={session.reconcileReason} espSoftFailure={session.espSoftFailure} />
{session.isHybridJoin && (
<span
className="px-2 inline-flex items-center gap-1 text-xs leading-5 font-semibold rounded-full bg-purple-100 text-purple-800"
@@ -31,6 +31,8 @@ interface SessionSummary {
completedAt: string | null;
status: string;
failureReason: string | null;
/** Succeeded despite an unresolved ESP failure (Continue-Anyway) — amber "with issues" badge. */
espSoftFailure?: boolean;
durationSeconds: number | null;
enrollmentType: string;
/** Session-wide average agent→backend HTTP round-trip (ms); absent for pre-feature agents. */
@@ -360,7 +362,7 @@ function LocationSessionsContent() {
{session.model || "—"}
</td>
<td className="px-4 py-3 text-sm">
<SessionStatusBadge status={session.status} failureReason={session.failureReason} />
<SessionStatusBadge status={session.status} failureReason={session.failureReason} espSoftFailure={session.espSoftFailure} />
</td>
<td className="px-4 py-3 text-sm text-gray-600">
{formatDuration(session.durationSeconds)}
@@ -59,7 +59,7 @@ export default function SessionInfoCard({ session, enrollmentDuration, displaySt
<InfoItem label="Duration" value={enrollmentDuration ?? `${Math.round(session.durationSeconds / 60)} min`} tooltip={lastContactTooltip} />
<InfoItem label="Events" value={session.eventCount.toString()} tooltip={lastContactTooltip} />
<InfoItem label="Reboots" value={(session.rebootCount ?? 0).toString()} tooltip="System reboots observed during enrollment (V2 only)" />
<InfoItem label="Status" value={<StatusBadge status={displayStatus} failureReason={session.failureReason} failureSource={session.failureSource} adminMarkedAction={session.adminMarkedAction} reconcileReason={session.reconcileReason} />} />
<InfoItem label="Status" value={<StatusBadge status={displayStatus} failureReason={session.failureReason} failureSource={session.failureSource} adminMarkedAction={session.adminMarkedAction} reconcileReason={session.reconcileReason} espSoftFailure={session.espSoftFailure} />} />
<InfoItem label="Enrollment Type" value={enrollmentTypeLabel(session, isGatherRulesSession)} />
<InfoItem label="Join Type" value={joinTypeLabel(session)} />
{validatedByLabel(session) && (
@@ -84,6 +84,19 @@ export default function SessionInfoCard({ session, enrollmentDuration, displaySt
</div>
</div>
)}
{displayStatus === "Succeeded" && session.espSoftFailure && (
<div className="mt-4 flex items-start gap-2 px-3 py-2 rounded-lg bg-amber-50 border border-amber-200 text-sm text-amber-800 dark:bg-amber-900/30 dark:border-amber-800 dark:text-amber-200">
<svg className="w-4 h-4 flex-shrink-0 mt-0.5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M12 9v2m0 4h.01m-6.938 4h13.856c1.54 0 2.502-1.667 1.732-2.5L13.732 4c-.77-.833-1.964-.833-2.732 0L4.082 16.5c-.77.833.192 2.5 1.732 2.5z" />
</svg>
<span>
<strong>Completed with issues:</strong> the ESP reported a terminal failure, but the user
reached a working desktop — most likely via the &quot;Continue anyway&quot; button. At least one
blocking item did not finish inside the ESP window; check the timeline and app list for the
item that held the ESP at its timeout wall.
</span>
</div>
)}
{ntpOffset && Math.abs(ntpOffset.offsetSeconds) > 30 && (
<div className="mt-4 flex items-center gap-2 px-3 py-2 rounded-lg bg-amber-50 border border-amber-200 text-sm text-amber-800">
<svg className="w-4 h-4 flex-shrink-0" fill="none" stroke="currentColor" viewBox="0 0 24 24">
@@ -250,12 +263,12 @@ function ReconcileTimingFacts({ session }: { session: Session }) {
);
}
function StatusBadge({ status, failureReason, failureSource, adminMarkedAction, reconcileReason }: { status: string; failureReason?: string; failureSource?: string; adminMarkedAction?: string; reconcileReason?: string }) {
function StatusBadge({ status, failureReason, failureSource, adminMarkedAction, reconcileReason, espSoftFailure }: { status: string; failureReason?: string; failureSource?: string; adminMarkedAction?: string; reconcileReason?: string; espSoftFailure?: boolean }) {
// Delegate the status pill (+ timeout affordance + "manual"/"reconciled" badges) to the shared,
// canonical SessionStatusBadge so this page picks up every status — incl. the AwaitingUser/Incomplete
// reclassification states — from one source instead of a divergent local map.
const badge = (
<SessionStatusBadge status={status} failureReason={failureReason} adminMarkedAction={adminMarkedAction} reconcileReason={reconcileReason} />
<SessionStatusBadge status={status} failureReason={failureReason} adminMarkedAction={adminMarkedAction} reconcileReason={reconcileReason} espSoftFailure={espSoftFailure} />
);
// Session-detail-only affordance: link a rule-attributed failure back to its analyze rule.
@@ -5,7 +5,9 @@
* failure gets a small ⏱️ affordance. When {@link adminMarkedAction} is set (an administrator manually flipped
* the terminal state), a small "manual" badge is appended. When {@link reconcileReason} is set (the BACKEND
* declared the success — timeout-sweep reconcile or late-completion upgrade), a small "reconciled" badge is
* appended with the justification as tooltip — otherwise the bare pill is returned unchanged.
* appended with the justification as tooltip. When {@link espSoftFailure} is set (the session completed while
* an ESP-failure advisory was unresolved — Continue-Anyway observation / advisory defang), an amber
* "with issues" badge is appended — otherwise the bare pill is returned unchanged.
*/
const STATUS_CONFIG: Record<string, { color: string; text: string }> = {
@@ -26,6 +28,7 @@ export function SessionStatusBadge({
failureReason,
adminMarkedAction,
reconcileReason,
espSoftFailure,
}: {
status: string;
failureReason?: string | null;
@@ -33,6 +36,8 @@ export function SessionStatusBadge({
adminMarkedAction?: string | null;
/** Set when the backend declared the success (sweep reconcile / late-completion upgrade) — appends a "reconciled" badge. */
reconcileReason?: string | null;
/** Set when the session succeeded despite an unresolved ESP failure (Continue-Anyway) — appends an amber "with issues" badge. */
espSoftFailure?: boolean | null;
}) {
const config = STATUS_CONFIG[status] || STATUS_CONFIG.Unknown;
// The ⏱️ affordance marks the silence/timeout family: a "timed out" Failed, or any Incomplete
@@ -54,8 +59,12 @@ export function SessionStatusBadge({
// Backend-declared success: adminMarkedAction wins (the "manual" badge already attributes the
// flip); a reconciled badge on top would double-attribute the same verdict.
const showReconciled = !adminMarkedAction && status === "Succeeded" && !!reconcileReason;
// Soft failure: the device reached a working desktop, but the ESP gave up on at least one
// blocking item first (user pressed "Continue anyway"). Amber to match the semantic weight —
// enrolled and usable, yet not everything the ESP tracked finished inside its window.
const showSoftFailure = status === "Succeeded" && !!espSoftFailure;
if (!adminMarkedAction && !showReconciled) return pill;
if (!adminMarkedAction && !showReconciled && !showSoftFailure) return pill;
return (
<span className="inline-flex items-center gap-1.5">
@@ -67,13 +76,21 @@ export function SessionStatusBadge({
>
manual
</span>
) : (
) : showReconciled ? (
<span
className="px-1.5 py-0.5 text-[10px] leading-4 font-semibold rounded border border-sky-300 bg-sky-50 text-sky-700 dark:border-sky-700 dark:bg-sky-900/40 dark:text-sky-300"
title={reconcileReason || undefined}
>
reconciled
</span>
) : null}
{showSoftFailure && (
<span
className="px-1.5 py-0.5 text-[10px] leading-4 font-semibold rounded border border-amber-300 bg-amber-50 text-amber-700 dark:border-amber-700 dark:bg-amber-900/40 dark:text-amber-300"
title="ESP reported a terminal failure, but the user most likely pressed 'Continue anyway' and reached the desktop — at least one blocking item did not finish inside the ESP window"
>
with issues
</span>
)}
</span>
);
@@ -18,6 +18,10 @@ export interface Session {
failureSource?: string;
/** Non-empty only when the BACKEND declared this session Succeeded (timeout-sweep reconcile or late-completion upgrade) — carries the justification so operators can tell it apart from an agent-reported success. */
reconcileReason?: string;
/** True only on a Succeeded session that completed while an ESP failure was unresolved — the user most likely pressed "Continue anyway". Rendered as the amber "with issues" badge. */
espSoftFailure?: boolean;
/** How a soft-failure completion was detected: "continue_anyway_observation" | "continue_anyway_post_accountsetup". Empty on clean successes. */
completionSource?: string;
/** Non-null only when an administrator flipped the session manually via the portal. Values: "Succeeded" | "Failed". */
adminMarkedAction?: string;
/** Backend device-validation path that accepted the device at registration: "AutopilotV1" | "CorporateIdentifier" | "DeviceAssociation" | "Bootstrap" — absent for sessions before this feature or tenants with device validation off. */