Compare commits

...

7 Commits

Author SHA1 Message Date
Oliver Kieselbach e8ef4f53ae feat(agent): OOBE-state observability via WinRT SystemSetupInfo
Sample Windows.System.Profile.SystemSetupInfo.OutOfBoxExperienceState
(build >= 17763, SYSTEM-callable, CLR-native WinRT projection — no NuGet)
as a purely observational signal, mirroring the bootstrap guard marker:

- oobeStateAtAgentStart on agent_started and agent_late_start payloads:
  makes late starts provable instead of heuristic ("completed" on a
  first_run = OOBE was already over when the agent arrived) and validates
  the bootstrap OOBE guard fleet-wide.
- One-shot oobe_state_completed trace event on the in_progress->completed
  flip, piggybacked on the DesktopArrivalDetector 30s poll: an
  owner-independent desktop corroboration for DAD-starved sessions
  (4d5a0b78 class — WTS+WMI owner resolution dead, desktop_arrived never
  fires). Often absent in healthy sessions since the DAD timer is
  disposed at desktop arrival, which is when the flip lands.

OobeStateReader is fail-soft (never throws, "unavailable" on missing
contract); the flip check has its own try/catch so it cannot pollute the
DAD wmiErrorCount liveness diagnostics, "unavailable" never overwrites
the baseline, and the one-shot flag survives ResetForRealUserSwitch.
No decision-engine involvement, no config/backend changes. Event type
registered in Constants.EventTypes, MCP resource-catalog (phase_events)
and web gather-rule autocomplete.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:39:10 +02:00
Oliver Kieselbach a0d10b8471 test(cleanup): end-to-end incident replays for the self-destruct script
Six integration tests execute the generated cleanup script with a real
powershell.exe against GUID-unique scratch trees — proving what the string
pins cannot:

- locked tree ⇒ complete abort, marker + binaries survive (the 2026-07-22
  regression: the old script force-deleted around the locks)
- full incident replay: a renamed second process holds a file in the tree,
  is found by NAME, force-killed after the wait deadline, cleanup then
  completes with nothing left behind
- fresh foreign lock ⇒ loser backs off without touching the tree; stale
  lock (>10min) is taken over and cleanup proceeds
- happy path + keepLogs variant

BuildCleanupScript gains test-only timing parameters (defaults unchanged
for production) so the replays run in seconds. Agent suite 1969/1969.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 10:06:29 +02:00
Oliver Kieselbach ad3632e9fb fix(emergency-break): survivable self-destruct, reliable break report, ops visibility
Incident 2026-07-22: the 48h emergency break left exe + loaded DLLs orphaned
forever — a second agent instance held the images while the cleanup script
(waiting on the launcher PID only) force-deleted marker + state around the
locks AFTER removing the Scheduled Task, destroying the retry path.

Agent — cleanup script (LIFE-F7):
- wait for ALL agent processes by name (PID-reuse-proof), then force-kill
  stragglers; single-flight lock file serializes concurrent cleanup scripts
- probe-rename (GUID target) BEFORE task removal; any failure renames back
  and aborts with task + marker + files intact for a next-boot retry; the
  destructive Remove-Item-on-original fallback is gone
- script generation extracted to BuildCleanupScript, pinned by 13 tests;
  all 4 variants parser-validated + E2E-smoked (happy path, locked tree)

Agent — break report delivery:
- EmergencyReporter counts DELIVERED reports: reservation rolls back on
  failure (mark-before-send permanently suppressed a key after its first
  transport failure); optional in-reservation retries
- break path: NIC wait <=15s, 3 attempts x 15s timeout (survives Functions
  cold starts observed as >5s 503s), 75s overall bound

Backend:
- backend-materialized events (agent_emergency_break, session_timeout) now
  upsert the cross-session EventType index — they were invisible to every
  search-by-eventType surface (portal + MCP)
- new AgentEmergencyBreak ops event (Agent/Warning, once per session) so
  operators see silently-lost agents; dual-registered in OPS_EVENT_TYPES
- materialization extracted to a static seam, covered by 4 new tests

Agent 1963/1963, Backend 3198/3198, tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 09:56:46 +02:00
Oliver Kieselbach 05e1569503 feat(agent): wildcard support for Local Admin Analyzer allowed accounts
Allowed Local Accounts entries may now contain glob wildcards (* / ?),
matching the hardware-whitelist semantics: Regex.Escape conversion,
case-insensitive, 100ms ReDoS timeout. Applies to both the account and
the profile-folder check. Entries without wildcard chars keep exact
matching; * and ? are invalid in Windows account names, so patterns and
literals cannot collide. UI documents the syntax and rejects patterns
with fewer than 2 literal characters (a bare * would disable the
analyzer). No schema change - old agents treat patterns as literals and
keep alerting (fail-safe).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 09:47:13 +02:00
Oliver Kieselbach ba3581e253 bootstrapper added download stopwatch 2026-07-23 09:35:30 +02:00
Oliver Kieselbach 829ec4c969 fix(release): stamp Cache-Control no-cache on every blob upload
The first release after the download-alias migration exposed Front Door
caching as a correctness hazard: the alias kept serving the previous
version.json (X-Cache: TCP_REMOTE_HIT) after the origin had rotated to
2.0.1370, pinning the portal's version badge to the old build. The dangerous
variant is worse than a stale badge -- manifest and ZIP are independent cache
entries, so a stale manifest paired with a fresh ZIP fails the bootstrap's
SHA-256 integrity check and the install dies.

Route caching on /agent/* has been disabled (infra side); this commit is the
defense-in-depth half: every upload now carries x-ms-blob-cache-control
no-cache -- the CI Send-Blob (eu and legacy mirror alike), the local build.ps1
counterpart (gitignored, changed in lockstep), and the versions/*.json
manifest publishes. Uniform no-cache rather than an immutable/mutable split
because every blob in the agent container rotates in place, including the
per-line AutopilotMonitor-Agent-v2.zip -- nothing there is safely cacheable.
Existing blobs get the header with the next release's re-upload.

Log/docs updated in url-registry.md; no code paths changed, YAML and ps1
parse-checked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 09:17:02 +02:00
Oliver Kieselbach b2d2d37c6f feat(realmjoin): detect aborted RJ ESP (missing 110) and release the completion gate
An interactive logon during the RealmJoin first deployment (phase 100/101)
makes RJ reclassify the run as a secondary-user deployment, whose branch
writes 200/210 and never CompletedFirstDeployment (110) — no crash, no
reboot request. The V2 completion gate waited for exactly 110, so these
sessions hung until the 60-min hard timeout (session 224b2087; siblings
558bf067/3cc18724 delayed ~50 min the same way).

- promote realmjoin_phase_changed to a typed RealmJoinPhaseChanged signal
  (dual-emitted by the adapter) so RealmJoinFacts.LastDeploymentPhase is
  persisted across agent restarts
- abort rule: prev in {100,101} and current in {200,210} with no outcome
  yet -> emit realmjoin_first_deployment_incomplete (Warning), cancel the
  hard-timeout deadline, release the gate with outcome
  FirstDeploymentIncomplete; also evaluated on the RealmJoinDetected
  replay path (restart swallows the transition)
- deliberately NOT triggered by 101->0 (RJ service restart may still
  complete the first deployment) nor by a first observation of 210
  (session 6f1959c0: RJ already completed before agent boot)
- fix realmjoin_timeout always claiming "last phase: 0" - the persisted
  phase now feeds the message
- backend EnrollmentTimeoutClassifier counts the new event as a gate
  terminal; MCP resource catalog dual-synced
- CompleteIfDeferredOrBookkeep widened to a leading-effect list

Verified by replaying session 224b2087's signal log: gate releases at
15:05:17 (101->200) instead of hanging until the 15:54 deadline.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 09:14:30 +02:00
41 changed files with 1980 additions and 152 deletions
+6 -1
View File
@@ -231,7 +231,12 @@ jobs:
param([string]$Local, [string]$BlobName, [string]$ContentType)
$url = "$containerUrl/$BlobName`?$sasToken"
$bytes = [System.IO.File]::ReadAllBytes($Local)
$headers = @{ "x-ms-blob-type" = "BlockBlob"; "Content-Type" = $ContentType }
# no-cache on EVERY blob: they all rotate in place (even the per-line ZIP is
# overwritten each build), and the download alias fronts them with Front Door.
# Route caching is disabled there, but a stale manifest paired with a fresh ZIP
# (independent cache entries) would fail the bootstrap SHA check -- this header
# keeps that class of bug impossible even if caching is ever re-enabled.
$headers = @{ "x-ms-blob-type" = "BlockBlob"; "Content-Type" = $ContentType; "x-ms-blob-cache-control" = "no-cache" }
if ($ContentType -eq "application/zip") {
$md5 = [System.Security.Cryptography.MD5]::Create()
$headers["Content-MD5"] = [Convert]::ToBase64String($md5.ComputeHash($bytes))
+2 -1
View File
@@ -117,6 +117,7 @@ jobs:
$sas = '${{ secrets.AZURE_BLOB_VERSIONS_SAS_TOKEN }}'.TrimStart('?')
$url = "${{ env.VERSIONS_CONTAINER }}/backend.json?$sas"
Invoke-RestMethod -Uri $url -Method Put -Headers @{ 'x-ms-blob-type' = 'BlockBlob' } -Body $manifest -ContentType 'application/json' | Out-Null
# no-cache: rotating pointer, must never be served stale by any fronting cache.
Invoke-RestMethod -Uri $url -Method Put -Headers @{ 'x-ms-blob-type' = 'BlockBlob'; 'x-ms-blob-cache-control' = 'no-cache' } -Body $manifest -ContentType 'application/json' | Out-Null
Write-Host "Published versions/backend.json"
Write-Host $manifest
+2 -1
View File
@@ -141,6 +141,7 @@ jobs:
$sas = '${{ secrets.AZURE_BLOB_VERSIONS_SAS_TOKEN }}'.TrimStart('?')
$url = "${{ env.VERSIONS_CONTAINER }}/mcp.json?$sas"
Invoke-RestMethod -Uri $url -Method Put -Headers @{ 'x-ms-blob-type' = 'BlockBlob' } -Body $manifest -ContentType 'application/json' | Out-Null
# no-cache: rotating pointer, must never be served stale by any fronting cache.
Invoke-RestMethod -Uri $url -Method Put -Headers @{ 'x-ms-blob-type' = 'BlockBlob'; 'x-ms-blob-cache-control' = 'no-cache' } -Body $manifest -ContentType 'application/json' | Out-Null
Write-Host "Published versions/mcp.json"
Write-Host $manifest
+11 -2
View File
@@ -137,8 +137,17 @@ false` and an arm holds), `EspExiting`, `ImeUserSessionCompleted`,
advisory backstop. All routes converge on `CompleteThroughFinalizingOrDefer`:
1. **Completion gates** (currently only the RealmJoin AND-gate): if RealmJoin was
detected and is neither resolved nor timed-out (hard 60-min timeout), completion is
*deferred* — the session stays in stage and emits `completion_waiting`.
detected and has no terminal outcome yet, completion is *deferred* — the session
stays in stage and emits `completion_waiting`. Three terminals open the gate:
`Resolved` (DeploymentPhase 110), `FirstDeploymentIncomplete` (phase left the
first-deployment window 100/101 for 200/210 without 110 — the RJ ESP was aborted,
typically by an interactive logon reclassifying the run as secondary-user; emits
the `realmjoin_first_deployment_incomplete` Warning, session 224b2087), and
`Timeout` (hard 60-min deadline). The abort rule reads the *persisted*
`RealmJoinFacts.LastDeploymentPhase` (updated by the typed `RealmJoinPhaseChanged`
signal), so it survives agent restarts — the `RealmJoinDetected` replay path
evaluates it too. A first observation of 210 does **not** release (RJ may have
completed before agent boot and deploy afterwards).
2. `Finalizing` stage + ~5 s `FinalizingGrace` deadline.
3. Grace fires → `Completed`, `EnrollmentComplete` outcome, `enrollment_complete`
timeline event with full audit payload.
+4
View File
@@ -2,6 +2,10 @@
## 2026-07-23
* **Update**: `url-registry.md` — the first release after the alias migration surfaced Front Door caching as a correctness hazard: the alias served the previous `version.json` (`X-Cache: TCP_REMOTE_HIT`) after the origin had rotated, and since manifest and ZIP are independent cache entries, a stale manifest with a fresh ZIP fails the bootstrap SHA-256 check. Route caching on `/agent/*` is now disabled and every blob upload stamps `Cache-Control: no-cache` (CI `Send-Blob`, local `build.ps1`, `versions/*.json` manifests) — all agent-container blobs rotate in place, so nothing there is safely cacheable.
* **Update**: `agent/decision-engine.md` — the RealmJoin completion gate has a third terminal, `FirstDeploymentIncomplete`. Session 224b2087 showed why: an interactive logon 17 s after DeploymentPhase 101 makes RealmJoin reclassify the run as a secondary-user deployment, whose branch writes 200/210 and never 110 — the gate stayed closed until the 60-min hard timeout although Hello and desktop were long resolved. `realmjoin_phase_changed` is now a typed `RealmJoinPhaseChanged` signal, so `RealmJoinFacts.LastDeploymentPhase` is persisted across agent restarts (this also fixes every `realmjoin_timeout` event claiming "last phase: 0"); a transition out of 100/101 into 200/210 releases the gate with the `realmjoin_first_deployment_incomplete` Warning, and the `RealmJoinDetected` replay path evaluates the same rule for restarts that swallow the transition. A first observation of 210 deliberately does not release — session 6f1959c0 started at 210 and only deployed afterwards.
* **Creation**: Added `url-registry.md` — all well-known hosts (own subdomains, both blob accounts, API origin, Graph, Entra login) are now defined once per component (`Constants.cs`, web `utils/config.ts`, MCP `src/config.ts`) and referenced everywhere else; three guard tests fail the build on any literal outside the registries. Written because the EU cutover missed two hardcoded copies of the legacy blob host — `LatestVersionsService` (feeds the portal's version badge) and `ValidateBootstrapCodeFunction` — which kept working only through the fail-soft legacy mirror in `build-agent.yml`; those two plus the agent `SelfUpdater` now use the `download.autopilotmonitor.com` alias (verified serving all four artifacts first). The portal's bootstrap allow-list accepts alias AND legacy host during the transition, which imposes a deploy order: web before (or with) backend. The sweep also surfaced a fourth copy of the www/portal host pair in `middleware.ts` (now derived from the registry via `hostRouting.ts`) and a backend short-link built on the bare apex domain, which only worked through a registrar-level 301.
## 2026-07-22
+11
View File
@@ -60,6 +60,17 @@ backend that serves alias URLs — an old portal build rejects the alias host an
the bootstrap flow dies client-side. Remove the legacy hostname from the
allow-list only after the legacy account is torn down.
**Front Door caching is a correctness hazard here, not a staleness nuisance.**
The first release after the migration exposed it: the alias kept serving the
previous `version.json` (`X-Cache: TCP_REMOTE_HIT`) while the origin was already
updated — and because manifest and ZIP are independent cache entries, a stale
manifest paired with a fresh ZIP fails the bootstrap's SHA-256 check. Two
defenses, both in place since 2026-07-23: route caching on `/agent/*` is
DISABLED, and every blob upload (CI `Send-Blob`, local `build.ps1`, the
`versions/*.json` manifests) stamps `Cache-Control: no-cache` — all these blobs
rotate in place, even the per-line ZIP. If caching is ever re-enabled, the
headers keep the mismatch class impossible.
Deliberately still on the legacy account: the `HealthCheckService` /
`MaintenanceService` legacy-keepalive probes and the fail-soft mirror in
`build-agent.yml` — they exist precisely to keep already-deployed customer
@@ -272,13 +272,16 @@ try {
$downloadAttempt++
try {
Write-Log "Download attempt ${downloadAttempt}/${maxDownloadAttempts}"
$downloadStopwatch = [System.Diagnostics.Stopwatch]::StartNew()
Invoke-WebRequest `
-Uri $AgentDownloadUrl `
-OutFile $zipPath `
-UseBasicParsing `
-TimeoutSec 30 `
-ErrorAction Stop
Write-Log "Downloaded agent to $zipPath"
$downloadStopwatch.Stop()
$downloadSeconds = [math]::Round($downloadStopwatch.Elapsed.TotalSeconds, 1)
Write-Log "Downloaded agent to $zipPath (took ${downloadSeconds}s)"
break
}
catch {
@@ -0,0 +1,71 @@
#nullable enable
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Analyzers
{
/// <summary>
/// LocalAdminAnalyzer.MatchesAllowedEntry — the allowed-list matching contract.
/// Entries without wildcard characters match exactly (case-insensitive); entries
/// containing * or ? are glob patterns with the same semantics as the hardware
/// whitelist. The same matcher is applied to local accounts and profile folders.
/// </summary>
public sealed class LocalAdminAnalyzerTests
{
// -------------------------------------------------------------- literal entries
[Theory]
[InlineData("SupportAdmin", "SupportAdmin")]
[InlineData("supportadmin", "SupportAdmin")]
[InlineData("SUPPORTADMIN", "supportadmin")]
public void Literal_entry_matches_exactly_case_insensitive(string name, string entry)
{
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
[Theory]
[InlineData("SupportAdmin2", "SupportAdmin")]
[InlineData("SupportAdmin", "SupportAdmin2")]
[InlineData("Support", "SupportAdmin")]
public void Literal_entry_does_not_substring_match(string name, string entry)
{
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
// -------------------------------------------------------------- glob patterns
[Theory]
[InlineData("adm-12345", "adm-*")]
[InlineData("adm-", "adm-*")]
[InlineData("ADM-9F3B2C", "adm-*")]
[InlineData("lapsAdmin01", "lapsAdmin??")]
[InlineData("svc-backup-01", "svc-*-01")]
public void Wildcard_entry_matches_glob(string name, string entry)
{
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
[Theory]
[InlineData("administrator2", "adm-*")] // no literal "adm-" prefix
[InlineData("xadm-123", "adm-*")] // pattern is anchored at start
[InlineData("lapsAdmin1", "lapsAdmin??")] // ? is exactly one char
[InlineData("adm.123", "adm-*")] // "-" is literal, not a regex class
public void Wildcard_entry_does_not_overmatch(string name, string entry)
{
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
[Fact]
public void Regex_metacharacters_in_entry_are_treated_literally()
{
// "adm.123" as an entry has no wildcard chars → exact match only,
// the dot must not act as a regex any-char.
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry("adm.123", "adm.123"));
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry("admX123", "adm.123"));
// Mixed: wildcard present, remaining metacharacters stay literal.
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry("adm.123", "adm.*"));
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry("admX123", "adm.*"));
}
}
}
@@ -405,5 +405,144 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
var started = trace.Events.Find(e => e.EventType == SharedEventTypes.DesktopDetectorStarted);
Assert.Equal(0, started.Data["noCandidateThresholdMinutes"]);
}
// ============================================================================
// OOBE-state flip (oobe_state_completed) — observational one-shot on the poll
// ============================================================================
// The WinRT OutOfBoxExperienceState flip in_progress->completed is an owner-
// independent desktop corroboration for DAD-starved sessions (4d5a0b78 class).
// Tests drive polls synchronously via PollOnceForTest after Stop() so the real
// timer cannot race, and pin the owner seams to an excluded user so a real
// explorer.exe on the test machine can never set _desktopArrived (which would
// short-circuit PollForDesktop before the flip check).
private static DesktopArrivalDetector BuildFlipDetector(TempDirectory tmp, TraceCapture trace, System.Func<string> oobeProvider)
{
var detector = new DesktopArrivalDetector(new AgentLogger(tmp.Path, AgentLogLevel.Info), noCandidateTimeoutMinutes: 0)
{
OnTraceEvent = trace.Sink,
OobeStateProvider = oobeProvider,
SessionOwnerResolver = _ => "SYSTEM",
ProcessOwnerResolver = _ => "SYSTEM",
};
return detector;
}
/// <summary>Returns the queued states in order, then repeats the last one forever.</summary>
private static System.Func<string> StateQueue(params string[] states)
{
var queue = new Queue<string>(states);
var last = states[states.Length - 1];
return () => queue.Count > 0 ? queue.Dequeue() : last;
}
[Fact]
public void OobeFlip_emits_exactly_once_with_payload()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
// Baseline sample happens synchronously inside Start()/ArmTimer.
using var detector = BuildFlipDetector(tmp, trace, StateQueue("in_progress", "completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest(); // reads "completed" -> flip
detector.PollOnceForTest(); // still "completed" -> no second emit
Assert.Equal(1, trace.CountOf(SharedEventTypes.OobeStateCompleted));
var flip = trace.Events.Find(e => e.EventType == SharedEventTypes.OobeStateCompleted);
Assert.Equal("in_progress", flip.Data["previousState"]);
Assert.True(flip.Data.ContainsKey("pollCount"));
Assert.True(flip.Data.ContainsKey("minutesSinceStart"));
Assert.False((bool)flip.Data["desktopArrived"]);
Assert.True(flip.Data.ContainsKey("wmiErrorCount"));
Assert.True(flip.Data.ContainsKey("wtsErrorCount"));
}
[Fact]
public void OobeFlip_does_not_emit_when_state_is_already_completed()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace, StateQueue("completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest();
detector.PollOnceForTest();
// Productive machines read "completed" from the very first sample — the flip
// event must never fire there (the agent_started field already records the state).
Assert.Equal(0, trace.CountOf(SharedEventTypes.OobeStateCompleted));
}
[Fact]
public void OobeFlip_does_not_emit_when_in_progress_was_never_seen()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace, StateQueue("unavailable", "completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest();
// Old builds without the WinRT contract report "unavailable"; a later
// "completed" without a positively observed "in_progress" is not a flip.
Assert.Equal(0, trace.CountOf(SharedEventTypes.OobeStateCompleted));
}
[Fact]
public void OobeFlip_survives_transient_unavailable_between_polls()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace, StateQueue("in_progress", "unavailable", "completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest(); // transient reflection failure — must NOT clear the baseline
detector.PollOnceForTest(); // "completed" -> flip still detected
Assert.Equal(1, trace.CountOf(SharedEventTypes.OobeStateCompleted));
}
[Fact]
public void OobeFlip_provider_throwing_never_emits_and_never_pollutes_wmi_error_count()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace, () => throw new System.InvalidOperationException("boom"));
detector.Start(); // baseline sample throws -> swallowed
detector.Stop();
detector.PollOnceForTest();
detector.PollOnceForTest();
Assert.Equal(0, trace.CountOf(SharedEventTypes.OobeStateCompleted));
// The flip check has its own try/catch — a throw must not fall into
// PollForDesktop's outer catch, which increments the WMI error counter the
// session-4d5a0b78 liveness diagnostics rely on. first_poll carries that counter.
var firstPoll = trace.Events.Find(e => e.EventType == SharedEventTypes.DesktopDetectorFirstPoll);
Assert.Equal(0, firstPoll.Data["wmiErrorCount"]);
}
[Fact]
public void OobeFlip_one_shot_flag_survives_ResetForRealUserSwitch()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace,
StateQueue("in_progress", "completed", "in_progress", "completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest(); // flip #1
detector.ResetForRealUserSwitch(); // re-arms + re-samples baseline ("in_progress")
detector.Stop();
detector.PollOnceForTest(); // "completed" again — must NOT re-emit
Assert.Equal(1, trace.CountOf(SharedEventTypes.OobeStateCompleted));
}
}
}
@@ -0,0 +1,40 @@
#nullable enable
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
{
/// <summary>
/// Smoke tests for the WinRT <c>SystemSetupInfo.OutOfBoxExperienceState</c> reader.
/// The actual state on the test machine is unknown (CI runners are post-OOBE; Server
/// SKUs may lack the UniversalApiContract entirely), so only the fail-soft contract
/// is pinned: never throw, always return a member of the known value set.
/// </summary>
public sealed class OobeStateReaderTests
{
[Fact]
public void Read_never_throws_and_returns_known_value()
{
var value = OobeStateReader.Read();
Assert.False(string.IsNullOrEmpty(value));
Assert.True(
value == "not_started"
|| value == "in_progress"
|| value == "completed"
|| value == OobeStateReader.Unavailable
|| value.StartsWith("unknown_"),
$"unexpected reader value '{value}'");
}
[Fact]
public void Read_is_stable_across_repeated_calls()
{
// Exercises the cached-PropertyInfo path (second call skips resolution).
var first = OobeStateReader.Read();
var second = OobeStateReader.Read();
Assert.Equal(first, second);
}
}
}
@@ -0,0 +1,212 @@
using System;
using System.Diagnostics;
using System.IO;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
using AutopilotMonitor.Agent.V2.Core.Security;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
{
/// <summary>
/// Executes the generated cleanup script against REAL scratch trees with a real
/// powershell.exe — end-to-end replays of the 2026-07-22 incident mechanics that the
/// string pins in <see cref="CleanupServiceScriptTests"/> cannot prove:
/// <list type="bullet">
/// <item>a locked tree must abort WITHOUT deleting anything (old script force-deleted
/// marker + state around the locks and orphaned the binaries),</item>
/// <item>a straggler agent process (the incident's second instance) must be found by
/// NAME, killed, and the cleanup must then complete,</item>
/// <item>concurrent cleanup scripts must be serialized by the lock file.</item>
/// </list>
/// Timings are dialed down via the test-only BuildCleanupScript parameters so each case
/// runs in seconds. Every test uses GUID-unique process/task/tree names so nothing real
/// (a dev-box agent, a real Scheduled Task) can ever be touched.
/// </summary>
[Collection("SerialThreading")]
public class CleanupServiceScriptIntegrationTests : IDisposable
{
private readonly string _root;
private readonly string _basePath;
private readonly string _taskName = $"AmTest-NoSuchTask-{Guid.NewGuid():N}";
private readonly string _fakeProcessName = $"AmTestAgent{Guid.NewGuid().ToString("N").Substring(0, 8)}";
public CleanupServiceScriptIntegrationTests()
{
_root = Path.Combine(Path.GetTempPath(), $"am-cleanup-it-{Guid.NewGuid():N}");
_basePath = Path.Combine(_root, "AutopilotMonitor");
Directory.CreateDirectory(Path.Combine(_basePath, "Agent"));
Directory.CreateDirectory(Path.Combine(_basePath, "State"));
Directory.CreateDirectory(Path.Combine(_basePath, "Logs"));
File.WriteAllText(Path.Combine(_basePath, "Agent", "fake.exe"), "x");
File.WriteAllText(Path.Combine(_basePath, "State", "enrollment-complete.marker"), "x");
File.WriteAllText(Path.Combine(_basePath, "Logs", "agent.log"), "x");
}
public void Dispose()
{
try { Directory.Delete(_root, recursive: true); } catch { }
}
private static string LockPath => Path.Combine(Path.GetTempPath(), "AutopilotMonitor-Cleanup.lock");
private string BuildScript(bool keepLogs = false, int processWaitSeconds = 1)
=> CleanupService.BuildCleanupScript(
agentProcessName: _fakeProcessName,
agentBasePath: _basePath,
keepLogs: keepLogs,
scheduledTaskName: _taskName,
rebootOnComplete: false,
initialDelaySeconds: 0,
processWaitSeconds: processWaitSeconds,
renameRetries: 2,
renameRetryDelaySeconds: 1);
/// <summary>Runs the script synchronously in a real powershell.exe. Returns the script path.</summary>
private string RunScript(string script)
{
var scriptPath = Path.Combine(_root, $"cleanup-{Guid.NewGuid():N}.ps1");
File.WriteAllText(scriptPath, script);
using (var proc = Process.Start(new ProcessStartInfo
{
FileName = SystemPaths.PowerShell,
Arguments = $"-NoProfile -ExecutionPolicy Bypass -File \"{scriptPath}\"",
UseShellExecute = false,
CreateNoWindow = true,
}))
{
Assert.True(proc.WaitForExit(90_000), "cleanup script did not finish within 90s");
}
return scriptPath;
}
[Fact]
public void Happy_path_deletes_tree_and_removes_lock_and_script()
{
var scriptPath = RunScript(BuildScript());
Assert.False(Directory.Exists(_basePath), "tree should be fully deleted");
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
Assert.False(File.Exists(scriptPath), "script should self-delete");
Assert.False(File.Exists(LockPath), "cleanup lock should be released");
}
[Fact]
public void Locked_tree_aborts_completely_marker_and_files_survive()
{
// The incident regression: with a file locked inside the tree the OLD script
// deleted everything else (incl. the enrollment-complete marker) and left the
// binaries orphaned with the task already gone. The hardened script must treat
// the failed probe-rename as "someone still holds this" and walk away entirely.
using (File.Open(Path.Combine(_basePath, "Agent", "fake.exe"),
FileMode.Open, FileAccess.Read, FileShare.None))
{
RunScript(BuildScript());
}
Assert.True(File.Exists(Path.Combine(_basePath, "Agent", "fake.exe")), "binaries must stay in place");
Assert.True(File.Exists(Path.Combine(_basePath, "State", "enrollment-complete.marker")),
"marker must survive so the next boot retries cleanup");
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
Assert.False(File.Exists(LockPath));
}
[Fact]
public void Straggler_process_holding_the_tree_is_killed_and_cleanup_completes()
{
// Full incident replay: a SECOND agent-named process (here: a renamed
// powershell.exe, matching how only the process NAME identifies agents) holds a
// file in the tree open. The old script waited on the launcher PID only and ran
// its force-delete against the live locks. The hardened script must find the
// straggler by name, kill it after the wait deadline, and then delete everything.
var fakeExe = Path.Combine(_root, _fakeProcessName + ".exe");
File.Copy(SystemPaths.PowerShell, fakeExe);
var lockTarget = Path.Combine(_basePath, "Agent", "fake.exe").Replace("'", "''");
Process straggler = null;
try
{
straggler = Process.Start(new ProcessStartInfo
{
FileName = fakeExe,
Arguments = "-NoProfile -Command \"$f = [IO.File]::Open('" + lockTarget +
"', 'Open', 'Read', 'None'); Start-Sleep -Seconds 120\"",
UseShellExecute = false,
CreateNoWindow = true,
});
// Give the straggler time to acquire its file handle before cleanup starts.
var acquireDeadline = DateTime.UtcNow.AddSeconds(20);
var acquired = false;
while (!acquired && DateTime.UtcNow < acquireDeadline)
{
try
{
using (File.Open(Path.Combine(_basePath, "Agent", "fake.exe"),
FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
}
}
catch (IOException)
{
acquired = true; // share violation ⇒ straggler holds the file
}
if (!acquired) System.Threading.Thread.Sleep(200);
}
Assert.True(acquired, "straggler never acquired its file lock");
RunScript(BuildScript(processWaitSeconds: 2));
Assert.True(straggler.HasExited, "straggler must be force-killed by the name-based backstop");
Assert.False(Directory.Exists(_basePath), "after the kill the tree must be fully deleted");
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
}
finally
{
try { if (straggler != null && !straggler.HasExited) straggler.Kill(); } catch { }
try { straggler?.Dispose(); } catch { }
}
}
[Fact]
public void Fresh_foreign_lock_backs_off_without_touching_the_tree()
{
File.WriteAllText(LockPath, "held by another cleanup");
try
{
var scriptPath = RunScript(BuildScript());
Assert.True(File.Exists(Path.Combine(_basePath, "Agent", "fake.exe")),
"loser script must not touch the tree — the lock holder owns the deletion");
Assert.True(File.Exists(LockPath), "foreign lock must not be deleted by the loser");
Assert.False(File.Exists(scriptPath), "loser still self-deletes its script");
}
finally
{
try { File.Delete(LockPath); } catch { }
}
}
[Fact]
public void Stale_foreign_lock_is_taken_over_and_cleanup_proceeds()
{
File.WriteAllText(LockPath, "crashed cleanup from a previous boot");
File.SetLastWriteTime(LockPath, DateTime.Now.AddMinutes(-15));
RunScript(BuildScript());
Assert.False(Directory.Exists(_basePath), "stale lock must be taken over, cleanup must run");
Assert.False(File.Exists(LockPath));
}
[Fact]
public void KeepLogs_variant_deletes_everything_except_logs()
{
RunScript(BuildScript(keepLogs: true));
Assert.True(File.Exists(Path.Combine(_basePath, "Logs", "agent.log")), "Logs must survive");
Assert.False(Directory.Exists(Path.Combine(_basePath, "Agent")));
Assert.False(Directory.Exists(Path.Combine(_basePath, "State")));
Assert.Empty(Directory.GetFileSystemEntries(_basePath, "*.del-*"));
}
}
}
@@ -0,0 +1,135 @@
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
{
/// <summary>
/// Pins the LIFE-F7 hardening invariants of the self-destruct cleanup script
/// (incident 2026-07-22: a second agent instance held exe + loaded DLLs while the
/// old script — waiting on the launcher PID only — force-deleted around the locks
/// AFTER removing the Scheduled Task, orphaning the binaries with no retry path).
/// These are string-level pins on <see cref="CleanupService.BuildCleanupScript"/>:
/// weakening any of them silently re-introduces the orphaned-binaries failure mode.
/// </summary>
public class CleanupServiceScriptTests
{
private const string ProcessName = "AutopilotMonitor.Agent";
private const string BasePath = @"C:\ProgramData\AutopilotMonitor";
private const string TaskName = "AutopilotMonitorAgent";
private static string Script(bool keepLogs = false, bool reboot = false)
=> CleanupService.BuildCleanupScript(ProcessName, BasePath, keepLogs, TaskName, reboot);
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Waits_by_process_name_not_launcher_pid(bool keepLogs)
{
var script = Script(keepLogs);
// LIFE-F7 #2: name-based wait covers marker-retry / self-update / manual instances
// and is immune to PID reuse. Wait-Process -Id saw exactly one PID and nothing else.
Assert.Contains($"Get-Process -Name '{ProcessName}'", script);
Assert.DoesNotContain("Wait-Process", script);
// Backstop: stragglers are force-killed — their images are deleted right after.
Assert.Contains("Stop-Process -Force", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Never_deletes_the_original_path(bool keepLogs)
{
var script = Script(keepLogs);
// LIFE-F7 #3: deletion happens exclusively on successfully renamed targets
// ($entry.Renamed). A Remove-Item against the original tree is what deleted
// marker + state around locked binaries in the incident.
Assert.DoesNotContain($"Remove-Item -Path '{BasePath}'", script);
Assert.Contains("Remove-Item -Path $entry.Renamed", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Task_removal_is_gated_on_probe_and_deletion_on_task_gone(bool keepLogs)
{
var script = Script(keepLogs);
// Probe-rename first; only a successful probe reaches task removal, and only a
// removed task reaches deletion. Both failure branches rename BACK so the next
// boot retries with task + marker + files intact.
Assert.Contains("if ($probeOk)", script);
Assert.Contains("if ($taskGone)", script);
Assert.Contains("Rename-Item -Path $entry.Renamed -NewName $entry.Original", script);
// LIFE-F6 task-removal loop + module-independent fallback stay intact.
Assert.Contains("Unregister-ScheduledTask -TaskName $taskName", script);
Assert.Contains("schtasks.exe /Delete /TN $taskName /F", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Concurrent_cleanup_scripts_are_serialized_by_lock_file(bool keepLogs)
{
var script = Script(keepLogs);
// LIFE-F7 #1: CreateNew is the atomic single-flight gate; the loser exits without
// touching the tree. Stale locks (>10 min) are taken over.
Assert.Contains("AutopilotMonitor-Cleanup.lock", script);
Assert.Contains("[System.IO.FileMode]::CreateNew", script);
Assert.Contains("$lockAge.TotalMinutes -gt 10", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Rename_targets_carry_a_guid_suffix(bool keepLogs)
{
// A fixed '.del' target collides with leftovers of earlier attempts and with a
// concurrent script's rename — the GUID makes every attempt's target unique.
Assert.Contains(".del-' + [Guid]::NewGuid().ToString('N')", Script(keepLogs));
}
[Fact]
public void KeepLogs_variant_excludes_logs_directory()
{
var script = Script(keepLogs: true);
Assert.Contains("-Exclude 'Logs'", script);
Assert.DoesNotContain($"Rename-Item -Path '{BasePath}' -NewName", script);
}
[Fact]
public void Full_variant_renames_the_whole_tree()
{
var script = Script(keepLogs: false);
Assert.Contains($"Rename-Item -Path '{BasePath}' -NewName $renamedPath", script);
Assert.DoesNotContain("-Exclude 'Logs'", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Reboot_flag_controls_shutdown_invocation(bool reboot)
{
var script = Script(keepLogs: false, reboot: reboot);
if (reboot)
Assert.Contains("shutdown.exe /r /t 10", script);
else
Assert.DoesNotContain("shutdown.exe", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Script_and_lock_are_released_in_finally(bool keepLogs)
{
var script = Script(keepLogs);
Assert.Contains("finally", script);
Assert.Contains("$lockStream.Dispose()", script);
Assert.Contains("Remove-Item -Path $scriptPath -Force", script);
}
}
}
@@ -0,0 +1,143 @@
using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Transport;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Transport
{
/// <summary>
/// Pins the anti-flood semantics of <see cref="EmergencyReporter.TrySendAsync"/> after the
/// delivery-hardening change (2026-07-23): the once-per-session dedup and the per-session
/// budget count DELIVERED reports. The previous mark-before-send permanently suppressed an
/// error key after its first transport failure — fatal for single-shot callers like the 48h
/// emergency break, whose process exits right after the attempt.
/// </summary>
public class EmergencyReporterTests
{
private sealed class FakeApiClient : BackendApiClient
{
public readonly List<TimeSpan?> Timeouts = new List<TimeSpan?>();
public readonly Queue<bool> Results = new Queue<bool>();
public FakeApiClient() : base(
httpClient: new HttpClient(),
baseUrl: "http://localhost",
manufacturer: string.Empty,
model: string.Empty,
serialNumber: string.Empty,
useBootstrapTokenAuth: false,
bootstrapToken: null,
agentVersion: "0.0.0",
logger: null)
{
}
public int Calls => Timeouts.Count;
public override Task<bool> ReportAgentErrorAsync(AgentErrorReport report, TimeSpan? timeout = null)
{
Timeouts.Add(timeout);
return Task.FromResult(Results.Count > 0 && Results.Dequeue());
}
}
private static EmergencyReporter Reporter(FakeApiClient client)
=> new EmergencyReporter(client, "session", "tenant", "0.0.0", logger: null);
[Fact]
public async Task Failed_send_rolls_back_reservation_so_a_later_call_retries()
{
var client = new FakeApiClient();
client.Results.Enqueue(false); // first call fails
client.Results.Enqueue(true); // second call succeeds
var reporter = Reporter(client);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
// Old behaviour: second call suppressed as "already reported" ⇒ 1 call.
Assert.Equal(2, client.Calls);
}
[Fact]
public async Task Delivered_send_is_deduplicated_per_key()
{
var client = new FakeApiClient();
client.Results.Enqueue(true);
var reporter = Reporter(client);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
Assert.Equal(1, client.Calls);
}
[Fact]
public async Task Retries_stay_inside_one_reservation_and_stop_on_success()
{
var client = new FakeApiClient();
client.Results.Enqueue(false);
client.Results.Enqueue(true); // second attempt delivers — third must not fire
var reporter = Reporter(client);
await reporter.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak, "m",
attempts: 3, perAttemptTimeout: TimeSpan.FromMilliseconds(50), retryDelay: TimeSpan.Zero);
Assert.Equal(2, client.Calls);
// Delivered ⇒ dedup holds for the session.
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
Assert.Equal(2, client.Calls);
}
[Fact]
public async Task Per_attempt_timeout_is_forwarded_to_the_api_client()
{
var client = new FakeApiClient();
client.Results.Enqueue(true);
var reporter = Reporter(client);
var timeout = TimeSpan.FromSeconds(15);
await reporter.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak, "m",
attempts: 1, perAttemptTimeout: timeout);
Assert.Equal(timeout, Assert.Single(client.Timeouts));
}
[Fact]
public async Task All_attempts_failing_rolls_back_after_exhausting_attempts()
{
var client = new FakeApiClient(); // queue empty ⇒ every attempt fails
var reporter = Reporter(client);
await reporter.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak, "m",
attempts: 3, retryDelay: TimeSpan.Zero);
Assert.Equal(3, client.Calls);
// Rolled back ⇒ a later call may try again (budget + dedup + cooldown restored).
client.Results.Enqueue(true);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
Assert.Equal(4, client.Calls);
}
[Fact]
public async Task Different_keys_do_not_suppress_each_other_after_rollback()
{
var client = new FakeApiClient();
client.Results.Enqueue(false); // break report fails → rollback (incl. cooldown restore)
client.Results.Enqueue(true); // unrelated report must still go out
var reporter = Reporter(client);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
await reporter.TrySendAsync(AgentErrorType.ConfigFetchFailed, "m");
Assert.Equal(2, client.Calls);
}
}
}
@@ -151,17 +151,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
}
[Fact]
public void TriggerPhaseChangedFromTest_emits_informational_only_no_decision_signal()
public void TriggerPhaseChangedFromTest_emits_typed_signal_and_informational_event()
{
using var f = new Fixture();
using var adapter = new RealmJoinWatcherAdapter(f.Watcher, f.Ingress, f.Clock);
adapter.TriggerPhaseChangedFromTest(prev: 100, curr: 200);
// Phase change is observability-only. There is no DecisionSignalKind.RealmJoinPhaseChanged.
// 1) Typed signal — persists LastDeploymentPhase and drives the aborted-RJ-ESP
// detection in the reducer (session 224b2087). Not Detected/Resolved.
Assert.DoesNotContain(f.Ingress.Posted, p => p.Kind == DecisionSignalKind.RealmJoinDetected);
Assert.DoesNotContain(f.Ingress.Posted, p => p.Kind == DecisionSignalKind.RealmJoinResolved);
var decision = f.Ingress.Posted.Single(p => p.Kind == DecisionSignalKind.RealmJoinPhaseChanged);
Assert.Equal("RealmJoinWatcher", decision.SourceOrigin);
Assert.Equal("200", decision.Payload![DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase]);
Assert.Equal("100", decision.Payload[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase]);
// 2) Dual-emit informational event keeps the existing timeline entry.
var info = f.Ingress.Posted.Single(p =>
p.Kind == DecisionSignalKind.InformationalEvent
&& p.Payload != null
@@ -61,6 +61,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
internal Func<int, string> SessionOwnerResolver { get; set; }
internal Func<int, string> ProcessOwnerResolver { get; set; }
// Test seam for the WinRT OOBE-state sample (null = OobeStateReader.Read).
internal Func<string> OobeStateProvider { get; set; }
// OOBE-state flip tracking (observational only — see OobeStateReader contract).
// _oobeCompletedEmitted is one-shot per agent lifetime and deliberately NOT reset
// by ResetForRealUserSwitch: the OOBE flip is a global OS transition, not per-user.
private string _lastOobeState;
private bool _oobeCompletedEmitted;
/// <summary>
/// System/service account names that should NOT be considered real users.
/// </summary>
@@ -161,6 +170,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
_startedAtUtc = DateTime.UtcNow;
// Baseline OOBE-state sample so the first poll can already detect a flip
// (self-contained try/catch inside, like the Emit* helpers).
CheckOobeStateFlip();
_pollingTimer?.Dispose();
_pollingTimer = new Timer(
PollForDesktop,
@@ -186,6 +199,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
_pollCount++;
CheckOobeStateFlip();
try
{
var explorerProcesses = Process.GetProcessesByName("explorer");
@@ -381,6 +396,66 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
catch (Exception ex) { _logger.Verbose($"DesktopArrivalDetector: detector-started emit failed: {ex.Message}"); }
}
/// <summary>
/// Samples the WinRT OOBE state (<see cref="OobeStateReader"/>) and emits the one-shot
/// <c>oobe_state_completed</c> trace event on the in_progress-&gt;completed flip.
/// Observational only — an owner-independent desktop corroboration for sessions where
/// WTS+WMI owner resolution starves and <c>desktop_arrived</c> never fires. Often absent
/// in healthy sessions: the polling timer is disposed at desktop arrival, which is when
/// the flip lands (empirical probe, session 9c404ae9).
/// <para>
/// Self-contained try/catch: a throw here must NOT reach <see cref="PollForDesktop"/>'s
/// outer catch, which would inflate <see cref="_wmiErrorCountSinceStart"/> and skew the
/// session-4d5a0b78 liveness diagnostics. "unavailable" readings never overwrite the last
/// known state, so a transient reflection failure between two polls cannot mask the flip.
/// One-shot flag survives <see cref="ResetForRealUserSwitch"/> — the flip is a global OS
/// transition, not per-user.
/// </para>
/// </summary>
private void CheckOobeStateFlip()
{
try
{
var current = (OobeStateProvider ?? OobeStateReader.Read)();
if (string.IsNullOrEmpty(current) || current == OobeStateReader.Unavailable)
return;
if (!_oobeCompletedEmitted
&& _lastOobeState == "in_progress"
&& current == "completed")
{
_oobeCompletedEmitted = true;
var minutesSinceStart = Math.Round((DateTime.UtcNow - _startedAtUtc).TotalMinutes, 1);
_logger.Info("DesktopArrivalDetector: OOBE state flipped in_progress->completed (WinRT SystemSetupInfo)");
OnTraceEvent?.Invoke(
SharedEventTypes.OobeStateCompleted,
$"OOBE state flipped in_progress->completed (WinRT SystemSetupInfo, poll {_pollCount}) — owner-independent desktop corroboration.",
new Dictionary<string, object>
{
{ "previousState", "in_progress" },
{ "pollCount", _pollCount },
{ "minutesSinceStart", minutesSinceStart },
{ "desktopArrived", _desktopArrived },
{ "wmiErrorCount", _wmiErrorCountSinceStart },
{ "wtsErrorCount", _wtsErrorCountSinceStart },
});
}
_lastOobeState = current;
}
catch (Exception ex)
{
_logger.Verbose($"DesktopArrivalDetector: OOBE state check failed: {ex.Message}");
}
}
/// <summary>
/// Drives a single synchronous poll for unit tests — the production poll is private and
/// timer-driven with a fixed 30 s interval (precedent: <see cref="ResolveOwner"/> is
/// likewise internal for direct testing).
/// </summary>
internal void PollOnceForTest() => PollForDesktop(null);
/// <summary>
/// Resolves the owning user of an explorer.exe candidate. Session 4d5a0b78 fix
/// (2026-06-11): the WMI <c>Win32_Process.GetOwner</c> call failed on every poll on
@@ -0,0 +1,72 @@
using System;
using System.Reflection;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Interop
{
/// <summary>
/// Reads WinRT <c>Windows.System.Profile.SystemSetupInfo.OutOfBoxExperienceState</c>
/// (NotStarted=0 / InProgress=1 / Completed=2) via the CLR's built-in WinRT projection —
/// no NuGet package, no winmd reference; the type is resolved at runtime with
/// <c>ContentType=WindowsRuntime</c>, the same mechanism the bootstrap script uses from
/// PowerShell 5.1.
/// <para>
/// <b>Contract:</b> observational only — never feed this into decision-engine logic.
/// Requires Windows 10 1809+ (build 17763, UniversalApiContract 7.0) and is callable
/// from the SYSTEM service context (empirically validated, probe session 9c404ae9).
/// On older builds or SKUs without the contract the type resolves to null and
/// <see cref="Read"/> returns <c>"unavailable"</c>. <see cref="Read"/> never throws.
/// </para>
/// <para>
/// Empirical timing (Win11 24H2 probe): the state flips InProgress→Completed at desktop
/// arrival — it is a corroboration signal, not an early one. WDP reseal and placeholder
/// (fooUser) flip semantics are unverified; another reason this must stay observational.
/// </para>
/// </summary>
internal static class OobeStateReader
{
public const string Unavailable = "unavailable";
// Resolved once per process. Benign race: concurrent first calls resolve the same
// PropertyInfo and the assignment is idempotent — no lock needed.
private static PropertyInfo _stateProperty;
private static bool _initTried;
/// <summary>
/// Returns <c>"not_started"</c> / <c>"in_progress"</c> / <c>"completed"</c>,
/// <c>"unknown_&lt;n&gt;"</c> for unexpected enum values, or <c>"unavailable"</c>
/// when the WinRT contract is absent or any part of the read fails. Never throws.
/// </summary>
public static string Read()
{
try
{
if (!_initTried)
{
var type = Type.GetType(
"Windows.System.Profile.SystemSetupInfo, Windows.System.Profile, ContentType=WindowsRuntime",
throwOnError: false);
_stateProperty = type?.GetProperty(
"OutOfBoxExperienceState",
BindingFlags.Public | BindingFlags.Static);
_initTried = true;
}
if (_stateProperty == null)
return Unavailable;
var value = Convert.ToInt32(_stateProperty.GetValue(null));
switch (value)
{
case 0: return "not_started";
case 1: return "in_progress";
case 2: return "completed";
default: return "unknown_" + value;
}
}
catch
{
return Unavailable;
}
}
}
}
@@ -31,84 +31,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
{
_logger.Info($"Executing FULL SELF-DESTRUCT (Scheduled Task + Files{(_configuration.RebootOnComplete ? " + Reboot" : "")})");
var currentProcessId = Process.GetCurrentProcess().Id;
var currentProcess = Process.GetCurrentProcess();
var agentBasePath = Path.GetFullPath(Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData),
"AutopilotMonitor"
));
var logDir = Path.GetFullPath(Path.Combine(agentBasePath, "Logs"));
var keepLogs = _configuration.KeepLogFile;
// Create a self-deleting PowerShell script
var cleanupScript = $@"
$scriptPath = $MyInvocation.MyCommand.Path
# Wait for agent process to exit
Start-Sleep -Seconds 2
Wait-Process -Id {currentProcessId} -ErrorAction SilentlyContinue -Timeout 30
Start-Sleep -Seconds 1
# Remove Scheduled Task — verify removal BEFORE deleting files (LIFE-F6). An orphan task firing
# against a deleted exe spams Task Scheduler errors every boot forever and, once the files (incl.
# the enrollment-complete marker) are gone, has no cleanup-retry path. So confirm the task is gone
# first; if it cannot be removed, SKIP the file deletion and leave everything intact for a clean
# cleanup retry on the next boot (the marker survives, so bootstrap re-runs cleanup).
$taskName = '{_configuration.ScheduledTaskName}'
$taskGone = $false
for ($i = 1; $i -le 5; $i++) {{
try {{ Stop-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
try {{ Unregister-ScheduledTask -TaskName $taskName -Confirm:$false -ErrorAction SilentlyContinue }} catch {{ }}
$still = $null
try {{ $still = Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
if (-not $still) {{ $taskGone = $true; break }}
Start-Sleep -Seconds 1
}}
if (-not $taskGone) {{
# Fallback: schtasks.exe /Delete does not depend on the ScheduledTasks PowerShell module.
try {{ & schtasks.exe /Delete /TN $taskName /F 2>$null | Out-Null }} catch {{ }}
try {{ $taskGone = -not (Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue) }} catch {{ }}
}}
{(keepLogs ? $@"
if ($taskGone) {{
# Delete everything except Logs directory
Get-ChildItem -Path '{agentBasePath}' -Exclude 'Logs' | ForEach-Object {{
$dest = $_.FullName + '.del'
try {{ Rename-Item -Path $_.FullName -NewName $dest -Force -ErrorAction SilentlyContinue }} catch {{ }}
Remove-Item -Path $dest -Recurse -Force -ErrorAction SilentlyContinue
}}
}}
" : $@"
if ($taskGone) {{
# Rename folder then delete. Retry a few times to let the OS release handles.
$renamedPath = '{agentBasePath}.del'
$renamed = $false
for ($i = 1; $i -le 10; $i++) {{
try {{
Rename-Item -Path '{agentBasePath}' -NewName $renamedPath -Force -ErrorAction Stop
$renamed = $true
break
}} catch {{
Start-Sleep -Seconds 2
}}
}}
if ($renamed) {{
Remove-Item -Path $renamedPath -Recurse -Force -ErrorAction SilentlyContinue
}} else {{
Remove-Item -Path '{agentBasePath}' -Recurse -Force -ErrorAction SilentlyContinue
}}
}}
")}
{(_configuration.RebootOnComplete ? @"
# shutdown.exe (NOT Restart-Computer): Restart-Computer has no -Comment parameter and -Delay is
# only honoured with -Wait, so the previous invocation failed parameter binding and the device
# silently never rebooted. shutdown.exe /r /t 10 mirrors the standalone reboot path and reboots
# after a 10 s delay (lets this script finish + the agent exit). Review LIFE-F2.
shutdown.exe /r /t 10 /c 'Autopilot enrollment completed - Autopilot Monitor is rebooting'
" : "")}
Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
";
var cleanupScript = BuildCleanupScript(
agentProcessName: currentProcess.ProcessName,
agentBasePath: agentBasePath,
keepLogs: _configuration.KeepLogFile,
scheduledTaskName: _configuration.ScheduledTaskName,
rebootOnComplete: _configuration.RebootOnComplete);
// Write cleanup script to temp location (outside of agent folder)
var tempScriptPath = Path.Combine(Path.GetTempPath(), $"AutopilotMonitor-Cleanup-{Guid.NewGuid()}.ps1");
@@ -143,5 +77,171 @@ Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
}
}
/// <summary>
/// Builds the self-deleting cleanup PowerShell script. Extracted (internal static, pure)
/// so tests can pin the hardened invariants without launching processes.
/// <para>
/// LIFE-F7 hardening (incident 2026-07-22 — exe + loaded DLLs orphaned forever):
/// </para>
/// <list type="number">
/// <item><b>Single-flight lock</b> — the enrollment-complete marker is written BEFORE
/// cleanup launches, so any agent start during an in-flight cleanup spawns a SECOND
/// cleanup script. Two unsynchronized scripts interleave their rename/delete phases
/// and leave partial trees behind. A CreateNew lock file in the SYSTEM temp dir
/// serializes them; a stale lock (&gt;10 min) is taken over.</item>
/// <item><b>Wait by process NAME, not launcher PID</b> — marker-retry instances,
/// self-update restarts and manual runs all load the same images but have PIDs the
/// old <c>Wait-Process -Id</c> never saw (and an already-freed PID can be reused by
/// an unrelated boot-time process). After the deadline any straggler is force-killed:
/// its images are about to be deleted anyway.</item>
/// <item><b>Probe-rename BEFORE task removal, never delete the original path</b> — the
/// rename is the atomic lock probe. If it fails, something still holds the tree: abort
/// with task + files + marker intact so the next boot retries (file-side symmetry to
/// the LIFE-F6 task-side rule). If the task cannot be removed afterwards, rename BACK
/// and abort for the same reason. Only a fully renamed tree with the task gone is
/// deleted — the old <c>Remove-Item</c>-on-original fallback deleted marker + state
/// around locked binaries AFTER the task was gone, converting a transient lock into a
/// permanently orphaned exe.</item>
/// </list>
/// </summary>
/// <remarks>
/// The timing parameters exist ONLY so integration tests can replay the incident
/// scenarios (locked tree, straggler process) in seconds instead of minutes —
/// production call sites always use the defaults.
/// </remarks>
internal static string BuildCleanupScript(
string agentProcessName,
string agentBasePath,
bool keepLogs,
string scheduledTaskName,
bool rebootOnComplete,
int initialDelaySeconds = 2,
int processWaitSeconds = 60,
int renameRetries = 10,
int renameRetryDelaySeconds = 2)
{
return $@"
$scriptPath = $MyInvocation.MyCommand.Path
# Single-flight lock (LIFE-F7 #1): serialize concurrent cleanup scripts. CreateNew is
# atomic; the loser backs off and lets the winner own the deletion.
$lockPath = Join-Path ([System.IO.Path]::GetTempPath()) 'AutopilotMonitor-Cleanup.lock'
$lockStream = $null
try {{
if (Test-Path $lockPath) {{
$lockAge = (Get-Date) - (Get-Item $lockPath -ErrorAction Stop).LastWriteTime
if ($lockAge.TotalMinutes -gt 10) {{ Remove-Item -Path $lockPath -Force -ErrorAction SilentlyContinue }}
}}
$lockStream = [System.IO.File]::Open($lockPath, [System.IO.FileMode]::CreateNew, [System.IO.FileAccess]::Write, [System.IO.FileShare]::None)
}} catch {{
Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
exit 0
}}
try {{
# Wait for EVERY agent process to exit, by NAME (LIFE-F7 #2) — covers marker-retry
# instances, self-update restarts and manual runs, and is immune to PID reuse.
Start-Sleep -Seconds {initialDelaySeconds}
$deadline = (Get-Date).AddSeconds({processWaitSeconds})
while ((Get-Date) -lt $deadline) {{
if (-not (Get-Process -Name '{agentProcessName}' -ErrorAction SilentlyContinue)) {{ break }}
Start-Sleep -Seconds 2
}}
# Backstop: force-kill stragglers — their images are about to be deleted anyway.
Get-Process -Name '{agentProcessName}' -ErrorAction SilentlyContinue | Stop-Process -Force -ErrorAction SilentlyContinue
Start-Sleep -Seconds 1
{(keepLogs ? $@"
# Probe-rename all children except Logs (LIFE-F7 #3). Any failure ⇒ rename back and
# abort with the Scheduled Task intact so the next boot retries cleanly.
$renamed = @()
$probeOk = $true
foreach ($item in @(Get-ChildItem -Path '{agentBasePath}' -Exclude 'Logs' -ErrorAction SilentlyContinue)) {{
$dest = $item.FullName + '.del-' + [Guid]::NewGuid().ToString('N')
$itemOk = $false
for ($i = 1; $i -le {renameRetries}; $i++) {{
try {{
Rename-Item -Path $item.FullName -NewName $dest -Force -ErrorAction Stop
$itemOk = $true
break
}} catch {{
Start-Sleep -Seconds {renameRetryDelaySeconds}
}}
}}
if ($itemOk) {{ $renamed += @{{ Original = $item.FullName; Renamed = $dest }} }}
else {{ $probeOk = $false; break }}
}}
" : $@"
# Probe-rename the whole tree (LIFE-F7 #3). Failure ⇒ abort with task + files + marker
# intact so the next boot retries cleanly. GUID suffix avoids collisions with leftovers
# of earlier attempts.
$renamedPath = '{agentBasePath}.del-' + [Guid]::NewGuid().ToString('N')
$probeOk = $false
for ($i = 1; $i -le {renameRetries}; $i++) {{
try {{
Rename-Item -Path '{agentBasePath}' -NewName $renamedPath -Force -ErrorAction Stop
$probeOk = $true
break
}} catch {{
Start-Sleep -Seconds {renameRetryDelaySeconds}
}}
}}
$renamed = @(@{{ Original = '{agentBasePath}'; Renamed = $renamedPath }})
")}
if ($probeOk) {{
# Remove Scheduled Task — verify removal BEFORE deleting files (LIFE-F6). An orphan
# task firing against a deleted exe spams Task Scheduler errors every boot forever
# and, once the files (incl. the enrollment-complete marker) are gone, has no
# cleanup-retry path.
$taskName = '{scheduledTaskName}'
$taskGone = $false
for ($i = 1; $i -le 5; $i++) {{
try {{ Stop-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
try {{ Unregister-ScheduledTask -TaskName $taskName -Confirm:$false -ErrorAction SilentlyContinue }} catch {{ }}
$still = $null
try {{ $still = Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
if (-not $still) {{ $taskGone = $true; break }}
Start-Sleep -Seconds 1
}}
if (-not $taskGone) {{
# Fallback: schtasks.exe /Delete does not depend on the ScheduledTasks PowerShell module.
try {{ & schtasks.exe /Delete /TN $taskName /F 2>$null | Out-Null }} catch {{ }}
try {{ $taskGone = -not (Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue) }} catch {{ }}
}}
if ($taskGone) {{
# Task gone + tree renamed ⇒ safe to delete. Never touches the original path.
foreach ($entry in $renamed) {{
Remove-Item -Path $entry.Renamed -Recurse -Force -ErrorAction SilentlyContinue
}}
}} else {{
# Task survived ⇒ rename back and leave everything for the next-boot retry
# (the marker survives, so bootstrap re-runs cleanup).
foreach ($entry in $renamed) {{
try {{ Rename-Item -Path $entry.Renamed -NewName $entry.Original -Force -ErrorAction SilentlyContinue }} catch {{ }}
}}
}}
}} else {{
# Probe failed ⇒ something still holds the tree. Rename back whatever succeeded and
# abort WITHOUT deleting anything — task + marker stay, next boot retries.
foreach ($entry in $renamed) {{
try {{ Rename-Item -Path $entry.Renamed -NewName $entry.Original -Force -ErrorAction SilentlyContinue }} catch {{ }}
}}
}}
{(rebootOnComplete ? @"
# shutdown.exe (NOT Restart-Computer): Restart-Computer has no -Comment parameter and -Delay is
# only honoured with -Wait, so the previous invocation failed parameter binding and the device
# silently never rebooted. shutdown.exe /r /t 10 mirrors the standalone reboot path and reboots
# after a 10 s delay (lets this script finish + the agent exit). Review LIFE-F2.
shutdown.exe /r /t 10 /c 'Autopilot enrollment completed - Autopilot Monitor is rebooting'
" : "")}
}} finally {{
if ($lockStream) {{ $lockStream.Dispose() }}
Remove-Item -Path $lockPath -Force -ErrorAction SilentlyContinue
Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
}}
";
}
}
}
@@ -3,6 +3,7 @@ using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Management;
using System.Text.RegularExpressions;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
@@ -28,6 +29,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
///
/// Emits a single "local_admin_analysis" event at startup and at shutdown,
/// enabling delta detection between pre- and post-enrollment state.
///
/// Tenant-supplied allowed accounts may contain glob wildcards (e.g. "adm-*");
/// see <see cref="MatchesAllowedEntry"/>.
/// </summary>
public class LocalAdminAnalyzer : IAgentAnalyzer
{
@@ -246,6 +250,43 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
}
}
// -----------------------------------------------------------------------
// Allowed-list matching
// -----------------------------------------------------------------------
/// <summary>
/// Matches an account or profile-folder name against one allowed-list entry.
/// Entries containing <c>*</c> or <c>?</c> are glob patterns (same semantics as the
/// hardware whitelist: <c>*</c> = any run, <c>?</c> = one char, case-insensitive);
/// all other entries match exactly. No ambiguity with literal names — <c>*</c> and
/// <c>?</c> are invalid characters in Windows account names.
/// Internal for direct unit-testing via InternalsVisibleTo.
/// </summary>
internal static bool MatchesAllowedEntry(string name, string entry)
{
if (entry.IndexOf('*') < 0 && entry.IndexOf('?') < 0)
return string.Equals(entry, name, StringComparison.OrdinalIgnoreCase);
var regexPattern = "^" + Regex.Escape(entry)
.Replace("\\*", ".*")
.Replace("\\?", ".") + "$";
try
{
// Timeout guards against pathological patterns (ReDoS)
return Regex.IsMatch(name, regexPattern, RegexOptions.IgnoreCase, TimeSpan.FromMilliseconds(100));
}
catch (RegexMatchTimeoutException)
{
return false;
}
}
private static bool IsAllowed(string name, List<string> allowedAccounts)
{
return allowedAccounts.Any(a => MatchesAllowedEntry(name, a));
}
// -----------------------------------------------------------------------
// Individual checks
// -----------------------------------------------------------------------
@@ -308,8 +349,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
if (disabled)
continue;
if (!allowedAccounts.Any(a =>
string.Equals(a, name, StringComparison.OrdinalIgnoreCase)))
if (!IsAllowed(name, allowedAccounts))
{
unexpected.Add(name);
_logger.Debug($"{Name}: Unexpected local account: {name}");
@@ -349,8 +389,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
allFound.Add(folderName);
if (!allowedAccounts.Any(a =>
string.Equals(a, folderName, StringComparison.OrdinalIgnoreCase)))
if (!IsAllowed(folderName, allowedAccounts))
{
unexpected.Add(folderName);
_logger.Debug($"{Name}: Unexpected profile folder: {folderName}");
@@ -169,9 +169,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
/// <summary>
/// Sends a critical error report to the emergency channel endpoint.
/// Fire-and-forget: swallows all exceptions so a failure here never cascades.
/// Uses a 5-second per-request timeout so it cannot block the upload loop.
/// Default 5-second per-request timeout so it cannot block the upload loop; the
/// emergency-break path passes a longer <paramref name="timeout"/> because its first
/// request of a freshly booted process pays cold TLS + DNS + a possible backend
/// cold start (observed 503s take well over 5 s).
/// Returns <c>true</c> when the HTTP round-trip completed (the endpoint always
/// answers 200), <c>false</c> on timeout/transport failure — the caller-side
/// anti-flood layer uses this to decide whether the report counts as delivered.
/// Virtual for tests.
/// </summary>
public async Task ReportAgentErrorAsync(AgentErrorReport report)
public virtual async Task<bool> ReportAgentErrorAsync(AgentErrorReport report, TimeSpan? timeout = null)
{
try
{
@@ -190,16 +197,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
httpRequest.Headers.Add("X-Tenant-Id", report.TenantId);
AddSecurityHeaders(httpRequest);
using (var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5)))
using (var cts = new CancellationTokenSource(timeout ?? TimeSpan.FromSeconds(5)))
using (var response = await _httpClient.SendAsync(httpRequest, cts.Token).ConfigureAwait(false))
{
// Response status is deliberately ignored — endpoint always returns 200.
return true;
}
}
}
catch (Exception ex)
{
_logger?.Debug($"ReportAgentErrorAsync: emergency channel failed: {ex.Message}");
return false;
}
}
@@ -57,19 +57,33 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
/// (intended to be called as fire-and-forget via <c>_ = reporter.TrySendAsync(...)</c>).
///
/// All anti-flood checks are applied synchronously before the async HTTP call,
/// so repeated calls are cheap when suppressed.
/// so repeated calls are cheap when suppressed. The anti-flood reservation is
/// taken up front (so concurrent same-key calls cannot double-send) but ROLLED
/// BACK when every attempt fails: the budget and the once-per-session dedup are
/// meant to count DELIVERED reports — the previous mark-before-send made the very
/// first transport failure permanently suppress that error key for the session
/// (found via the 48h emergency-break report, which is a single shot at boot).
///
/// <paramref name="attempts"/>/<paramref name="perAttemptTimeout"/>/<paramref name="retryDelay"/>
/// exist for callers whose process is about to exit and cannot come back later
/// (emergency break): retries stay INSIDE the one reservation, so they never
/// multiply against the per-session budget.
/// </summary>
public virtual async Task TrySendAsync(
AgentErrorType errorType,
string message,
int? httpStatusCode = null,
long? sequenceNumber = null)
long? sequenceNumber = null,
int attempts = 1,
TimeSpan? perAttemptTimeout = null,
TimeSpan? retryDelay = null)
{
// Deduplicate by error type + status code: same failure category sent only once per session.
// All anti-flood checks are protected by a lock to prevent race conditions when
// TrySendAsync is called concurrently (fire-and-forget from multiple threads).
var key = $"{errorType}:{httpStatusCode}";
int currentReportCount;
DateTime? previousLastReportTime;
lock (_antiFloodLock)
{
@@ -92,6 +106,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
return;
}
previousLastReportTime = _lastReportTime;
_reportedKeys.Add(key);
_reportCount++;
_lastReportTime = DateTime.UtcNow;
@@ -113,15 +128,40 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
Timestamp = DateTime.UtcNow
};
var delivered = false;
var totalAttempts = attempts < 1 ? 1 : attempts;
try
{
await _apiClient.ReportAgentErrorAsync(report);
for (var attempt = 1; attempt <= totalAttempts && !delivered; attempt++)
{
if (attempt > 1 && retryDelay.HasValue)
{
await Task.Delay(retryDelay.Value).ConfigureAwait(false);
}
delivered = await _apiClient.ReportAgentErrorAsync(report, perAttemptTimeout).ConfigureAwait(false);
if (!delivered && totalAttempts > 1)
{
_logger?.Debug($"[EmergencyChannel] Attempt {attempt}/{totalAttempts} failed: {key}");
}
}
}
catch (Exception ex)
{
// Swallow — emergency channel must never cascade failures into the caller
_logger?.Debug($"[EmergencyChannel] Failed to send report: {ex.Message}");
}
if (!delivered)
{
lock (_antiFloodLock)
{
_reportedKeys.Remove(key);
_reportCount--;
_lastReportTime = previousLastReportTime;
}
_logger?.Debug($"[EmergencyChannel] Report not delivered — reservation rolled back: {key}");
}
}
}
}
@@ -137,7 +137,24 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
private void OnPhaseChanged(object? sender, RealmJoinPhaseChangedEventArgs e)
{
// Phase change is observability-only: no DecisionSignalKind, just a timeline event.
// Dual-emission like OnDetected/OnResolved: (1) the typed signal persists the phase
// into RealmJoinFacts.LastDeploymentPhase (restart-safe) and lets the reducer detect
// the aborted-RJ-ESP shape (phase left 100/101 for 200/210 without 110 — session
// 224b2087); (2) the InformationalEvent keeps the existing timeline entry.
var payload = new Dictionary<string, string>(StringComparer.Ordinal)
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = e.CurrentPhase.ToString(),
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = e.PreviousPhase.ToString(),
};
_ingress.Post(
kind: DecisionSignalKind.RealmJoinPhaseChanged,
occurredAtUtc: _clock.UtcNow,
sourceOrigin: SourceLabel,
evidence: BuildEvidence(
"realmjoin-phase-changed-v1",
$"RealmJoin DeploymentPhase {e.PreviousPhase} -> {e.CurrentPhase}"),
payload: payload);
// Both raw int (deploymentPhase / previousPhase) and human-readable name
// (deploymentPhaseName / previousPhaseName) flow through — int stays for downstream
// filters / KQL, name fields make the message + UI rows readable without a lookup.
@@ -6,6 +6,7 @@ using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Configuration;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Runtime;
@@ -93,6 +94,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
// correction between agent start and termination no longer skews bootToStartSeconds.
private readonly ulong _bootAnchorTickMs;
private readonly DateTime _bootAnchorUtc;
// WinRT OOBE state at construction (handler is built at agent start) — corroborates
// the late-start heuristic: "completed" here means OOBE was already over on arrival.
private readonly string _oobeStateAtAgentStart;
private int _handled;
public EnrollmentTerminationHandler(
@@ -129,6 +133,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
_bootAnchorTickMs = ObservationCoverage.CurrentTickMs();
_bootAnchorUtc = DateTime.UtcNow;
_oobeStateAtAgentStart = OobeStateReader.Read();
}
/// <summary>
@@ -658,6 +663,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
{ "agentUptimeSeconds", Math.Round(uptimeSeconds, 1) },
{ "deviceBootUtc", deviceBootUtc.ToString("o") },
{ "agentStartUtc", _agentStartTimeUtc.ToString("o") },
{ "oobeStateAtAgentStart", _oobeStateAtAgentStart },
{ "outcome", args.Outcome.ToString() },
{ "note", "The agent's bootstrap (an Intune platform script) ran only near the end of the enrollment, so the agent observed the already-decided end-state rather than the failure window. Treat the diagnosis as a post-mortem; check for a platform/remediation script that hung ahead of the bootstrap (see script_timeout_suspected)." },
},
@@ -1,4 +1,6 @@
using System;
using System.Net.NetworkInformation;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Configuration;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Shared.Models;
@@ -15,28 +17,78 @@ namespace AutopilotMonitor.Agent.V2.Runtime
/// classifier terminalize the session instead of waiting out the grace. Fully swallowed — a send
/// failure (no network, no cert) must never delay the cleanup/exit the break exists to perform.
/// See tasks/enrollment-status-reclassification.md.
/// <para>
/// Delivery hardening (2026-07-23): this is the FIRST request of a freshly booted process and
/// the process never comes back — a miss is gone forever. The old single shot with a 5 s cap
/// died on boot-time realities (NIC still associating, cold TLS/DNS, backend cold starts
/// observed as &gt;5 s 503s). Now: wait briefly for the NIC, then up to
/// <see cref="SendAttempts"/> attempts with a per-attempt timeout that survives a Functions
/// cold start. The session is 48 h dead — spending up to ~75 s here is irrelevant, losing the
/// only signal that the agent gave up is not.
/// </para>
/// </summary>
internal static class EmergencyBreakReporter
{
private static readonly TimeSpan NetworkWaitMax = TimeSpan.FromSeconds(15);
private const int SendAttempts = 3;
private static readonly TimeSpan PerAttemptTimeout = TimeSpan.FromSeconds(15);
private static readonly TimeSpan RetryDelay = TimeSpan.FromSeconds(5);
// Upper bound for the blocking wait: NIC wait + 3×15s attempts + 2×5s delays + slack.
private static readonly TimeSpan OverallBudget = TimeSpan.FromSeconds(75);
public static void TrySend(AgentConfiguration agentConfig, string agentVersion, AgentLogger logger)
{
try
{
WaitForNetwork(logger);
var auth = BackendClientFactory.BuildAuthClients(agentConfig, agentVersion, logger);
var message =
$"Agent absolute session-age emergency break fired (cap {agentConfig.AbsoluteMaxSessionHours}h) — cleaning up and exiting.";
// The process is about to exit, so block briefly on the otherwise fire-and-forget send
// rather than let it be abandoned mid-flight. TrySendAsync swallows its own HTTP errors;
// the auth bundle's HttpClients are intentionally left for process teardown.
// The process is about to exit, so block on the otherwise fire-and-forget send
// rather than let it be abandoned mid-flight. TrySendAsync swallows its own HTTP
// errors and keeps the retries inside one anti-flood reservation; the auth
// bundle's HttpClients are intentionally left for process teardown.
auth.EmergencyReporter
.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, message)
.Wait(TimeSpan.FromSeconds(5));
.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak,
message,
attempts: SendAttempts,
perAttemptTimeout: PerAttemptTimeout,
retryDelay: RetryDelay)
.Wait(OverallBudget);
}
catch (Exception ex)
{
logger?.Debug($"Emergency-break report send failed (best-effort): {ex.Message}");
}
}
/// <summary>
/// Boot-time NIC grace: the break fires during bootstrap, often seconds after boot while
/// Wi-Fi is still associating. Polls the cheap link-level signal — it cannot prove backend
/// reachability (the send attempts handle that), it only avoids burning the first attempts
/// into a known-dead link. Best-effort: probe errors abort the wait, never the send.
/// </summary>
private static void WaitForNetwork(AgentLogger logger)
{
try
{
if (NetworkInterface.GetIsNetworkAvailable()) return;
logger?.Info($"Emergency-break report: no network link yet — waiting up to {NetworkWaitMax.TotalSeconds:F0}s.");
var deadline = DateTime.UtcNow + NetworkWaitMax;
while (DateTime.UtcNow < deadline)
{
Thread.Sleep(TimeSpan.FromSeconds(1));
if (NetworkInterface.GetIsNetworkAvailable()) return;
}
}
catch (Exception ex)
{
logger?.Debug($"Emergency-break report: network probe failed ({ex.Message}) — sending anyway.");
}
}
}
}
@@ -2,6 +2,7 @@ using System;
using System.Collections.Generic;
using AutopilotMonitor.Agent.V2.Core.Configuration;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Security;
@@ -68,6 +69,10 @@ namespace AutopilotMonitor.Agent.V2.Runtime
{ "configVersion", configVersion },
{ "remoteConfigFetched", outcome == RemoteConfigFetchOutcome.Succeeded },
{ "remoteConfigOutcome", outcome.ToString() },
// WinRT OOBE state sampled at this start. Together with previousExitType
// this makes late starts provable: "completed" on a first_run means OOBE
// was already over when the agent arrived (e.g. IME queue blocked).
{ "oobeStateAtAgentStart", OobeStateReader.Read() },
};
if (!string.IsNullOrEmpty(previousExit?.CrashExceptionType))
@@ -241,6 +241,199 @@ namespace AutopilotMonitor.DecisionCore.Tests
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_timeout");
}
// ============================================================== Aborted first deployment (session 224b2087)
[Fact]
public void PhaseChanged_leaving_first_deployment_for_200_releases_gate_with_warning()
{
// Session 224b2087: interactive logon 17 s after phase 101 → RJ reclassifies the run
// as secondary-user deployment and writes 200/210, never 110. The 101 -> 200
// transition is impossible in a healthy first deployment, so it releases the gate
// with Outcome=FirstDeploymentIncomplete and a Warning timeline entry.
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101" })).NewState;
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.DesktopArrived, T0.AddMinutes(6))).NewState;
Assert.NotEqual(SessionStage.Finalizing, state.Stage);
var step = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(10),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "200",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "101",
}));
Assert.Equal(SessionStage.Finalizing, step.NewState.Stage);
Assert.Equal("FirstDeploymentIncomplete", step.NewState.RealmJoinFacts.Outcome!.Value);
Assert.NotNull(step.NewState.RealmJoinFacts.ResolvedUtc);
Assert.Equal(200, step.NewState.RealmJoinFacts.LastDeploymentPhase!.Value);
// Hard-timeout deadline cancelled in state AND as a scheduler effect.
Assert.DoesNotContain(step.NewState.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
Assert.Contains(step.Effects, e =>
e.Kind == DecisionEffectKind.CancelDeadline &&
e.CancelDeadlineName == DeadlineNames.RealmJoinTimeout);
// Warning timeline entry names both phases.
var warning = Assert.Single(step.Effects, e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
e.Parameters != null &&
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_first_deployment_incomplete");
Assert.Equal("Warning", warning.Parameters!["severity"]);
Assert.Equal("101", warning.Parameters["previousPhase"]);
Assert.Equal("200", warning.Parameters["deploymentPhase"]);
Assert.Contains("110", warning.Parameters["message"]);
}
[Fact]
public void PhaseChanged_within_first_deployment_window_is_bookkeeping_only()
{
// 100 -> 101 stays inside the first-deployment window — no release, but the phase
// fact must advance (it is the restart-safe witness for the abort rule).
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "100" })).NewState;
var step = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(6),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "100",
}));
Assert.Equal(101, step.NewState.RealmJoinFacts.LastDeploymentPhase!.Value);
Assert.Null(step.NewState.RealmJoinFacts.Outcome);
Assert.Contains(step.NewState.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
Assert.Empty(step.Effects);
}
[Fact]
public void PhaseChanged_from_completed_deployment_start_does_not_release()
{
// Session 6f1959c0 protection: RJ already stood at CompletedDeployment (210) when
// the agent booted and only deployed AFTERWARDS (210 -> 200 -> 210). Neither
// transition starts from the first-deployment window, so the abort rule must not
// fire — this ambiguous shape keeps waiting for 110 / the hard timeout (status quo).
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "210" })).NewState;
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(6),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "200",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "210",
})).NewState;
Assert.Null(state.RealmJoinFacts.Outcome);
var step = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(7),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "210",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "200",
}));
Assert.Null(step.NewState.RealmJoinFacts.Outcome);
Assert.Contains(step.NewState.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
}
[Fact]
public void Detected_replay_after_restart_with_deployment_phase_releases_gate()
{
// Agent restarted between the 101 and 200 observations: the watcher re-fires
// Detected with the CURRENT phase (200) and no PhaseChanged ever carries the
// transition. The persisted LastDeploymentPhase=101 is the witness — the Detected
// replay path must release the gate exactly like the PhaseChanged path.
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101" })).NewState;
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.DesktopArrived, T0.AddMinutes(6))).NewState;
Assert.NotEqual(SessionStage.Finalizing, state.Stage);
Assert.Equal(101, state.RealmJoinFacts.LastDeploymentPhase!.Value);
var step = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(12),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "200" }));
Assert.Equal(SessionStage.Finalizing, step.NewState.Stage);
Assert.Equal("FirstDeploymentIncomplete", step.NewState.RealmJoinFacts.Outcome!.Value);
Assert.Contains(step.Effects, e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
e.Parameters != null &&
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_first_deployment_incomplete");
}
[Fact]
public void Timeout_message_reports_persisted_last_phase_instead_of_zero()
{
// Before RealmJoinPhaseChanged became a typed signal, LastDeploymentPhase was only
// written at detection time — every realmjoin_timeout claimed "last phase: 0" no
// matter how far RJ actually got, hiding this whole failure class from ops.
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "100" })).NewState;
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(6),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "100",
})).NewState;
state = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.DesktopArrived, T0.AddMinutes(7))).NewState;
var step = engine.Reduce(state, MakeSignal(8, DecisionSignalKind.DeadlineFired, T0.AddMinutes(65),
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout }));
var timeout = Assert.Single(step.Effects, e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
e.Parameters != null &&
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_timeout");
Assert.Contains("last phase: 101", timeout.Parameters!["message"]);
Assert.Equal("101", timeout.Parameters["lastSeenPhase"]);
}
[Fact]
public void SelfDeploying_deferred_terminal_released_by_first_deployment_incomplete()
{
// Same deferred-release contract as RealmJoinResolved: when the SelfDeploying
// terminal was deferred on the closed RJ gate, the aborted-first-deployment release
// must complete the session directly (Completed + enrollment_complete), with the
// Warning entry leading the effects.
var engine = new DecisionEngine();
var state = DecisionState.CreateInitial("rj-sd-3", "rj-tenant", T0);
state = engine.Reduce(state, MakeSignal(0, DecisionSignalKind.SessionStarted, T0)).NewState;
state = engine.Reduce(state, MakeSignal(1, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(1),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "DeviceSetup" })).NewState;
state = engine.Reduce(state, MakeSignal(2, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(2),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101" })).NewState;
state = engine.Reduce(state, MakeSignal(3, DecisionSignalKind.DeviceSetupProvisioningComplete, T0.AddMinutes(3))).NewState;
state = engine.Reduce(state, MakeSignal(4, DecisionSignalKind.DeadlineFired, T0.AddMinutes(8),
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.DeviceOnlyEspDetection })).NewState;
Assert.True(state.RealmJoinFacts.SelfDeployingDeferredCompletion?.Value);
var step = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(10),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "200",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "101",
}));
Assert.Equal(SessionStage.Completed, step.NewState.Stage);
Assert.Equal(SessionOutcome.EnrollmentComplete, step.NewState.Outcome);
Assert.Equal("FirstDeploymentIncomplete", step.NewState.RealmJoinFacts.Outcome!.Value);
Assert.Contains(step.Effects, e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
e.Parameters != null &&
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_first_deployment_incomplete");
Assert.Contains(step.Effects, e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
e.Parameters != null &&
e.Parameters.TryGetValue("eventType", out var et) && et == "enrollment_complete");
}
// ============================================================== SelfDeploying flow
[Fact]
@@ -113,9 +113,11 @@ public class EnrollmentTimeoutClassifierTests
// Negative Hello terminals leave the agent waiting — must NOT count as resolved.
Assert.False(EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("hello_provisioning_failed") }).HelloResolved);
Assert.False(EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("hello_completion_timeout") }).HelloResolved);
// Both RealmJoin gate terminals count as resolved (phase 110 or 60-min hard timeout).
// All RealmJoin gate terminals count as resolved (phase 110, aborted first
// deployment — session 224b2087 — or 60-min hard timeout).
Assert.True(EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("realmjoin_resolved") }).RealmJoinResolved);
Assert.True(EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("realmjoin_timeout") }).RealmJoinResolved);
Assert.True(EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("realmjoin_first_deployment_incomplete") }).RealmJoinResolved);
}
[Fact]
@@ -1,7 +1,15 @@
using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Functions.Ingest;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Functions.Services.Notifications;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
namespace AutopilotMonitor.Functions.Tests;
@@ -80,4 +88,116 @@ public class ReportAgentErrorFunctionTests
var evt = ReportAgentErrorFunction.BuildAgentEmergencyBreakEvent(Report(Now, message: ""), TenantId, null!, Now);
Assert.Contains("emergency break", evt.Message, StringComparison.OrdinalIgnoreCase);
}
// =========================================================================
// MaterializeEmergencyBreakArtifactsAsync — the artifact set per break report
// (2026-07-23 hardening): timeline event + cross-session EventType index row +
// AgentEmergencyBreak ops event, once per session. The index upsert is load-bearing:
// StoreEventsBatchAsync alone leaves the event invisible to every search-by-eventType
// surface (found in the 2026-07-22 incident analysis).
// =========================================================================
private sealed class Harness
{
public readonly Mock<ISessionRepository> SessionRepo = new();
public readonly List<OpsEventEntry> OpsEvents = new();
public readonly OpsEventService OpsService;
public Harness(List<EnrollmentEvent>? existingEvents = null)
{
SessionRepo.Setup(r => r.GetSessionEventsAsync(TenantId, SessionId, It.IsAny<int>()))
.ReturnsAsync(existingEvents ?? new List<EnrollmentEvent>());
SessionRepo.Setup(r => r.StoreEventsBatchAsync(It.IsAny<List<EnrollmentEvent>>()))
.ReturnsAsync((List<EnrollmentEvent> e) => e);
SessionRepo.Setup(r => r.UpsertEventTypeIndexBatchAsync(
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<IEnumerable<EnrollmentEvent>>()))
.Returns(Task.CompletedTask);
var opsRepo = new Mock<IOpsEventRepository>();
opsRepo.Setup(r => r.SaveOpsEventAsync(It.IsAny<OpsEventEntry>()))
.Callback<OpsEventEntry>(e => { lock (OpsEvents) OpsEvents.Add(e); })
.Returns(Task.CompletedTask);
var adminConfig = new Mock<AdminConfigurationService>(
Mock.Of<IConfigRepository>(), NullLogger<AdminConfigurationService>.Instance,
new MemoryCache(new MemoryCacheOptions()));
var alertDispatch = new OpsAlertDispatchService(
adminConfig.Object,
new TelegramNotificationService(new HttpClient(), Mock.Of<IConfigRepository>(),
NullLogger<TelegramNotificationService>.Instance),
new WebhookNotificationService(new HttpClient(),
NullLogger<WebhookNotificationService>.Instance),
NullLogger<OpsAlertDispatchService>.Instance);
OpsService = new OpsEventService(opsRepo.Object, NullLogger<OpsEventService>.Instance, alertDispatch);
}
public Task RunAsync(AgentErrorReport report)
=> ReportAgentErrorFunction.MaterializeEmergencyBreakArtifactsAsync(
report, TenantId, SessionRepo.Object, OpsService, NullLogger<ReportAgentErrorFunction>.Instance);
}
[Fact]
public async Task First_break_report_materializes_event_index_row_and_ops_event()
{
var h = new Harness();
await h.RunAsync(Report(Now));
h.SessionRepo.Verify(r => r.StoreEventsBatchAsync(
It.Is<List<EnrollmentEvent>>(e => e.Count == 1
&& e[0].EventType == AutopilotMonitor.Shared.Constants.EventTypes.AgentEmergencyBreak)), Times.Once);
h.SessionRepo.Verify(r => r.UpsertEventTypeIndexBatchAsync(
TenantId, SessionId, It.IsAny<IEnumerable<EnrollmentEvent>>()), Times.Once);
var ops = Assert.Single(h.OpsEvents);
Assert.Equal("AgentEmergencyBreak", ops.EventType);
Assert.Equal(TenantId, ops.TenantId);
}
[Fact]
public async Task Already_materialized_session_produces_no_second_artifact_set()
{
var h = new Harness(new List<EnrollmentEvent>
{
new() { EventType = AutopilotMonitor.Shared.Constants.EventTypes.AgentEmergencyBreak, Sequence = 45 },
});
await h.RunAsync(Report(Now));
h.SessionRepo.Verify(r => r.StoreEventsBatchAsync(It.IsAny<List<EnrollmentEvent>>()), Times.Never);
h.SessionRepo.Verify(r => r.UpsertEventTypeIndexBatchAsync(
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<IEnumerable<EnrollmentEvent>>()), Times.Never);
Assert.Empty(h.OpsEvents);
}
[Fact]
public async Task Non_break_error_types_and_missing_session_id_are_ignored()
{
var h = new Harness();
var other = Report(Now);
other.ErrorType = AgentErrorType.ConfigFetchFailed;
await h.RunAsync(other);
var noSession = Report(Now);
noSession.SessionId = null!;
await h.RunAsync(noSession);
h.SessionRepo.Verify(r => r.GetSessionEventsAsync(
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<int>()), Times.Never);
Assert.Empty(h.OpsEvents);
}
[Fact]
public async Task Storage_failure_is_swallowed_and_never_throws()
{
var h = new Harness();
h.SessionRepo.Setup(r => r.StoreEventsBatchAsync(It.IsAny<List<EnrollmentEvent>>()))
.ThrowsAsync(new InvalidOperationException("storage down"));
// Must not throw — the always-200 emergency channel must never turn into a retry loop.
await h.RunAsync(Report(Now));
Assert.Empty(h.OpsEvents);
}
}
@@ -37,6 +37,7 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
private readonly TelemetryClient _telemetryClient;
private readonly BootstrapSessionService _bootstrapSessionService;
private readonly ISessionRepository _sessionRepo;
private readonly OpsEventService _opsEventService;
public ReportAgentErrorFunction(
ILogger<ReportAgentErrorFunction> logger,
@@ -48,7 +49,8 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
DeviceAssociationValidator deviceAssociationValidator,
TelemetryClient telemetryClient,
BootstrapSessionService bootstrapSessionService,
ISessionRepository sessionRepo)
ISessionRepository sessionRepo,
OpsEventService opsEventService)
{
_logger = logger;
_configService = configService;
@@ -60,6 +62,7 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
_telemetryClient = telemetryClient;
_bootstrapSessionService = bootstrapSessionService;
_sessionRepo = sessionRepo;
_opsEventService = opsEventService;
}
[Function("ReportAgentError")]
@@ -169,35 +172,71 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
["AgentTimestamp"] = report.Timestamp.ToString("O"),
});
// Materialize the agent's silent 48h emergency break as a timeline event so it shows in the
// session and the timeout classifier can see it (tasks/enrollment-status-reclassification.md).
// Best-effort — a failure here must never turn the always-200 emergency channel into a retry loop.
if (report.ErrorType == AgentErrorType.SessionAgeEmergencyBreak
&& !string.IsNullOrEmpty(report.SessionId))
{
try
{
var existing = await _sessionRepo.GetSessionEventsAsync(tenantId, report.SessionId, maxResults: 1000);
// Idempotency: the agent's emergency channel can send up to a few reports per session;
// only ever materialize one timeline event.
var alreadyMaterialized = existing.Any(e =>
string.Equals(e.EventType, Constants.EventTypes.AgentEmergencyBreak, StringComparison.OrdinalIgnoreCase));
if (!alreadyMaterialized)
{
var evt = BuildAgentEmergencyBreakEvent(report, tenantId, existing, DateTime.UtcNow);
await _sessionRepo.StoreEventsBatchAsync(new List<EnrollmentEvent> { evt });
}
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"ReportAgentError: failed to materialize agent_emergency_break event for session {SessionId}", report.SessionId);
}
}
await MaterializeEmergencyBreakArtifactsAsync(report, tenantId, _sessionRepo, _opsEventService, _logger);
return req.CreateResponse(HttpStatusCode.OK);
}
/// <summary>
/// Materializes the agent's silent 48h emergency break as (1) a timeline event so it shows
/// in the session and the timeout classifier can see it, (2) the cross-session EventType
/// index row, and (3) an <c>AgentEmergencyBreak</c> ops event for operator visibility
/// (tasks/enrollment-status-reclassification.md). Static seam with explicit dependencies so
/// tests can pin the artifact set without booting the Functions HTTP stack.
/// Best-effort — a failure here must never turn the always-200 emergency channel into a
/// retry loop.
/// </summary>
internal static async Task MaterializeEmergencyBreakArtifactsAsync(
AgentErrorReport report,
string tenantId,
ISessionRepository sessionRepo,
OpsEventService opsEventService,
ILogger logger)
{
if (report.ErrorType != AgentErrorType.SessionAgeEmergencyBreak
|| string.IsNullOrEmpty(report.SessionId))
{
return;
}
try
{
var existing = await sessionRepo.GetSessionEventsAsync(tenantId, report.SessionId, maxResults: 1000);
// Idempotency: the agent's emergency channel can send up to a few reports per session;
// only ever materialize one timeline event (and one ops event).
var alreadyMaterialized = existing.Any(e =>
string.Equals(e.EventType, Constants.EventTypes.AgentEmergencyBreak, StringComparison.OrdinalIgnoreCase));
if (alreadyMaterialized)
{
return;
}
var evt = BuildAgentEmergencyBreakEvent(report, tenantId, existing, DateTime.UtcNow);
await sessionRepo.StoreEventsBatchAsync(new List<EnrollmentEvent> { evt });
// StoreEventsBatchAsync writes the Events partition only — the cross-session
// EventType index is normally written by EventIngestProcessor, which this
// backend-materialized event never passes through. Without the upsert the event
// exists on the session timeline but is invisible to every search-by-eventType
// surface (portal cross-session search, MCP search_sessions_by_event /
// query_raw_events) — found the hard way in the 2026-07-22 incident analysis.
await sessionRepo.UpsertEventTypeIndexBatchAsync(
tenantId, report.SessionId, new List<EnrollmentEvent> { evt });
// Operator visibility: an emergency break means an agent silently gave up at its
// absolute age cap — exactly the "are we losing agents?" signal. Emitted only on
// first materialization so repeat reports for the same session cannot flood the
// ops feed. OpsEventService never throws.
await opsEventService.RecordAgentEmergencyBreakAsync(
tenantId, report.SessionId, report.AgentVersion, evt.Message);
}
catch (Exception ex)
{
logger.LogWarning(ex,
"ReportAgentError: failed to materialize agent_emergency_break event for session {SessionId}", report.SessionId);
}
}
/// <summary>
/// Builds the backend-materialized <c>agent_emergency_break</c> timeline event from the agent's
/// best-effort emergency report. Static + pure (analog to
@@ -147,9 +147,12 @@ namespace AutopilotMonitor.Functions.Services
helloResolved = true;
else if (Eq(type, "realmjoin_detected"))
realmJoinDetected = true;
// Either terminal opens the agent-side RealmJoin gate: phase 110 or the 60-min
// hard timeout (both dual-emitted to the timeline by the agent/engine).
else if (Eq(type, "realmjoin_resolved") || Eq(type, "realmjoin_timeout"))
// Any terminal opens the agent-side RealmJoin gate: phase 110, the aborted
// first deployment (phase left 100/101 for 200/210 without 110 — session
// 224b2087) or the 60-min hard timeout (all emitted to the timeline by the
// agent/engine).
else if (Eq(type, "realmjoin_resolved") || Eq(type, "realmjoin_timeout")
|| Eq(type, "realmjoin_first_deployment_incomplete"))
realmJoinResolved = true;
// Policy facts for the Hello-disabled / User-ESP-skipped completion mirror.
// Contradicting observations are resolved pessimistically below (both-seen →
@@ -439,6 +439,13 @@ namespace AutopilotMonitor.Functions.Services
var timeoutEvent = BuildSessionTimeoutEvent(session, timeoutHours, sessionEvents, now);
await _sessionRepo.StoreEventsBatchAsync(new List<EnrollmentEvent> { timeoutEvent });
// Backend-materialized events bypass EventIngestProcessor and thus the
// cross-session EventType index — without this upsert session_timeout is
// invisible to every search-by-eventType surface (portal cross-session
// search, MCP search_sessions_by_event / query_raw_events).
await _sessionRepo.UpsertEventTypeIndexBatchAsync(
session.TenantId, session.SessionId, new List<EnrollmentEvent> { timeoutEvent });
// StoreEventsBatchAsync only bumps the orphan side-index, not the
// session's EventCount, and the terminal reconcile already ran inside
// UpdateSessionStatusAsync *before* this synthetic event existed. Recount
@@ -493,6 +493,20 @@ namespace AutopilotMonitor.Functions.Services
$"{sessionCount} session(s) timed out after {timeoutHours}h",
tenantId, "System.Maintenance", new { sessionCount, timeoutHours });
/// <summary>
/// An agent reported its absolute session-age emergency break (48h cap) over the
/// emergency channel — it cleaned itself up and exited on a session that never reached
/// a terminal state. This is the "are we silently losing agents?" signal; emitted by
/// <see cref="Functions.Ingest.ReportAgentErrorFunction"/> once per session (guarded by
/// the timeline-event idempotency check). Warning-tier so operators can wire a Telegram
/// rule; if real-world volume turns out noisy, downgrade or remove — the timeline event
/// on the session is the durable record.
/// </summary>
public Task RecordAgentEmergencyBreakAsync(string tenantId, string sessionId, string? agentVersion, string message)
=> WriteAsync(OpsEventCategory.Agent, "AgentEmergencyBreak", OpsEventSeverity.Warning,
$"Agent emergency break on session {sessionId} (agent {agentVersion ?? "?"}): {message}",
tenantId, "System.EmergencyChannel", new { sessionId, agentVersion });
public Task RecordExcessiveSessionEventsAsync(string tenantId, string sessionId, int eventCount, int threshold)
=> WriteAsync(OpsEventCategory.Agent, "ExcessiveSessionEvents", OpsEventSeverity.Warning,
$"Session {sessionId} has {eventCount} events (threshold {threshold}) — likely agent loop bug",
@@ -34,6 +34,7 @@ export const EVENT_TYPES_CATALOG = {
'desktop_detector_started',
'desktop_detector_first_poll',
'desktop_detector_no_candidate',
'oobe_state_completed',
],
whiteglove_events: [
'whiteglove_complete',
@@ -115,6 +116,7 @@ export const EVENT_TYPES_CATALOG = {
'realmjoin_phase_changed',
'realmjoin_resolved',
'realmjoin_timeout',
'realmjoin_first_deployment_incomplete',
'realmjoin_package_started',
'realmjoin_package_completed',
],
@@ -9,11 +9,12 @@ using SharedConstants = AutopilotMonitor.Shared.Constants;
namespace AutopilotMonitor.DecisionCore.Engine
{
// RealmJoin (RJ) deployment-tracking handlers. Plan: tasks/zany-gathering-oasis plan.
// The agent posts five new DecisionSignalKinds when it observes the
// The agent posts six DecisionSignalKinds when it observes the
// HKLM\SYSTEM\CurrentControlSet\Services\realmjoin\Parameters key + HKLM\SOFTWARE\RealmJoin\Packages
// (and the HKU\<sid>\... user-scope counterpart). Detection arms a 60-min hard timeout
// and gates the enrollment-completion AND-gate. Resolution (phase 110) or timeout
// releases the gate so the session can complete.
// and gates the enrollment-completion AND-gate. Resolution (phase 110), the
// aborted-first-deployment release (phase left 100/101 for 200/210 without 110 —
// session 224b2087) or the timeout releases the gate so the session can complete.
public sealed partial class DecisionEngine
{
// Hard 60-minute deadline from RJ-detected. Not configurable (per design choice —
@@ -25,6 +26,10 @@ namespace AutopilotMonitor.DecisionCore.Engine
public static class RealmJoinPayloadKeys
{
public const string DeploymentPhase = "deploymentPhase";
// Set on RealmJoinPhaseChanged: the phase the watcher observed immediately before
// the transition. The reducer prefers the PERSISTED RealmJoinFacts.LastDeploymentPhase
// (restart-safe) and only falls back to this process-local value.
public const string PreviousPhase = "previousPhase";
public const string PackageId = "packageId";
public const string DisplayName = "displayName";
public const string Version = "version";
@@ -107,6 +112,25 @@ namespace AutopilotMonitor.DecisionCore.Engine
}
builder.RealmJoinFacts = updatedFacts;
// Replay path (agent restarted between the 101 and 200 observations): the watcher
// re-fires Detected with the CURRENT phase and no PhaseChanged ever carries the
// transition — the persisted LastDeploymentPhase is the only witness of the aborted
// first deployment. Evaluate the abort rule here too so the gate does not stay
// closed until the hard timeout.
if (alreadyDetected
&& phase.HasValue
&& IsFirstDeploymentAbortTransition(state.RealmJoinFacts, state.RealmJoinFacts.LastDeploymentPhase?.Value, phase.Value))
{
return ReleaseFirstDeploymentIncomplete(
state: state,
signal: signal,
builder: builder,
nextStep: nextStep,
previousPhase: state.RealmJoinFacts.LastDeploymentPhase!.Value,
currentPhase: phase.Value,
triggerBase: nameof(DecisionSignalKind.RealmJoinDetected));
}
var effects = Array.Empty<DecisionEffect>();
if (!alreadyDetected)
{
@@ -177,7 +201,7 @@ namespace AutopilotMonitor.DecisionCore.Engine
preparedBuilder: builder,
nextStepIndex: nextStep,
trigger: nameof(DecisionSignalKind.RealmJoinResolved),
leadingEffect: cancelEffect);
leadingEffects: cancelEffect != null ? new[] { cancelEffect } : null);
}
/// <summary>
@@ -237,7 +261,156 @@ namespace AutopilotMonitor.DecisionCore.Engine
preparedBuilder: builder,
nextStepIndex: nextStep,
trigger: $"DeadlineFired:{DeadlineNames.RealmJoinTimeout}",
leadingEffect: timeoutEffect);
leadingEffects: new[] { timeoutEffect });
}
// RJ DeploymentPhase enum values relevant to the abort rule (mirrors
// RealmJoin.Core.SoftwarePackaging.DeploymentPhase; the agent-side RealmJoinInfo
// constants are not referencable from this netstandard shared lib).
private const int RjPhaseRunningFirstDeployment = 100;
private const int RjPhaseRunningFirstDeploymentAuto = 101;
private const int RjPhaseCompletedFirstDeployment = 110;
private const int RjPhaseRunningDeployment = 200;
private const int RjPhaseCompletedDeployment = 210;
private static bool IsRealmJoinFirstDeploymentPhase(int phase) =>
phase == RjPhaseRunningFirstDeployment || phase == RjPhaseRunningFirstDeploymentAuto;
/// <summary>
/// The aborted-RJ-ESP shape (session 224b2087): the phase left the first-deployment
/// window (100/101) for a regular deployment phase (200/210) — a transition that cannot
/// occur in a healthy first deployment, whose only exit is CompletedFirstDeployment
/// (110). Root cause observed in the RJ logs: an interactive logon lands seconds after
/// 101, RJ reclassifies the run as a secondary-user deployment
/// (isFirstMachineDeployment=False) and that branch writes 200/210, never 110.
/// <para>
/// Deliberately NOT triggered by 101 → 0 (RJ service restart may still complete the
/// first deployment) nor by a first observation of 210 (RJ already completed before
/// the agent booted — session 6f1959c0 starts 210 → 200; releasing there would be
/// wrong). <paramref name="previousPhase"/> comes from the persisted
/// <see cref="RealmJoinFacts.LastDeploymentPhase"/> where available so the rule
/// survives an agent restart between the 101 and 200 observations.
/// </para>
/// </summary>
private static bool IsFirstDeploymentAbortTransition(RealmJoinFacts facts, int? previousPhase, int currentPhase) =>
facts.DetectedUtc != null
&& facts.Outcome == null
&& previousPhase.HasValue
&& IsRealmJoinFirstDeploymentPhase(previousPhase.Value)
&& (currentPhase == RjPhaseRunningDeployment || currentPhase == RjPhaseCompletedDeployment);
/// <summary>
/// Phase-transition observation from the watcher. Always persists the current phase
/// into <see cref="RealmJoinFacts.LastDeploymentPhase"/> (restart-safe; also fixes the
/// <c>realmjoin_timeout</c> event reporting "last phase: 0" regardless of how far RJ
/// actually got). When <see cref="IsFirstDeploymentAbortTransition"/> matches, emits
/// the <c>realmjoin_first_deployment_incomplete</c> Warning, cancels the hard-timeout
/// deadline and releases the completion gate — for us RJ is finished at that point.
/// </summary>
private DecisionStep HandleRealmJoinPhaseChangedV1(DecisionState state, DecisionSignal signal)
{
var currentPhase = TryReadPhase(signal);
if (currentPhase == null)
{
var deadEnd = BumpStepBookkeeping(state, signal);
var deadEndTransition = BuildDeadEndTransition(
state: state,
signal: signal,
nextStepIndex: deadEnd.StepIndex,
trigger: nameof(DecisionSignalKind.RealmJoinPhaseChanged),
deadEndReason: "realmjoin_phase_changed_missing_phase");
return new DecisionStep(deadEnd, deadEndTransition, Array.Empty<DecisionEffect>());
}
// Persisted fact first (survives agent restarts), watcher's process-local value
// as fallback for the very first phase observation of a session.
var previousPhase = state.RealmJoinFacts.LastDeploymentPhase?.Value
?? TryReadInt(signal, RealmJoinPayloadKeys.PreviousPhase);
var nextStep = state.StepIndex + 1;
var builder = state.ToBuilder()
.WithStepIndex(nextStep)
.WithLastAppliedSignalOrdinal(signal.SessionSignalOrdinal);
builder.RealmJoinFacts = state.RealmJoinFacts.WithLastPhase(currentPhase.Value, signal.SessionSignalOrdinal);
if (!IsFirstDeploymentAbortTransition(state.RealmJoinFacts, previousPhase, currentPhase.Value))
{
var bookkept = builder.Build();
var transition = BuildTakenTransition(
before: state,
signal: signal,
toStage: state.Stage,
nextStepIndex: nextStep,
trigger: nameof(DecisionSignalKind.RealmJoinPhaseChanged));
return new DecisionStep(bookkept, transition, Array.Empty<DecisionEffect>());
}
return ReleaseFirstDeploymentIncomplete(
state: state,
signal: signal,
builder: builder,
nextStep: nextStep,
previousPhase: previousPhase!.Value,
currentPhase: currentPhase.Value,
triggerBase: nameof(DecisionSignalKind.RealmJoinPhaseChanged));
}
/// <summary>
/// Shared gate-release for the aborted-first-deployment rule — called from
/// <see cref="HandleRealmJoinPhaseChangedV1"/> and from the
/// <see cref="HandleRealmJoinDetectedV1"/> replay path (agent restarted between the
/// 101 and 200 observations: the watcher re-fires Detected with the CURRENT phase and
/// no PhaseChanged ever carries the transition, so the persisted fact is the only
/// witness). Mirrors <see cref="HandleRealmJoinResolvedV1"/>'s release shape with a
/// Warning timeline entry ahead of the completion effects.
/// </summary>
private DecisionStep ReleaseFirstDeploymentIncomplete(
DecisionState state,
DecisionSignal signal,
DecisionStateBuilder builder,
int nextStep,
int previousPhase,
int currentPhase,
string triggerBase)
{
builder.CancelDeadline(DeadlineNames.RealmJoinTimeout);
builder.RealmJoinFacts = builder.RealmJoinFacts.WithFirstDeploymentIncomplete(
signal.OccurredAtUtc, currentPhase, signal.SessionSignalOrdinal);
var leadingEffects = new List<DecisionEffect>(capacity: 2);
var cancelEffect = BuildRealmJoinTimeoutCancelEffectIfArmed(state);
if (cancelEffect != null) leadingEffects.Add(cancelEffect);
leadingEffects.Add(BuildRealmJoinFirstDeploymentIncompleteEvent(previousPhase, currentPhase));
return CompleteIfDeferredOrBookkeep(
state: state,
signal: signal,
preparedBuilder: builder,
nextStepIndex: nextStep,
trigger: triggerBase + ":FirstDeploymentIncomplete",
leadingEffects: leadingEffects);
}
/// <summary>
/// Build the <c>realmjoin_first_deployment_incomplete</c> Warning timeline entry.
/// Reducer-owned (like <see cref="BuildRealmJoinTimeoutEvent"/>) — the agent adapter
/// only dual-emits the neutral <c>realmjoin_phase_changed</c> observation; the abort
/// interpretation lives here where the persisted facts are.
/// </summary>
private static DecisionEffect BuildRealmJoinFirstDeploymentIncompleteEvent(int previousPhase, int currentPhase)
{
return new DecisionEffect(
kind: DecisionEffectKind.EmitEventTimelineEntry,
parameters: new Dictionary<string, string>(StringComparer.Ordinal)
{
[SignalPayloadKeys.EventType] = SharedConstants.EventTypes.RealmJoinFirstDeploymentIncomplete,
[SignalPayloadKeys.Source] = "DecisionEngine",
[SignalPayloadKeys.Severity] = "Warning",
[SignalPayloadKeys.Message] =
$"RealmJoin left first deployment (phase {previousPhase} -> {currentPhase}) without CompletedFirstDeployment (110) — the RealmJoin ESP likely aborted. Treating the RealmJoin deployment as finished.",
["previousPhase"] = previousPhase.ToString(CultureInfo.InvariantCulture),
["deploymentPhase"] = currentPhase.ToString(CultureInfo.InvariantCulture),
});
}
/// <summary>
@@ -360,7 +533,7 @@ namespace AutopilotMonitor.DecisionCore.Engine
DecisionStateBuilder preparedBuilder,
int nextStepIndex,
string trigger,
DecisionEffect? leadingEffect)
IReadOnlyList<DecisionEffect>? leadingEffects)
{
var selfDeployingDeferred = state.RealmJoinFacts.SelfDeployingDeferredCompletion?.Value == true;
var classicReady = state.HelloResolvedUtc != null && state.DesktopArrivedUtc != null;
@@ -421,8 +594,8 @@ namespace AutopilotMonitor.DecisionCore.Engine
// Plan v9 Phase 4 — UI phase coverage: emit FinalizingSetup + Complete phase
// declarations BEFORE enrollment_complete so the Web timeline opens both bars
// for RJ-deferred-completion just like the direct SelfDeploying-terminal path.
var effects = new List<DecisionEffect>(capacity: 4);
if (leadingEffect != null) effects.Add(leadingEffect);
var effects = new List<DecisionEffect>(capacity: (leadingEffects?.Count ?? 0) + 3);
if (leadingEffects != null && leadingEffects.Count > 0) effects.AddRange(leadingEffects);
effects.Add(BuildPhaseTransitionEffect(EnrollmentPhase.FinalizingSetup, completedState, trigger + ":SelfDeployingDeferred"));
effects.Add(BuildPhaseTransitionEffect(EnrollmentPhase.Complete, completedState, trigger + ":SelfDeployingDeferred"));
effects.Add(BuildEnrollmentCompleteEffect(completedState, trigger + ":SelfDeployingDeferred"));
@@ -434,19 +607,19 @@ namespace AutopilotMonitor.DecisionCore.Engine
if (classicReady)
{
// This IS the gate-release path: the RealmJoin gate has just opened (the caller
// wrote WithResolved / WithTimeoutOutcome into preparedBuilder), so complete
// directly. Note the completion gates read `state` (pre-resolution), where RJ is
// still closed — routing this through CompleteThroughFinalizingOrDefer would
// re-defer on the very gate we just released. A future second gate that must
// re-block here would re-check the *post* state (WDP-v2 follow-up, ARCH-F1).
var extra = leadingEffect != null ? new[] { leadingEffect } : null;
// wrote WithResolved / WithTimeoutOutcome / WithFirstDeploymentIncomplete into
// preparedBuilder), so complete directly. Note the completion gates read `state`
// (pre-resolution), where RJ is still closed — routing this through
// CompleteThroughFinalizingOrDefer would re-defer on the very gate we just
// released. A future second gate that must re-block here would re-check the
// *post* state (WDP-v2 follow-up, ARCH-F1).
return TransitionToFinalizing(
state: state,
signal: signal,
preparedBuilder: preparedBuilder,
nextStepIndex: nextStepIndex,
trigger: trigger,
extraLeadingEffects: extra);
extraLeadingEffects: leadingEffects);
}
// Bookkeeping only — defer to the next Hello / Desktop / SelfDeploying signal.
@@ -458,10 +631,7 @@ namespace AutopilotMonitor.DecisionCore.Engine
nextStepIndex: nextStepIndex,
trigger: trigger);
var bookkeepEffects = leadingEffect != null
? new[] { leadingEffect }
: Array.Empty<DecisionEffect>();
return new DecisionStep(newState, transition, bookkeepEffects);
return new DecisionStep(newState, transition, MaterializeEffects(leadingEffects));
}
/// <summary>
@@ -143,6 +143,7 @@ namespace AutopilotMonitor.DecisionCore.Engine
// ----- RealmJoin deployment tracking (DecisionEngine.RealmJoin.cs) -----
(DecisionSignalKind.RealmJoinDetected, 1) => HandleRealmJoinDetectedV1(state, signal),
(DecisionSignalKind.RealmJoinResolved, 1) => HandleRealmJoinResolvedV1(state, signal),
(DecisionSignalKind.RealmJoinPhaseChanged, 1) => HandleRealmJoinPhaseChangedV1(state, signal),
(DecisionSignalKind.RealmJoinPackageStarted, 1) => HandleRealmJoinPackageStartedV1(state, signal),
(DecisionSignalKind.RealmJoinPackageCompleted, 1) => HandleRealmJoinPackageCompletedV1(state, signal),
// Note: RealmJoinTimeout is a deadline-only kind (no direct adapter signal) —
@@ -118,5 +118,12 @@ namespace AutopilotMonitor.DecisionCore.Signals
RealmJoinTimeout,
RealmJoinPackageStarted,
RealmJoinPackageCompleted,
// Session 224b2087 (2026-07-22): DeploymentPhase transition observed by the watcher.
// Promoted from InformationalEvent-only so RealmJoinFacts.LastDeploymentPhase is
// persisted across agent restarts AND so the reducer can detect the aborted-RJ-ESP
// shape: an interactive logon during the first deployment (100/101) makes RJ reclassify
// the run as a secondary-user deployment, which writes 200/210 and NEVER 110 — the
// gate would otherwise stay closed until the 60-min hard timeout.
RealmJoinPhaseChanged,
}
}
@@ -14,8 +14,8 @@ namespace AutopilotMonitor.DecisionCore.State
/// <list type="bullet">
/// <item>Immutable; <see cref="With"/>-methods return new instances.</item>
/// <item><see cref="Packages"/> is capped at <see cref="MaxPackages"/>; overflow is discarded.</item>
/// <item><see cref="ResolvedUtc"/> is set once on observing phase 110; later phase readings update <see cref="LastDeploymentPhase"/> only.</item>
/// <item><see cref="Outcome"/> is one of <c>"Resolved"</c> / <c>"Timeout"</c>; null while RJ is still running.</item>
/// <item><see cref="ResolvedUtc"/> is set once on observing phase 110 (or the aborted-first-deployment release); later phase readings update <see cref="LastDeploymentPhase"/> only.</item>
/// <item><see cref="Outcome"/> is one of <c>"Resolved"</c> / <c>"Timeout"</c> / <c>"FirstDeploymentIncomplete"</c>; null while RJ is still running.</item>
/// </list>
/// </remarks>
public sealed class RealmJoinFacts
@@ -25,6 +25,11 @@ namespace AutopilotMonitor.DecisionCore.State
public const string OutcomeResolved = "Resolved";
public const string OutcomeTimeout = "Timeout";
// Session 224b2087 (2026-07-22): the RJ ESP was aborted (interactive logon during the
// first deployment reclassified the run as secondary-user) — DeploymentPhase left
// 100/101 for 200/210 without ever writing CompletedFirstDeployment (110). The gate
// releases with this distinct outcome so reporting can tell the three cases apart.
public const string OutcomeFirstDeploymentIncomplete = "FirstDeploymentIncomplete";
public static readonly RealmJoinFacts Empty = new RealmJoinFacts(
detectedUtc: null,
@@ -161,6 +166,27 @@ namespace AutopilotMonitor.DecisionCore.State
packages: Packages);
}
/// <summary>
/// Terminal outcome for the aborted-RJ-ESP case: DeploymentPhase left the
/// first-deployment window (100/101) for a regular deployment phase (200/210) without
/// ever reaching CompletedFirstDeployment (110). Sets <see cref="ResolvedUtc"/> (this
/// IS our resolution moment — RJ counts as finished for the completion gate) with
/// <see cref="OutcomeFirstDeploymentIncomplete"/>. No-op when an outcome is already set.
/// </summary>
public RealmJoinFacts WithFirstDeploymentIncomplete(DateTime utc, int phase, long sourceSignalOrdinal)
{
if (Outcome != null) return this; // resolved/timeout already wins
return new RealmJoinFacts(
detectedUtc: DetectedUtc,
resolvedUtc: new SignalFact<DateTime>(utc, sourceSignalOrdinal),
lastDeploymentPhase: new SignalFact<int>(phase, sourceSignalOrdinal),
outcome: new SignalFact<string>(OutcomeFirstDeploymentIncomplete, sourceSignalOrdinal),
selfDeployingDeferredCompletion: SelfDeployingDeferredCompletion,
productVersion: ProductVersion,
releaseChannel: ReleaseChannel,
packages: Packages);
}
public RealmJoinFacts WithLastPhase(int phase, long sourceSignalOrdinal)
{
if (LastDeploymentPhase != null && LastDeploymentPhase.Value == phase) return this;
@@ -317,6 +317,12 @@ namespace AutopilotMonitor.Shared
public const string DesktopDetectorStarted = "desktop_detector_started"; // 1x on Start() / ResetForRealUserSwitch()
public const string DesktopDetectorFirstPoll = "desktop_detector_first_poll"; // 1x after first PollForDesktop() completes
public const string DesktopDetectorNoCandidate = "desktop_detector_no_candidate"; // 1x after threshold polls without resolution (configurable)
// 1x when WinRT SystemSetupInfo.OutOfBoxExperienceState flips InProgress->Completed,
// sampled on the DAD 30s poll. Observational only (no decision-engine involvement) —
// an owner-independent desktop corroboration for sessions where WTS+WMI owner
// resolution starves and desktop_arrived never fires. Often absent in healthy
// sessions: the DAD timer is disposed at desktop arrival, which is when the flip lands.
public const string OobeStateCompleted = "oobe_state_completed";
public const string CompletionCheck = "completion_check";
// Liveness plan PR2 — emitted (state-change-only, deduped via the
// CompletionWaitingFingerprint state fact) whenever the DecisionEngine blocks or
@@ -463,6 +469,11 @@ namespace AutopilotMonitor.Shared
public const string RealmJoinPhaseChanged = "realmjoin_phase_changed";
public const string RealmJoinResolved = "realmjoin_resolved";
public const string RealmJoinTimeout = "realmjoin_timeout";
// Warning: DeploymentPhase left the first-deployment window (100/101) for 200/210
// without CompletedFirstDeployment (110) — the RJ ESP was aborted (typically an
// interactive logon during the first deployment reclassified the run as
// secondary-user). The completion gate releases at this point.
public const string RealmJoinFirstDeploymentIncomplete = "realmjoin_first_deployment_incomplete";
public const string RealmJoinPackageStarted = "realmjoin_package_started";
public const string RealmJoinPackageCompleted = "realmjoin_package_completed";
@@ -95,7 +95,17 @@ const OPS_EVENT_TYPES: Record<string, string[]> = {
"TenantTrialExpiring",
"TenantTrialExpired",
],
Agent: ["BlobStorageMissing", "BlobStorageUnreachable", "NewImeVersionDetected", "ExcessiveSessionEvents"],
Agent: [
"BlobStorageMissing",
"BlobStorageUnreachable",
"NewImeVersionDetected",
"ExcessiveSessionEvents",
// 48h session-age emergency break reported over the agent's emergency channel — the
// "are we silently losing agents?" signal. Emitted once per session by
// ReportAgentErrorFunction (RecordAgentEmergencyBreakAsync). Dual-register per memory
// feedback_ops_event_types_dual_register.
"AgentEmergencyBreak",
],
};
const SEVERITIES = ["Info", "Warning", "Error", "Critical"];
@@ -32,6 +32,8 @@ export const KNOWN_EVENT_TYPES: EventTypeEntry[] = [
description: "Pre-Provisioning Part 2 resumed after the user powered on the device." },
{ value: "desktop_arrived", label: "desktop_arrived", category: "enrollment",
description: "User desktop detected (explorer.exe under real user)." },
{ value: "oobe_state_completed", label: "oobe_state_completed", category: "enrollment",
description: "Windows OOBE state flipped InProgress→Completed (WinRT SystemSetupInfo, sampled on the desktop-arrival poll). One-shot and observational only; valuable as an owner-independent desktop corroboration in sessions where desktop_arrived never fires." },
{ value: "completion_check", label: "completion_check", category: "enrollment",
description: "Throttled state-machine snapshot during completion evaluation." },
{ value: "completion_waiting", label: "completion_waiting", category: "enrollment",
@@ -64,9 +64,13 @@ export default function AgentAnalyzersSection({
trimmed !== "" &&
(localAdminAllowedAccounts.some((a) => a.toLowerCase() === trimmed.toLowerCase()) ||
BUILTIN_ACCOUNTS.some((a) => a.toLowerCase() === trimmed.toLowerCase()));
// Wildcard patterns (* / ?) need at least 2 literal characters — a bare "*"
// would silently disable the analyzer for the whole tenant.
const isTooBroad =
/[*?]/.test(trimmed) && trimmed.replace(/[*?]/g, "").length < 2;
const addAccount = () => {
if (!trimmed || isDuplicate) return;
if (!trimmed || isDuplicate || isTooBroad) return;
setLocalAdminAllowedAccounts([...localAdminAllowedAccounts, trimmed]);
setNewAllowedAccount("");
};
@@ -114,6 +118,10 @@ export default function AgentAnalyzersSection({
<p className="font-medium text-gray-900 mb-1">Allowed Local Accounts</p>
<p className="text-sm text-gray-500 mb-3">
Accounts listed here are considered expected on enrolled devices and will not trigger alerts. These are merged with the built-in defaults on the agent.
Wildcards are supported: <code className="px-1 bg-gray-100 rounded">*</code> matches any sequence of characters,{" "}
<code className="px-1 bg-gray-100 rounded">?</code> exactly one — e.g.{" "}
<code className="px-1 bg-gray-100 rounded">adm-*</code> allows generated accounts like <code className="px-1 bg-gray-100 rounded">adm-83921</code>.
Matching is case-insensitive and applies to both account names and profile folders.
</p>
{/* Built-in accounts (read-only) */}
@@ -155,7 +163,7 @@ export default function AgentAnalyzersSection({
<div className="flex gap-2 mt-2">
<input
type="text"
placeholder="Account name (e.g. SupportAdmin)"
placeholder="Account name or pattern (e.g. SupportAdmin, adm-*)"
value={newAllowedAccount}
onChange={(e) => setNewAllowedAccount(e.target.value)}
onKeyDown={(e) => { if (e.key === "Enter") { e.preventDefault(); addAccount(); } }}
@@ -163,7 +171,7 @@ export default function AgentAnalyzersSection({
/>
<button
onClick={addAccount}
disabled={!trimmed || isDuplicate}
disabled={!trimmed || isDuplicate || isTooBroad}
className="px-4 py-1.5 bg-rose-600 text-white rounded-lg text-sm font-medium hover:bg-rose-700 disabled:opacity-50 disabled:cursor-not-allowed transition-colors whitespace-nowrap"
>
Add
@@ -172,6 +180,11 @@ export default function AgentAnalyzersSection({
{isDuplicate && (
<p className="text-xs text-red-500 mt-1">This account is already in the list.</p>
)}
{isTooBroad && (
<p className="text-xs text-red-500 mt-1">
Pattern is too broad — it needs at least 2 characters besides <code>*</code>/<code>?</code>, otherwise the analyzer would be effectively disabled.
</p>
)}
</div>
{/* ── Divider ─────────────────────────────────────── */}
@@ -0,0 +1,13 @@
# RealmJoinPhaseChanged v1 — RealmJoin DeploymentPhase transition.
# Posted by RealmJoinWatcherAdapter on every observed DeploymentPhase change (dual-emitted
# alongside the existing realmjoin_phase_changed InformationalEvent). Payload carries the
# current and previous phase DWORDs.
# Reducer: HandleRealmJoinPhaseChangedV1 → always updates RealmJoinFacts.LastDeploymentPhase
# (restart-safe persistence; fixes realmjoin_timeout reporting "last phase: 0"). When the
# phase leaves the first-deployment window (100/101) for a regular deployment phase (200/210)
# without CompletedFirstDeployment (110) ever being observed, the RJ ESP was aborted (e.g.
# interactive logon during first deployment reclassified the run as secondary-user): emits
# realmjoin_first_deployment_incomplete (Warning), cancels the RealmJoinTimeout deadline and
# releases the completion gate via CompleteIfDeferredOrBookkeep with
# Outcome="FirstDeploymentIncomplete".
{"SessionSignalOrdinal":43,"SessionTraceOrdinal":43,"Kind":"RealmJoinPhaseChanged","KindSchemaVersion":1,"OccurredAtUtc":"2026-05-21T10:26:00.0000000Z","SourceOrigin":"RealmJoinWatcher","Evidence":{"Kind":"Derived","Identifier":"realmjoin-phase-changed-v1","Summary":"RealmJoin DeploymentPhase 101 -> 200","DerivationInputs":{"registryKey":"HKLM\\SYSTEM\\CurrentControlSet\\Services\\realmjoin\\Parameters"}},"Payload":{"deploymentPhase":"200","previousPhase":"101"}}