Compare commits

...

5 Commits

Author SHA1 Message Date
Oliver Kieselbach e8ef4f53ae feat(agent): OOBE-state observability via WinRT SystemSetupInfo
Sample Windows.System.Profile.SystemSetupInfo.OutOfBoxExperienceState
(build >= 17763, SYSTEM-callable, CLR-native WinRT projection — no NuGet)
as a purely observational signal, mirroring the bootstrap guard marker:

- oobeStateAtAgentStart on agent_started and agent_late_start payloads:
  makes late starts provable instead of heuristic ("completed" on a
  first_run = OOBE was already over when the agent arrived) and validates
  the bootstrap OOBE guard fleet-wide.
- One-shot oobe_state_completed trace event on the in_progress->completed
  flip, piggybacked on the DesktopArrivalDetector 30s poll: an
  owner-independent desktop corroboration for DAD-starved sessions
  (4d5a0b78 class — WTS+WMI owner resolution dead, desktop_arrived never
  fires). Often absent in healthy sessions since the DAD timer is
  disposed at desktop arrival, which is when the flip lands.

OobeStateReader is fail-soft (never throws, "unavailable" on missing
contract); the flip check has its own try/catch so it cannot pollute the
DAD wmiErrorCount liveness diagnostics, "unavailable" never overwrites
the baseline, and the one-shot flag survives ResetForRealUserSwitch.
No decision-engine involvement, no config/backend changes. Event type
registered in Constants.EventTypes, MCP resource-catalog (phase_events)
and web gather-rule autocomplete.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:39:10 +02:00
Oliver Kieselbach a0d10b8471 test(cleanup): end-to-end incident replays for the self-destruct script
Six integration tests execute the generated cleanup script with a real
powershell.exe against GUID-unique scratch trees — proving what the string
pins cannot:

- locked tree ⇒ complete abort, marker + binaries survive (the 2026-07-22
  regression: the old script force-deleted around the locks)
- full incident replay: a renamed second process holds a file in the tree,
  is found by NAME, force-killed after the wait deadline, cleanup then
  completes with nothing left behind
- fresh foreign lock ⇒ loser backs off without touching the tree; stale
  lock (>10min) is taken over and cleanup proceeds
- happy path + keepLogs variant

BuildCleanupScript gains test-only timing parameters (defaults unchanged
for production) so the replays run in seconds. Agent suite 1969/1969.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 10:06:29 +02:00
Oliver Kieselbach ad3632e9fb fix(emergency-break): survivable self-destruct, reliable break report, ops visibility
Incident 2026-07-22: the 48h emergency break left exe + loaded DLLs orphaned
forever — a second agent instance held the images while the cleanup script
(waiting on the launcher PID only) force-deleted marker + state around the
locks AFTER removing the Scheduled Task, destroying the retry path.

Agent — cleanup script (LIFE-F7):
- wait for ALL agent processes by name (PID-reuse-proof), then force-kill
  stragglers; single-flight lock file serializes concurrent cleanup scripts
- probe-rename (GUID target) BEFORE task removal; any failure renames back
  and aborts with task + marker + files intact for a next-boot retry; the
  destructive Remove-Item-on-original fallback is gone
- script generation extracted to BuildCleanupScript, pinned by 13 tests;
  all 4 variants parser-validated + E2E-smoked (happy path, locked tree)

Agent — break report delivery:
- EmergencyReporter counts DELIVERED reports: reservation rolls back on
  failure (mark-before-send permanently suppressed a key after its first
  transport failure); optional in-reservation retries
- break path: NIC wait <=15s, 3 attempts x 15s timeout (survives Functions
  cold starts observed as >5s 503s), 75s overall bound

Backend:
- backend-materialized events (agent_emergency_break, session_timeout) now
  upsert the cross-session EventType index — they were invisible to every
  search-by-eventType surface (portal + MCP)
- new AgentEmergencyBreak ops event (Agent/Warning, once per session) so
  operators see silently-lost agents; dual-registered in OPS_EVENT_TYPES
- materialization extracted to a static seam, covered by 4 new tests

Agent 1963/1963, Backend 3198/3198, tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 09:56:46 +02:00
Oliver Kieselbach 05e1569503 feat(agent): wildcard support for Local Admin Analyzer allowed accounts
Allowed Local Accounts entries may now contain glob wildcards (* / ?),
matching the hardware-whitelist semantics: Regex.Escape conversion,
case-insensitive, 100ms ReDoS timeout. Applies to both the account and
the profile-folder check. Entries without wildcard chars keep exact
matching; * and ? are invalid in Windows account names, so patterns and
literals cannot collide. UI documents the syntax and rejects patterns
with fewer than 2 literal characters (a bare * would disable the
analyzer). No schema change - old agents treat patterns as literals and
keep alerting (fail-safe).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 09:47:13 +02:00
Oliver Kieselbach ba3581e253 bootstrapper added download stopwatch 2026-07-23 09:35:30 +02:00
25 changed files with 1472 additions and 119 deletions
@@ -272,13 +272,16 @@ try {
$downloadAttempt++
try {
Write-Log "Download attempt ${downloadAttempt}/${maxDownloadAttempts}"
$downloadStopwatch = [System.Diagnostics.Stopwatch]::StartNew()
Invoke-WebRequest `
-Uri $AgentDownloadUrl `
-OutFile $zipPath `
-UseBasicParsing `
-TimeoutSec 30 `
-ErrorAction Stop
Write-Log "Downloaded agent to $zipPath"
$downloadStopwatch.Stop()
$downloadSeconds = [math]::Round($downloadStopwatch.Elapsed.TotalSeconds, 1)
Write-Log "Downloaded agent to $zipPath (took ${downloadSeconds}s)"
break
}
catch {
@@ -0,0 +1,71 @@
#nullable enable
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Analyzers
{
/// <summary>
/// LocalAdminAnalyzer.MatchesAllowedEntry — the allowed-list matching contract.
/// Entries without wildcard characters match exactly (case-insensitive); entries
/// containing * or ? are glob patterns with the same semantics as the hardware
/// whitelist. The same matcher is applied to local accounts and profile folders.
/// </summary>
public sealed class LocalAdminAnalyzerTests
{
// -------------------------------------------------------------- literal entries
[Theory]
[InlineData("SupportAdmin", "SupportAdmin")]
[InlineData("supportadmin", "SupportAdmin")]
[InlineData("SUPPORTADMIN", "supportadmin")]
public void Literal_entry_matches_exactly_case_insensitive(string name, string entry)
{
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
[Theory]
[InlineData("SupportAdmin2", "SupportAdmin")]
[InlineData("SupportAdmin", "SupportAdmin2")]
[InlineData("Support", "SupportAdmin")]
public void Literal_entry_does_not_substring_match(string name, string entry)
{
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
// -------------------------------------------------------------- glob patterns
[Theory]
[InlineData("adm-12345", "adm-*")]
[InlineData("adm-", "adm-*")]
[InlineData("ADM-9F3B2C", "adm-*")]
[InlineData("lapsAdmin01", "lapsAdmin??")]
[InlineData("svc-backup-01", "svc-*-01")]
public void Wildcard_entry_matches_glob(string name, string entry)
{
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
[Theory]
[InlineData("administrator2", "adm-*")] // no literal "adm-" prefix
[InlineData("xadm-123", "adm-*")] // pattern is anchored at start
[InlineData("lapsAdmin1", "lapsAdmin??")] // ? is exactly one char
[InlineData("adm.123", "adm-*")] // "-" is literal, not a regex class
public void Wildcard_entry_does_not_overmatch(string name, string entry)
{
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
[Fact]
public void Regex_metacharacters_in_entry_are_treated_literally()
{
// "adm.123" as an entry has no wildcard chars → exact match only,
// the dot must not act as a regex any-char.
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry("adm.123", "adm.123"));
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry("admX123", "adm.123"));
// Mixed: wildcard present, remaining metacharacters stay literal.
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry("adm.123", "adm.*"));
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry("admX123", "adm.*"));
}
}
}
@@ -405,5 +405,144 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
var started = trace.Events.Find(e => e.EventType == SharedEventTypes.DesktopDetectorStarted);
Assert.Equal(0, started.Data["noCandidateThresholdMinutes"]);
}
// ============================================================================
// OOBE-state flip (oobe_state_completed) — observational one-shot on the poll
// ============================================================================
// The WinRT OutOfBoxExperienceState flip in_progress->completed is an owner-
// independent desktop corroboration for DAD-starved sessions (4d5a0b78 class).
// Tests drive polls synchronously via PollOnceForTest after Stop() so the real
// timer cannot race, and pin the owner seams to an excluded user so a real
// explorer.exe on the test machine can never set _desktopArrived (which would
// short-circuit PollForDesktop before the flip check).
private static DesktopArrivalDetector BuildFlipDetector(TempDirectory tmp, TraceCapture trace, System.Func<string> oobeProvider)
{
var detector = new DesktopArrivalDetector(new AgentLogger(tmp.Path, AgentLogLevel.Info), noCandidateTimeoutMinutes: 0)
{
OnTraceEvent = trace.Sink,
OobeStateProvider = oobeProvider,
SessionOwnerResolver = _ => "SYSTEM",
ProcessOwnerResolver = _ => "SYSTEM",
};
return detector;
}
/// <summary>Returns the queued states in order, then repeats the last one forever.</summary>
private static System.Func<string> StateQueue(params string[] states)
{
var queue = new Queue<string>(states);
var last = states[states.Length - 1];
return () => queue.Count > 0 ? queue.Dequeue() : last;
}
[Fact]
public void OobeFlip_emits_exactly_once_with_payload()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
// Baseline sample happens synchronously inside Start()/ArmTimer.
using var detector = BuildFlipDetector(tmp, trace, StateQueue("in_progress", "completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest(); // reads "completed" -> flip
detector.PollOnceForTest(); // still "completed" -> no second emit
Assert.Equal(1, trace.CountOf(SharedEventTypes.OobeStateCompleted));
var flip = trace.Events.Find(e => e.EventType == SharedEventTypes.OobeStateCompleted);
Assert.Equal("in_progress", flip.Data["previousState"]);
Assert.True(flip.Data.ContainsKey("pollCount"));
Assert.True(flip.Data.ContainsKey("minutesSinceStart"));
Assert.False((bool)flip.Data["desktopArrived"]);
Assert.True(flip.Data.ContainsKey("wmiErrorCount"));
Assert.True(flip.Data.ContainsKey("wtsErrorCount"));
}
[Fact]
public void OobeFlip_does_not_emit_when_state_is_already_completed()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace, StateQueue("completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest();
detector.PollOnceForTest();
// Productive machines read "completed" from the very first sample — the flip
// event must never fire there (the agent_started field already records the state).
Assert.Equal(0, trace.CountOf(SharedEventTypes.OobeStateCompleted));
}
[Fact]
public void OobeFlip_does_not_emit_when_in_progress_was_never_seen()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace, StateQueue("unavailable", "completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest();
// Old builds without the WinRT contract report "unavailable"; a later
// "completed" without a positively observed "in_progress" is not a flip.
Assert.Equal(0, trace.CountOf(SharedEventTypes.OobeStateCompleted));
}
[Fact]
public void OobeFlip_survives_transient_unavailable_between_polls()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace, StateQueue("in_progress", "unavailable", "completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest(); // transient reflection failure — must NOT clear the baseline
detector.PollOnceForTest(); // "completed" -> flip still detected
Assert.Equal(1, trace.CountOf(SharedEventTypes.OobeStateCompleted));
}
[Fact]
public void OobeFlip_provider_throwing_never_emits_and_never_pollutes_wmi_error_count()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace, () => throw new System.InvalidOperationException("boom"));
detector.Start(); // baseline sample throws -> swallowed
detector.Stop();
detector.PollOnceForTest();
detector.PollOnceForTest();
Assert.Equal(0, trace.CountOf(SharedEventTypes.OobeStateCompleted));
// The flip check has its own try/catch — a throw must not fall into
// PollForDesktop's outer catch, which increments the WMI error counter the
// session-4d5a0b78 liveness diagnostics rely on. first_poll carries that counter.
var firstPoll = trace.Events.Find(e => e.EventType == SharedEventTypes.DesktopDetectorFirstPoll);
Assert.Equal(0, firstPoll.Data["wmiErrorCount"]);
}
[Fact]
public void OobeFlip_one_shot_flag_survives_ResetForRealUserSwitch()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace,
StateQueue("in_progress", "completed", "in_progress", "completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest(); // flip #1
detector.ResetForRealUserSwitch(); // re-arms + re-samples baseline ("in_progress")
detector.Stop();
detector.PollOnceForTest(); // "completed" again — must NOT re-emit
Assert.Equal(1, trace.CountOf(SharedEventTypes.OobeStateCompleted));
}
}
}
@@ -0,0 +1,40 @@
#nullable enable
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
{
/// <summary>
/// Smoke tests for the WinRT <c>SystemSetupInfo.OutOfBoxExperienceState</c> reader.
/// The actual state on the test machine is unknown (CI runners are post-OOBE; Server
/// SKUs may lack the UniversalApiContract entirely), so only the fail-soft contract
/// is pinned: never throw, always return a member of the known value set.
/// </summary>
public sealed class OobeStateReaderTests
{
[Fact]
public void Read_never_throws_and_returns_known_value()
{
var value = OobeStateReader.Read();
Assert.False(string.IsNullOrEmpty(value));
Assert.True(
value == "not_started"
|| value == "in_progress"
|| value == "completed"
|| value == OobeStateReader.Unavailable
|| value.StartsWith("unknown_"),
$"unexpected reader value '{value}'");
}
[Fact]
public void Read_is_stable_across_repeated_calls()
{
// Exercises the cached-PropertyInfo path (second call skips resolution).
var first = OobeStateReader.Read();
var second = OobeStateReader.Read();
Assert.Equal(first, second);
}
}
}
@@ -0,0 +1,212 @@
using System;
using System.Diagnostics;
using System.IO;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
using AutopilotMonitor.Agent.V2.Core.Security;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
{
/// <summary>
/// Executes the generated cleanup script against REAL scratch trees with a real
/// powershell.exe — end-to-end replays of the 2026-07-22 incident mechanics that the
/// string pins in <see cref="CleanupServiceScriptTests"/> cannot prove:
/// <list type="bullet">
/// <item>a locked tree must abort WITHOUT deleting anything (old script force-deleted
/// marker + state around the locks and orphaned the binaries),</item>
/// <item>a straggler agent process (the incident's second instance) must be found by
/// NAME, killed, and the cleanup must then complete,</item>
/// <item>concurrent cleanup scripts must be serialized by the lock file.</item>
/// </list>
/// Timings are dialed down via the test-only BuildCleanupScript parameters so each case
/// runs in seconds. Every test uses GUID-unique process/task/tree names so nothing real
/// (a dev-box agent, a real Scheduled Task) can ever be touched.
/// </summary>
[Collection("SerialThreading")]
public class CleanupServiceScriptIntegrationTests : IDisposable
{
private readonly string _root;
private readonly string _basePath;
private readonly string _taskName = $"AmTest-NoSuchTask-{Guid.NewGuid():N}";
private readonly string _fakeProcessName = $"AmTestAgent{Guid.NewGuid().ToString("N").Substring(0, 8)}";
public CleanupServiceScriptIntegrationTests()
{
_root = Path.Combine(Path.GetTempPath(), $"am-cleanup-it-{Guid.NewGuid():N}");
_basePath = Path.Combine(_root, "AutopilotMonitor");
Directory.CreateDirectory(Path.Combine(_basePath, "Agent"));
Directory.CreateDirectory(Path.Combine(_basePath, "State"));
Directory.CreateDirectory(Path.Combine(_basePath, "Logs"));
File.WriteAllText(Path.Combine(_basePath, "Agent", "fake.exe"), "x");
File.WriteAllText(Path.Combine(_basePath, "State", "enrollment-complete.marker"), "x");
File.WriteAllText(Path.Combine(_basePath, "Logs", "agent.log"), "x");
}
public void Dispose()
{
try { Directory.Delete(_root, recursive: true); } catch { }
}
private static string LockPath => Path.Combine(Path.GetTempPath(), "AutopilotMonitor-Cleanup.lock");
private string BuildScript(bool keepLogs = false, int processWaitSeconds = 1)
=> CleanupService.BuildCleanupScript(
agentProcessName: _fakeProcessName,
agentBasePath: _basePath,
keepLogs: keepLogs,
scheduledTaskName: _taskName,
rebootOnComplete: false,
initialDelaySeconds: 0,
processWaitSeconds: processWaitSeconds,
renameRetries: 2,
renameRetryDelaySeconds: 1);
/// <summary>Runs the script synchronously in a real powershell.exe. Returns the script path.</summary>
private string RunScript(string script)
{
var scriptPath = Path.Combine(_root, $"cleanup-{Guid.NewGuid():N}.ps1");
File.WriteAllText(scriptPath, script);
using (var proc = Process.Start(new ProcessStartInfo
{
FileName = SystemPaths.PowerShell,
Arguments = $"-NoProfile -ExecutionPolicy Bypass -File \"{scriptPath}\"",
UseShellExecute = false,
CreateNoWindow = true,
}))
{
Assert.True(proc.WaitForExit(90_000), "cleanup script did not finish within 90s");
}
return scriptPath;
}
[Fact]
public void Happy_path_deletes_tree_and_removes_lock_and_script()
{
var scriptPath = RunScript(BuildScript());
Assert.False(Directory.Exists(_basePath), "tree should be fully deleted");
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
Assert.False(File.Exists(scriptPath), "script should self-delete");
Assert.False(File.Exists(LockPath), "cleanup lock should be released");
}
[Fact]
public void Locked_tree_aborts_completely_marker_and_files_survive()
{
// The incident regression: with a file locked inside the tree the OLD script
// deleted everything else (incl. the enrollment-complete marker) and left the
// binaries orphaned with the task already gone. The hardened script must treat
// the failed probe-rename as "someone still holds this" and walk away entirely.
using (File.Open(Path.Combine(_basePath, "Agent", "fake.exe"),
FileMode.Open, FileAccess.Read, FileShare.None))
{
RunScript(BuildScript());
}
Assert.True(File.Exists(Path.Combine(_basePath, "Agent", "fake.exe")), "binaries must stay in place");
Assert.True(File.Exists(Path.Combine(_basePath, "State", "enrollment-complete.marker")),
"marker must survive so the next boot retries cleanup");
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
Assert.False(File.Exists(LockPath));
}
[Fact]
public void Straggler_process_holding_the_tree_is_killed_and_cleanup_completes()
{
// Full incident replay: a SECOND agent-named process (here: a renamed
// powershell.exe, matching how only the process NAME identifies agents) holds a
// file in the tree open. The old script waited on the launcher PID only and ran
// its force-delete against the live locks. The hardened script must find the
// straggler by name, kill it after the wait deadline, and then delete everything.
var fakeExe = Path.Combine(_root, _fakeProcessName + ".exe");
File.Copy(SystemPaths.PowerShell, fakeExe);
var lockTarget = Path.Combine(_basePath, "Agent", "fake.exe").Replace("'", "''");
Process straggler = null;
try
{
straggler = Process.Start(new ProcessStartInfo
{
FileName = fakeExe,
Arguments = "-NoProfile -Command \"$f = [IO.File]::Open('" + lockTarget +
"', 'Open', 'Read', 'None'); Start-Sleep -Seconds 120\"",
UseShellExecute = false,
CreateNoWindow = true,
});
// Give the straggler time to acquire its file handle before cleanup starts.
var acquireDeadline = DateTime.UtcNow.AddSeconds(20);
var acquired = false;
while (!acquired && DateTime.UtcNow < acquireDeadline)
{
try
{
using (File.Open(Path.Combine(_basePath, "Agent", "fake.exe"),
FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
}
}
catch (IOException)
{
acquired = true; // share violation ⇒ straggler holds the file
}
if (!acquired) System.Threading.Thread.Sleep(200);
}
Assert.True(acquired, "straggler never acquired its file lock");
RunScript(BuildScript(processWaitSeconds: 2));
Assert.True(straggler.HasExited, "straggler must be force-killed by the name-based backstop");
Assert.False(Directory.Exists(_basePath), "after the kill the tree must be fully deleted");
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
}
finally
{
try { if (straggler != null && !straggler.HasExited) straggler.Kill(); } catch { }
try { straggler?.Dispose(); } catch { }
}
}
[Fact]
public void Fresh_foreign_lock_backs_off_without_touching_the_tree()
{
File.WriteAllText(LockPath, "held by another cleanup");
try
{
var scriptPath = RunScript(BuildScript());
Assert.True(File.Exists(Path.Combine(_basePath, "Agent", "fake.exe")),
"loser script must not touch the tree — the lock holder owns the deletion");
Assert.True(File.Exists(LockPath), "foreign lock must not be deleted by the loser");
Assert.False(File.Exists(scriptPath), "loser still self-deletes its script");
}
finally
{
try { File.Delete(LockPath); } catch { }
}
}
[Fact]
public void Stale_foreign_lock_is_taken_over_and_cleanup_proceeds()
{
File.WriteAllText(LockPath, "crashed cleanup from a previous boot");
File.SetLastWriteTime(LockPath, DateTime.Now.AddMinutes(-15));
RunScript(BuildScript());
Assert.False(Directory.Exists(_basePath), "stale lock must be taken over, cleanup must run");
Assert.False(File.Exists(LockPath));
}
[Fact]
public void KeepLogs_variant_deletes_everything_except_logs()
{
RunScript(BuildScript(keepLogs: true));
Assert.True(File.Exists(Path.Combine(_basePath, "Logs", "agent.log")), "Logs must survive");
Assert.False(Directory.Exists(Path.Combine(_basePath, "Agent")));
Assert.False(Directory.Exists(Path.Combine(_basePath, "State")));
Assert.Empty(Directory.GetFileSystemEntries(_basePath, "*.del-*"));
}
}
}
@@ -0,0 +1,135 @@
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
{
/// <summary>
/// Pins the LIFE-F7 hardening invariants of the self-destruct cleanup script
/// (incident 2026-07-22: a second agent instance held exe + loaded DLLs while the
/// old script — waiting on the launcher PID only — force-deleted around the locks
/// AFTER removing the Scheduled Task, orphaning the binaries with no retry path).
/// These are string-level pins on <see cref="CleanupService.BuildCleanupScript"/>:
/// weakening any of them silently re-introduces the orphaned-binaries failure mode.
/// </summary>
public class CleanupServiceScriptTests
{
private const string ProcessName = "AutopilotMonitor.Agent";
private const string BasePath = @"C:\ProgramData\AutopilotMonitor";
private const string TaskName = "AutopilotMonitorAgent";
private static string Script(bool keepLogs = false, bool reboot = false)
=> CleanupService.BuildCleanupScript(ProcessName, BasePath, keepLogs, TaskName, reboot);
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Waits_by_process_name_not_launcher_pid(bool keepLogs)
{
var script = Script(keepLogs);
// LIFE-F7 #2: name-based wait covers marker-retry / self-update / manual instances
// and is immune to PID reuse. Wait-Process -Id saw exactly one PID and nothing else.
Assert.Contains($"Get-Process -Name '{ProcessName}'", script);
Assert.DoesNotContain("Wait-Process", script);
// Backstop: stragglers are force-killed — their images are deleted right after.
Assert.Contains("Stop-Process -Force", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Never_deletes_the_original_path(bool keepLogs)
{
var script = Script(keepLogs);
// LIFE-F7 #3: deletion happens exclusively on successfully renamed targets
// ($entry.Renamed). A Remove-Item against the original tree is what deleted
// marker + state around locked binaries in the incident.
Assert.DoesNotContain($"Remove-Item -Path '{BasePath}'", script);
Assert.Contains("Remove-Item -Path $entry.Renamed", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Task_removal_is_gated_on_probe_and_deletion_on_task_gone(bool keepLogs)
{
var script = Script(keepLogs);
// Probe-rename first; only a successful probe reaches task removal, and only a
// removed task reaches deletion. Both failure branches rename BACK so the next
// boot retries with task + marker + files intact.
Assert.Contains("if ($probeOk)", script);
Assert.Contains("if ($taskGone)", script);
Assert.Contains("Rename-Item -Path $entry.Renamed -NewName $entry.Original", script);
// LIFE-F6 task-removal loop + module-independent fallback stay intact.
Assert.Contains("Unregister-ScheduledTask -TaskName $taskName", script);
Assert.Contains("schtasks.exe /Delete /TN $taskName /F", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Concurrent_cleanup_scripts_are_serialized_by_lock_file(bool keepLogs)
{
var script = Script(keepLogs);
// LIFE-F7 #1: CreateNew is the atomic single-flight gate; the loser exits without
// touching the tree. Stale locks (>10 min) are taken over.
Assert.Contains("AutopilotMonitor-Cleanup.lock", script);
Assert.Contains("[System.IO.FileMode]::CreateNew", script);
Assert.Contains("$lockAge.TotalMinutes -gt 10", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Rename_targets_carry_a_guid_suffix(bool keepLogs)
{
// A fixed '.del' target collides with leftovers of earlier attempts and with a
// concurrent script's rename — the GUID makes every attempt's target unique.
Assert.Contains(".del-' + [Guid]::NewGuid().ToString('N')", Script(keepLogs));
}
[Fact]
public void KeepLogs_variant_excludes_logs_directory()
{
var script = Script(keepLogs: true);
Assert.Contains("-Exclude 'Logs'", script);
Assert.DoesNotContain($"Rename-Item -Path '{BasePath}' -NewName", script);
}
[Fact]
public void Full_variant_renames_the_whole_tree()
{
var script = Script(keepLogs: false);
Assert.Contains($"Rename-Item -Path '{BasePath}' -NewName $renamedPath", script);
Assert.DoesNotContain("-Exclude 'Logs'", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Reboot_flag_controls_shutdown_invocation(bool reboot)
{
var script = Script(keepLogs: false, reboot: reboot);
if (reboot)
Assert.Contains("shutdown.exe /r /t 10", script);
else
Assert.DoesNotContain("shutdown.exe", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Script_and_lock_are_released_in_finally(bool keepLogs)
{
var script = Script(keepLogs);
Assert.Contains("finally", script);
Assert.Contains("$lockStream.Dispose()", script);
Assert.Contains("Remove-Item -Path $scriptPath -Force", script);
}
}
}
@@ -0,0 +1,143 @@
using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Transport;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Transport
{
/// <summary>
/// Pins the anti-flood semantics of <see cref="EmergencyReporter.TrySendAsync"/> after the
/// delivery-hardening change (2026-07-23): the once-per-session dedup and the per-session
/// budget count DELIVERED reports. The previous mark-before-send permanently suppressed an
/// error key after its first transport failure — fatal for single-shot callers like the 48h
/// emergency break, whose process exits right after the attempt.
/// </summary>
public class EmergencyReporterTests
{
private sealed class FakeApiClient : BackendApiClient
{
public readonly List<TimeSpan?> Timeouts = new List<TimeSpan?>();
public readonly Queue<bool> Results = new Queue<bool>();
public FakeApiClient() : base(
httpClient: new HttpClient(),
baseUrl: "http://localhost",
manufacturer: string.Empty,
model: string.Empty,
serialNumber: string.Empty,
useBootstrapTokenAuth: false,
bootstrapToken: null,
agentVersion: "0.0.0",
logger: null)
{
}
public int Calls => Timeouts.Count;
public override Task<bool> ReportAgentErrorAsync(AgentErrorReport report, TimeSpan? timeout = null)
{
Timeouts.Add(timeout);
return Task.FromResult(Results.Count > 0 && Results.Dequeue());
}
}
private static EmergencyReporter Reporter(FakeApiClient client)
=> new EmergencyReporter(client, "session", "tenant", "0.0.0", logger: null);
[Fact]
public async Task Failed_send_rolls_back_reservation_so_a_later_call_retries()
{
var client = new FakeApiClient();
client.Results.Enqueue(false); // first call fails
client.Results.Enqueue(true); // second call succeeds
var reporter = Reporter(client);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
// Old behaviour: second call suppressed as "already reported" ⇒ 1 call.
Assert.Equal(2, client.Calls);
}
[Fact]
public async Task Delivered_send_is_deduplicated_per_key()
{
var client = new FakeApiClient();
client.Results.Enqueue(true);
var reporter = Reporter(client);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
Assert.Equal(1, client.Calls);
}
[Fact]
public async Task Retries_stay_inside_one_reservation_and_stop_on_success()
{
var client = new FakeApiClient();
client.Results.Enqueue(false);
client.Results.Enqueue(true); // second attempt delivers — third must not fire
var reporter = Reporter(client);
await reporter.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak, "m",
attempts: 3, perAttemptTimeout: TimeSpan.FromMilliseconds(50), retryDelay: TimeSpan.Zero);
Assert.Equal(2, client.Calls);
// Delivered ⇒ dedup holds for the session.
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
Assert.Equal(2, client.Calls);
}
[Fact]
public async Task Per_attempt_timeout_is_forwarded_to_the_api_client()
{
var client = new FakeApiClient();
client.Results.Enqueue(true);
var reporter = Reporter(client);
var timeout = TimeSpan.FromSeconds(15);
await reporter.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak, "m",
attempts: 1, perAttemptTimeout: timeout);
Assert.Equal(timeout, Assert.Single(client.Timeouts));
}
[Fact]
public async Task All_attempts_failing_rolls_back_after_exhausting_attempts()
{
var client = new FakeApiClient(); // queue empty ⇒ every attempt fails
var reporter = Reporter(client);
await reporter.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak, "m",
attempts: 3, retryDelay: TimeSpan.Zero);
Assert.Equal(3, client.Calls);
// Rolled back ⇒ a later call may try again (budget + dedup + cooldown restored).
client.Results.Enqueue(true);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
Assert.Equal(4, client.Calls);
}
[Fact]
public async Task Different_keys_do_not_suppress_each_other_after_rollback()
{
var client = new FakeApiClient();
client.Results.Enqueue(false); // break report fails → rollback (incl. cooldown restore)
client.Results.Enqueue(true); // unrelated report must still go out
var reporter = Reporter(client);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
await reporter.TrySendAsync(AgentErrorType.ConfigFetchFailed, "m");
Assert.Equal(2, client.Calls);
}
}
}
@@ -61,6 +61,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
internal Func<int, string> SessionOwnerResolver { get; set; }
internal Func<int, string> ProcessOwnerResolver { get; set; }
// Test seam for the WinRT OOBE-state sample (null = OobeStateReader.Read).
internal Func<string> OobeStateProvider { get; set; }
// OOBE-state flip tracking (observational only — see OobeStateReader contract).
// _oobeCompletedEmitted is one-shot per agent lifetime and deliberately NOT reset
// by ResetForRealUserSwitch: the OOBE flip is a global OS transition, not per-user.
private string _lastOobeState;
private bool _oobeCompletedEmitted;
/// <summary>
/// System/service account names that should NOT be considered real users.
/// </summary>
@@ -161,6 +170,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
_startedAtUtc = DateTime.UtcNow;
// Baseline OOBE-state sample so the first poll can already detect a flip
// (self-contained try/catch inside, like the Emit* helpers).
CheckOobeStateFlip();
_pollingTimer?.Dispose();
_pollingTimer = new Timer(
PollForDesktop,
@@ -186,6 +199,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
_pollCount++;
CheckOobeStateFlip();
try
{
var explorerProcesses = Process.GetProcessesByName("explorer");
@@ -381,6 +396,66 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
catch (Exception ex) { _logger.Verbose($"DesktopArrivalDetector: detector-started emit failed: {ex.Message}"); }
}
/// <summary>
/// Samples the WinRT OOBE state (<see cref="OobeStateReader"/>) and emits the one-shot
/// <c>oobe_state_completed</c> trace event on the in_progress-&gt;completed flip.
/// Observational only — an owner-independent desktop corroboration for sessions where
/// WTS+WMI owner resolution starves and <c>desktop_arrived</c> never fires. Often absent
/// in healthy sessions: the polling timer is disposed at desktop arrival, which is when
/// the flip lands (empirical probe, session 9c404ae9).
/// <para>
/// Self-contained try/catch: a throw here must NOT reach <see cref="PollForDesktop"/>'s
/// outer catch, which would inflate <see cref="_wmiErrorCountSinceStart"/> and skew the
/// session-4d5a0b78 liveness diagnostics. "unavailable" readings never overwrite the last
/// known state, so a transient reflection failure between two polls cannot mask the flip.
/// One-shot flag survives <see cref="ResetForRealUserSwitch"/> — the flip is a global OS
/// transition, not per-user.
/// </para>
/// </summary>
private void CheckOobeStateFlip()
{
try
{
var current = (OobeStateProvider ?? OobeStateReader.Read)();
if (string.IsNullOrEmpty(current) || current == OobeStateReader.Unavailable)
return;
if (!_oobeCompletedEmitted
&& _lastOobeState == "in_progress"
&& current == "completed")
{
_oobeCompletedEmitted = true;
var minutesSinceStart = Math.Round((DateTime.UtcNow - _startedAtUtc).TotalMinutes, 1);
_logger.Info("DesktopArrivalDetector: OOBE state flipped in_progress->completed (WinRT SystemSetupInfo)");
OnTraceEvent?.Invoke(
SharedEventTypes.OobeStateCompleted,
$"OOBE state flipped in_progress->completed (WinRT SystemSetupInfo, poll {_pollCount}) — owner-independent desktop corroboration.",
new Dictionary<string, object>
{
{ "previousState", "in_progress" },
{ "pollCount", _pollCount },
{ "minutesSinceStart", minutesSinceStart },
{ "desktopArrived", _desktopArrived },
{ "wmiErrorCount", _wmiErrorCountSinceStart },
{ "wtsErrorCount", _wtsErrorCountSinceStart },
});
}
_lastOobeState = current;
}
catch (Exception ex)
{
_logger.Verbose($"DesktopArrivalDetector: OOBE state check failed: {ex.Message}");
}
}
/// <summary>
/// Drives a single synchronous poll for unit tests — the production poll is private and
/// timer-driven with a fixed 30 s interval (precedent: <see cref="ResolveOwner"/> is
/// likewise internal for direct testing).
/// </summary>
internal void PollOnceForTest() => PollForDesktop(null);
/// <summary>
/// Resolves the owning user of an explorer.exe candidate. Session 4d5a0b78 fix
/// (2026-06-11): the WMI <c>Win32_Process.GetOwner</c> call failed on every poll on
@@ -0,0 +1,72 @@
using System;
using System.Reflection;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Interop
{
/// <summary>
/// Reads WinRT <c>Windows.System.Profile.SystemSetupInfo.OutOfBoxExperienceState</c>
/// (NotStarted=0 / InProgress=1 / Completed=2) via the CLR's built-in WinRT projection —
/// no NuGet package, no winmd reference; the type is resolved at runtime with
/// <c>ContentType=WindowsRuntime</c>, the same mechanism the bootstrap script uses from
/// PowerShell 5.1.
/// <para>
/// <b>Contract:</b> observational only — never feed this into decision-engine logic.
/// Requires Windows 10 1809+ (build 17763, UniversalApiContract 7.0) and is callable
/// from the SYSTEM service context (empirically validated, probe session 9c404ae9).
/// On older builds or SKUs without the contract the type resolves to null and
/// <see cref="Read"/> returns <c>"unavailable"</c>. <see cref="Read"/> never throws.
/// </para>
/// <para>
/// Empirical timing (Win11 24H2 probe): the state flips InProgress→Completed at desktop
/// arrival — it is a corroboration signal, not an early one. WDP reseal and placeholder
/// (fooUser) flip semantics are unverified; another reason this must stay observational.
/// </para>
/// </summary>
internal static class OobeStateReader
{
public const string Unavailable = "unavailable";
// Resolved once per process. Benign race: concurrent first calls resolve the same
// PropertyInfo and the assignment is idempotent — no lock needed.
private static PropertyInfo _stateProperty;
private static bool _initTried;
/// <summary>
/// Returns <c>"not_started"</c> / <c>"in_progress"</c> / <c>"completed"</c>,
/// <c>"unknown_&lt;n&gt;"</c> for unexpected enum values, or <c>"unavailable"</c>
/// when the WinRT contract is absent or any part of the read fails. Never throws.
/// </summary>
public static string Read()
{
try
{
if (!_initTried)
{
var type = Type.GetType(
"Windows.System.Profile.SystemSetupInfo, Windows.System.Profile, ContentType=WindowsRuntime",
throwOnError: false);
_stateProperty = type?.GetProperty(
"OutOfBoxExperienceState",
BindingFlags.Public | BindingFlags.Static);
_initTried = true;
}
if (_stateProperty == null)
return Unavailable;
var value = Convert.ToInt32(_stateProperty.GetValue(null));
switch (value)
{
case 0: return "not_started";
case 1: return "in_progress";
case 2: return "completed";
default: return "unknown_" + value;
}
}
catch
{
return Unavailable;
}
}
}
}
@@ -31,84 +31,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
{
_logger.Info($"Executing FULL SELF-DESTRUCT (Scheduled Task + Files{(_configuration.RebootOnComplete ? " + Reboot" : "")})");
var currentProcessId = Process.GetCurrentProcess().Id;
var currentProcess = Process.GetCurrentProcess();
var agentBasePath = Path.GetFullPath(Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData),
"AutopilotMonitor"
));
var logDir = Path.GetFullPath(Path.Combine(agentBasePath, "Logs"));
var keepLogs = _configuration.KeepLogFile;
// Create a self-deleting PowerShell script
var cleanupScript = $@"
$scriptPath = $MyInvocation.MyCommand.Path
# Wait for agent process to exit
Start-Sleep -Seconds 2
Wait-Process -Id {currentProcessId} -ErrorAction SilentlyContinue -Timeout 30
Start-Sleep -Seconds 1
# Remove Scheduled Task — verify removal BEFORE deleting files (LIFE-F6). An orphan task firing
# against a deleted exe spams Task Scheduler errors every boot forever and, once the files (incl.
# the enrollment-complete marker) are gone, has no cleanup-retry path. So confirm the task is gone
# first; if it cannot be removed, SKIP the file deletion and leave everything intact for a clean
# cleanup retry on the next boot (the marker survives, so bootstrap re-runs cleanup).
$taskName = '{_configuration.ScheduledTaskName}'
$taskGone = $false
for ($i = 1; $i -le 5; $i++) {{
try {{ Stop-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
try {{ Unregister-ScheduledTask -TaskName $taskName -Confirm:$false -ErrorAction SilentlyContinue }} catch {{ }}
$still = $null
try {{ $still = Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
if (-not $still) {{ $taskGone = $true; break }}
Start-Sleep -Seconds 1
}}
if (-not $taskGone) {{
# Fallback: schtasks.exe /Delete does not depend on the ScheduledTasks PowerShell module.
try {{ & schtasks.exe /Delete /TN $taskName /F 2>$null | Out-Null }} catch {{ }}
try {{ $taskGone = -not (Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue) }} catch {{ }}
}}
{(keepLogs ? $@"
if ($taskGone) {{
# Delete everything except Logs directory
Get-ChildItem -Path '{agentBasePath}' -Exclude 'Logs' | ForEach-Object {{
$dest = $_.FullName + '.del'
try {{ Rename-Item -Path $_.FullName -NewName $dest -Force -ErrorAction SilentlyContinue }} catch {{ }}
Remove-Item -Path $dest -Recurse -Force -ErrorAction SilentlyContinue
}}
}}
" : $@"
if ($taskGone) {{
# Rename folder then delete. Retry a few times to let the OS release handles.
$renamedPath = '{agentBasePath}.del'
$renamed = $false
for ($i = 1; $i -le 10; $i++) {{
try {{
Rename-Item -Path '{agentBasePath}' -NewName $renamedPath -Force -ErrorAction Stop
$renamed = $true
break
}} catch {{
Start-Sleep -Seconds 2
}}
}}
if ($renamed) {{
Remove-Item -Path $renamedPath -Recurse -Force -ErrorAction SilentlyContinue
}} else {{
Remove-Item -Path '{agentBasePath}' -Recurse -Force -ErrorAction SilentlyContinue
}}
}}
")}
{(_configuration.RebootOnComplete ? @"
# shutdown.exe (NOT Restart-Computer): Restart-Computer has no -Comment parameter and -Delay is
# only honoured with -Wait, so the previous invocation failed parameter binding and the device
# silently never rebooted. shutdown.exe /r /t 10 mirrors the standalone reboot path and reboots
# after a 10 s delay (lets this script finish + the agent exit). Review LIFE-F2.
shutdown.exe /r /t 10 /c 'Autopilot enrollment completed - Autopilot Monitor is rebooting'
" : "")}
Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
";
var cleanupScript = BuildCleanupScript(
agentProcessName: currentProcess.ProcessName,
agentBasePath: agentBasePath,
keepLogs: _configuration.KeepLogFile,
scheduledTaskName: _configuration.ScheduledTaskName,
rebootOnComplete: _configuration.RebootOnComplete);
// Write cleanup script to temp location (outside of agent folder)
var tempScriptPath = Path.Combine(Path.GetTempPath(), $"AutopilotMonitor-Cleanup-{Guid.NewGuid()}.ps1");
@@ -143,5 +77,171 @@ Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
}
}
/// <summary>
/// Builds the self-deleting cleanup PowerShell script. Extracted (internal static, pure)
/// so tests can pin the hardened invariants without launching processes.
/// <para>
/// LIFE-F7 hardening (incident 2026-07-22 — exe + loaded DLLs orphaned forever):
/// </para>
/// <list type="number">
/// <item><b>Single-flight lock</b> — the enrollment-complete marker is written BEFORE
/// cleanup launches, so any agent start during an in-flight cleanup spawns a SECOND
/// cleanup script. Two unsynchronized scripts interleave their rename/delete phases
/// and leave partial trees behind. A CreateNew lock file in the SYSTEM temp dir
/// serializes them; a stale lock (&gt;10 min) is taken over.</item>
/// <item><b>Wait by process NAME, not launcher PID</b> — marker-retry instances,
/// self-update restarts and manual runs all load the same images but have PIDs the
/// old <c>Wait-Process -Id</c> never saw (and an already-freed PID can be reused by
/// an unrelated boot-time process). After the deadline any straggler is force-killed:
/// its images are about to be deleted anyway.</item>
/// <item><b>Probe-rename BEFORE task removal, never delete the original path</b> — the
/// rename is the atomic lock probe. If it fails, something still holds the tree: abort
/// with task + files + marker intact so the next boot retries (file-side symmetry to
/// the LIFE-F6 task-side rule). If the task cannot be removed afterwards, rename BACK
/// and abort for the same reason. Only a fully renamed tree with the task gone is
/// deleted — the old <c>Remove-Item</c>-on-original fallback deleted marker + state
/// around locked binaries AFTER the task was gone, converting a transient lock into a
/// permanently orphaned exe.</item>
/// </list>
/// </summary>
/// <remarks>
/// The timing parameters exist ONLY so integration tests can replay the incident
/// scenarios (locked tree, straggler process) in seconds instead of minutes —
/// production call sites always use the defaults.
/// </remarks>
internal static string BuildCleanupScript(
string agentProcessName,
string agentBasePath,
bool keepLogs,
string scheduledTaskName,
bool rebootOnComplete,
int initialDelaySeconds = 2,
int processWaitSeconds = 60,
int renameRetries = 10,
int renameRetryDelaySeconds = 2)
{
return $@"
$scriptPath = $MyInvocation.MyCommand.Path
# Single-flight lock (LIFE-F7 #1): serialize concurrent cleanup scripts. CreateNew is
# atomic; the loser backs off and lets the winner own the deletion.
$lockPath = Join-Path ([System.IO.Path]::GetTempPath()) 'AutopilotMonitor-Cleanup.lock'
$lockStream = $null
try {{
if (Test-Path $lockPath) {{
$lockAge = (Get-Date) - (Get-Item $lockPath -ErrorAction Stop).LastWriteTime
if ($lockAge.TotalMinutes -gt 10) {{ Remove-Item -Path $lockPath -Force -ErrorAction SilentlyContinue }}
}}
$lockStream = [System.IO.File]::Open($lockPath, [System.IO.FileMode]::CreateNew, [System.IO.FileAccess]::Write, [System.IO.FileShare]::None)
}} catch {{
Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
exit 0
}}
try {{
# Wait for EVERY agent process to exit, by NAME (LIFE-F7 #2) — covers marker-retry
# instances, self-update restarts and manual runs, and is immune to PID reuse.
Start-Sleep -Seconds {initialDelaySeconds}
$deadline = (Get-Date).AddSeconds({processWaitSeconds})
while ((Get-Date) -lt $deadline) {{
if (-not (Get-Process -Name '{agentProcessName}' -ErrorAction SilentlyContinue)) {{ break }}
Start-Sleep -Seconds 2
}}
# Backstop: force-kill stragglers — their images are about to be deleted anyway.
Get-Process -Name '{agentProcessName}' -ErrorAction SilentlyContinue | Stop-Process -Force -ErrorAction SilentlyContinue
Start-Sleep -Seconds 1
{(keepLogs ? $@"
# Probe-rename all children except Logs (LIFE-F7 #3). Any failure ⇒ rename back and
# abort with the Scheduled Task intact so the next boot retries cleanly.
$renamed = @()
$probeOk = $true
foreach ($item in @(Get-ChildItem -Path '{agentBasePath}' -Exclude 'Logs' -ErrorAction SilentlyContinue)) {{
$dest = $item.FullName + '.del-' + [Guid]::NewGuid().ToString('N')
$itemOk = $false
for ($i = 1; $i -le {renameRetries}; $i++) {{
try {{
Rename-Item -Path $item.FullName -NewName $dest -Force -ErrorAction Stop
$itemOk = $true
break
}} catch {{
Start-Sleep -Seconds {renameRetryDelaySeconds}
}}
}}
if ($itemOk) {{ $renamed += @{{ Original = $item.FullName; Renamed = $dest }} }}
else {{ $probeOk = $false; break }}
}}
" : $@"
# Probe-rename the whole tree (LIFE-F7 #3). Failure ⇒ abort with task + files + marker
# intact so the next boot retries cleanly. GUID suffix avoids collisions with leftovers
# of earlier attempts.
$renamedPath = '{agentBasePath}.del-' + [Guid]::NewGuid().ToString('N')
$probeOk = $false
for ($i = 1; $i -le {renameRetries}; $i++) {{
try {{
Rename-Item -Path '{agentBasePath}' -NewName $renamedPath -Force -ErrorAction Stop
$probeOk = $true
break
}} catch {{
Start-Sleep -Seconds {renameRetryDelaySeconds}
}}
}}
$renamed = @(@{{ Original = '{agentBasePath}'; Renamed = $renamedPath }})
")}
if ($probeOk) {{
# Remove Scheduled Task — verify removal BEFORE deleting files (LIFE-F6). An orphan
# task firing against a deleted exe spams Task Scheduler errors every boot forever
# and, once the files (incl. the enrollment-complete marker) are gone, has no
# cleanup-retry path.
$taskName = '{scheduledTaskName}'
$taskGone = $false
for ($i = 1; $i -le 5; $i++) {{
try {{ Stop-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
try {{ Unregister-ScheduledTask -TaskName $taskName -Confirm:$false -ErrorAction SilentlyContinue }} catch {{ }}
$still = $null
try {{ $still = Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
if (-not $still) {{ $taskGone = $true; break }}
Start-Sleep -Seconds 1
}}
if (-not $taskGone) {{
# Fallback: schtasks.exe /Delete does not depend on the ScheduledTasks PowerShell module.
try {{ & schtasks.exe /Delete /TN $taskName /F 2>$null | Out-Null }} catch {{ }}
try {{ $taskGone = -not (Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue) }} catch {{ }}
}}
if ($taskGone) {{
# Task gone + tree renamed ⇒ safe to delete. Never touches the original path.
foreach ($entry in $renamed) {{
Remove-Item -Path $entry.Renamed -Recurse -Force -ErrorAction SilentlyContinue
}}
}} else {{
# Task survived ⇒ rename back and leave everything for the next-boot retry
# (the marker survives, so bootstrap re-runs cleanup).
foreach ($entry in $renamed) {{
try {{ Rename-Item -Path $entry.Renamed -NewName $entry.Original -Force -ErrorAction SilentlyContinue }} catch {{ }}
}}
}}
}} else {{
# Probe failed ⇒ something still holds the tree. Rename back whatever succeeded and
# abort WITHOUT deleting anything — task + marker stay, next boot retries.
foreach ($entry in $renamed) {{
try {{ Rename-Item -Path $entry.Renamed -NewName $entry.Original -Force -ErrorAction SilentlyContinue }} catch {{ }}
}}
}}
{(rebootOnComplete ? @"
# shutdown.exe (NOT Restart-Computer): Restart-Computer has no -Comment parameter and -Delay is
# only honoured with -Wait, so the previous invocation failed parameter binding and the device
# silently never rebooted. shutdown.exe /r /t 10 mirrors the standalone reboot path and reboots
# after a 10 s delay (lets this script finish + the agent exit). Review LIFE-F2.
shutdown.exe /r /t 10 /c 'Autopilot enrollment completed - Autopilot Monitor is rebooting'
" : "")}
}} finally {{
if ($lockStream) {{ $lockStream.Dispose() }}
Remove-Item -Path $lockPath -Force -ErrorAction SilentlyContinue
Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
}}
";
}
}
}
@@ -3,6 +3,7 @@ using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Management;
using System.Text.RegularExpressions;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
@@ -28,6 +29,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
///
/// Emits a single "local_admin_analysis" event at startup and at shutdown,
/// enabling delta detection between pre- and post-enrollment state.
///
/// Tenant-supplied allowed accounts may contain glob wildcards (e.g. "adm-*");
/// see <see cref="MatchesAllowedEntry"/>.
/// </summary>
public class LocalAdminAnalyzer : IAgentAnalyzer
{
@@ -246,6 +250,43 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
}
}
// -----------------------------------------------------------------------
// Allowed-list matching
// -----------------------------------------------------------------------
/// <summary>
/// Matches an account or profile-folder name against one allowed-list entry.
/// Entries containing <c>*</c> or <c>?</c> are glob patterns (same semantics as the
/// hardware whitelist: <c>*</c> = any run, <c>?</c> = one char, case-insensitive);
/// all other entries match exactly. No ambiguity with literal names — <c>*</c> and
/// <c>?</c> are invalid characters in Windows account names.
/// Internal for direct unit-testing via InternalsVisibleTo.
/// </summary>
internal static bool MatchesAllowedEntry(string name, string entry)
{
if (entry.IndexOf('*') < 0 && entry.IndexOf('?') < 0)
return string.Equals(entry, name, StringComparison.OrdinalIgnoreCase);
var regexPattern = "^" + Regex.Escape(entry)
.Replace("\\*", ".*")
.Replace("\\?", ".") + "$";
try
{
// Timeout guards against pathological patterns (ReDoS)
return Regex.IsMatch(name, regexPattern, RegexOptions.IgnoreCase, TimeSpan.FromMilliseconds(100));
}
catch (RegexMatchTimeoutException)
{
return false;
}
}
private static bool IsAllowed(string name, List<string> allowedAccounts)
{
return allowedAccounts.Any(a => MatchesAllowedEntry(name, a));
}
// -----------------------------------------------------------------------
// Individual checks
// -----------------------------------------------------------------------
@@ -308,8 +349,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
if (disabled)
continue;
if (!allowedAccounts.Any(a =>
string.Equals(a, name, StringComparison.OrdinalIgnoreCase)))
if (!IsAllowed(name, allowedAccounts))
{
unexpected.Add(name);
_logger.Debug($"{Name}: Unexpected local account: {name}");
@@ -349,8 +389,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
allFound.Add(folderName);
if (!allowedAccounts.Any(a =>
string.Equals(a, folderName, StringComparison.OrdinalIgnoreCase)))
if (!IsAllowed(folderName, allowedAccounts))
{
unexpected.Add(folderName);
_logger.Debug($"{Name}: Unexpected profile folder: {folderName}");
@@ -169,9 +169,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
/// <summary>
/// Sends a critical error report to the emergency channel endpoint.
/// Fire-and-forget: swallows all exceptions so a failure here never cascades.
/// Uses a 5-second per-request timeout so it cannot block the upload loop.
/// Default 5-second per-request timeout so it cannot block the upload loop; the
/// emergency-break path passes a longer <paramref name="timeout"/> because its first
/// request of a freshly booted process pays cold TLS + DNS + a possible backend
/// cold start (observed 503s take well over 5 s).
/// Returns <c>true</c> when the HTTP round-trip completed (the endpoint always
/// answers 200), <c>false</c> on timeout/transport failure — the caller-side
/// anti-flood layer uses this to decide whether the report counts as delivered.
/// Virtual for tests.
/// </summary>
public async Task ReportAgentErrorAsync(AgentErrorReport report)
public virtual async Task<bool> ReportAgentErrorAsync(AgentErrorReport report, TimeSpan? timeout = null)
{
try
{
@@ -190,16 +197,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
httpRequest.Headers.Add("X-Tenant-Id", report.TenantId);
AddSecurityHeaders(httpRequest);
using (var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5)))
using (var cts = new CancellationTokenSource(timeout ?? TimeSpan.FromSeconds(5)))
using (var response = await _httpClient.SendAsync(httpRequest, cts.Token).ConfigureAwait(false))
{
// Response status is deliberately ignored — endpoint always returns 200.
return true;
}
}
}
catch (Exception ex)
{
_logger?.Debug($"ReportAgentErrorAsync: emergency channel failed: {ex.Message}");
return false;
}
}
@@ -57,19 +57,33 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
/// (intended to be called as fire-and-forget via <c>_ = reporter.TrySendAsync(...)</c>).
///
/// All anti-flood checks are applied synchronously before the async HTTP call,
/// so repeated calls are cheap when suppressed.
/// so repeated calls are cheap when suppressed. The anti-flood reservation is
/// taken up front (so concurrent same-key calls cannot double-send) but ROLLED
/// BACK when every attempt fails: the budget and the once-per-session dedup are
/// meant to count DELIVERED reports — the previous mark-before-send made the very
/// first transport failure permanently suppress that error key for the session
/// (found via the 48h emergency-break report, which is a single shot at boot).
///
/// <paramref name="attempts"/>/<paramref name="perAttemptTimeout"/>/<paramref name="retryDelay"/>
/// exist for callers whose process is about to exit and cannot come back later
/// (emergency break): retries stay INSIDE the one reservation, so they never
/// multiply against the per-session budget.
/// </summary>
public virtual async Task TrySendAsync(
AgentErrorType errorType,
string message,
int? httpStatusCode = null,
long? sequenceNumber = null)
long? sequenceNumber = null,
int attempts = 1,
TimeSpan? perAttemptTimeout = null,
TimeSpan? retryDelay = null)
{
// Deduplicate by error type + status code: same failure category sent only once per session.
// All anti-flood checks are protected by a lock to prevent race conditions when
// TrySendAsync is called concurrently (fire-and-forget from multiple threads).
var key = $"{errorType}:{httpStatusCode}";
int currentReportCount;
DateTime? previousLastReportTime;
lock (_antiFloodLock)
{
@@ -92,6 +106,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
return;
}
previousLastReportTime = _lastReportTime;
_reportedKeys.Add(key);
_reportCount++;
_lastReportTime = DateTime.UtcNow;
@@ -113,15 +128,40 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
Timestamp = DateTime.UtcNow
};
var delivered = false;
var totalAttempts = attempts < 1 ? 1 : attempts;
try
{
await _apiClient.ReportAgentErrorAsync(report);
for (var attempt = 1; attempt <= totalAttempts && !delivered; attempt++)
{
if (attempt > 1 && retryDelay.HasValue)
{
await Task.Delay(retryDelay.Value).ConfigureAwait(false);
}
delivered = await _apiClient.ReportAgentErrorAsync(report, perAttemptTimeout).ConfigureAwait(false);
if (!delivered && totalAttempts > 1)
{
_logger?.Debug($"[EmergencyChannel] Attempt {attempt}/{totalAttempts} failed: {key}");
}
}
}
catch (Exception ex)
{
// Swallow — emergency channel must never cascade failures into the caller
_logger?.Debug($"[EmergencyChannel] Failed to send report: {ex.Message}");
}
if (!delivered)
{
lock (_antiFloodLock)
{
_reportedKeys.Remove(key);
_reportCount--;
_lastReportTime = previousLastReportTime;
}
_logger?.Debug($"[EmergencyChannel] Report not delivered — reservation rolled back: {key}");
}
}
}
}
@@ -6,6 +6,7 @@ using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Configuration;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Runtime;
@@ -93,6 +94,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
// correction between agent start and termination no longer skews bootToStartSeconds.
private readonly ulong _bootAnchorTickMs;
private readonly DateTime _bootAnchorUtc;
// WinRT OOBE state at construction (handler is built at agent start) — corroborates
// the late-start heuristic: "completed" here means OOBE was already over on arrival.
private readonly string _oobeStateAtAgentStart;
private int _handled;
public EnrollmentTerminationHandler(
@@ -129,6 +133,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
_bootAnchorTickMs = ObservationCoverage.CurrentTickMs();
_bootAnchorUtc = DateTime.UtcNow;
_oobeStateAtAgentStart = OobeStateReader.Read();
}
/// <summary>
@@ -658,6 +663,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
{ "agentUptimeSeconds", Math.Round(uptimeSeconds, 1) },
{ "deviceBootUtc", deviceBootUtc.ToString("o") },
{ "agentStartUtc", _agentStartTimeUtc.ToString("o") },
{ "oobeStateAtAgentStart", _oobeStateAtAgentStart },
{ "outcome", args.Outcome.ToString() },
{ "note", "The agent's bootstrap (an Intune platform script) ran only near the end of the enrollment, so the agent observed the already-decided end-state rather than the failure window. Treat the diagnosis as a post-mortem; check for a platform/remediation script that hung ahead of the bootstrap (see script_timeout_suspected)." },
},
@@ -1,4 +1,6 @@
using System;
using System.Net.NetworkInformation;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Configuration;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Shared.Models;
@@ -15,28 +17,78 @@ namespace AutopilotMonitor.Agent.V2.Runtime
/// classifier terminalize the session instead of waiting out the grace. Fully swallowed — a send
/// failure (no network, no cert) must never delay the cleanup/exit the break exists to perform.
/// See tasks/enrollment-status-reclassification.md.
/// <para>
/// Delivery hardening (2026-07-23): this is the FIRST request of a freshly booted process and
/// the process never comes back — a miss is gone forever. The old single shot with a 5 s cap
/// died on boot-time realities (NIC still associating, cold TLS/DNS, backend cold starts
/// observed as &gt;5 s 503s). Now: wait briefly for the NIC, then up to
/// <see cref="SendAttempts"/> attempts with a per-attempt timeout that survives a Functions
/// cold start. The session is 48 h dead — spending up to ~75 s here is irrelevant, losing the
/// only signal that the agent gave up is not.
/// </para>
/// </summary>
internal static class EmergencyBreakReporter
{
private static readonly TimeSpan NetworkWaitMax = TimeSpan.FromSeconds(15);
private const int SendAttempts = 3;
private static readonly TimeSpan PerAttemptTimeout = TimeSpan.FromSeconds(15);
private static readonly TimeSpan RetryDelay = TimeSpan.FromSeconds(5);
// Upper bound for the blocking wait: NIC wait + 3×15s attempts + 2×5s delays + slack.
private static readonly TimeSpan OverallBudget = TimeSpan.FromSeconds(75);
public static void TrySend(AgentConfiguration agentConfig, string agentVersion, AgentLogger logger)
{
try
{
WaitForNetwork(logger);
var auth = BackendClientFactory.BuildAuthClients(agentConfig, agentVersion, logger);
var message =
$"Agent absolute session-age emergency break fired (cap {agentConfig.AbsoluteMaxSessionHours}h) — cleaning up and exiting.";
// The process is about to exit, so block briefly on the otherwise fire-and-forget send
// rather than let it be abandoned mid-flight. TrySendAsync swallows its own HTTP errors;
// the auth bundle's HttpClients are intentionally left for process teardown.
// The process is about to exit, so block on the otherwise fire-and-forget send
// rather than let it be abandoned mid-flight. TrySendAsync swallows its own HTTP
// errors and keeps the retries inside one anti-flood reservation; the auth
// bundle's HttpClients are intentionally left for process teardown.
auth.EmergencyReporter
.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, message)
.Wait(TimeSpan.FromSeconds(5));
.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak,
message,
attempts: SendAttempts,
perAttemptTimeout: PerAttemptTimeout,
retryDelay: RetryDelay)
.Wait(OverallBudget);
}
catch (Exception ex)
{
logger?.Debug($"Emergency-break report send failed (best-effort): {ex.Message}");
}
}
/// <summary>
/// Boot-time NIC grace: the break fires during bootstrap, often seconds after boot while
/// Wi-Fi is still associating. Polls the cheap link-level signal — it cannot prove backend
/// reachability (the send attempts handle that), it only avoids burning the first attempts
/// into a known-dead link. Best-effort: probe errors abort the wait, never the send.
/// </summary>
private static void WaitForNetwork(AgentLogger logger)
{
try
{
if (NetworkInterface.GetIsNetworkAvailable()) return;
logger?.Info($"Emergency-break report: no network link yet — waiting up to {NetworkWaitMax.TotalSeconds:F0}s.");
var deadline = DateTime.UtcNow + NetworkWaitMax;
while (DateTime.UtcNow < deadline)
{
Thread.Sleep(TimeSpan.FromSeconds(1));
if (NetworkInterface.GetIsNetworkAvailable()) return;
}
}
catch (Exception ex)
{
logger?.Debug($"Emergency-break report: network probe failed ({ex.Message}) — sending anyway.");
}
}
}
}
@@ -2,6 +2,7 @@ using System;
using System.Collections.Generic;
using AutopilotMonitor.Agent.V2.Core.Configuration;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Security;
@@ -68,6 +69,10 @@ namespace AutopilotMonitor.Agent.V2.Runtime
{ "configVersion", configVersion },
{ "remoteConfigFetched", outcome == RemoteConfigFetchOutcome.Succeeded },
{ "remoteConfigOutcome", outcome.ToString() },
// WinRT OOBE state sampled at this start. Together with previousExitType
// this makes late starts provable: "completed" on a first_run means OOBE
// was already over when the agent arrived (e.g. IME queue blocked).
{ "oobeStateAtAgentStart", OobeStateReader.Read() },
};
if (!string.IsNullOrEmpty(previousExit?.CrashExceptionType))
@@ -1,7 +1,15 @@
using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Functions.Ingest;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Functions.Services.Notifications;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
namespace AutopilotMonitor.Functions.Tests;
@@ -80,4 +88,116 @@ public class ReportAgentErrorFunctionTests
var evt = ReportAgentErrorFunction.BuildAgentEmergencyBreakEvent(Report(Now, message: ""), TenantId, null!, Now);
Assert.Contains("emergency break", evt.Message, StringComparison.OrdinalIgnoreCase);
}
// =========================================================================
// MaterializeEmergencyBreakArtifactsAsync — the artifact set per break report
// (2026-07-23 hardening): timeline event + cross-session EventType index row +
// AgentEmergencyBreak ops event, once per session. The index upsert is load-bearing:
// StoreEventsBatchAsync alone leaves the event invisible to every search-by-eventType
// surface (found in the 2026-07-22 incident analysis).
// =========================================================================
private sealed class Harness
{
public readonly Mock<ISessionRepository> SessionRepo = new();
public readonly List<OpsEventEntry> OpsEvents = new();
public readonly OpsEventService OpsService;
public Harness(List<EnrollmentEvent>? existingEvents = null)
{
SessionRepo.Setup(r => r.GetSessionEventsAsync(TenantId, SessionId, It.IsAny<int>()))
.ReturnsAsync(existingEvents ?? new List<EnrollmentEvent>());
SessionRepo.Setup(r => r.StoreEventsBatchAsync(It.IsAny<List<EnrollmentEvent>>()))
.ReturnsAsync((List<EnrollmentEvent> e) => e);
SessionRepo.Setup(r => r.UpsertEventTypeIndexBatchAsync(
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<IEnumerable<EnrollmentEvent>>()))
.Returns(Task.CompletedTask);
var opsRepo = new Mock<IOpsEventRepository>();
opsRepo.Setup(r => r.SaveOpsEventAsync(It.IsAny<OpsEventEntry>()))
.Callback<OpsEventEntry>(e => { lock (OpsEvents) OpsEvents.Add(e); })
.Returns(Task.CompletedTask);
var adminConfig = new Mock<AdminConfigurationService>(
Mock.Of<IConfigRepository>(), NullLogger<AdminConfigurationService>.Instance,
new MemoryCache(new MemoryCacheOptions()));
var alertDispatch = new OpsAlertDispatchService(
adminConfig.Object,
new TelegramNotificationService(new HttpClient(), Mock.Of<IConfigRepository>(),
NullLogger<TelegramNotificationService>.Instance),
new WebhookNotificationService(new HttpClient(),
NullLogger<WebhookNotificationService>.Instance),
NullLogger<OpsAlertDispatchService>.Instance);
OpsService = new OpsEventService(opsRepo.Object, NullLogger<OpsEventService>.Instance, alertDispatch);
}
public Task RunAsync(AgentErrorReport report)
=> ReportAgentErrorFunction.MaterializeEmergencyBreakArtifactsAsync(
report, TenantId, SessionRepo.Object, OpsService, NullLogger<ReportAgentErrorFunction>.Instance);
}
[Fact]
public async Task First_break_report_materializes_event_index_row_and_ops_event()
{
var h = new Harness();
await h.RunAsync(Report(Now));
h.SessionRepo.Verify(r => r.StoreEventsBatchAsync(
It.Is<List<EnrollmentEvent>>(e => e.Count == 1
&& e[0].EventType == AutopilotMonitor.Shared.Constants.EventTypes.AgentEmergencyBreak)), Times.Once);
h.SessionRepo.Verify(r => r.UpsertEventTypeIndexBatchAsync(
TenantId, SessionId, It.IsAny<IEnumerable<EnrollmentEvent>>()), Times.Once);
var ops = Assert.Single(h.OpsEvents);
Assert.Equal("AgentEmergencyBreak", ops.EventType);
Assert.Equal(TenantId, ops.TenantId);
}
[Fact]
public async Task Already_materialized_session_produces_no_second_artifact_set()
{
var h = new Harness(new List<EnrollmentEvent>
{
new() { EventType = AutopilotMonitor.Shared.Constants.EventTypes.AgentEmergencyBreak, Sequence = 45 },
});
await h.RunAsync(Report(Now));
h.SessionRepo.Verify(r => r.StoreEventsBatchAsync(It.IsAny<List<EnrollmentEvent>>()), Times.Never);
h.SessionRepo.Verify(r => r.UpsertEventTypeIndexBatchAsync(
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<IEnumerable<EnrollmentEvent>>()), Times.Never);
Assert.Empty(h.OpsEvents);
}
[Fact]
public async Task Non_break_error_types_and_missing_session_id_are_ignored()
{
var h = new Harness();
var other = Report(Now);
other.ErrorType = AgentErrorType.ConfigFetchFailed;
await h.RunAsync(other);
var noSession = Report(Now);
noSession.SessionId = null!;
await h.RunAsync(noSession);
h.SessionRepo.Verify(r => r.GetSessionEventsAsync(
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<int>()), Times.Never);
Assert.Empty(h.OpsEvents);
}
[Fact]
public async Task Storage_failure_is_swallowed_and_never_throws()
{
var h = new Harness();
h.SessionRepo.Setup(r => r.StoreEventsBatchAsync(It.IsAny<List<EnrollmentEvent>>()))
.ThrowsAsync(new InvalidOperationException("storage down"));
// Must not throw — the always-200 emergency channel must never turn into a retry loop.
await h.RunAsync(Report(Now));
Assert.Empty(h.OpsEvents);
}
}
@@ -37,6 +37,7 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
private readonly TelemetryClient _telemetryClient;
private readonly BootstrapSessionService _bootstrapSessionService;
private readonly ISessionRepository _sessionRepo;
private readonly OpsEventService _opsEventService;
public ReportAgentErrorFunction(
ILogger<ReportAgentErrorFunction> logger,
@@ -48,7 +49,8 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
DeviceAssociationValidator deviceAssociationValidator,
TelemetryClient telemetryClient,
BootstrapSessionService bootstrapSessionService,
ISessionRepository sessionRepo)
ISessionRepository sessionRepo,
OpsEventService opsEventService)
{
_logger = logger;
_configService = configService;
@@ -60,6 +62,7 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
_telemetryClient = telemetryClient;
_bootstrapSessionService = bootstrapSessionService;
_sessionRepo = sessionRepo;
_opsEventService = opsEventService;
}
[Function("ReportAgentError")]
@@ -169,35 +172,71 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
["AgentTimestamp"] = report.Timestamp.ToString("O"),
});
// Materialize the agent's silent 48h emergency break as a timeline event so it shows in the
// session and the timeout classifier can see it (tasks/enrollment-status-reclassification.md).
// Best-effort — a failure here must never turn the always-200 emergency channel into a retry loop.
if (report.ErrorType == AgentErrorType.SessionAgeEmergencyBreak
&& !string.IsNullOrEmpty(report.SessionId))
{
try
{
var existing = await _sessionRepo.GetSessionEventsAsync(tenantId, report.SessionId, maxResults: 1000);
// Idempotency: the agent's emergency channel can send up to a few reports per session;
// only ever materialize one timeline event.
var alreadyMaterialized = existing.Any(e =>
string.Equals(e.EventType, Constants.EventTypes.AgentEmergencyBreak, StringComparison.OrdinalIgnoreCase));
if (!alreadyMaterialized)
{
var evt = BuildAgentEmergencyBreakEvent(report, tenantId, existing, DateTime.UtcNow);
await _sessionRepo.StoreEventsBatchAsync(new List<EnrollmentEvent> { evt });
}
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"ReportAgentError: failed to materialize agent_emergency_break event for session {SessionId}", report.SessionId);
}
}
await MaterializeEmergencyBreakArtifactsAsync(report, tenantId, _sessionRepo, _opsEventService, _logger);
return req.CreateResponse(HttpStatusCode.OK);
}
/// <summary>
/// Materializes the agent's silent 48h emergency break as (1) a timeline event so it shows
/// in the session and the timeout classifier can see it, (2) the cross-session EventType
/// index row, and (3) an <c>AgentEmergencyBreak</c> ops event for operator visibility
/// (tasks/enrollment-status-reclassification.md). Static seam with explicit dependencies so
/// tests can pin the artifact set without booting the Functions HTTP stack.
/// Best-effort — a failure here must never turn the always-200 emergency channel into a
/// retry loop.
/// </summary>
internal static async Task MaterializeEmergencyBreakArtifactsAsync(
AgentErrorReport report,
string tenantId,
ISessionRepository sessionRepo,
OpsEventService opsEventService,
ILogger logger)
{
if (report.ErrorType != AgentErrorType.SessionAgeEmergencyBreak
|| string.IsNullOrEmpty(report.SessionId))
{
return;
}
try
{
var existing = await sessionRepo.GetSessionEventsAsync(tenantId, report.SessionId, maxResults: 1000);
// Idempotency: the agent's emergency channel can send up to a few reports per session;
// only ever materialize one timeline event (and one ops event).
var alreadyMaterialized = existing.Any(e =>
string.Equals(e.EventType, Constants.EventTypes.AgentEmergencyBreak, StringComparison.OrdinalIgnoreCase));
if (alreadyMaterialized)
{
return;
}
var evt = BuildAgentEmergencyBreakEvent(report, tenantId, existing, DateTime.UtcNow);
await sessionRepo.StoreEventsBatchAsync(new List<EnrollmentEvent> { evt });
// StoreEventsBatchAsync writes the Events partition only — the cross-session
// EventType index is normally written by EventIngestProcessor, which this
// backend-materialized event never passes through. Without the upsert the event
// exists on the session timeline but is invisible to every search-by-eventType
// surface (portal cross-session search, MCP search_sessions_by_event /
// query_raw_events) — found the hard way in the 2026-07-22 incident analysis.
await sessionRepo.UpsertEventTypeIndexBatchAsync(
tenantId, report.SessionId, new List<EnrollmentEvent> { evt });
// Operator visibility: an emergency break means an agent silently gave up at its
// absolute age cap — exactly the "are we losing agents?" signal. Emitted only on
// first materialization so repeat reports for the same session cannot flood the
// ops feed. OpsEventService never throws.
await opsEventService.RecordAgentEmergencyBreakAsync(
tenantId, report.SessionId, report.AgentVersion, evt.Message);
}
catch (Exception ex)
{
logger.LogWarning(ex,
"ReportAgentError: failed to materialize agent_emergency_break event for session {SessionId}", report.SessionId);
}
}
/// <summary>
/// Builds the backend-materialized <c>agent_emergency_break</c> timeline event from the agent's
/// best-effort emergency report. Static + pure (analog to
@@ -439,6 +439,13 @@ namespace AutopilotMonitor.Functions.Services
var timeoutEvent = BuildSessionTimeoutEvent(session, timeoutHours, sessionEvents, now);
await _sessionRepo.StoreEventsBatchAsync(new List<EnrollmentEvent> { timeoutEvent });
// Backend-materialized events bypass EventIngestProcessor and thus the
// cross-session EventType index — without this upsert session_timeout is
// invisible to every search-by-eventType surface (portal cross-session
// search, MCP search_sessions_by_event / query_raw_events).
await _sessionRepo.UpsertEventTypeIndexBatchAsync(
session.TenantId, session.SessionId, new List<EnrollmentEvent> { timeoutEvent });
// StoreEventsBatchAsync only bumps the orphan side-index, not the
// session's EventCount, and the terminal reconcile already ran inside
// UpdateSessionStatusAsync *before* this synthetic event existed. Recount
@@ -493,6 +493,20 @@ namespace AutopilotMonitor.Functions.Services
$"{sessionCount} session(s) timed out after {timeoutHours}h",
tenantId, "System.Maintenance", new { sessionCount, timeoutHours });
/// <summary>
/// An agent reported its absolute session-age emergency break (48h cap) over the
/// emergency channel — it cleaned itself up and exited on a session that never reached
/// a terminal state. This is the "are we silently losing agents?" signal; emitted by
/// <see cref="Functions.Ingest.ReportAgentErrorFunction"/> once per session (guarded by
/// the timeline-event idempotency check). Warning-tier so operators can wire a Telegram
/// rule; if real-world volume turns out noisy, downgrade or remove — the timeline event
/// on the session is the durable record.
/// </summary>
public Task RecordAgentEmergencyBreakAsync(string tenantId, string sessionId, string? agentVersion, string message)
=> WriteAsync(OpsEventCategory.Agent, "AgentEmergencyBreak", OpsEventSeverity.Warning,
$"Agent emergency break on session {sessionId} (agent {agentVersion ?? "?"}): {message}",
tenantId, "System.EmergencyChannel", new { sessionId, agentVersion });
public Task RecordExcessiveSessionEventsAsync(string tenantId, string sessionId, int eventCount, int threshold)
=> WriteAsync(OpsEventCategory.Agent, "ExcessiveSessionEvents", OpsEventSeverity.Warning,
$"Session {sessionId} has {eventCount} events (threshold {threshold}) — likely agent loop bug",
@@ -34,6 +34,7 @@ export const EVENT_TYPES_CATALOG = {
'desktop_detector_started',
'desktop_detector_first_poll',
'desktop_detector_no_candidate',
'oobe_state_completed',
],
whiteglove_events: [
'whiteglove_complete',
@@ -317,6 +317,12 @@ namespace AutopilotMonitor.Shared
public const string DesktopDetectorStarted = "desktop_detector_started"; // 1x on Start() / ResetForRealUserSwitch()
public const string DesktopDetectorFirstPoll = "desktop_detector_first_poll"; // 1x after first PollForDesktop() completes
public const string DesktopDetectorNoCandidate = "desktop_detector_no_candidate"; // 1x after threshold polls without resolution (configurable)
// 1x when WinRT SystemSetupInfo.OutOfBoxExperienceState flips InProgress->Completed,
// sampled on the DAD 30s poll. Observational only (no decision-engine involvement) —
// an owner-independent desktop corroboration for sessions where WTS+WMI owner
// resolution starves and desktop_arrived never fires. Often absent in healthy
// sessions: the DAD timer is disposed at desktop arrival, which is when the flip lands.
public const string OobeStateCompleted = "oobe_state_completed";
public const string CompletionCheck = "completion_check";
// Liveness plan PR2 — emitted (state-change-only, deduped via the
// CompletionWaitingFingerprint state fact) whenever the DecisionEngine blocks or
@@ -95,7 +95,17 @@ const OPS_EVENT_TYPES: Record<string, string[]> = {
"TenantTrialExpiring",
"TenantTrialExpired",
],
Agent: ["BlobStorageMissing", "BlobStorageUnreachable", "NewImeVersionDetected", "ExcessiveSessionEvents"],
Agent: [
"BlobStorageMissing",
"BlobStorageUnreachable",
"NewImeVersionDetected",
"ExcessiveSessionEvents",
// 48h session-age emergency break reported over the agent's emergency channel — the
// "are we silently losing agents?" signal. Emitted once per session by
// ReportAgentErrorFunction (RecordAgentEmergencyBreakAsync). Dual-register per memory
// feedback_ops_event_types_dual_register.
"AgentEmergencyBreak",
],
};
const SEVERITIES = ["Info", "Warning", "Error", "Critical"];
@@ -32,6 +32,8 @@ export const KNOWN_EVENT_TYPES: EventTypeEntry[] = [
description: "Pre-Provisioning Part 2 resumed after the user powered on the device." },
{ value: "desktop_arrived", label: "desktop_arrived", category: "enrollment",
description: "User desktop detected (explorer.exe under real user)." },
{ value: "oobe_state_completed", label: "oobe_state_completed", category: "enrollment",
description: "Windows OOBE state flipped InProgress→Completed (WinRT SystemSetupInfo, sampled on the desktop-arrival poll). One-shot and observational only; valuable as an owner-independent desktop corroboration in sessions where desktop_arrived never fires." },
{ value: "completion_check", label: "completion_check", category: "enrollment",
description: "Throttled state-machine snapshot during completion evaluation." },
{ value: "completion_waiting", label: "completion_waiting", category: "enrollment",
@@ -64,9 +64,13 @@ export default function AgentAnalyzersSection({
trimmed !== "" &&
(localAdminAllowedAccounts.some((a) => a.toLowerCase() === trimmed.toLowerCase()) ||
BUILTIN_ACCOUNTS.some((a) => a.toLowerCase() === trimmed.toLowerCase()));
// Wildcard patterns (* / ?) need at least 2 literal characters — a bare "*"
// would silently disable the analyzer for the whole tenant.
const isTooBroad =
/[*?]/.test(trimmed) && trimmed.replace(/[*?]/g, "").length < 2;
const addAccount = () => {
if (!trimmed || isDuplicate) return;
if (!trimmed || isDuplicate || isTooBroad) return;
setLocalAdminAllowedAccounts([...localAdminAllowedAccounts, trimmed]);
setNewAllowedAccount("");
};
@@ -114,6 +118,10 @@ export default function AgentAnalyzersSection({
<p className="font-medium text-gray-900 mb-1">Allowed Local Accounts</p>
<p className="text-sm text-gray-500 mb-3">
Accounts listed here are considered expected on enrolled devices and will not trigger alerts. These are merged with the built-in defaults on the agent.
Wildcards are supported: <code className="px-1 bg-gray-100 rounded">*</code> matches any sequence of characters,{" "}
<code className="px-1 bg-gray-100 rounded">?</code> exactly one — e.g.{" "}
<code className="px-1 bg-gray-100 rounded">adm-*</code> allows generated accounts like <code className="px-1 bg-gray-100 rounded">adm-83921</code>.
Matching is case-insensitive and applies to both account names and profile folders.
</p>
{/* Built-in accounts (read-only) */}
@@ -155,7 +163,7 @@ export default function AgentAnalyzersSection({
<div className="flex gap-2 mt-2">
<input
type="text"
placeholder="Account name (e.g. SupportAdmin)"
placeholder="Account name or pattern (e.g. SupportAdmin, adm-*)"
value={newAllowedAccount}
onChange={(e) => setNewAllowedAccount(e.target.value)}
onKeyDown={(e) => { if (e.key === "Enter") { e.preventDefault(); addAccount(); } }}
@@ -163,7 +171,7 @@ export default function AgentAnalyzersSection({
/>
<button
onClick={addAccount}
disabled={!trimmed || isDuplicate}
disabled={!trimmed || isDuplicate || isTooBroad}
className="px-4 py-1.5 bg-rose-600 text-white rounded-lg text-sm font-medium hover:bg-rose-700 disabled:opacity-50 disabled:cursor-not-allowed transition-colors whitespace-nowrap"
>
Add
@@ -172,6 +180,11 @@ export default function AgentAnalyzersSection({
{isDuplicate && (
<p className="text-xs text-red-500 mt-1">This account is already in the list.</p>
)}
{isTooBroad && (
<p className="text-xs text-red-500 mt-1">
Pattern is too broad — it needs at least 2 characters besides <code>*</code>/<code>?</code>, otherwise the analyzer would be effectively disabled.
</p>
)}
</div>
{/* ── Divider ─────────────────────────────────────── */}