fix(classification): self-deploying profiles reconcile to Succeeded after Device Setup, never AwaitingUser

Kiosk tenant audit 2026-08-23: 836 of ~970 self-deploying sessions in one week
were parked AwaitingUser ("Account Setup 0/5") and graduated to Incomplete 51h
later although the devices were finished — the profile has no user phase, so the
classifier's rule 5 was asking for evidence that can never arrive.

- EnrollmentTimeoutClassifier: new rule 1c IsSelfDeployingProvisioned (profile flag +
  Device Setup all-succeeded + no explicit failure) => Succeeded reconcile with an
  honest reason; isSelfDeployingProfile passed by every caller (sweep stage 1+2,
  max-lifetime verdict, late-telemetry heal, legacy retro mode).
- MaintenanceService stage 1: TryReconcileSelfDeployingAsync gate before the Stalled
  marker (mirrors the WhiteGlove Part-2 gate), audit counter SessionsSelfDeployingReconciled.
- LegacyReclassificationService: new admin mode self_deploying_silent (heal-only,
  dry-run by default) + GetSelfDeployingSilentSessionsAsync for the existing backlog.
- Agent SessionRegistrationHelper: wait up to 15s for a network link and retry 6x
  (~62s) instead of 5x (~30s) so the BootTrigger relaunch on Wi-Fi no longer exits 7
  before the link is back (the mechanism that left these sessions silent).
- Tests: 6 classifier cases (gate, within/past grace, user-driven guard, failure,
  pre-Device-Setup), 2 registration tests; docs/backend/silent-session-classification.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Oliver Kieselbach
2026-08-23 09:55:06 +02:00
parent 87b7a20223
commit 98f8114d90
16 changed files with 541 additions and 19 deletions
@@ -0,0 +1,110 @@
---
type: Concept
title: Silent-Session Classification (Sweep, Max-Lifetime, Retro-Reconcile)
description: How a session whose agent went silent gets its honest status — the two-stage maintenance sweep (2h Stalled marker, 5h timeout classification), the EnrollmentTimeoutClassifier rule order incl. the WhiteGlove Part-2 and self-deploying gates, which ingest paths share the classifier, and the admin retro-reconcile modes.
resource: src/Backend/AutopilotMonitor.Functions/Services/EnrollmentTimeoutClassifier.cs
tags:
- backend
- maintenance
- session-status
- classification
- whiteglove
- self-deploying
timestamp: 2026-08-23T18:00:00+02:00
---
# Silent-Session Classification
A session only ever becomes terminal through evidence. When the agent stops reporting
(power-off, reboot without relaunch, Wi-Fi never re-associating, max-lifetime watchdog),
the backend must decide the status from what it last saw — it never hard-fails a session
for being silent. One pure function holds that decision,
`EnrollmentTimeoutClassifier.ClassifyTimedOutSession`, and every silent-session path calls it
so a session gets the same verdict regardless of which path noticed the silence first.
# Schema
## Inputs (`EspProvisioningRollup` + session flags)
`ExtractRollup(events)` distills the event stream into facts: Device/Account Setup
subcategory counts (incl. the 30s fallback-confirmed messages), `HasExplicitFailure`,
`HasEnrollmentComplete` vs. `HasTerminalComplete` (the Part-1 `whiteglove_complete`),
`HasAgentEmergencyBreak`, `DesktopArrived`, `HelloResolved`, `HelloPolicyDisabled`,
`SkipUserEsp`, RealmJoin detection/resolution. Session-row flags are passed alongside:
`IsPreProvisioned` + `ResumedAt` (WhiteGlove Part 2) and `IsSelfDeployingProfile`
(registry-confirmed `CloudAssignedOobeConfig 0x20|0x40`, set at registration).
## Rule order (first match wins)
| # | Condition | Verdict |
| --- | --- | --- |
| 1 | `HasExplicitFailure` | Failed |
| 1b | `IsWhiteGloveAwaitingUser`: pre-provisioned + resumed + Device Setup done + **no** user evidence since resume + no explicit failure/enrollment_complete | AwaitingUser within grace; Succeeded (honest WG reason) past grace |
| 1c | `IsSelfDeployingProvisioned`: self-deploying profile + Device Setup done + no explicit failure | **Succeeded** (reconcile) — no grace, no AwaitingUser: the profile has no user phase |
| 2 | Account Setup all-succeeded, or `enrollment_complete`, or `whiteglove_complete` on a **non-resumed** session | Succeeded |
| 3 | `HasAgentEmergencyBreak` | Incomplete immediately (agent self-destructed, nothing can arrive) |
| 4 | `DesktopArrived` + (Hello terminal, or Hello disabled + SkipUserEsp) | Succeeded (user provably finished) |
| 5 | Device Setup done | AwaitingUser within grace, Incomplete past grace |
| 6 | otherwise | Incomplete |
Grace = `ResolveGraceHours(SessionGraceHours, AbsoluteMaxSessionHours)` (~51h default),
measured from `ResumedAt ?? StartedAt`. Every Succeeded-reconcile reason carries the
silence-transparency clause (`AppendReconcileTiming`: last agent contact, silence, verdict time).
## Why rule 1c exists (kiosk tenant audit 2026-08-23)
On a self-deploying profile Device ESP all-succeeded IS the enrollment's end; the agent's
own SelfDeploying terminal only adds a 5-min confirmation window
(`DecisionEngine.SelfDeploying.s_deviceOnlyEspDetectionWindow`). Tenant `aebdce78` (school,
Wi-Fi ThinkBooks, a language-change app forcing a reboot right after Device ESP) lost that
window on 836 of ~970 sessions in one week: the agent died 0–4 min after
`DeviceSetup fallback_confirmed`, the reboot relaunch never registered (30s registration
budget on a Wi-Fi link still associating), and rule 5 labeled finished kiosks
"awaiting user / Account Setup 0/5" → Incomplete 51h later. Nobody signs in on these devices
(prepared during the holidays, boxed, in service) — AwaitingUser was factually wrong and
Incomplete turned a working rollout into a red wall. Other self-deploying tenants completed
normally in the same window, so this is a classification gap, not a profile-wide agent bug.
Agent-side companion fix: `SessionRegistrationHelper` now waits up to 15s for a network link
and retries 6× (~62s backoff) instead of 5× (~30s).
## Callers (all pass both gate inputs)
| Path | File | Notes |
| --- | --- | --- |
| Sweep stage 1 (2h agent-silent marker) | `MaintenanceService.MarkStalledSessionsAsTimedOutAsync` | Default: InProgress → Stalled. Gates run BEFORE the Stalled write: `TryMarkWhiteGloveAwaitingUserAsync` (AwaitingUser) and `TryReconcileSelfDeployingAsync` (Succeeded). Only flagged sessions pay the event read. |
| Sweep stage 2 (SessionTimeoutHours, 5h default) | same | InProgress/Stalled/AwaitingUser past the window and past the 2h silence cutoff → full classification; AwaitingUser within grace is skipped without an event read. |
| Agent max-lifetime (`enrollment_failed`/max_lifetime, `agent_shutting_down`/max_lifetime) | `EventIngestProcessor.ApplyMaxLifetimeVerdictAsync` | Watchdog = "agent stopped waiting", not a verdict; classified from evidence. Pending sessions exempt. |
| Late-telemetry reconcile | `EventIngestProcessor.TryLateTelemetryReconcileAsync` | Heal-only: applies a Succeeded verdict to Incomplete/AwaitingUser rows when straggler events arrive. |
| Agent stall probe (`session_stalled`) | `EventIngestProcessor.TryMapStallToWhiteGloveAwaitingUserAsync` | Only the WG gate (1b); a stall probe on a self-deploying device with Device Setup done means the agent is alive and something else blocks — stays Stalled. |
| Admin retro-reconcile | `LegacyReclassificationService` | Modes `legacy_timeouts`, `pending_orphans`, `self_deploying_silent` (Incomplete/AwaitingUser/Stalled rows with the profile flag, heal-only to Succeeded, admin-marked rows skipped). `POST /api/maintenance/reclassify-legacy?mode=…&dryRun=false&tenantId=…&maxSessions=…`, dry-run by default. |
## Known gap (decided, not built)
After max-lifetime the agent writes `enrollment-complete.marker` and self-destructs, so an
AwaitingUser verdict from that path can never heal — it is a 51h countdown to Incomplete.
The emergency-break path (rule 3) already terminalizes immediately; max-lifetime still goes
through the grace window by design ("user simply absent"). Revisit if customers read
AwaitingUser as "still possible".
# Examples
* Session `195593e2` (tenant aebdce78): `DeviceSetup 4/4` + `fallback_confirmed`, `AccountSetup 0/5`
(IME false positive), Hello disabled, SkipUser=True, silent after 13:01 → rule 1c →
Succeeded "self-deploying profile — Device Setup fully provisioned … agent went silent
before confirming completion. Agent last reported 2026-08-21 13:01 UTC …".
* Same event shape on a user-driven profile → rule 5 → AwaitingUser (within grace) /
Incomplete (past grace). Guarded by
`Classify_same_stream_without_self_deploying_flag_keeps_user_phase_verdicts`.
* fairstone WG Part 2 reseal-reboot, technician powers off at logon screen → rule 1b →
AwaitingUser; user unboxes days later → ingest completion heals to Succeeded.
# Citations
* `src/Backend/AutopilotMonitor.Functions/Services/EnrollmentTimeoutClassifier.cs` — rollup + rule order, `IsWhiteGloveAwaitingUser`, `IsSelfDeployingProvisioned`, reason helpers.
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.cs` — `MarkStalledSessionsAsTimedOutAsync`, `TryMarkWhiteGloveAwaitingUserAsync`, `TryReconcileSelfDeployingAsync`.
* `src/Backend/AutopilotMonitor.Functions/Services/EventIngestProcessor.Classification.cs` — max-lifetime verdict, late-telemetry reconcile, stall mapping.
* `src/Backend/AutopilotMonitor.Functions/Services/LegacyReclassificationService.cs`, `Functions/Admin/ReclassifyLegacySessionsFunction.cs` — retro modes.
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.Maintenance.cs` — `GetAgentSilentSessionsAsync`, `GetStalledSessionsAsync`, `GetSelfDeployingSilentSessionsAsync`.
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Security/SessionRegistrationHelper.cs` — link wait + 6-attempt registration budget.
* `src/Shared/AutopilotMonitor.DecisionCore/Engine/DecisionEngine.SelfDeploying.cs` — the agent-side 5-min SelfDeploying terminal window.
* `src/Backend/AutopilotMonitor.Functions.Tests/EnrollmentTimeoutClassifierTests.cs` — rule coverage incl. the 2026-08-23 self-deploying block.
+1
View File
@@ -48,6 +48,7 @@ https://docs.autopilotmonitor.com (separate repository).
* [Business Timestamps](backend/business-timestamps.md) - Why the Azure Tables system Timestamp is never authoritative (migrations reset it), and the OccurredUtc + RowKey-decode compensation for AuditLogs/OpsEvents/Events after the 2026-07-18 migration. * [Business Timestamps](backend/business-timestamps.md) - Why the Azure Tables system Timestamp is never authoritative (migrations reset it), and the OccurredUtc + RowKey-decode compensation for AuditLogs/OpsEvents/Events after the 2026-07-18 migration.
* [Send-Time Frame Separation](backend/send-time-frame-separation.md) - One device-clock send timestamp per upload batch (X-Send-Time-Utc → SentAt on Events) splits ReceivedAt − OccurredUtc into pure spool delay and pure device-vs-server clock offset; the clock_skew condition measures SentAt batches directly (spool-immune), the CMTrace tripwire deliberately stays on its differential math. * [Send-Time Frame Separation](backend/send-time-frame-separation.md) - One device-clock send timestamp per upload batch (X-Send-Time-Utc → SentAt on Events) splits ReceivedAt − OccurredUtc into pure spool delay and pure device-vs-server clock offset; the clock_skew condition measures SentAt batches directly (spool-immune), the CMTrace tripwire deliberately stays on its differential math.
* [OOBE Bootstrap go-Code Flow](backend/oobe-bootstrap-go-flow.md) - The bootstrap short-code lifecycle (portal-created session → `irm go.autopilotmonitor.com/CODE | iex` at Shift+F10 → GetBootstrapScriptFunction), the always-HTTP-200 text/plain contract, the two-line injection defense (GUID/URL/expiry validator + single-quoted PS literals, golden-pinned against the legacy TS template), and the per-IP rate-limit story behind Front Door. * [OOBE Bootstrap go-Code Flow](backend/oobe-bootstrap-go-flow.md) - The bootstrap short-code lifecycle (portal-created session → `irm go.autopilotmonitor.com/CODE | iex` at Shift+F10 → GetBootstrapScriptFunction), the always-HTTP-200 text/plain contract, the two-line injection defense (GUID/URL/expiry validator + single-quoted PS literals, golden-pinned against the legacy TS template), and the per-IP rate-limit story behind Front Door.
* [Silent-Session Classification](backend/silent-session-classification.md) - How a session whose agent went silent gets its honest status: the two-stage maintenance sweep (2h Stalled marker with the WhiteGlove Part-2 AwaitingUser gate and the self-deploying Succeeded gate, 5h timeout classification), the EnrollmentTimeoutClassifier rule order (explicit failure → WG Part 2 → self-deploying → completion evidence → emergency break → desktop+Hello → Device Setup/grace), the ingest paths sharing it (max-lifetime, late-telemetry heal, stall probe) and the admin retro-reconcile modes incl. self_deploying_silent.
* [Device Journeys & First-Time-Right](backend/device-journeys.md) - How a device's terminal enrollment attempts are grouped into journeys (trim+casefold device key with junk-serial exclusion, first-success end, 30-day gap, WhiteGlove = one attempt), the DeviceHistories chain with its tombstone-driven deleted-session pruning, the additive daily FTR aggregates, and the banner/fleet/MCP surfaces. * [Device Journeys & First-Time-Right](backend/device-journeys.md) - How a device's terminal enrollment attempts are grouped into journeys (trim+casefold device key with junk-serial exclusion, first-success end, 30-day gap, WhiteGlove = one attempt), the DeviceHistories chain with its tombstone-driven deleted-session pruning, the additive daily FTR aggregates, and the banner/fleet/MCP surfaces.
* [Enrollment Time Attribution](backend/time-attribution.md) - How a terminal session's authoritative wall clock (DurationSeconds, WhiteGlove pause excluded) is partitioned into named segments with an explicit unattributed remainder, ESP-blocking apps as positive evidence with "up to" what-if bounds, reboot spans from event gaps, and the breakdown/aggregate tables the sweep maintains. * [Enrollment Time Attribution](backend/time-attribution.md) - How a terminal session's authoritative wall clock (DurationSeconds, WhiteGlove pause excluded) is partitioned into named segments with an explicit unattributed remainder, ESP-blocking apps as positive evidence with "up to" what-if bounds, reboot spans from event gaps, and the breakdown/aggregate tables the sweep maintains.
* [Attempt-Scoped App Install Durations](backend/app-install-duration-attempt.md) - AppInstallSummary.DurationSeconds = status-defining (last) install attempt via the LastAttemptStartedAt max-fold anchor instead of the first-observation→last-terminal span that billed multi-pass IME gaps to trivial apps; InstallPassCount, the weaker-terminal guard, the radar's cutover gate, and time attribution's union-of-attempt-segments occupancy. * [Attempt-Scoped App Install Durations](backend/app-install-duration-attempt.md) - AppInstallSummary.DurationSeconds = status-defining (last) install attempt via the LastAttemptStartedAt max-fold anchor instead of the first-observation→last-terminal span that billed multi-pass IME gaps to trivial apps; InstallPassCount, the weaker-terminal guard, the radar's cutover gate, and time attribution's union-of-attempt-segments occupancy.
+1
View File
@@ -2,6 +2,7 @@
## 2026-08-23 ## 2026-08-23
* **New**: `backend/silent-session-classification.md` — one place for the silent-session verdict: `EnrollmentTimeoutClassifier` rule order incl. the new rule 1c `IsSelfDeployingProvisioned` (self-deploying profile + Device Setup all-succeeded + no explicit failure ⇒ Succeeded reconcile, no grace/AwaitingUser — kiosk tenant aebdce78 audit: 836 finished devices parked AwaitingUser/Incomplete), the sweep stage-1 gates (`TryMarkWhiteGloveAwaitingUserAsync`, new `TryReconcileSelfDeployingAsync`), the ingest callers (max-lifetime, late-telemetry heal, stall probe) and the new admin retro mode `self_deploying_silent` (`GetSelfDeployingSilentSessionsAsync`). Agent companion: `SessionRegistrationHelper` waits up to 15s for a network link and retries 6× (~62s) instead of 5× (~30s) so the reboot relaunch on Wi-Fi no longer exits 7 before the link is back.
* **New**: `agent/diagnostics-package.md` — the diagnostics ZIP's built-in collection is now ONE catalog in Shared (`DiagnosticsBuiltInSections.All`): the agent iterates it in order and the backend serves the same list to the portal (`GET diagnostics/paths`, MemberRead), so what administrators see is exactly what the agent collects. New conditional sections: RealmJoin client/package/Chocolatey/per-user logs behind the tenant's RealmJoin Watcher toggle (mirrored onto `AgentConfiguration.EnableRealmJoinWatcher` by the merger) and the Device Preparation bootstrapper event log behind the deterministic WDP marker (probe once per build, fail toward Classic); every skip is a manifest line. Built-ins bypass `DiagnosticsPathGuards` by design (reviewed code, `%ProgramData%\RealmJoin` is not allowlisted); caps/reparse/evtx export still apply. `MergeDiagnosticsLogPaths` dedups global ∪ tenant paths. Portal: collapsed read-only catalog + one-line rows in both cards, tenant context dropped the GA-only global fetch. * **New**: `agent/diagnostics-package.md` — the diagnostics ZIP's built-in collection is now ONE catalog in Shared (`DiagnosticsBuiltInSections.All`): the agent iterates it in order and the backend serves the same list to the portal (`GET diagnostics/paths`, MemberRead), so what administrators see is exactly what the agent collects. New conditional sections: RealmJoin client/package/Chocolatey/per-user logs behind the tenant's RealmJoin Watcher toggle (mirrored onto `AgentConfiguration.EnableRealmJoinWatcher` by the merger) and the Device Preparation bootstrapper event log behind the deterministic WDP marker (probe once per build, fail toward Classic); every skip is a manifest line. Built-ins bypass `DiagnosticsPathGuards` by design (reviewed code, `%ProgramData%\RealmJoin` is not allowlisted); caps/reparse/evtx export still apply. `MergeDiagnosticsLogPaths` dedups global ∪ tenant paths. Portal: collapsed read-only catalog + one-line rows in both cards, tenant context dropped the GA-only global fetch.
* **Update**: `rules/gather-rule-guardrails.md` — `DiagnosticsPathGuards` scope clarified: configured diagnostics paths only, never the built-in catalog. * **Update**: `rules/gather-rule-guardrails.md` — `DiagnosticsPathGuards` scope clarified: configured diagnostics paths only, never the built-in catalog.
* **Update**: `backend/tenant-activation-welcome-mail.md` — new "Template overrides" section: Global Admins can replace the welcome/farewell HTML in the portal (Settings → Email Templates); `EmailTemplateService` resolves override-or-built-in per send (PreviewConfig table, partition `EmailTemplates`, 5-min cache, `{{domainName}}` placeholder, 30k limit, fail-soft send path), GA endpoints `global/email-templates/{kind}` (+`/test` to the caller's tenant contact address). Also: the mail header is image-free (CSS-drawn brand mark, gradient stripes and clip-path edge as progressive enhancement). * **Update**: `backend/tenant-activation-welcome-mail.md` — new "Template overrides" section: Global Admins can replace the welcome/farewell HTML in the portal (Settings → Email Templates); `EmailTemplateService` resolves override-or-built-in per send (PreviewConfig table, partition `EmailTemplates`, 5-min cache, `{{domainName}}` placeholder, 30k limit, fail-soft send path), GA endpoints `global/email-templates/{kind}` (+`/test` to the caller's tenant contact address). Also: the mail header is image-free (CSS-drawn brand mark, gradient stripes and clip-path edge as progressive enhancement).
@@ -97,7 +97,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Security
} }
[Fact] [Fact]
public async Task Fails_after_five_exceptions_without_auth_error() public async Task Fails_after_six_exceptions_without_auth_error()
{ {
using var tmp = new TempDirectory(); using var tmp = new TempDirectory();
var logger = NewLogger(tmp); var logger = NewLogger(tmp);
@@ -111,8 +111,50 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Security
backoffDelay: _ => Task.CompletedTask); backoffDelay: _ => Task.CompletedTask);
Assert.Equal(SessionRegistrationOutcome.Failed, result.Outcome); Assert.Equal(SessionRegistrationOutcome.Failed, result.Outcome);
Assert.Equal(5, apiClient.CallCount); Assert.Equal(6, apiClient.CallCount);
Assert.Contains("transient attempt 5", result.ErrorMessage); Assert.Contains("transient attempt 6", result.ErrorMessage);
}
[Fact]
public async Task Waits_for_network_link_before_first_attempt()
{
using var tmp = new TempDirectory();
var logger = NewLogger(tmp);
var config = NewConfig();
var order = new System.Collections.Generic.List<string>();
var apiClient = new FakeApiClient(_ =>
{
order.Add("register");
return new RegisterSessionResponse { Success = true, SessionId = config.SessionId };
});
var result = await SessionRegistrationHelper.RegisterWithRetryAsync(
apiClient, config, agentVersion: "0.0.0", logger: logger,
backoffDelay: _ => Task.CompletedTask,
networkLinkWait: _ => { order.Add("link-wait"); return Task.CompletedTask; });
Assert.Equal(SessionRegistrationOutcome.Succeeded, result.Outcome);
Assert.Equal(new[] { "link-wait", "register" }, order);
}
[Fact]
public async Task Network_link_wait_failure_does_not_block_registration()
{
using var tmp = new TempDirectory();
var logger = NewLogger(tmp);
var config = NewConfig();
var apiClient = new FakeApiClient(_ =>
new RegisterSessionResponse { Success = true, SessionId = config.SessionId });
var result = await SessionRegistrationHelper.RegisterWithRetryAsync(
apiClient, config, agentVersion: "0.0.0", logger: logger,
backoffDelay: _ => Task.CompletedTask,
networkLinkWait: _ => throw new InvalidOperationException("probe broken"));
Assert.Equal(SessionRegistrationOutcome.Succeeded, result.Outcome);
Assert.Equal(1, apiClient.CallCount);
} }
[Fact] [Fact]
@@ -1,4 +1,5 @@
using System; using System;
using System.Net.NetworkInformation;
using System.Threading.Tasks; using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Configuration; using AutopilotMonitor.Agent.V2.Core.Configuration;
using AutopilotMonitor.Agent.V2.Core.Logging; using AutopilotMonitor.Agent.V2.Core.Logging;
@@ -19,7 +20,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
/// <b>Retry contract (V1 <c>MonitoringService.RegisterSessionAsync</c>):</b> /// <b>Retry contract (V1 <c>MonitoringService.RegisterSessionAsync</c>):</b>
/// </para> /// </para>
/// <list type="bullet"> /// <list type="bullet">
/// <item>5 attempts total; between attempts <c>2^attempt</c> seconds (2s, 4s, 8s, 16s).</item> /// <item>Boot-time NIC grace before the first attempt: when no network link is up yet
/// (BootTrigger relaunch after a mid-enrollment reboot, Wi-Fi still associating) wait
/// up to <see cref="NetworkLinkWaitMax"/> for the link-level signal. Best-effort and
/// free on the normal path — a live link returns immediately.</item>
/// <item>6 attempts total; between attempts <c>2^attempt</c> seconds (2s, 4s, 8s, 16s, 32s —
/// ~62 s of backoff). The original V1 budget of 5 attempts / ~30 s was too short for the
/// reboot-relaunch on Wi-Fi kiosks: the relaunched agent gave up and exited 7 before
/// the network came back, leaving the session silent forever (tenant aebdce78,
/// 2026-08-23 audit). A relaunch that misses here is lost — nothing retries later.</item>
/// <item>On <c>response.Success == true</c> → stop immediately.</item> /// <item>On <c>response.Success == true</c> → stop immediately.</item>
/// <item>On <see cref="BackendAuthException"/> (401/403) → report to the /// <item>On <see cref="BackendAuthException"/> (401/403) → report to the
/// <see cref="AuthFailureTracker"/> (which fires the first-failure distress + may trip /// <see cref="AuthFailureTracker"/> (which fires the first-failure distress + may trip
@@ -37,16 +46,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
/// </summary> /// </summary>
public static class SessionRegistrationHelper public static class SessionRegistrationHelper
{ {
private const int MaxAttempts = 5; private const int MaxAttempts = 6;
/// <summary>Upper bound for the pre-registration link wait (see class remarks).</summary>
internal static readonly TimeSpan NetworkLinkWaitMax = TimeSpan.FromSeconds(15);
/// <summary> /// <summary>
/// Executes the 5-retry register-session handshake against the backend. Returns the /// Executes the retrying register-session handshake against the backend. Returns the
/// response + classified outcome. Never throws. /// response + classified outcome. Never throws.
/// </summary> /// </summary>
/// <param name="backoffDelay"> /// <param name="backoffDelay">
/// Optional delay-provider used by tests to avoid real-time waits. Production callers /// Optional delay-provider used by tests to avoid real-time waits. Production callers
/// pass <c>null</c> → V1 parity <c>Task.Delay(2^attempt * 1000)</c> between attempts. /// pass <c>null</c> → V1 parity <c>Task.Delay(2^attempt * 1000)</c> between attempts.
/// </param> /// </param>
/// <param name="networkLinkWait">
/// Optional pre-registration link wait used by tests. Production callers pass
/// <c>null</c> → <see cref="WaitForNetworkLinkAsync"/> (bounded by <see cref="NetworkLinkWaitMax"/>).
/// </param>
public static async Task<SessionRegistrationResult> RegisterWithRetryAsync( public static async Task<SessionRegistrationResult> RegisterWithRetryAsync(
BackendApiClient apiClient, BackendApiClient apiClient,
AgentConfiguration agentConfig, AgentConfiguration agentConfig,
@@ -56,14 +72,19 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
EmergencyReporter emergencyReporter = null, EmergencyReporter emergencyReporter = null,
Func<int, Task> backoffDelay = null, Func<int, Task> backoffDelay = null,
Func<Exception, Task> onTerminalTransportFailure = null, Func<Exception, Task> onTerminalTransportFailure = null,
(string Manufacturer, string Model, string SerialNumber)? deviceHardware = null) (string Manufacturer, string Model, string SerialNumber)? deviceHardware = null,
Func<AgentLogger, Task> networkLinkWait = null)
{ {
if (apiClient == null) throw new ArgumentNullException(nameof(apiClient)); if (apiClient == null) throw new ArgumentNullException(nameof(apiClient));
if (agentConfig == null) throw new ArgumentNullException(nameof(agentConfig)); if (agentConfig == null) throw new ArgumentNullException(nameof(agentConfig));
if (logger == null) throw new ArgumentNullException(nameof(logger)); if (logger == null) throw new ArgumentNullException(nameof(logger));
backoffDelay ??= DefaultBackoffDelay; backoffDelay ??= DefaultBackoffDelay;
networkLinkWait ??= WaitForNetworkLinkAsync;
var registration = BuildRegistration(agentConfig, agentVersion, deviceHardware); var registration = BuildRegistration(agentConfig, agentVersion, deviceHardware);
try { await networkLinkWait(logger).ConfigureAwait(false); }
catch (Exception ex) { logger.Debug($"Pre-registration network wait failed ({ex.Message}) — registering anyway."); }
string lastError = null; string lastError = null;
Exception lastException = null; Exception lastException = null;
@@ -113,7 +134,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
if (attempt < MaxAttempts) if (attempt < MaxAttempts)
{ {
var delaySeconds = (int)Math.Pow(2, attempt); // 2, 4, 8, 16 var delaySeconds = (int)Math.Pow(2, attempt); // 2, 4, 8, 16, 32
logger.Info($"Retrying session registration in {delaySeconds}s"); logger.Info($"Retrying session registration in {delaySeconds}s");
await backoffDelay(attempt).ConfigureAwait(false); await backoffDelay(attempt).ConfigureAwait(false);
} }
@@ -143,6 +164,31 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
private static Task DefaultBackoffDelay(int attempt) private static Task DefaultBackoffDelay(int attempt)
=> Task.Delay(((int)Math.Pow(2, attempt)) * 1000); => Task.Delay(((int)Math.Pow(2, attempt)) * 1000);
/// <summary>
/// Boot-time NIC grace (same pattern as the emergency-break reporter): polls the cheap
/// link-level signal once per second for at most <see cref="NetworkLinkWaitMax"/>. It
/// cannot prove backend reachability — the retry loop handles that — it only keeps the
/// first attempts from being burned into a link that is provably still down. Probe
/// errors end the wait, never the registration.
/// </summary>
internal static async Task WaitForNetworkLinkAsync(AgentLogger logger)
{
if (NetworkInterface.GetIsNetworkAvailable()) return;
logger?.Info($"Session registration: no network link yet — waiting up to {NetworkLinkWaitMax.TotalSeconds:F0}s.");
var deadline = DateTime.UtcNow + NetworkLinkWaitMax;
while (DateTime.UtcNow < deadline)
{
await Task.Delay(1000).ConfigureAwait(false);
if (NetworkInterface.GetIsNetworkAvailable())
{
logger?.Info("Session registration: network link is up.");
return;
}
}
logger?.Warning($"Session registration: still no network link after {NetworkLinkWaitMax.TotalSeconds:F0}s — attempting anyway.");
}
private static SessionRegistration BuildRegistration( private static SessionRegistration BuildRegistration(
AgentConfiguration agentConfig, AgentConfiguration agentConfig,
string agentVersion, string agentVersion,
@@ -194,7 +240,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
/// <summary>401/403 from the backend — cert/token rejected. No retry was attempted.</summary> /// <summary>401/403 from the backend — cert/token rejected. No retry was attempted.</summary>
AuthFailed = 1, AuthFailed = 1,
/// <summary>All 5 attempts failed with non-auth errors (network, 5xx, malformed response).</summary> /// <summary>All attempts failed with non-auth errors (network, 5xx, malformed response).</summary>
Failed = 2, Failed = 2,
} }
@@ -11,7 +11,7 @@ namespace AutopilotMonitor.Agent.V2.Runtime
{ {
/// <summary> /// <summary>
/// Phase 6 of <see cref="Program"/>'s <c>RunAgent</c>: blocking /// Phase 6 of <see cref="Program"/>'s <c>RunAgent</c>: blocking
/// <c>POST /api/agent/register-session</c> with 5-retry exponential backoff before the /// <c>POST /api/agent/register-session</c> with link-wait + 6-retry exponential backoff before the
/// orchestrator starts. V1 parity (MonitoringService.RegisterSessionAsync) — without /// orchestrator starts. V1 parity (MonitoringService.RegisterSessionAsync) — without
/// this call the backend's Sessions table never gets a row for this session, so /// this call the backend's Sessions table never gets a row for this session, so
/// <c>IncrementSessionEventCountAsync</c> / <c>UpdateSessionStatusAsync</c> silently /// <c>IncrementSessionEventCountAsync</c> / <c>UpdateSessionStatusAsync</c> silently
@@ -615,4 +615,84 @@ public class EnrollmentTimeoutClassifierTests
rollup, Start, Start.AddHours(6), 51, isPreProvisioned: true, resumedAt: null); rollup, Start, Start.AddHours(6), 51, isPreProvisioned: true, resumedAt: null);
Assert.Equal(SessionStatus.Succeeded, status); Assert.Equal(SessionStatus.Succeeded, status);
} }
// -------- Self-deploying profile gate (kiosk tenant aebdce78, audit 2026-08-23) --------
private static (SessionStatus, string) ClassifySelfDeploying(
IReadOnlyList<EnrollmentEvent> events, double hoursSinceStart = 3, int grace = 51)
{
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
return EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, Start, Start.AddHours(hoursSinceStart), grace,
lastEventAtUtc: Start.AddMinutes(4), isSelfDeployingProfile: true);
}
// Session 195593e2 replay: DeviceSetup 4/4 (fallback-confirmed), AccountSetup registry
// at 0/5 (the IME false positive — user ESP never runs on this profile), Hello disabled,
// SkipUser=True, then silence after the post-ESP reboot. No user will ever sign in.
private static readonly EnrollmentEvent[] KioskSilentStream =
{
Esp(DeviceSetup44), Esp(AccountSetup05), Esp(DeviceSetupFallback),
HelloPolicy(enabled: false), EspConfig(skipUser: true),
};
[Fact]
public void IsSelfDeployingProvisioned_requires_flag_device_setup_and_no_failure()
{
var provisioned = EnrollmentTimeoutClassifier.ExtractRollup(KioskSilentStream);
Assert.True(EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned(provisioned, isSelfDeployingProfile: true));
Assert.False(EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned(provisioned, isSelfDeployingProfile: false));
var beforeDeviceSetup = EnrollmentTimeoutClassifier.ExtractRollup(new[] { Esp(AccountSetup05) });
Assert.False(EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned(beforeDeviceSetup, isSelfDeployingProfile: true));
var failed = EnrollmentTimeoutClassifier.ExtractRollup(new[] { Esp(DeviceSetup44), Evt("enrollment_failed") });
Assert.False(EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned(failed, isSelfDeployingProfile: true));
}
[Fact]
public void Classify_self_deploying_silent_after_device_setup_reconciles_to_Succeeded_within_grace()
{
var (status, reason) = ClassifySelfDeploying(KioskSilentStream, hoursSinceStart: 3);
Assert.Equal(SessionStatus.Succeeded, status);
Assert.Contains("self-deploying profile", reason);
Assert.Contains("no user / Account Setup phase", reason);
Assert.Contains("Agent last reported 2026-07-06 15:04 UTC", reason);
Assert.DoesNotContain("Account Setup completed", reason);
}
[Fact]
public void Classify_self_deploying_silent_after_device_setup_reconciles_to_Succeeded_past_grace()
{
// Grace is a user-phase concept; it does not apply when there is no user phase.
var (status, _) = ClassifySelfDeploying(KioskSilentStream, hoursSinceStart: 60, grace: 51);
Assert.Equal(SessionStatus.Succeeded, status);
}
[Fact]
public void Classify_same_stream_without_self_deploying_flag_keeps_user_phase_verdicts()
{
// Guard against the gate leaking into user-driven sessions: the identical event shape
// on a user-driven profile is still "awaiting user" within grace / Incomplete past it.
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(KioskSilentStream);
var (within, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(rollup, Start, Start.AddHours(3), 51);
var (past, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(rollup, Start, Start.AddHours(60), 51);
Assert.Equal(SessionStatus.AwaitingUser, within);
Assert.Equal(SessionStatus.Incomplete, past);
}
[Fact]
public void Classify_self_deploying_explicit_failure_stays_Failed()
{
var (status, _) = ClassifySelfDeploying(new[] { Esp(DeviceSetup44), Evt("enrollment_failed") });
Assert.Equal(SessionStatus.Failed, status);
}
[Fact]
public void Classify_self_deploying_silent_before_device_setup_is_Incomplete()
{
// The profile flag alone proves nothing — Device ESP must have finished.
var (status, _) = ClassifySelfDeploying(new[] { Esp(AccountSetup05), Evt("agent_started") });
Assert.Equal(SessionStatus.Incomplete, status);
}
} }
@@ -115,6 +115,7 @@ public class OffboardingSessionEnumeratorTests
public Task<List<SessionSummary>> GetGeoWindowSessionsAsync(DateTime startDate, DateTime endDate, string? tenantId = null) => throw new NotImplementedException(); public Task<List<SessionSummary>> GetGeoWindowSessionsAsync(DateTime startDate, DateTime endDate, string? tenantId = null) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetStalledSessionsAsync(string tenantId, DateTime cutoffTime) => throw new NotImplementedException(); public Task<List<SessionSummary>> GetStalledSessionsAsync(string tenantId, DateTime cutoffTime) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetLegacyTimeoutFailedSessionsAsync(string tenantId, int maxResults) => throw new NotImplementedException(); public Task<List<SessionSummary>> GetLegacyTimeoutFailedSessionsAsync(string tenantId, int maxResults) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetSelfDeployingSilentSessionsAsync(string tenantId, int maxResults) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetSessionsLeanAsync(string tenantId) => throw new NotImplementedException(); public Task<List<SessionSummary>> GetSessionsLeanAsync(string tenantId) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetAgentSilentSessionsAsync(string tenantId, DateTime silenceCutoff, DateTime hardCutoff) => throw new NotImplementedException(); public Task<List<SessionSummary>> GetAgentSilentSessionsAsync(string tenantId, DateTime silenceCutoff, DateTime hardCutoff) => throw new NotImplementedException();
public Task<List<string>> GetAllTenantIdsAsync() => throw new NotImplementedException(); public Task<List<string>> GetAllTenantIdsAsync() => throw new NotImplementedException();
@@ -72,6 +72,9 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
public Task<List<SessionSummary>> GetLegacyTimeoutFailedSessionsAsync(string tenantId, int maxResults) public Task<List<SessionSummary>> GetLegacyTimeoutFailedSessionsAsync(string tenantId, int maxResults)
=> _storage.GetLegacyTimeoutFailedSessionsAsync(tenantId, maxResults); => _storage.GetLegacyTimeoutFailedSessionsAsync(tenantId, maxResults);
public Task<List<SessionSummary>> GetSelfDeployingSilentSessionsAsync(string tenantId, int maxResults)
=> _storage.GetSelfDeployingSilentSessionsAsync(tenantId, maxResults);
public Task<List<SessionSummary>> GetSessionsLeanAsync(string tenantId) public Task<List<SessionSummary>> GetSessionsLeanAsync(string tenantId)
=> _storage.GetSessionsLeanAsync(tenantId); => _storage.GetSessionsLeanAsync(tenantId);
@@ -35,6 +35,8 @@ namespace AutopilotMonitor.Functions.Functions.Admin
/// Query parameters: /// Query parameters:
/// - mode: "legacy_timeouts" (default) — re-classify pre-classifier "Session timed out" Failed rows; /// - mode: "legacy_timeouts" (default) — re-classify pre-classifier "Session timed out" Failed rows;
/// "pending_orphans" — resolve Pending rows superseded by a later session of the same device. /// "pending_orphans" — resolve Pending rows superseded by a later session of the same device.
/// "self_deploying_silent" — heal self-deploying (kiosk) rows parked Incomplete/AwaitingUser/Stalled
/// after Device ESP provisioned to Succeeded (classifier rule 1c, 2026-08-23).
/// - dryRun: "false" to actually write; anything else (or absent) reports only. /// - dryRun: "false" to actually write; anything else (or absent) reports only.
/// - tenantId: optional single-tenant scope (GUID); absent = all tenants. /// - tenantId: optional single-tenant scope (GUID); absent = all tenants.
/// - maxSessions: per-invocation examination cap (default 200, max 2000). Re-run to continue. /// - maxSessions: per-invocation examination cap (default 200, max 2000). Re-run to continue.
@@ -82,6 +84,8 @@ namespace AutopilotMonitor.Functions.Functions.Admin
await _service.ReclassifyLegacyTimeoutsAsync(tenantIdScope, dryRun, maxSessions, userEmail), await _service.ReclassifyLegacyTimeoutsAsync(tenantIdScope, dryRun, maxSessions, userEmail),
LegacyReclassificationService.ModePendingOrphans => LegacyReclassificationService.ModePendingOrphans =>
await _service.ResolvePendingOrphansAsync(tenantIdScope, dryRun, maxSessions, userEmail), await _service.ResolvePendingOrphansAsync(tenantIdScope, dryRun, maxSessions, userEmail),
LegacyReclassificationService.ModeSelfDeployingSilent =>
await _service.ReconcileSelfDeployingSilentAsync(tenantIdScope, dryRun, maxSessions, userEmail),
_ => null, _ => null,
}; };
@@ -90,7 +94,7 @@ namespace AutopilotMonitor.Functions.Functions.Admin
var bad = req.CreateResponse(HttpStatusCode.BadRequest); var bad = req.CreateResponse(HttpStatusCode.BadRequest);
await bad.WriteAsJsonAsync(new await bad.WriteAsJsonAsync(new
{ {
error = $"Unknown mode '{mode}'. Valid: {LegacyReclassificationService.ModeLegacyTimeouts}, {LegacyReclassificationService.ModePendingOrphans}" error = $"Unknown mode '{mode}'. Valid: {LegacyReclassificationService.ModeLegacyTimeouts}, {LegacyReclassificationService.ModePendingOrphans}, {LegacyReclassificationService.ModeSelfDeployingSilent}"
}); });
return bad; return bad;
} }
@@ -273,6 +273,34 @@ namespace AutopilotMonitor.Functions.Services
$"device presumably sealed or powered off awaiting the end user{acct}"; $"device presumably sealed or powered off awaiting the end user{acct}";
} }
/// <summary>
/// Self-deploying (kiosk / shared-device) profile gate. Such a profile has NO user phase:
/// Device ESP reaching all-succeeded IS the enrollment's end, and the agent's own
/// SelfDeploying terminal only adds a 5-min confirmation window on top of it. When the
/// agent goes silent after that point (reboot into the kiosk autologon, powered off and
/// boxed for the school term, WiFi never re-associating for the relaunch — tenant
/// aebdce78, 836 sessions 2026-08-17..21), "awaiting user / Account Setup" is a factual
/// error and the later Incomplete("Account Setup 0/5") is worse: nobody will ever sign
/// in, the devices are finished and in service. Demands the registration-time
/// registry-confirmed profile flag (CloudAssignedOobeConfig 0x20|0x40), Device Setup
/// all-succeeded and no explicit failure. Shared by the sweep (stage 1 + 2), the
/// max-lifetime ingest verdict and the retro-reclassification so one evidence rule applies.
/// </summary>
public static bool IsSelfDeployingProvisioned(in EspProvisioningRollup rollup, bool isSelfDeployingProfile)
=> isSelfDeployingProfile
&& rollup.DeviceSetupAllSucceeded
&& !rollup.HasExplicitFailure;
/// <summary>
/// Customer-facing Succeeded-reconcile reason for <see cref="IsSelfDeployingProvisioned"/>.
/// Carries the silence-transparency clause like every other reconcile reason.
/// </summary>
public static string SelfDeployingReconcileReason(DateTime startedAtUtc, DateTime nowUtc, DateTime? lastEventAtUtc)
=> AppendReconcileTiming(
"Reconciled: self-deploying profile — Device Setup fully provisioned (this profile has no user / Account Setup phase); " +
"agent went silent before confirming completion",
startedAtUtc, nowUtc, lastEventAtUtc);
/// <summary> /// <summary>
/// Decide the honest terminal (or non-terminal) state for a timed-out session. See the /// Decide the honest terminal (or non-terminal) state for a timed-out session. See the
/// decision table in the design note. Returns the target status and a human-readable /// decision table in the design note. Returns the target status and a human-readable
@@ -291,9 +319,11 @@ namespace AutopilotMonitor.Functions.Services
/// </param> /// </param>
/// <param name="isPreProvisioned">Session's IsPreProvisioned flag (WhiteGlove Part-2 gate input).</param> /// <param name="isPreProvisioned">Session's IsPreProvisioned flag (WhiteGlove Part-2 gate input).</param>
/// <param name="resumedAt">Session's ResumedAt — set exclusively by the whiteglove_resumed ingest.</param> /// <param name="resumedAt">Session's ResumedAt — set exclusively by the whiteglove_resumed ingest.</param>
/// <param name="isSelfDeployingProfile">Session's IsSelfDeployingProfile flag (rule 1c gate input).</param>
public static (SessionStatus Status, string Reason) ClassifyTimedOutSession( public static (SessionStatus Status, string Reason) ClassifyTimedOutSession(
EspProvisioningRollup rollup, DateTime startedAtUtc, DateTime nowUtc, int graceHours, EspProvisioningRollup rollup, DateTime startedAtUtc, DateTime nowUtc, int graceHours,
DateTime? lastEventAtUtc = null, bool isPreProvisioned = false, DateTime? resumedAt = null) DateTime? lastEventAtUtc = null, bool isPreProvisioned = false, DateTime? resumedAt = null,
bool isSelfDeployingProfile = false)
{ {
// 1. An explicit terminal failure event is a real failure (defensive — such a session // 1. An explicit terminal failure event is a real failure (defensive — such a session
// would normally already be terminal via ingest and not reach the sweep). // would normally already be terminal via ingest and not reach the sweep).
@@ -318,6 +348,15 @@ namespace AutopilotMonitor.Functions.Services
startedAtUtc, nowUtc, lastEventAtUtc)); startedAtUtc, nowUtc, lastEventAtUtc));
} }
// 1c. Self-deploying profile with Device Setup fully provisioned: there is no user phase
// to wait for, so neither AwaitingUser nor the grace window apply — the device is
// done (see IsSelfDeployingProvisioned). Runs before rule 2 so the reason names the
// profile instead of claiming "Account Setup completed"; real user evidence cannot
// contradict it (a kiosk autologon is not a user) and an explicit failure already
// returned in rule 1.
if (IsSelfDeployingProvisioned(rollup, isSelfDeployingProfile))
return (SessionStatus.Succeeded, SelfDeployingReconcileReason(startedAtUtc, nowUtc, lastEventAtUtc));
// 2. Account Setup fully succeeded (or a terminal completion event) → reconcile to success. // 2. Account Setup fully succeeded (or a terminal completion event) → reconcile to success.
// Once Part 2 has resumed, the Part-1 whiteglove_complete in the same stream no longer // Once Part 2 has resumed, the Part-1 whiteglove_complete in the same stream no longer
// proves the session's outcome — only a real enrollment_complete does. Without this, // proves the session's outcome — only a real enrollment_complete does. Without this,
@@ -288,7 +288,8 @@ namespace AutopilotMonitor.Functions.Services
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(sessionEvents); var rollup = EnrollmentTimeoutClassifier.ExtractRollup(sessionEvents);
var (targetStatus, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession( var (targetStatus, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, effectiveStart, now, graceHours, session?.LastEventAt ?? c.LatestEventTimestamp, rollup, effectiveStart, now, graceHours, session?.LastEventAt ?? c.LatestEventTimestamp,
isPreProvisioned: session?.IsPreProvisioned == true, resumedAt: session?.ResumedAt); isPreProvisioned: session?.IsPreProvisioned == true, resumedAt: session?.ResumedAt,
isSelfDeployingProfile: session?.IsSelfDeployingProfile == true);
// Keep the max-lifetime trigger visible: the Succeeded reasons already carry their own // Keep the max-lifetime trigger visible: the Succeeded reasons already carry their own
// silence-transparency clause, everything else gets the watchdog context appended. // silence-transparency clause, everything else gets the watchdog context appended.
@@ -358,7 +359,8 @@ namespace AutopilotMonitor.Functions.Services
var effectiveStart = session.ResumedAt ?? session.StartedAt; var effectiveStart = session.ResumedAt ?? session.StartedAt;
var (targetStatus, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession( var (targetStatus, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, effectiveStart, now, graceHours, session.LastEventAt ?? c.LatestEventTimestamp, rollup, effectiveStart, now, graceHours, session.LastEventAt ?? c.LatestEventTimestamp,
isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt); isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt,
isSelfDeployingProfile: session.IsSelfDeployingProfile);
if (targetStatus != SessionStatus.Succeeded) if (targetStatus != SessionStatus.Succeeded)
return; return;
@@ -18,6 +18,11 @@ namespace AutopilotMonitor.Functions.Services
/// registered a LATER session (Part 2 under a fresh session id, or re-provisioning) are /// registered a LATER session (Part 2 under a fresh session id, or re-provisioning) are
/// resolved as Incomplete("Superseded by ..."). Forward-looking coverage lives in the /// resolved as Incomplete("Superseded by ..."). Forward-looking coverage lives in the
/// registration supersede pass; this mode clears the existing backlog.</item> /// registration supersede pass; this mode clears the existing backlog.</item>
/// <item><b>self_deploying_silent</b> — self-deploying-profile rows parked as
/// Incomplete / AwaitingUser / Stalled by the pre-2026-08-23 classifier (which had no
/// notion of a profile without a user phase) are re-run through the classifier; only a
/// Succeeded verdict (<see cref="EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned"/>)
/// is applied, everything else keeps its verdict.</item>
/// </list> /// </list>
/// Hand-marked sessions (AdminMarkedAction) are never touched. /// Hand-marked sessions (AdminMarkedAction) are never touched.
/// </summary> /// </summary>
@@ -30,6 +35,7 @@ namespace AutopilotMonitor.Functions.Services
public const string ModeLegacyTimeouts = "legacy_timeouts"; public const string ModeLegacyTimeouts = "legacy_timeouts";
public const string ModePendingOrphans = "pending_orphans"; public const string ModePendingOrphans = "pending_orphans";
public const string ModeSelfDeployingSilent = "self_deploying_silent";
private const int MaxSamplesInResponse = 100; private const int MaxSamplesInResponse = 100;
public LegacyReclassificationService( public LegacyReclassificationService(
@@ -110,7 +116,8 @@ namespace AutopilotMonitor.Functions.Services
var effectiveStart = session.ResumedAt ?? session.StartedAt; var effectiveStart = session.ResumedAt ?? session.StartedAt;
var (target, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession( var (target, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, effectiveStart, now, graceHours, session.LastEventAt, rollup, effectiveStart, now, graceHours, session.LastEventAt,
isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt); isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt,
isSelfDeployingProfile: session.IsSelfDeployingProfile);
if (target == SessionStatus.Failed) if (target == SessionStatus.Failed)
{ {
@@ -165,6 +172,99 @@ namespace AutopilotMonitor.Functions.Services
return result; return result;
} }
public async Task<ReclassificationResult> ReconcileSelfDeployingSilentAsync(
string? tenantIdScope, bool dryRun, int maxSessions, string triggeredBy)
{
var result = new ReclassificationResult { Mode = ModeSelfDeployingSilent, DryRun = dryRun };
var now = DateTime.UtcNow;
foreach (var tenantId in await ResolveTenantsAsync(tenantIdScope))
{
if (result.SessionsExamined >= maxSessions) { result.CapReached = true; break; }
result.TenantsExamined++;
int? tenantGraceHours = null, absoluteMaxHours = null;
try
{
var config = await _configService.GetConfigurationAsync(tenantId);
tenantGraceHours = config?.SessionGraceHours;
absoluteMaxHours = config?.AbsoluteMaxSessionHours;
}
catch { /* defaults */ }
var graceHours = EnrollmentTimeoutClassifier.ResolveGraceHours(tenantGraceHours, absoluteMaxHours);
var candidates = await _maintenanceRepo.GetSelfDeployingSilentSessionsAsync(
tenantId, maxSessions - result.SessionsExamined + 1);
if (candidates.Count > maxSessions - result.SessionsExamined)
{
result.CapReached = true;
candidates = candidates.Take(maxSessions - result.SessionsExamined).ToList();
}
foreach (var session in candidates)
{
result.SessionsExamined++;
try
{
if (!string.IsNullOrEmpty(session.AdminMarkedAction))
{
result.Skipped++;
continue;
}
var events = await _sessionRepo.GetSessionEventsAsync(tenantId, session.SessionId, maxResults: 1000);
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
var effectiveStart = session.ResumedAt ?? session.StartedAt;
var (target, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, effectiveStart, now, graceHours, session.LastEventAt,
isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt,
isSelfDeployingProfile: session.IsSelfDeployingProfile);
// Heal-only: a verdict other than Succeeded means the device never reached
// Device ESP all-succeeded (or failed explicitly) — the existing label stands.
if (target != SessionStatus.Succeeded)
{
result.Skipped++;
continue;
}
RecordSample(result, tenantId, session, target, reason, oldStatus: session.Status.ToString());
if (dryRun)
{
result.WouldChange++;
continue;
}
// Succeeded may upgrade Incomplete / AwaitingUser / Stalled through the normal
// reconcile gate (no terminal-reclassification override needed); the storage
// layer additionally refuses admin-marked rows.
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
tenantId, session.SessionId, SessionStatus.Succeeded,
failureReason: reason);
if (transitioned)
{
result.Changed++;
result.ToSucceeded++;
}
else
{
result.Skipped++;
}
}
catch (Exception ex)
{
result.Errors++;
_logger.LogWarning(ex, "Self-deploying retro-reconcile failed for session {SessionId} (tenant {TenantId})",
session.SessionId, tenantId);
}
}
}
await WriteAuditAsync(result, tenantIdScope, triggeredBy);
return result;
}
public async Task<ReclassificationResult> ResolvePendingOrphansAsync( public async Task<ReclassificationResult> ResolvePendingOrphansAsync(
string? tenantIdScope, bool dryRun, int maxSessions, string triggeredBy) string? tenantIdScope, bool dryRun, int maxSessions, string triggeredBy)
{ {
@@ -350,6 +350,7 @@ namespace AutopilotMonitor.Functions.Services
int silentMarked = 0; int silentMarked = 0;
int silentAwaitingUser = 0; int silentAwaitingUser = 0;
int silentSelfDeployingReconciled = 0;
foreach (var silent in silentSessions) foreach (var silent in silentSessions)
{ {
// The query is a permissive two-frame pre-filter; the decision is made // The query is a permissive two-frame pre-filter; the decision is made
@@ -374,6 +375,18 @@ namespace AutopilotMonitor.Functions.Services
continue; continue;
} }
// Self-deploying profile gate (kiosk tenant aebdce78, 2026-08-23): a
// silent agent after Device ESP all-succeeded is a finished device
// (rebooted into the kiosk autologon / boxed), not a stall and never
// "awaiting user" — reconcile to Succeeded right here instead of
// routing it through Stalled → AwaitingUser → Incomplete.
if (silent.IsSelfDeployingProfile
&& await TryReconcileSelfDeployingAsync(silent, lastContactAt, now))
{
silentSelfDeployingReconciled++;
continue;
}
var silentMinutes = (int)(now - lastContactAt).TotalMinutes; var silentMinutes = (int)(now - lastContactAt).TotalMinutes;
await _sessionRepo.UpdateSessionStatusAsync( await _sessionRepo.UpdateSessionStatusAsync(
silent.TenantId, silent.TenantId,
@@ -384,20 +397,21 @@ namespace AutopilotMonitor.Functions.Services
silentMarked++; silentMarked++;
} }
if (silentMarked > 0 || silentAwaitingUser > 0) if (silentMarked > 0 || silentAwaitingUser > 0 || silentSelfDeployingReconciled > 0)
{ {
totalSessionsMarkedStalled += silentMarked; totalSessionsMarkedStalled += silentMarked;
_logger.LogInformation($"Tenant {tenantId}: Marked {silentMarked} agent-silent sessions as Stalled, {silentAwaitingUser} as AwaitingUser (WhiteGlove Part 2; silence threshold: {agentSilenceHours}h)"); _logger.LogInformation($"Tenant {tenantId}: Marked {silentMarked} agent-silent sessions as Stalled, {silentAwaitingUser} as AwaitingUser (WhiteGlove Part 2), {silentSelfDeployingReconciled} reconciled to Succeeded (self-deploying; silence threshold: {agentSilenceHours}h)");
await _maintenanceRepo.LogAuditEntryAsync( await _maintenanceRepo.LogAuditEntryAsync(
tenantId, tenantId,
"SessionStalled", "SessionStalled",
"Session", "Session",
$"{silentMarked + silentAwaitingUser} sessions", $"{silentMarked + silentAwaitingUser + silentSelfDeployingReconciled} sessions",
"System.Maintenance", "System.Maintenance",
new Dictionary<string, string> new Dictionary<string, string>
{ {
{ "SessionsMarkedStalled", silentMarked.ToString() }, { "SessionsMarkedStalled", silentMarked.ToString() },
{ "SessionsAwaitingUser", silentAwaitingUser.ToString() }, { "SessionsAwaitingUser", silentAwaitingUser.ToString() },
{ "SessionsSelfDeployingReconciled", silentSelfDeployingReconciled.ToString() },
{ "AgentSilenceHours", agentSilenceHours.ToString() }, { "AgentSilenceHours", agentSilenceHours.ToString() },
{ "SilenceCutoff", silenceCutoff.ToString("yyyy-MM-ddTHH:mm:ss") } { "SilenceCutoff", silenceCutoff.ToString("yyyy-MM-ddTHH:mm:ss") }
}); });
@@ -470,7 +484,8 @@ namespace AutopilotMonitor.Functions.Services
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(sessionEvents); var rollup = EnrollmentTimeoutClassifier.ExtractRollup(sessionEvents);
var (targetStatus, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession( var (targetStatus, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, effectiveStart, now, graceHours, session.LastEventAt, rollup, effectiveStart, now, graceHours, session.LastEventAt,
isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt); isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt,
isSelfDeployingProfile: session.IsSelfDeployingProfile);
// No-op if the verdict equals the current (non-terminal) state. // No-op if the verdict equals the current (non-terminal) state.
if (targetStatus == session.Status) if (targetStatus == session.Status)
@@ -623,6 +638,43 @@ namespace AutopilotMonitor.Functions.Services
} }
} }
/// <summary>
/// Applies the self-deploying gate to one agent-silent candidate (the caller pre-filters on
/// <see cref="SessionSummary.IsSelfDeployingProfile"/> so only those pay the event read).
/// When <see cref="EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned"/> confirms Device
/// ESP all-succeeded with no explicit failure, reconciles the session to Succeeded with the
/// shared honest reason. Returns true when the write happened; false (fail-soft, also on
/// read errors) lets the caller fall through to the normal Stalled marker.
/// </summary>
private async Task<bool> TryReconcileSelfDeployingAsync(SessionSummary silent, DateTime lastContactAt, DateTime now)
{
try
{
var events = await _sessionRepo.GetSessionEventsAsync(
silent.TenantId, silent.SessionId, maxResults: 1000);
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
if (!EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned(rollup, silent.IsSelfDeployingProfile))
return false;
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
silent.TenantId,
silent.SessionId,
SessionStatus.Succeeded,
failureReason: EnrollmentTimeoutClassifier.SelfDeployingReconcileReason(
silent.ResumedAt ?? silent.StartedAt, now, lastContactAt));
if (transitioned)
_logger.LogInformation(
$"Session {silent.SessionId}: self-deploying profile silent after Device Setup provisioned — reconciled to Succeeded instead of Stalled");
return transitioned;
}
catch (Exception ex)
{
_logger.LogWarning(ex,
$"Self-deploying reconcile gate failed for session {silent.SessionId}; falling back to Stalled marker");
return false;
}
}
/// <summary> /// <summary>
/// Builds the server-authored <c>session_timeout</c> event injected into the stream when the /// Builds the server-authored <c>session_timeout</c> event injected into the stream when the
/// maintenance sweep graduates a stalled session to terminal Failed. Static + pure so the field /// maintenance sweep graduates a stalled session to terminal Failed. Static + pure so the field
@@ -767,6 +767,39 @@ namespace AutopilotMonitor.Functions.Services
} }
} }
/// <summary>
/// Self-deploying-profile sessions sitting in Incomplete / AwaitingUser / Stalled — the
/// shapes the pre-2026-08-23 classifier produced for a finished kiosk device whose agent
/// went silent after Device ESP. Candidates for the retro-reconcile; the classifier
/// decides per session, this is only the cheap server-side pre-filter.
/// </summary>
public async Task<List<SessionSummary>> GetSelfDeployingSilentSessionsAsync(string tenantId, int maxResults)
{
SecurityValidator.EnsureValidGuid(tenantId, nameof(tenantId));
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.Sessions);
var filter = $"PartitionKey eq '{tenantId}' " +
$"and IsSelfDeployingProfile eq true " +
$"and (Status eq 'Incomplete' or Status eq 'AwaitingUser' or Status eq 'Stalled')";
var sessions = new List<SessionSummary>();
await foreach (var entity in tableClient.QueryAsync<TableEntity>(filter: filter))
{
sessions.Add(MapToSessionSummary(entity));
if (sessions.Count >= maxResults)
break;
}
return sessions;
}
catch (Exception ex)
{
_logger.LogError(ex, $"Failed to get self-deploying silent sessions for tenant {tenantId}");
return new List<SessionSummary>();
}
}
/// <summary> /// <summary>
/// Narrow projection of every session row in the tenant partition — one scan feeding the /// Narrow projection of every session row in the tenant partition — one scan feeding the
/// in-memory Pending-orphan matching (misclassification audit 2026-07-16). Fields outside /// in-memory Pending-orphan matching (misclassification audit 2026-07-16). Fields outside
@@ -92,6 +92,14 @@ namespace AutopilotMonitor.Shared.DataAccess
/// </summary> /// </summary>
Task<List<SessionSummary>> GetLegacyTimeoutFailedSessionsAsync(string tenantId, int maxResults); Task<List<SessionSummary>> GetLegacyTimeoutFailedSessionsAsync(string tenantId, int maxResults);
/// <summary>
/// Self-deploying-profile sessions parked in a non-success, non-failure state
/// (Incomplete / AwaitingUser / Stalled) — candidates for the one-time retro-reconcile
/// through <c>EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned</c> (kiosk tenant
/// audit 2026-08-23). Server-side filter, capped at <paramref name="maxResults"/>.
/// </summary>
Task<List<SessionSummary>> GetSelfDeployingSilentSessionsAsync(string tenantId, int maxResults);
/// <summary> /// <summary>
/// Narrow projection of every session row in the tenant partition (id, status, timing, /// Narrow projection of every session row in the tenant partition (id, status, timing,
/// serial). Used by the Pending-orphan resolution to match superseding sessions of the /// serial). Used by the Pending-orphan resolution to match superseding sessions of the