mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-24 12:42:38 +00:00
fix(classification): self-deploying profiles reconcile to Succeeded after Device Setup, never AwaitingUser
Kiosk tenant audit 2026-08-23: 836 of ~970 self-deploying sessions in one week
were parked AwaitingUser ("Account Setup 0/5") and graduated to Incomplete 51h
later although the devices were finished — the profile has no user phase, so the
classifier's rule 5 was asking for evidence that can never arrive.
- EnrollmentTimeoutClassifier: new rule 1c IsSelfDeployingProvisioned (profile flag +
Device Setup all-succeeded + no explicit failure) => Succeeded reconcile with an
honest reason; isSelfDeployingProfile passed by every caller (sweep stage 1+2,
max-lifetime verdict, late-telemetry heal, legacy retro mode).
- MaintenanceService stage 1: TryReconcileSelfDeployingAsync gate before the Stalled
marker (mirrors the WhiteGlove Part-2 gate), audit counter SessionsSelfDeployingReconciled.
- LegacyReclassificationService: new admin mode self_deploying_silent (heal-only,
dry-run by default) + GetSelfDeployingSilentSessionsAsync for the existing backlog.
- Agent SessionRegistrationHelper: wait up to 15s for a network link and retry 6x
(~62s) instead of 5x (~30s) so the BootTrigger relaunch on Wi-Fi no longer exits 7
before the link is back (the mechanism that left these sessions silent).
- Tests: 6 classifier cases (gate, within/past grace, user-driven guard, failure,
pre-Device-Setup), 2 registration tests; docs/backend/silent-session-classification.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,110 @@
|
||||
---
|
||||
type: Concept
|
||||
title: Silent-Session Classification (Sweep, Max-Lifetime, Retro-Reconcile)
|
||||
description: How a session whose agent went silent gets its honest status — the two-stage maintenance sweep (2h Stalled marker, 5h timeout classification), the EnrollmentTimeoutClassifier rule order incl. the WhiteGlove Part-2 and self-deploying gates, which ingest paths share the classifier, and the admin retro-reconcile modes.
|
||||
resource: src/Backend/AutopilotMonitor.Functions/Services/EnrollmentTimeoutClassifier.cs
|
||||
tags:
|
||||
- backend
|
||||
- maintenance
|
||||
- session-status
|
||||
- classification
|
||||
- whiteglove
|
||||
- self-deploying
|
||||
timestamp: 2026-08-23T18:00:00+02:00
|
||||
---
|
||||
|
||||
# Silent-Session Classification
|
||||
|
||||
A session only ever becomes terminal through evidence. When the agent stops reporting
|
||||
(power-off, reboot without relaunch, Wi-Fi never re-associating, max-lifetime watchdog),
|
||||
the backend must decide the status from what it last saw — it never hard-fails a session
|
||||
for being silent. One pure function holds that decision,
|
||||
`EnrollmentTimeoutClassifier.ClassifyTimedOutSession`, and every silent-session path calls it
|
||||
so a session gets the same verdict regardless of which path noticed the silence first.
|
||||
|
||||
# Schema
|
||||
|
||||
## Inputs (`EspProvisioningRollup` + session flags)
|
||||
|
||||
`ExtractRollup(events)` distills the event stream into facts: Device/Account Setup
|
||||
subcategory counts (incl. the 30s fallback-confirmed messages), `HasExplicitFailure`,
|
||||
`HasEnrollmentComplete` vs. `HasTerminalComplete` (the Part-1 `whiteglove_complete`),
|
||||
`HasAgentEmergencyBreak`, `DesktopArrived`, `HelloResolved`, `HelloPolicyDisabled`,
|
||||
`SkipUserEsp`, RealmJoin detection/resolution. Session-row flags are passed alongside:
|
||||
`IsPreProvisioned` + `ResumedAt` (WhiteGlove Part 2) and `IsSelfDeployingProfile`
|
||||
(registry-confirmed `CloudAssignedOobeConfig 0x20|0x40`, set at registration).
|
||||
|
||||
## Rule order (first match wins)
|
||||
|
||||
| # | Condition | Verdict |
|
||||
| --- | --- | --- |
|
||||
| 1 | `HasExplicitFailure` | Failed |
|
||||
| 1b | `IsWhiteGloveAwaitingUser`: pre-provisioned + resumed + Device Setup done + **no** user evidence since resume + no explicit failure/enrollment_complete | AwaitingUser within grace; Succeeded (honest WG reason) past grace |
|
||||
| 1c | `IsSelfDeployingProvisioned`: self-deploying profile + Device Setup done + no explicit failure | **Succeeded** (reconcile) — no grace, no AwaitingUser: the profile has no user phase |
|
||||
| 2 | Account Setup all-succeeded, or `enrollment_complete`, or `whiteglove_complete` on a **non-resumed** session | Succeeded |
|
||||
| 3 | `HasAgentEmergencyBreak` | Incomplete immediately (agent self-destructed, nothing can arrive) |
|
||||
| 4 | `DesktopArrived` + (Hello terminal, or Hello disabled + SkipUserEsp) | Succeeded (user provably finished) |
|
||||
| 5 | Device Setup done | AwaitingUser within grace, Incomplete past grace |
|
||||
| 6 | otherwise | Incomplete |
|
||||
|
||||
Grace = `ResolveGraceHours(SessionGraceHours, AbsoluteMaxSessionHours)` (~51h default),
|
||||
measured from `ResumedAt ?? StartedAt`. Every Succeeded-reconcile reason carries the
|
||||
silence-transparency clause (`AppendReconcileTiming`: last agent contact, silence, verdict time).
|
||||
|
||||
## Why rule 1c exists (kiosk tenant audit 2026-08-23)
|
||||
|
||||
On a self-deploying profile Device ESP all-succeeded IS the enrollment's end; the agent's
|
||||
own SelfDeploying terminal only adds a 5-min confirmation window
|
||||
(`DecisionEngine.SelfDeploying.s_deviceOnlyEspDetectionWindow`). Tenant `aebdce78` (school,
|
||||
Wi-Fi ThinkBooks, a language-change app forcing a reboot right after Device ESP) lost that
|
||||
window on 836 of ~970 sessions in one week: the agent died 0–4 min after
|
||||
`DeviceSetup fallback_confirmed`, the reboot relaunch never registered (30s registration
|
||||
budget on a Wi-Fi link still associating), and rule 5 labeled finished kiosks
|
||||
"awaiting user / Account Setup 0/5" → Incomplete 51h later. Nobody signs in on these devices
|
||||
(prepared during the holidays, boxed, in service) — AwaitingUser was factually wrong and
|
||||
Incomplete turned a working rollout into a red wall. Other self-deploying tenants completed
|
||||
normally in the same window, so this is a classification gap, not a profile-wide agent bug.
|
||||
Agent-side companion fix: `SessionRegistrationHelper` now waits up to 15s for a network link
|
||||
and retries 6× (~62s backoff) instead of 5× (~30s).
|
||||
|
||||
## Callers (all pass both gate inputs)
|
||||
|
||||
| Path | File | Notes |
|
||||
| --- | --- | --- |
|
||||
| Sweep stage 1 (2h agent-silent marker) | `MaintenanceService.MarkStalledSessionsAsTimedOutAsync` | Default: InProgress → Stalled. Gates run BEFORE the Stalled write: `TryMarkWhiteGloveAwaitingUserAsync` (AwaitingUser) and `TryReconcileSelfDeployingAsync` (Succeeded). Only flagged sessions pay the event read. |
|
||||
| Sweep stage 2 (SessionTimeoutHours, 5h default) | same | InProgress/Stalled/AwaitingUser past the window and past the 2h silence cutoff → full classification; AwaitingUser within grace is skipped without an event read. |
|
||||
| Agent max-lifetime (`enrollment_failed`/max_lifetime, `agent_shutting_down`/max_lifetime) | `EventIngestProcessor.ApplyMaxLifetimeVerdictAsync` | Watchdog = "agent stopped waiting", not a verdict; classified from evidence. Pending sessions exempt. |
|
||||
| Late-telemetry reconcile | `EventIngestProcessor.TryLateTelemetryReconcileAsync` | Heal-only: applies a Succeeded verdict to Incomplete/AwaitingUser rows when straggler events arrive. |
|
||||
| Agent stall probe (`session_stalled`) | `EventIngestProcessor.TryMapStallToWhiteGloveAwaitingUserAsync` | Only the WG gate (1b); a stall probe on a self-deploying device with Device Setup done means the agent is alive and something else blocks — stays Stalled. |
|
||||
| Admin retro-reconcile | `LegacyReclassificationService` | Modes `legacy_timeouts`, `pending_orphans`, `self_deploying_silent` (Incomplete/AwaitingUser/Stalled rows with the profile flag, heal-only to Succeeded, admin-marked rows skipped). `POST /api/maintenance/reclassify-legacy?mode=…&dryRun=false&tenantId=…&maxSessions=…`, dry-run by default. |
|
||||
|
||||
## Known gap (decided, not built)
|
||||
|
||||
After max-lifetime the agent writes `enrollment-complete.marker` and self-destructs, so an
|
||||
AwaitingUser verdict from that path can never heal — it is a 51h countdown to Incomplete.
|
||||
The emergency-break path (rule 3) already terminalizes immediately; max-lifetime still goes
|
||||
through the grace window by design ("user simply absent"). Revisit if customers read
|
||||
AwaitingUser as "still possible".
|
||||
|
||||
# Examples
|
||||
|
||||
* Session `195593e2` (tenant aebdce78): `DeviceSetup 4/4` + `fallback_confirmed`, `AccountSetup 0/5`
|
||||
(IME false positive), Hello disabled, SkipUser=True, silent after 13:01 → rule 1c →
|
||||
Succeeded "self-deploying profile — Device Setup fully provisioned … agent went silent
|
||||
before confirming completion. Agent last reported 2026-08-21 13:01 UTC …".
|
||||
* Same event shape on a user-driven profile → rule 5 → AwaitingUser (within grace) /
|
||||
Incomplete (past grace). Guarded by
|
||||
`Classify_same_stream_without_self_deploying_flag_keeps_user_phase_verdicts`.
|
||||
* fairstone WG Part 2 reseal-reboot, technician powers off at logon screen → rule 1b →
|
||||
AwaitingUser; user unboxes days later → ingest completion heals to Succeeded.
|
||||
|
||||
# Citations
|
||||
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/EnrollmentTimeoutClassifier.cs` — rollup + rule order, `IsWhiteGloveAwaitingUser`, `IsSelfDeployingProvisioned`, reason helpers.
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.cs` — `MarkStalledSessionsAsTimedOutAsync`, `TryMarkWhiteGloveAwaitingUserAsync`, `TryReconcileSelfDeployingAsync`.
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/EventIngestProcessor.Classification.cs` — max-lifetime verdict, late-telemetry reconcile, stall mapping.
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/LegacyReclassificationService.cs`, `Functions/Admin/ReclassifyLegacySessionsFunction.cs` — retro modes.
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.Maintenance.cs` — `GetAgentSilentSessionsAsync`, `GetStalledSessionsAsync`, `GetSelfDeployingSilentSessionsAsync`.
|
||||
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Security/SessionRegistrationHelper.cs` — link wait + 6-attempt registration budget.
|
||||
* `src/Shared/AutopilotMonitor.DecisionCore/Engine/DecisionEngine.SelfDeploying.cs` — the agent-side 5-min SelfDeploying terminal window.
|
||||
* `src/Backend/AutopilotMonitor.Functions.Tests/EnrollmentTimeoutClassifierTests.cs` — rule coverage incl. the 2026-08-23 self-deploying block.
|
||||
@@ -48,6 +48,7 @@ https://docs.autopilotmonitor.com (separate repository).
|
||||
* [Business Timestamps](backend/business-timestamps.md) - Why the Azure Tables system Timestamp is never authoritative (migrations reset it), and the OccurredUtc + RowKey-decode compensation for AuditLogs/OpsEvents/Events after the 2026-07-18 migration.
|
||||
* [Send-Time Frame Separation](backend/send-time-frame-separation.md) - One device-clock send timestamp per upload batch (X-Send-Time-Utc → SentAt on Events) splits ReceivedAt − OccurredUtc into pure spool delay and pure device-vs-server clock offset; the clock_skew condition measures SentAt batches directly (spool-immune), the CMTrace tripwire deliberately stays on its differential math.
|
||||
* [OOBE Bootstrap go-Code Flow](backend/oobe-bootstrap-go-flow.md) - The bootstrap short-code lifecycle (portal-created session → `irm go.autopilotmonitor.com/CODE | iex` at Shift+F10 → GetBootstrapScriptFunction), the always-HTTP-200 text/plain contract, the two-line injection defense (GUID/URL/expiry validator + single-quoted PS literals, golden-pinned against the legacy TS template), and the per-IP rate-limit story behind Front Door.
|
||||
* [Silent-Session Classification](backend/silent-session-classification.md) - How a session whose agent went silent gets its honest status: the two-stage maintenance sweep (2h Stalled marker with the WhiteGlove Part-2 AwaitingUser gate and the self-deploying Succeeded gate, 5h timeout classification), the EnrollmentTimeoutClassifier rule order (explicit failure → WG Part 2 → self-deploying → completion evidence → emergency break → desktop+Hello → Device Setup/grace), the ingest paths sharing it (max-lifetime, late-telemetry heal, stall probe) and the admin retro-reconcile modes incl. self_deploying_silent.
|
||||
* [Device Journeys & First-Time-Right](backend/device-journeys.md) - How a device's terminal enrollment attempts are grouped into journeys (trim+casefold device key with junk-serial exclusion, first-success end, 30-day gap, WhiteGlove = one attempt), the DeviceHistories chain with its tombstone-driven deleted-session pruning, the additive daily FTR aggregates, and the banner/fleet/MCP surfaces.
|
||||
* [Enrollment Time Attribution](backend/time-attribution.md) - How a terminal session's authoritative wall clock (DurationSeconds, WhiteGlove pause excluded) is partitioned into named segments with an explicit unattributed remainder, ESP-blocking apps as positive evidence with "up to" what-if bounds, reboot spans from event gaps, and the breakdown/aggregate tables the sweep maintains.
|
||||
* [Attempt-Scoped App Install Durations](backend/app-install-duration-attempt.md) - AppInstallSummary.DurationSeconds = status-defining (last) install attempt via the LastAttemptStartedAt max-fold anchor instead of the first-observation→last-terminal span that billed multi-pass IME gaps to trivial apps; InstallPassCount, the weaker-terminal guard, the radar's cutover gate, and time attribution's union-of-attempt-segments occupancy.
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
## 2026-08-23
|
||||
|
||||
* **New**: `backend/silent-session-classification.md` — one place for the silent-session verdict: `EnrollmentTimeoutClassifier` rule order incl. the new rule 1c `IsSelfDeployingProvisioned` (self-deploying profile + Device Setup all-succeeded + no explicit failure ⇒ Succeeded reconcile, no grace/AwaitingUser — kiosk tenant aebdce78 audit: 836 finished devices parked AwaitingUser/Incomplete), the sweep stage-1 gates (`TryMarkWhiteGloveAwaitingUserAsync`, new `TryReconcileSelfDeployingAsync`), the ingest callers (max-lifetime, late-telemetry heal, stall probe) and the new admin retro mode `self_deploying_silent` (`GetSelfDeployingSilentSessionsAsync`). Agent companion: `SessionRegistrationHelper` waits up to 15s for a network link and retries 6× (~62s) instead of 5× (~30s) so the reboot relaunch on Wi-Fi no longer exits 7 before the link is back.
|
||||
* **New**: `agent/diagnostics-package.md` — the diagnostics ZIP's built-in collection is now ONE catalog in Shared (`DiagnosticsBuiltInSections.All`): the agent iterates it in order and the backend serves the same list to the portal (`GET diagnostics/paths`, MemberRead), so what administrators see is exactly what the agent collects. New conditional sections: RealmJoin client/package/Chocolatey/per-user logs behind the tenant's RealmJoin Watcher toggle (mirrored onto `AgentConfiguration.EnableRealmJoinWatcher` by the merger) and the Device Preparation bootstrapper event log behind the deterministic WDP marker (probe once per build, fail toward Classic); every skip is a manifest line. Built-ins bypass `DiagnosticsPathGuards` by design (reviewed code, `%ProgramData%\RealmJoin` is not allowlisted); caps/reparse/evtx export still apply. `MergeDiagnosticsLogPaths` dedups global ∪ tenant paths. Portal: collapsed read-only catalog + one-line rows in both cards, tenant context dropped the GA-only global fetch.
|
||||
* **Update**: `rules/gather-rule-guardrails.md` — `DiagnosticsPathGuards` scope clarified: configured diagnostics paths only, never the built-in catalog.
|
||||
* **Update**: `backend/tenant-activation-welcome-mail.md` — new "Template overrides" section: Global Admins can replace the welcome/farewell HTML in the portal (Settings → Email Templates); `EmailTemplateService` resolves override-or-built-in per send (PreviewConfig table, partition `EmailTemplates`, 5-min cache, `{{domainName}}` placeholder, 30k limit, fail-soft send path), GA endpoints `global/email-templates/{kind}` (+`/test` to the caller's tenant contact address). Also: the mail header is image-free (CSS-drawn brand mark, gradient stripes and clip-path edge as progressive enhancement).
|
||||
|
||||
+45
-3
@@ -97,7 +97,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Security
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Fails_after_five_exceptions_without_auth_error()
|
||||
public async Task Fails_after_six_exceptions_without_auth_error()
|
||||
{
|
||||
using var tmp = new TempDirectory();
|
||||
var logger = NewLogger(tmp);
|
||||
@@ -111,8 +111,50 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Security
|
||||
backoffDelay: _ => Task.CompletedTask);
|
||||
|
||||
Assert.Equal(SessionRegistrationOutcome.Failed, result.Outcome);
|
||||
Assert.Equal(5, apiClient.CallCount);
|
||||
Assert.Contains("transient attempt 5", result.ErrorMessage);
|
||||
Assert.Equal(6, apiClient.CallCount);
|
||||
Assert.Contains("transient attempt 6", result.ErrorMessage);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Waits_for_network_link_before_first_attempt()
|
||||
{
|
||||
using var tmp = new TempDirectory();
|
||||
var logger = NewLogger(tmp);
|
||||
var config = NewConfig();
|
||||
|
||||
var order = new System.Collections.Generic.List<string>();
|
||||
var apiClient = new FakeApiClient(_ =>
|
||||
{
|
||||
order.Add("register");
|
||||
return new RegisterSessionResponse { Success = true, SessionId = config.SessionId };
|
||||
});
|
||||
|
||||
var result = await SessionRegistrationHelper.RegisterWithRetryAsync(
|
||||
apiClient, config, agentVersion: "0.0.0", logger: logger,
|
||||
backoffDelay: _ => Task.CompletedTask,
|
||||
networkLinkWait: _ => { order.Add("link-wait"); return Task.CompletedTask; });
|
||||
|
||||
Assert.Equal(SessionRegistrationOutcome.Succeeded, result.Outcome);
|
||||
Assert.Equal(new[] { "link-wait", "register" }, order);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Network_link_wait_failure_does_not_block_registration()
|
||||
{
|
||||
using var tmp = new TempDirectory();
|
||||
var logger = NewLogger(tmp);
|
||||
var config = NewConfig();
|
||||
|
||||
var apiClient = new FakeApiClient(_ =>
|
||||
new RegisterSessionResponse { Success = true, SessionId = config.SessionId });
|
||||
|
||||
var result = await SessionRegistrationHelper.RegisterWithRetryAsync(
|
||||
apiClient, config, agentVersion: "0.0.0", logger: logger,
|
||||
backoffDelay: _ => Task.CompletedTask,
|
||||
networkLinkWait: _ => throw new InvalidOperationException("probe broken"));
|
||||
|
||||
Assert.Equal(SessionRegistrationOutcome.Succeeded, result.Outcome);
|
||||
Assert.Equal(1, apiClient.CallCount);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
using System;
|
||||
using System.Net.NetworkInformation;
|
||||
using System.Threading.Tasks;
|
||||
using AutopilotMonitor.Agent.V2.Core.Configuration;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
@@ -19,7 +20,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
|
||||
/// <b>Retry contract (V1 <c>MonitoringService.RegisterSessionAsync</c>):</b>
|
||||
/// </para>
|
||||
/// <list type="bullet">
|
||||
/// <item>5 attempts total; between attempts <c>2^attempt</c> seconds (2s, 4s, 8s, 16s).</item>
|
||||
/// <item>Boot-time NIC grace before the first attempt: when no network link is up yet
|
||||
/// (BootTrigger relaunch after a mid-enrollment reboot, Wi-Fi still associating) wait
|
||||
/// up to <see cref="NetworkLinkWaitMax"/> for the link-level signal. Best-effort and
|
||||
/// free on the normal path — a live link returns immediately.</item>
|
||||
/// <item>6 attempts total; between attempts <c>2^attempt</c> seconds (2s, 4s, 8s, 16s, 32s —
|
||||
/// ~62 s of backoff). The original V1 budget of 5 attempts / ~30 s was too short for the
|
||||
/// reboot-relaunch on Wi-Fi kiosks: the relaunched agent gave up and exited 7 before
|
||||
/// the network came back, leaving the session silent forever (tenant aebdce78,
|
||||
/// 2026-08-23 audit). A relaunch that misses here is lost — nothing retries later.</item>
|
||||
/// <item>On <c>response.Success == true</c> → stop immediately.</item>
|
||||
/// <item>On <see cref="BackendAuthException"/> (401/403) → report to the
|
||||
/// <see cref="AuthFailureTracker"/> (which fires the first-failure distress + may trip
|
||||
@@ -37,16 +46,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
|
||||
/// </summary>
|
||||
public static class SessionRegistrationHelper
|
||||
{
|
||||
private const int MaxAttempts = 5;
|
||||
private const int MaxAttempts = 6;
|
||||
|
||||
/// <summary>Upper bound for the pre-registration link wait (see class remarks).</summary>
|
||||
internal static readonly TimeSpan NetworkLinkWaitMax = TimeSpan.FromSeconds(15);
|
||||
|
||||
/// <summary>
|
||||
/// Executes the 5-retry register-session handshake against the backend. Returns the
|
||||
/// Executes the retrying register-session handshake against the backend. Returns the
|
||||
/// response + classified outcome. Never throws.
|
||||
/// </summary>
|
||||
/// <param name="backoffDelay">
|
||||
/// Optional delay-provider used by tests to avoid real-time waits. Production callers
|
||||
/// pass <c>null</c> → V1 parity <c>Task.Delay(2^attempt * 1000)</c> between attempts.
|
||||
/// </param>
|
||||
/// <param name="networkLinkWait">
|
||||
/// Optional pre-registration link wait used by tests. Production callers pass
|
||||
/// <c>null</c> → <see cref="WaitForNetworkLinkAsync"/> (bounded by <see cref="NetworkLinkWaitMax"/>).
|
||||
/// </param>
|
||||
public static async Task<SessionRegistrationResult> RegisterWithRetryAsync(
|
||||
BackendApiClient apiClient,
|
||||
AgentConfiguration agentConfig,
|
||||
@@ -56,14 +72,19 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
|
||||
EmergencyReporter emergencyReporter = null,
|
||||
Func<int, Task> backoffDelay = null,
|
||||
Func<Exception, Task> onTerminalTransportFailure = null,
|
||||
(string Manufacturer, string Model, string SerialNumber)? deviceHardware = null)
|
||||
(string Manufacturer, string Model, string SerialNumber)? deviceHardware = null,
|
||||
Func<AgentLogger, Task> networkLinkWait = null)
|
||||
{
|
||||
if (apiClient == null) throw new ArgumentNullException(nameof(apiClient));
|
||||
if (agentConfig == null) throw new ArgumentNullException(nameof(agentConfig));
|
||||
if (logger == null) throw new ArgumentNullException(nameof(logger));
|
||||
|
||||
backoffDelay ??= DefaultBackoffDelay;
|
||||
networkLinkWait ??= WaitForNetworkLinkAsync;
|
||||
var registration = BuildRegistration(agentConfig, agentVersion, deviceHardware);
|
||||
|
||||
try { await networkLinkWait(logger).ConfigureAwait(false); }
|
||||
catch (Exception ex) { logger.Debug($"Pre-registration network wait failed ({ex.Message}) — registering anyway."); }
|
||||
string lastError = null;
|
||||
Exception lastException = null;
|
||||
|
||||
@@ -113,7 +134,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
|
||||
|
||||
if (attempt < MaxAttempts)
|
||||
{
|
||||
var delaySeconds = (int)Math.Pow(2, attempt); // 2, 4, 8, 16
|
||||
var delaySeconds = (int)Math.Pow(2, attempt); // 2, 4, 8, 16, 32
|
||||
logger.Info($"Retrying session registration in {delaySeconds}s");
|
||||
await backoffDelay(attempt).ConfigureAwait(false);
|
||||
}
|
||||
@@ -143,6 +164,31 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
|
||||
private static Task DefaultBackoffDelay(int attempt)
|
||||
=> Task.Delay(((int)Math.Pow(2, attempt)) * 1000);
|
||||
|
||||
/// <summary>
|
||||
/// Boot-time NIC grace (same pattern as the emergency-break reporter): polls the cheap
|
||||
/// link-level signal once per second for at most <see cref="NetworkLinkWaitMax"/>. It
|
||||
/// cannot prove backend reachability — the retry loop handles that — it only keeps the
|
||||
/// first attempts from being burned into a link that is provably still down. Probe
|
||||
/// errors end the wait, never the registration.
|
||||
/// </summary>
|
||||
internal static async Task WaitForNetworkLinkAsync(AgentLogger logger)
|
||||
{
|
||||
if (NetworkInterface.GetIsNetworkAvailable()) return;
|
||||
|
||||
logger?.Info($"Session registration: no network link yet — waiting up to {NetworkLinkWaitMax.TotalSeconds:F0}s.");
|
||||
var deadline = DateTime.UtcNow + NetworkLinkWaitMax;
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
await Task.Delay(1000).ConfigureAwait(false);
|
||||
if (NetworkInterface.GetIsNetworkAvailable())
|
||||
{
|
||||
logger?.Info("Session registration: network link is up.");
|
||||
return;
|
||||
}
|
||||
}
|
||||
logger?.Warning($"Session registration: still no network link after {NetworkLinkWaitMax.TotalSeconds:F0}s — attempting anyway.");
|
||||
}
|
||||
|
||||
private static SessionRegistration BuildRegistration(
|
||||
AgentConfiguration agentConfig,
|
||||
string agentVersion,
|
||||
@@ -194,7 +240,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
|
||||
/// <summary>401/403 from the backend — cert/token rejected. No retry was attempted.</summary>
|
||||
AuthFailed = 1,
|
||||
|
||||
/// <summary>All 5 attempts failed with non-auth errors (network, 5xx, malformed response).</summary>
|
||||
/// <summary>All attempts failed with non-auth errors (network, 5xx, malformed response).</summary>
|
||||
Failed = 2,
|
||||
}
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ namespace AutopilotMonitor.Agent.V2.Runtime
|
||||
{
|
||||
/// <summary>
|
||||
/// Phase 6 of <see cref="Program"/>'s <c>RunAgent</c>: blocking
|
||||
/// <c>POST /api/agent/register-session</c> with 5-retry exponential backoff before the
|
||||
/// <c>POST /api/agent/register-session</c> with link-wait + 6-retry exponential backoff before the
|
||||
/// orchestrator starts. V1 parity (MonitoringService.RegisterSessionAsync) — without
|
||||
/// this call the backend's Sessions table never gets a row for this session, so
|
||||
/// <c>IncrementSessionEventCountAsync</c> / <c>UpdateSessionStatusAsync</c> silently
|
||||
|
||||
@@ -615,4 +615,84 @@ public class EnrollmentTimeoutClassifierTests
|
||||
rollup, Start, Start.AddHours(6), 51, isPreProvisioned: true, resumedAt: null);
|
||||
Assert.Equal(SessionStatus.Succeeded, status);
|
||||
}
|
||||
|
||||
// -------- Self-deploying profile gate (kiosk tenant aebdce78, audit 2026-08-23) --------
|
||||
|
||||
private static (SessionStatus, string) ClassifySelfDeploying(
|
||||
IReadOnlyList<EnrollmentEvent> events, double hoursSinceStart = 3, int grace = 51)
|
||||
{
|
||||
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
|
||||
return EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
|
||||
rollup, Start, Start.AddHours(hoursSinceStart), grace,
|
||||
lastEventAtUtc: Start.AddMinutes(4), isSelfDeployingProfile: true);
|
||||
}
|
||||
|
||||
// Session 195593e2 replay: DeviceSetup 4/4 (fallback-confirmed), AccountSetup registry
|
||||
// at 0/5 (the IME false positive — user ESP never runs on this profile), Hello disabled,
|
||||
// SkipUser=True, then silence after the post-ESP reboot. No user will ever sign in.
|
||||
private static readonly EnrollmentEvent[] KioskSilentStream =
|
||||
{
|
||||
Esp(DeviceSetup44), Esp(AccountSetup05), Esp(DeviceSetupFallback),
|
||||
HelloPolicy(enabled: false), EspConfig(skipUser: true),
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public void IsSelfDeployingProvisioned_requires_flag_device_setup_and_no_failure()
|
||||
{
|
||||
var provisioned = EnrollmentTimeoutClassifier.ExtractRollup(KioskSilentStream);
|
||||
Assert.True(EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned(provisioned, isSelfDeployingProfile: true));
|
||||
Assert.False(EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned(provisioned, isSelfDeployingProfile: false));
|
||||
|
||||
var beforeDeviceSetup = EnrollmentTimeoutClassifier.ExtractRollup(new[] { Esp(AccountSetup05) });
|
||||
Assert.False(EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned(beforeDeviceSetup, isSelfDeployingProfile: true));
|
||||
|
||||
var failed = EnrollmentTimeoutClassifier.ExtractRollup(new[] { Esp(DeviceSetup44), Evt("enrollment_failed") });
|
||||
Assert.False(EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned(failed, isSelfDeployingProfile: true));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Classify_self_deploying_silent_after_device_setup_reconciles_to_Succeeded_within_grace()
|
||||
{
|
||||
var (status, reason) = ClassifySelfDeploying(KioskSilentStream, hoursSinceStart: 3);
|
||||
Assert.Equal(SessionStatus.Succeeded, status);
|
||||
Assert.Contains("self-deploying profile", reason);
|
||||
Assert.Contains("no user / Account Setup phase", reason);
|
||||
Assert.Contains("Agent last reported 2026-07-06 15:04 UTC", reason);
|
||||
Assert.DoesNotContain("Account Setup completed", reason);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Classify_self_deploying_silent_after_device_setup_reconciles_to_Succeeded_past_grace()
|
||||
{
|
||||
// Grace is a user-phase concept; it does not apply when there is no user phase.
|
||||
var (status, _) = ClassifySelfDeploying(KioskSilentStream, hoursSinceStart: 60, grace: 51);
|
||||
Assert.Equal(SessionStatus.Succeeded, status);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Classify_same_stream_without_self_deploying_flag_keeps_user_phase_verdicts()
|
||||
{
|
||||
// Guard against the gate leaking into user-driven sessions: the identical event shape
|
||||
// on a user-driven profile is still "awaiting user" within grace / Incomplete past it.
|
||||
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(KioskSilentStream);
|
||||
var (within, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(rollup, Start, Start.AddHours(3), 51);
|
||||
var (past, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(rollup, Start, Start.AddHours(60), 51);
|
||||
Assert.Equal(SessionStatus.AwaitingUser, within);
|
||||
Assert.Equal(SessionStatus.Incomplete, past);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Classify_self_deploying_explicit_failure_stays_Failed()
|
||||
{
|
||||
var (status, _) = ClassifySelfDeploying(new[] { Esp(DeviceSetup44), Evt("enrollment_failed") });
|
||||
Assert.Equal(SessionStatus.Failed, status);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Classify_self_deploying_silent_before_device_setup_is_Incomplete()
|
||||
{
|
||||
// The profile flag alone proves nothing — Device ESP must have finished.
|
||||
var (status, _) = ClassifySelfDeploying(new[] { Esp(AccountSetup05), Evt("agent_started") });
|
||||
Assert.Equal(SessionStatus.Incomplete, status);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -115,6 +115,7 @@ public class OffboardingSessionEnumeratorTests
|
||||
public Task<List<SessionSummary>> GetGeoWindowSessionsAsync(DateTime startDate, DateTime endDate, string? tenantId = null) => throw new NotImplementedException();
|
||||
public Task<List<SessionSummary>> GetStalledSessionsAsync(string tenantId, DateTime cutoffTime) => throw new NotImplementedException();
|
||||
public Task<List<SessionSummary>> GetLegacyTimeoutFailedSessionsAsync(string tenantId, int maxResults) => throw new NotImplementedException();
|
||||
public Task<List<SessionSummary>> GetSelfDeployingSilentSessionsAsync(string tenantId, int maxResults) => throw new NotImplementedException();
|
||||
public Task<List<SessionSummary>> GetSessionsLeanAsync(string tenantId) => throw new NotImplementedException();
|
||||
public Task<List<SessionSummary>> GetAgentSilentSessionsAsync(string tenantId, DateTime silenceCutoff, DateTime hardCutoff) => throw new NotImplementedException();
|
||||
public Task<List<string>> GetAllTenantIdsAsync() => throw new NotImplementedException();
|
||||
|
||||
+3
@@ -72,6 +72,9 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
public Task<List<SessionSummary>> GetLegacyTimeoutFailedSessionsAsync(string tenantId, int maxResults)
|
||||
=> _storage.GetLegacyTimeoutFailedSessionsAsync(tenantId, maxResults);
|
||||
|
||||
public Task<List<SessionSummary>> GetSelfDeployingSilentSessionsAsync(string tenantId, int maxResults)
|
||||
=> _storage.GetSelfDeployingSilentSessionsAsync(tenantId, maxResults);
|
||||
|
||||
public Task<List<SessionSummary>> GetSessionsLeanAsync(string tenantId)
|
||||
=> _storage.GetSessionsLeanAsync(tenantId);
|
||||
|
||||
|
||||
+5
-1
@@ -35,6 +35,8 @@ namespace AutopilotMonitor.Functions.Functions.Admin
|
||||
/// Query parameters:
|
||||
/// - mode: "legacy_timeouts" (default) — re-classify pre-classifier "Session timed out" Failed rows;
|
||||
/// "pending_orphans" — resolve Pending rows superseded by a later session of the same device.
|
||||
/// "self_deploying_silent" — heal self-deploying (kiosk) rows parked Incomplete/AwaitingUser/Stalled
|
||||
/// after Device ESP provisioned to Succeeded (classifier rule 1c, 2026-08-23).
|
||||
/// - dryRun: "false" to actually write; anything else (or absent) reports only.
|
||||
/// - tenantId: optional single-tenant scope (GUID); absent = all tenants.
|
||||
/// - maxSessions: per-invocation examination cap (default 200, max 2000). Re-run to continue.
|
||||
@@ -82,6 +84,8 @@ namespace AutopilotMonitor.Functions.Functions.Admin
|
||||
await _service.ReclassifyLegacyTimeoutsAsync(tenantIdScope, dryRun, maxSessions, userEmail),
|
||||
LegacyReclassificationService.ModePendingOrphans =>
|
||||
await _service.ResolvePendingOrphansAsync(tenantIdScope, dryRun, maxSessions, userEmail),
|
||||
LegacyReclassificationService.ModeSelfDeployingSilent =>
|
||||
await _service.ReconcileSelfDeployingSilentAsync(tenantIdScope, dryRun, maxSessions, userEmail),
|
||||
_ => null,
|
||||
};
|
||||
|
||||
@@ -90,7 +94,7 @@ namespace AutopilotMonitor.Functions.Functions.Admin
|
||||
var bad = req.CreateResponse(HttpStatusCode.BadRequest);
|
||||
await bad.WriteAsJsonAsync(new
|
||||
{
|
||||
error = $"Unknown mode '{mode}'. Valid: {LegacyReclassificationService.ModeLegacyTimeouts}, {LegacyReclassificationService.ModePendingOrphans}"
|
||||
error = $"Unknown mode '{mode}'. Valid: {LegacyReclassificationService.ModeLegacyTimeouts}, {LegacyReclassificationService.ModePendingOrphans}, {LegacyReclassificationService.ModeSelfDeployingSilent}"
|
||||
});
|
||||
return bad;
|
||||
}
|
||||
|
||||
@@ -273,6 +273,34 @@ namespace AutopilotMonitor.Functions.Services
|
||||
$"device presumably sealed or powered off awaiting the end user{acct}";
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Self-deploying (kiosk / shared-device) profile gate. Such a profile has NO user phase:
|
||||
/// Device ESP reaching all-succeeded IS the enrollment's end, and the agent's own
|
||||
/// SelfDeploying terminal only adds a 5-min confirmation window on top of it. When the
|
||||
/// agent goes silent after that point (reboot into the kiosk autologon, powered off and
|
||||
/// boxed for the school term, WiFi never re-associating for the relaunch — tenant
|
||||
/// aebdce78, 836 sessions 2026-08-17..21), "awaiting user / Account Setup" is a factual
|
||||
/// error and the later Incomplete("Account Setup 0/5") is worse: nobody will ever sign
|
||||
/// in, the devices are finished and in service. Demands the registration-time
|
||||
/// registry-confirmed profile flag (CloudAssignedOobeConfig 0x20|0x40), Device Setup
|
||||
/// all-succeeded and no explicit failure. Shared by the sweep (stage 1 + 2), the
|
||||
/// max-lifetime ingest verdict and the retro-reclassification so one evidence rule applies.
|
||||
/// </summary>
|
||||
public static bool IsSelfDeployingProvisioned(in EspProvisioningRollup rollup, bool isSelfDeployingProfile)
|
||||
=> isSelfDeployingProfile
|
||||
&& rollup.DeviceSetupAllSucceeded
|
||||
&& !rollup.HasExplicitFailure;
|
||||
|
||||
/// <summary>
|
||||
/// Customer-facing Succeeded-reconcile reason for <see cref="IsSelfDeployingProvisioned"/>.
|
||||
/// Carries the silence-transparency clause like every other reconcile reason.
|
||||
/// </summary>
|
||||
public static string SelfDeployingReconcileReason(DateTime startedAtUtc, DateTime nowUtc, DateTime? lastEventAtUtc)
|
||||
=> AppendReconcileTiming(
|
||||
"Reconciled: self-deploying profile — Device Setup fully provisioned (this profile has no user / Account Setup phase); " +
|
||||
"agent went silent before confirming completion",
|
||||
startedAtUtc, nowUtc, lastEventAtUtc);
|
||||
|
||||
/// <summary>
|
||||
/// Decide the honest terminal (or non-terminal) state for a timed-out session. See the
|
||||
/// decision table in the design note. Returns the target status and a human-readable
|
||||
@@ -291,9 +319,11 @@ namespace AutopilotMonitor.Functions.Services
|
||||
/// </param>
|
||||
/// <param name="isPreProvisioned">Session's IsPreProvisioned flag (WhiteGlove Part-2 gate input).</param>
|
||||
/// <param name="resumedAt">Session's ResumedAt — set exclusively by the whiteglove_resumed ingest.</param>
|
||||
/// <param name="isSelfDeployingProfile">Session's IsSelfDeployingProfile flag (rule 1c gate input).</param>
|
||||
public static (SessionStatus Status, string Reason) ClassifyTimedOutSession(
|
||||
EspProvisioningRollup rollup, DateTime startedAtUtc, DateTime nowUtc, int graceHours,
|
||||
DateTime? lastEventAtUtc = null, bool isPreProvisioned = false, DateTime? resumedAt = null)
|
||||
DateTime? lastEventAtUtc = null, bool isPreProvisioned = false, DateTime? resumedAt = null,
|
||||
bool isSelfDeployingProfile = false)
|
||||
{
|
||||
// 1. An explicit terminal failure event is a real failure (defensive — such a session
|
||||
// would normally already be terminal via ingest and not reach the sweep).
|
||||
@@ -318,6 +348,15 @@ namespace AutopilotMonitor.Functions.Services
|
||||
startedAtUtc, nowUtc, lastEventAtUtc));
|
||||
}
|
||||
|
||||
// 1c. Self-deploying profile with Device Setup fully provisioned: there is no user phase
|
||||
// to wait for, so neither AwaitingUser nor the grace window apply — the device is
|
||||
// done (see IsSelfDeployingProvisioned). Runs before rule 2 so the reason names the
|
||||
// profile instead of claiming "Account Setup completed"; real user evidence cannot
|
||||
// contradict it (a kiosk autologon is not a user) and an explicit failure already
|
||||
// returned in rule 1.
|
||||
if (IsSelfDeployingProvisioned(rollup, isSelfDeployingProfile))
|
||||
return (SessionStatus.Succeeded, SelfDeployingReconcileReason(startedAtUtc, nowUtc, lastEventAtUtc));
|
||||
|
||||
// 2. Account Setup fully succeeded (or a terminal completion event) → reconcile to success.
|
||||
// Once Part 2 has resumed, the Part-1 whiteglove_complete in the same stream no longer
|
||||
// proves the session's outcome — only a real enrollment_complete does. Without this,
|
||||
|
||||
+4
-2
@@ -288,7 +288,8 @@ namespace AutopilotMonitor.Functions.Services
|
||||
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(sessionEvents);
|
||||
var (targetStatus, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
|
||||
rollup, effectiveStart, now, graceHours, session?.LastEventAt ?? c.LatestEventTimestamp,
|
||||
isPreProvisioned: session?.IsPreProvisioned == true, resumedAt: session?.ResumedAt);
|
||||
isPreProvisioned: session?.IsPreProvisioned == true, resumedAt: session?.ResumedAt,
|
||||
isSelfDeployingProfile: session?.IsSelfDeployingProfile == true);
|
||||
|
||||
// Keep the max-lifetime trigger visible: the Succeeded reasons already carry their own
|
||||
// silence-transparency clause, everything else gets the watchdog context appended.
|
||||
@@ -358,7 +359,8 @@ namespace AutopilotMonitor.Functions.Services
|
||||
var effectiveStart = session.ResumedAt ?? session.StartedAt;
|
||||
var (targetStatus, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
|
||||
rollup, effectiveStart, now, graceHours, session.LastEventAt ?? c.LatestEventTimestamp,
|
||||
isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt);
|
||||
isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt,
|
||||
isSelfDeployingProfile: session.IsSelfDeployingProfile);
|
||||
|
||||
if (targetStatus != SessionStatus.Succeeded)
|
||||
return;
|
||||
|
||||
@@ -18,6 +18,11 @@ namespace AutopilotMonitor.Functions.Services
|
||||
/// registered a LATER session (Part 2 under a fresh session id, or re-provisioning) are
|
||||
/// resolved as Incomplete("Superseded by ..."). Forward-looking coverage lives in the
|
||||
/// registration supersede pass; this mode clears the existing backlog.</item>
|
||||
/// <item><b>self_deploying_silent</b> — self-deploying-profile rows parked as
|
||||
/// Incomplete / AwaitingUser / Stalled by the pre-2026-08-23 classifier (which had no
|
||||
/// notion of a profile without a user phase) are re-run through the classifier; only a
|
||||
/// Succeeded verdict (<see cref="EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned"/>)
|
||||
/// is applied, everything else keeps its verdict.</item>
|
||||
/// </list>
|
||||
/// Hand-marked sessions (AdminMarkedAction) are never touched.
|
||||
/// </summary>
|
||||
@@ -30,6 +35,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
|
||||
public const string ModeLegacyTimeouts = "legacy_timeouts";
|
||||
public const string ModePendingOrphans = "pending_orphans";
|
||||
public const string ModeSelfDeployingSilent = "self_deploying_silent";
|
||||
private const int MaxSamplesInResponse = 100;
|
||||
|
||||
public LegacyReclassificationService(
|
||||
@@ -110,7 +116,8 @@ namespace AutopilotMonitor.Functions.Services
|
||||
var effectiveStart = session.ResumedAt ?? session.StartedAt;
|
||||
var (target, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
|
||||
rollup, effectiveStart, now, graceHours, session.LastEventAt,
|
||||
isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt);
|
||||
isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt,
|
||||
isSelfDeployingProfile: session.IsSelfDeployingProfile);
|
||||
|
||||
if (target == SessionStatus.Failed)
|
||||
{
|
||||
@@ -165,6 +172,99 @@ namespace AutopilotMonitor.Functions.Services
|
||||
return result;
|
||||
}
|
||||
|
||||
public async Task<ReclassificationResult> ReconcileSelfDeployingSilentAsync(
|
||||
string? tenantIdScope, bool dryRun, int maxSessions, string triggeredBy)
|
||||
{
|
||||
var result = new ReclassificationResult { Mode = ModeSelfDeployingSilent, DryRun = dryRun };
|
||||
var now = DateTime.UtcNow;
|
||||
|
||||
foreach (var tenantId in await ResolveTenantsAsync(tenantIdScope))
|
||||
{
|
||||
if (result.SessionsExamined >= maxSessions) { result.CapReached = true; break; }
|
||||
result.TenantsExamined++;
|
||||
|
||||
int? tenantGraceHours = null, absoluteMaxHours = null;
|
||||
try
|
||||
{
|
||||
var config = await _configService.GetConfigurationAsync(tenantId);
|
||||
tenantGraceHours = config?.SessionGraceHours;
|
||||
absoluteMaxHours = config?.AbsoluteMaxSessionHours;
|
||||
}
|
||||
catch { /* defaults */ }
|
||||
var graceHours = EnrollmentTimeoutClassifier.ResolveGraceHours(tenantGraceHours, absoluteMaxHours);
|
||||
|
||||
var candidates = await _maintenanceRepo.GetSelfDeployingSilentSessionsAsync(
|
||||
tenantId, maxSessions - result.SessionsExamined + 1);
|
||||
if (candidates.Count > maxSessions - result.SessionsExamined)
|
||||
{
|
||||
result.CapReached = true;
|
||||
candidates = candidates.Take(maxSessions - result.SessionsExamined).ToList();
|
||||
}
|
||||
|
||||
foreach (var session in candidates)
|
||||
{
|
||||
result.SessionsExamined++;
|
||||
try
|
||||
{
|
||||
if (!string.IsNullOrEmpty(session.AdminMarkedAction))
|
||||
{
|
||||
result.Skipped++;
|
||||
continue;
|
||||
}
|
||||
|
||||
var events = await _sessionRepo.GetSessionEventsAsync(tenantId, session.SessionId, maxResults: 1000);
|
||||
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
|
||||
var effectiveStart = session.ResumedAt ?? session.StartedAt;
|
||||
var (target, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
|
||||
rollup, effectiveStart, now, graceHours, session.LastEventAt,
|
||||
isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt,
|
||||
isSelfDeployingProfile: session.IsSelfDeployingProfile);
|
||||
|
||||
// Heal-only: a verdict other than Succeeded means the device never reached
|
||||
// Device ESP all-succeeded (or failed explicitly) — the existing label stands.
|
||||
if (target != SessionStatus.Succeeded)
|
||||
{
|
||||
result.Skipped++;
|
||||
continue;
|
||||
}
|
||||
|
||||
RecordSample(result, tenantId, session, target, reason, oldStatus: session.Status.ToString());
|
||||
|
||||
if (dryRun)
|
||||
{
|
||||
result.WouldChange++;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Succeeded may upgrade Incomplete / AwaitingUser / Stalled through the normal
|
||||
// reconcile gate (no terminal-reclassification override needed); the storage
|
||||
// layer additionally refuses admin-marked rows.
|
||||
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
|
||||
tenantId, session.SessionId, SessionStatus.Succeeded,
|
||||
failureReason: reason);
|
||||
if (transitioned)
|
||||
{
|
||||
result.Changed++;
|
||||
result.ToSucceeded++;
|
||||
}
|
||||
else
|
||||
{
|
||||
result.Skipped++;
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
result.Errors++;
|
||||
_logger.LogWarning(ex, "Self-deploying retro-reconcile failed for session {SessionId} (tenant {TenantId})",
|
||||
session.SessionId, tenantId);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
await WriteAuditAsync(result, tenantIdScope, triggeredBy);
|
||||
return result;
|
||||
}
|
||||
|
||||
public async Task<ReclassificationResult> ResolvePendingOrphansAsync(
|
||||
string? tenantIdScope, bool dryRun, int maxSessions, string triggeredBy)
|
||||
{
|
||||
|
||||
@@ -350,6 +350,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
|
||||
int silentMarked = 0;
|
||||
int silentAwaitingUser = 0;
|
||||
int silentSelfDeployingReconciled = 0;
|
||||
foreach (var silent in silentSessions)
|
||||
{
|
||||
// The query is a permissive two-frame pre-filter; the decision is made
|
||||
@@ -374,6 +375,18 @@ namespace AutopilotMonitor.Functions.Services
|
||||
continue;
|
||||
}
|
||||
|
||||
// Self-deploying profile gate (kiosk tenant aebdce78, 2026-08-23): a
|
||||
// silent agent after Device ESP all-succeeded is a finished device
|
||||
// (rebooted into the kiosk autologon / boxed), not a stall and never
|
||||
// "awaiting user" — reconcile to Succeeded right here instead of
|
||||
// routing it through Stalled → AwaitingUser → Incomplete.
|
||||
if (silent.IsSelfDeployingProfile
|
||||
&& await TryReconcileSelfDeployingAsync(silent, lastContactAt, now))
|
||||
{
|
||||
silentSelfDeployingReconciled++;
|
||||
continue;
|
||||
}
|
||||
|
||||
var silentMinutes = (int)(now - lastContactAt).TotalMinutes;
|
||||
await _sessionRepo.UpdateSessionStatusAsync(
|
||||
silent.TenantId,
|
||||
@@ -384,20 +397,21 @@ namespace AutopilotMonitor.Functions.Services
|
||||
silentMarked++;
|
||||
}
|
||||
|
||||
if (silentMarked > 0 || silentAwaitingUser > 0)
|
||||
if (silentMarked > 0 || silentAwaitingUser > 0 || silentSelfDeployingReconciled > 0)
|
||||
{
|
||||
totalSessionsMarkedStalled += silentMarked;
|
||||
_logger.LogInformation($"Tenant {tenantId}: Marked {silentMarked} agent-silent sessions as Stalled, {silentAwaitingUser} as AwaitingUser (WhiteGlove Part 2; silence threshold: {agentSilenceHours}h)");
|
||||
_logger.LogInformation($"Tenant {tenantId}: Marked {silentMarked} agent-silent sessions as Stalled, {silentAwaitingUser} as AwaitingUser (WhiteGlove Part 2), {silentSelfDeployingReconciled} reconciled to Succeeded (self-deploying; silence threshold: {agentSilenceHours}h)");
|
||||
await _maintenanceRepo.LogAuditEntryAsync(
|
||||
tenantId,
|
||||
"SessionStalled",
|
||||
"Session",
|
||||
$"{silentMarked + silentAwaitingUser} sessions",
|
||||
$"{silentMarked + silentAwaitingUser + silentSelfDeployingReconciled} sessions",
|
||||
"System.Maintenance",
|
||||
new Dictionary<string, string>
|
||||
{
|
||||
{ "SessionsMarkedStalled", silentMarked.ToString() },
|
||||
{ "SessionsAwaitingUser", silentAwaitingUser.ToString() },
|
||||
{ "SessionsSelfDeployingReconciled", silentSelfDeployingReconciled.ToString() },
|
||||
{ "AgentSilenceHours", agentSilenceHours.ToString() },
|
||||
{ "SilenceCutoff", silenceCutoff.ToString("yyyy-MM-ddTHH:mm:ss") }
|
||||
});
|
||||
@@ -470,7 +484,8 @@ namespace AutopilotMonitor.Functions.Services
|
||||
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(sessionEvents);
|
||||
var (targetStatus, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
|
||||
rollup, effectiveStart, now, graceHours, session.LastEventAt,
|
||||
isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt);
|
||||
isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt,
|
||||
isSelfDeployingProfile: session.IsSelfDeployingProfile);
|
||||
|
||||
// No-op if the verdict equals the current (non-terminal) state.
|
||||
if (targetStatus == session.Status)
|
||||
@@ -623,6 +638,43 @@ namespace AutopilotMonitor.Functions.Services
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Applies the self-deploying gate to one agent-silent candidate (the caller pre-filters on
|
||||
/// <see cref="SessionSummary.IsSelfDeployingProfile"/> so only those pay the event read).
|
||||
/// When <see cref="EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned"/> confirms Device
|
||||
/// ESP all-succeeded with no explicit failure, reconciles the session to Succeeded with the
|
||||
/// shared honest reason. Returns true when the write happened; false (fail-soft, also on
|
||||
/// read errors) lets the caller fall through to the normal Stalled marker.
|
||||
/// </summary>
|
||||
private async Task<bool> TryReconcileSelfDeployingAsync(SessionSummary silent, DateTime lastContactAt, DateTime now)
|
||||
{
|
||||
try
|
||||
{
|
||||
var events = await _sessionRepo.GetSessionEventsAsync(
|
||||
silent.TenantId, silent.SessionId, maxResults: 1000);
|
||||
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
|
||||
if (!EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned(rollup, silent.IsSelfDeployingProfile))
|
||||
return false;
|
||||
|
||||
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
|
||||
silent.TenantId,
|
||||
silent.SessionId,
|
||||
SessionStatus.Succeeded,
|
||||
failureReason: EnrollmentTimeoutClassifier.SelfDeployingReconcileReason(
|
||||
silent.ResumedAt ?? silent.StartedAt, now, lastContactAt));
|
||||
if (transitioned)
|
||||
_logger.LogInformation(
|
||||
$"Session {silent.SessionId}: self-deploying profile silent after Device Setup provisioned — reconciled to Succeeded instead of Stalled");
|
||||
return transitioned;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex,
|
||||
$"Self-deploying reconcile gate failed for session {silent.SessionId}; falling back to Stalled marker");
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Builds the server-authored <c>session_timeout</c> event injected into the stream when the
|
||||
/// maintenance sweep graduates a stalled session to terminal Failed. Static + pure so the field
|
||||
|
||||
@@ -767,6 +767,39 @@ namespace AutopilotMonitor.Functions.Services
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Self-deploying-profile sessions sitting in Incomplete / AwaitingUser / Stalled — the
|
||||
/// shapes the pre-2026-08-23 classifier produced for a finished kiosk device whose agent
|
||||
/// went silent after Device ESP. Candidates for the retro-reconcile; the classifier
|
||||
/// decides per session, this is only the cheap server-side pre-filter.
|
||||
/// </summary>
|
||||
public async Task<List<SessionSummary>> GetSelfDeployingSilentSessionsAsync(string tenantId, int maxResults)
|
||||
{
|
||||
SecurityValidator.EnsureValidGuid(tenantId, nameof(tenantId));
|
||||
|
||||
try
|
||||
{
|
||||
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.Sessions);
|
||||
var filter = $"PartitionKey eq '{tenantId}' " +
|
||||
$"and IsSelfDeployingProfile eq true " +
|
||||
$"and (Status eq 'Incomplete' or Status eq 'AwaitingUser' or Status eq 'Stalled')";
|
||||
|
||||
var sessions = new List<SessionSummary>();
|
||||
await foreach (var entity in tableClient.QueryAsync<TableEntity>(filter: filter))
|
||||
{
|
||||
sessions.Add(MapToSessionSummary(entity));
|
||||
if (sessions.Count >= maxResults)
|
||||
break;
|
||||
}
|
||||
return sessions;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, $"Failed to get self-deploying silent sessions for tenant {tenantId}");
|
||||
return new List<SessionSummary>();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Narrow projection of every session row in the tenant partition — one scan feeding the
|
||||
/// in-memory Pending-orphan matching (misclassification audit 2026-07-16). Fields outside
|
||||
|
||||
@@ -92,6 +92,14 @@ namespace AutopilotMonitor.Shared.DataAccess
|
||||
/// </summary>
|
||||
Task<List<SessionSummary>> GetLegacyTimeoutFailedSessionsAsync(string tenantId, int maxResults);
|
||||
|
||||
/// <summary>
|
||||
/// Self-deploying-profile sessions parked in a non-success, non-failure state
|
||||
/// (Incomplete / AwaitingUser / Stalled) — candidates for the one-time retro-reconcile
|
||||
/// through <c>EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned</c> (kiosk tenant
|
||||
/// audit 2026-08-23). Server-side filter, capped at <paramref name="maxResults"/>.
|
||||
/// </summary>
|
||||
Task<List<SessionSummary>> GetSelfDeployingSilentSessionsAsync(string tenantId, int maxResults);
|
||||
|
||||
/// <summary>
|
||||
/// Narrow projection of every session row in the tenant partition (id, status, timing,
|
||||
/// serial). Used by the Pending-orphan resolution to match superseding sessions of the
|
||||
|
||||
Reference in New Issue
Block a user