Compare commits

...

8 Commits

Author SHA1 Message Date
Oliver Kieselbach 517041a51c WDP detection: probe DevicePreparation BootstrapperAgent orchestration key
Field-verified on a live WDP enrollment (session 7b072755): WDP devices
carry NO Autopilot profile, so both CloudAssigned* v2 signals are
structurally absent and every WDP session was classified v1. The reliable
marker is the Device Preparation orchestration tree
AutopilotSettings\DevicePreparation\BootstrapperAgent, whose
ExecutionContext JSON carries the WDP policyId - present on WDP, absent on
classic v1 (verified there too; classic only has ESP category VALUES named
DevicePreparation.* on the parent key, hence a subkey probe).

CloudAssigned* checks stay as defence in depth. Profile seeding and the
downgrade guards from earlier today take over once registration carries v2.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-17 19:08:26 +02:00
Oliver Kieselbach eeaeb9ea7e Corp identifier: match in Intune-normalized space
Field-verified against the portal list: Intune normalizes corporate
identifiers on upload - uppercased, all non-alphanumerics stripped per
component ("Microsoft Corporation,Virtual Machine,7801-5131-..." is stored
as "MICROSOFTCORPORATION,VIRTUALMACHINE,78015131...") and the device
matched + enrolled against that normalized entry. Our exact-match against
the raw device values (and the contains() filter using the dashed serial)
could therefore never hit.

Both sides now normalize before comparison, and the contains() narrowing
uses the normalized serial. Raw-form entries created directly via Graph
still match. The not-found message shows raw AND normalized form so
rejection logs line up with the portal list.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-17 18:25:05 +02:00
Oliver Kieselbach 3983402983 Corp identifier: read importedDeviceIdentities via GET, no new Graph scope
The searchExistingIdentities ACTION requires
DeviceManagementServiceConfig.ReadWrite.All, which the app deliberately does
not hold - the feature was broken in every tenant. Listing the same data via
GET works with Read.All, which the app already has for
windowsAutopilotDeviceIdentities. No new permission, no customer re-consent.

- contains(importedDeviceIdentifier,serial) filter for server-side
  narrowing (mirrors AutopilotDeviceValidator), exact case-insensitive
  match client-side on manufacturerModelSerial identities only
- 400 filter rejection falls back to an unfiltered paged scan
  (60 x 1000 covers the portal maximum of 10 CSVs x 5000 rows);
  page-cap overrun is transient, never a cached false negative
- Graph 401/403 stays a definitive, negative-cached failure with an
  actionable message (now naming Read.All)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-17 17:47:25 +02:00
Oliver Kieselbach 4deb5e6a04 Corp identifier: MMS-only matching, Graph 401/403 = definitive failure
Two changes from the WDP live-test follow-up:

- Remove the serialNumber-type probe again (decision: only the documented
  Windows manufacturerModelSerial identifier authorizes; a serial-type entry
  is an Intune-side admin misconfiguration and must not count).
- Classify Graph 401/403 on searchExistingIdentities as DEFINITIVE with an
  actionable message naming the missing permission
  (DeviceManagementServiceConfig.ReadWrite.All / admin consent) and
  negative-cache it. Field case: the missing consent was classified
  transient and trapped the agent in an endless 503 Retry-After loop.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-17 17:36:21 +02:00
Oliver Kieselbach 5734eb576e Correct corp identifier dual-probe rationale in comments
Docs research showed the serial-only corporate identifier type is NOT the
documented WDP path (Windows only supports manufacturer,model,serial; Intune
ignores serial-type for Windows ownership). The dual probe stays because the
serial type is the only one the portal offers for MANUAL entry, so admins
land there by mistake and an uploaded serial still expresses authorization
intent. Comments and test docs now state that rationale instead of the
incorrect "WDP guidance uses serial type" claim. No behavior change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-17 17:25:35 +02:00
Oliver Kieselbach 7fbfea8f4e Corp identifier validation: probe serialNumber type + unmask validator errors
First live WDP enrollment 403d at device validation. Root causes:

- CorporateIdentifierValidator only searched the manufacturerModelSerial
  identifier type, but the Device Preparation guidance has admins upload
  SERIAL-type corporate identifiers - those never matched. The Graph
  searchExistingIdentities call now probes both types in one round-trip.
- SecurityValidator overwrote deviceValidationError per validator, so the
  403 Details only showed the LAST failure (CloudPc) and hid the corporate
  identifier miss. Errors now aggregate in chain order and the transient
  flag is sticky-OR (any transient failure yields 503 Retry-After instead
  of a definitive 403).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-17 16:41:57 +02:00
Oliver Kieselbach e5f955354d Hourly SessionSweep interleave: stalled-session sweep every hour instead of 2h
New SessionSweepFunction (cron 0 30 * * * *, offset from the 2h Maintenance
timer at minute 0) runs only MarkStalledSessionsAsTimedOutAsync, halving the
sweep-induced classification latency; thresholds unchanged. The sweep now
returns a SessionSweepResult (counts + catastrophic-failure message) so the
new timer emits SessionSweepCompleted heartbeats / SessionSweepFailed errors
(dual-registered in the ops-alert web catalog). No lease: the sweep is
idempotent, overlap with the 2h chain is a no-op.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-17 14:33:26 +02:00
Oliver Kieselbach 65651dd599 ANALYZE-ID-004 v1.2.0: pre-logon VPN + Hello-as-symptom self-service guidance
From the 3b7ba31f (succeeded) vs gardner/esa (parked) hybrid session comparison:
explanation now covers the cached-credentials/pre-logon-VPN variant and why the
Hello wizard never appearing is downstream of the missing PRT; new remediation
block for VPN machine tunnels, cloud Kerberos trust, explicit Hello disable,
update deferral and pre-provisioning. Links the new docs article.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-17 13:36:10 +02:00
12 changed files with 636 additions and 70 deletions
+16 -2
View File
@@ -5,7 +5,7 @@
"description": "On a hybrid-joined device the real user reached the desktop, but the Autopilot provisioning placeholder stayed active in JoinInfo and the agent repeatedly reported the hybrid sign-in as overdue. The user session never establishes Entra user affinity (no PRT), so the Account Setup ESP re-runs on every logon and the enrollment never completes.",
"severity": "high",
"category": "identity",
"version": "1.1.0",
"version": "1.2.0",
"author": "Autopilot Monitor",
"enabled": true,
"trigger": "correlation",
@@ -51,7 +51,7 @@
{ "signal": "hybrid_login_pending", "condition": "count >= 3", "weight": 10 },
{ "signal": "aad_placeholder_user_detected", "condition": "count >= 3", "weight": 10 }
],
"explanation": "A real user signed in and reached the desktop, but the device kept reporting the **Autopilot provisioning placeholder** as the active identity in JoinInfo and the hybrid sign-in as **overdue** — across multiple reboots. This means the signed-in user never established **Microsoft Entra user affinity** (no Primary Refresh Token), so:\n\n- the Account Setup ESP page re-appears on every logon,\n- user-targeted required apps and policies cannot settle,\n- the enrollment never reaches completion.\n\nAfter this many hours this is not normal Entra Connect sync latency — something is structurally preventing the user (or the computer object) from registering:\n\n- the computer object's `userCertificate` attribute is not syncing to Entra ID (sync scope / OU filtering),\n- the signing-in user is not synced, or their UPN does not match a verified Entra domain,\n- the device has no line of sight to a domain controller **at logon time**, so the hybrid PRT flow cannot finish,\n- time skew or a proxy blocking `login.microsoftonline.com` / `enterpriseregistration.windows.net` from the user context.",
"explanation": "A real user signed in and reached the desktop, but the device kept reporting the **Autopilot provisioning placeholder** as the active identity in JoinInfo and the hybrid sign-in as **overdue** — across multiple reboots. This means the signed-in user never established **Microsoft Entra user affinity** (no Primary Refresh Token), so:\n\n- the Account Setup ESP page re-appears on every logon,\n- user-targeted required apps and policies cannot settle,\n- the enrollment never reaches completion.\n\nAfter this many hours this is not normal Entra Connect sync latency — something is structurally preventing the user (or the computer object) from registering:\n\n- the computer object's `userCertificate` attribute is not syncing to Entra ID (sync scope / OU filtering),\n- the signing-in user is not synced, or their UPN does not match a verified Entra domain,\n- the device has no line of sight to a domain controller **at logon time**, so the hybrid PRT flow cannot finish,\n- time skew or a proxy blocking `login.microsoftonline.com` / `enterpriseregistration.windows.net` from the user context.\n\nA frequent variant: the user's **first sign-in happens without domain-controller reachability** — for example when a pre-logon VPN tunnel (GlobalProtect, Always On VPN, Zscaler ZPA) is not established yet at the logon screen. Windows falls back to cached credentials, the hybrid PRT flow silently never runs, and later reboots do not fix it — sessions in this state typically show repeated reboots and network flapping while the placeholder identity stays active.\n\nA visible side effect: **the Windows Hello for Business wizard never appears**, because Hello provisioning requires an Entra user token (PRT). Hello is a downstream symptom here, not the cause.",
"remediation": [
{
"title": "Verify user affinity on the device",
@@ -70,9 +70,23 @@
"Ensure domain-controller line of sight at logon from the provisioning network/VLAN (hybrid PRT requires a reachable DC).",
"Check proxy/firewall rules for login.microsoftonline.com, device.login.microsoftonline.com and enterpriseregistration.windows.net from the user context."
]
},
{
"title": "Address VPN and Windows Hello dependencies",
"steps": [
"If a VPN is required to reach a domain controller, it must be established before or at the logon screen (machine tunnel / pre-logon tunnel). A user-context tunnel that starts after sign-in is too late for the hybrid PRT flow.",
"If you deploy Windows Hello for Business to hybrid-joined devices, use cloud Kerberos trust — it removes the on-premises certificate/key-trust dependencies that commonly stall Hello on hybrid devices.",
"If you do not use Windows Hello, disable it explicitly via a device-scoped Intune policy instead of leaving it unconfigured — the enrollment then resolves the Hello step deterministically instead of waiting for a wizard that never comes.",
"Avoid feature/quality updates mid-enrollment where possible — an OS update with forced reboots during enrollment aggravates this pattern. Consider a short Windows Update deferral covering the enrollment window.",
"For hybrid deployments, pre-provisioning (White Glove) materially shrinks the user-facing phase: the device phase runs on a known-good technician network, and the user phase reduces to sign-in, user ESP, and Hello."
]
}
],
"relatedDocs": [
{
"title": "Autopilot Monitor: Hybrid join — stuck in Account Setup (no user affinity)",
"url": "https://docs.autopilotmonitor.com/troubleshooting/hybrid-join-stuck-in-account-setup"
},
{
"title": "Autopilot Monitor: Hybrid join — late device registration",
"url": "https://docs.autopilotmonitor.com/troubleshooting/hybrid-join-late-device-registration"
+16 -2
View File
@@ -2526,7 +2526,7 @@
"description": "On a hybrid-joined device the real user reached the desktop, but the Autopilot provisioning placeholder stayed active in JoinInfo and the agent repeatedly reported the hybrid sign-in as overdue. The user session never establishes Entra user affinity (no PRT), so the Account Setup ESP re-runs on every logon and the enrollment never completes.",
"severity": "high",
"category": "identity",
"version": "1.1.0",
"version": "1.2.0",
"author": "Autopilot Monitor",
"enabled": true,
"trigger": "correlation",
@@ -2587,7 +2587,7 @@
"weight": 10
}
],
"explanation": "A real user signed in and reached the desktop, but the device kept reporting the **Autopilot provisioning placeholder** as the active identity in JoinInfo and the hybrid sign-in as **overdue** — across multiple reboots. This means the signed-in user never established **Microsoft Entra user affinity** (no Primary Refresh Token), so:\n\n- the Account Setup ESP page re-appears on every logon,\n- user-targeted required apps and policies cannot settle,\n- the enrollment never reaches completion.\n\nAfter this many hours this is not normal Entra Connect sync latency — something is structurally preventing the user (or the computer object) from registering:\n\n- the computer object's `userCertificate` attribute is not syncing to Entra ID (sync scope / OU filtering),\n- the signing-in user is not synced, or their UPN does not match a verified Entra domain,\n- the device has no line of sight to a domain controller **at logon time**, so the hybrid PRT flow cannot finish,\n- time skew or a proxy blocking `login.microsoftonline.com` / `enterpriseregistration.windows.net` from the user context.",
"explanation": "A real user signed in and reached the desktop, but the device kept reporting the **Autopilot provisioning placeholder** as the active identity in JoinInfo and the hybrid sign-in as **overdue** — across multiple reboots. This means the signed-in user never established **Microsoft Entra user affinity** (no Primary Refresh Token), so:\n\n- the Account Setup ESP page re-appears on every logon,\n- user-targeted required apps and policies cannot settle,\n- the enrollment never reaches completion.\n\nAfter this many hours this is not normal Entra Connect sync latency — something is structurally preventing the user (or the computer object) from registering:\n\n- the computer object's `userCertificate` attribute is not syncing to Entra ID (sync scope / OU filtering),\n- the signing-in user is not synced, or their UPN does not match a verified Entra domain,\n- the device has no line of sight to a domain controller **at logon time**, so the hybrid PRT flow cannot finish,\n- time skew or a proxy blocking `login.microsoftonline.com` / `enterpriseregistration.windows.net` from the user context.\n\nA frequent variant: the user's **first sign-in happens without domain-controller reachability** — for example when a pre-logon VPN tunnel (GlobalProtect, Always On VPN, Zscaler ZPA) is not established yet at the logon screen. Windows falls back to cached credentials, the hybrid PRT flow silently never runs, and later reboots do not fix it — sessions in this state typically show repeated reboots and network flapping while the placeholder identity stays active.\n\nA visible side effect: **the Windows Hello for Business wizard never appears**, because Hello provisioning requires an Entra user token (PRT). Hello is a downstream symptom here, not the cause.",
"remediation": [
{
"title": "Verify user affinity on the device",
@@ -2606,9 +2606,23 @@
"Ensure domain-controller line of sight at logon from the provisioning network/VLAN (hybrid PRT requires a reachable DC).",
"Check proxy/firewall rules for login.microsoftonline.com, device.login.microsoftonline.com and enterpriseregistration.windows.net from the user context."
]
},
{
"title": "Address VPN and Windows Hello dependencies",
"steps": [
"If a VPN is required to reach a domain controller, it must be established before or at the logon screen (machine tunnel / pre-logon tunnel). A user-context tunnel that starts after sign-in is too late for the hybrid PRT flow.",
"If you deploy Windows Hello for Business to hybrid-joined devices, use cloud Kerberos trust — it removes the on-premises certificate/key-trust dependencies that commonly stall Hello on hybrid devices.",
"If you do not use Windows Hello, disable it explicitly via a device-scoped Intune policy instead of leaving it unconfigured — the enrollment then resolves the Hello step deterministically instead of waiting for a wizard that never comes.",
"Avoid feature/quality updates mid-enrollment where possible — an OS update with forced reboots during enrollment aggravates this pattern. Consider a short Windows Update deferral covering the enrollment window.",
"For hybrid deployments, pre-provisioning (White Glove) materially shrinks the user-facing phase: the device phase runs on a known-good technician network, and the user phase reduces to sign-in, user ESP, and Hello."
]
}
],
"relatedDocs": [
{
"title": "Autopilot Monitor: Hybrid join — stuck in Account Setup (no user affinity)",
"url": "https://docs.autopilotmonitor.com/troubleshooting/hybrid-join-stuck-in-account-setup"
},
{
"title": "Autopilot Monitor: Hybrid join — late device registration",
"url": "https://docs.autopilotmonitor.com/troubleshooting/hybrid-join-late-device-registration"
@@ -170,8 +170,17 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
/// <summary>
/// Classifies the Autopilot flow based on the <c>AutopilotSettings</c> registry:
/// <list type="bullet">
/// <item><c>CloudAssignedDeviceRegistration == 2</c> → <c>"v2"</c> (Windows Device Preparation).</item>
/// <item><c>CloudAssignedEspEnabled == 0</c> → <c>"v2"</c> (no ESP, WDP indicator).</item>
/// <item><c>DevicePreparation\BootstrapperAgent</c> SUBKEY with a non-empty
/// <c>ExecutionContext</c> → <c>"v2"</c>. This is the Device Preparation policy
/// orchestration tree (ExecutionContext JSON carries the WDP policyId and the
/// provider batch list). Field-verified 2026-08-17: present on a WDP device,
/// absent on classic v1 — which only has ESP category VALUES named
/// <c>DevicePreparation.*</c> on the AutopilotSettings key itself, hence the
/// subkey probe, never a value-name probe.</item>
/// <item><c>CloudAssignedDeviceRegistration == 2</c> → <c>"v2"</c> (legacy signal).</item>
/// <item><c>CloudAssignedEspEnabled == 0</c> → <c>"v2"</c> (no ESP, legacy signal).
/// Both CloudAssigned* values are profile-derived and structurally ABSENT on WDP
/// (no Autopilot profile), so they never fire there — kept for defence in depth.</item>
/// <item>Anything else → <c>"v1"</c> (Classic Autopilot / ESP).</item>
/// </list>
/// Defaults to <c>"v1"</c> on any error so a flaky registry does not misclassify a
@@ -181,6 +190,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
{
try
{
using (var wdpKey = Registry.LocalMachine.OpenSubKey(
AutopilotSettingsKey + @"\DevicePreparation\BootstrapperAgent"))
{
var executionContext = wdpKey?.GetValue("ExecutionContext")?.ToString();
if (!string.IsNullOrEmpty(executionContext)) return "v2";
}
using (var key = Registry.LocalMachine.OpenSubKey(AutopilotSettingsKey))
{
if (key == null) return "v1";
@@ -0,0 +1,250 @@
using System.Net;
using AutopilotMonitor.Functions.Security;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.DataAccess;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Graph contract tests for <see cref="CorporateIdentifierValidator"/>. Field case 2026-08-17
/// (first live Device Preparation enrollment): the searchExistingIdentities ACTION requires
/// DeviceManagementServiceConfig.ReadWrite.All, which the app does not hold, so validation
/// 403'd in every tenant — and the transient classification trapped the agent in an endless
/// 503 Retry-After loop. The validator now READS importedDeviceIdentities (Read.All suffices),
/// matches only manufacturerModelSerial identities exactly (client-side, case-insensitive),
/// and classifies Graph 401/403 as a definitive, actionable failure.
/// </summary>
public class CorporateIdentifierValidatorTests
{
private const string TenantId = "11111111-1111-1111-1111-111111111111";
private const string Serial = "7801-5131-4473-3387-5637-4002-18";
private const string MmsIdentifier = $"Microsoft Corporation,Virtual Machine,{Serial}";
// The form Intune actually stores (field-verified 2026-08-17 in the portal list):
// uppercased, spaces and dashes stripped per component.
private const string StoredIdentifier = "MICROSOFTCORPORATION,VIRTUALMACHINE,78015131447333875637400218";
private const string StoredSerial = "78015131447333875637400218";
private sealed class SequenceHandler : HttpMessageHandler
{
private readonly Queue<HttpResponseMessage> _responses;
public List<HttpRequestMessage> Requests { get; } = new();
public SequenceHandler(params HttpResponseMessage[] responses) => _responses = new Queue<HttpResponseMessage>(responses);
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken ct)
{
Requests.Add(request);
// Last response is sticky so retry loops in the SUT cannot dequeue past the script.
var response = _responses.Count > 1 ? _responses.Dequeue() : _responses.Peek();
return Task.FromResult(response);
}
}
private static HttpResponseMessage Json(string body, HttpStatusCode status = HttpStatusCode.OK) =>
new(status) { Content = new StringContent(body) };
private static (CorporateIdentifierValidator Sut, SequenceHandler Handler) BuildSut(params HttpResponseMessage[] responses)
{
var handler = new SequenceHandler(responses);
var factory = new Mock<IHttpClientFactory>();
factory.Setup(f => f.CreateClient(It.IsAny<string>())).Returns(new HttpClient(handler));
var cache = new MemoryCache(new MemoryCacheOptions());
var configuration = new ConfigurationBuilder().AddInMemoryCollection(new Dictionary<string, string?>
{
["EntraId:ClientId"] = "aaaaaaaa-0000-0000-0000-000000000001",
["EntraId:ClientSecret"] = "secret",
}).Build();
var registry = new EntraAppRegistry(configuration, NullLogger<EntraAppRegistry>.Instance);
var tenantConfig = new Mock<TenantConfigurationService>(
Mock.Of<IConfigRepository>(), NullLogger<TenantConfigurationService>.Instance, cache)
{ CallBase = false };
var tokenService = new Mock<GraphTokenService>(
NullLogger<GraphTokenService>.Instance, Mock.Of<IHttpClientFactory>(), cache,
configuration, registry, tenantConfig.Object)
{ CallBase = false };
tokenService.Setup(t => t.GetAccessTokenAsync(TenantId, It.IsAny<CancellationToken>()))
.ReturnsAsync(GraphTokenResult.Success("token"));
var sut = new CorporateIdentifierValidator(
NullLogger<CorporateIdentifierValidator>.Instance, factory.Object, cache, tokenService.Object);
return (sut, handler);
}
[Fact]
public async Task ValidateAsync_usesGet_withContainsFilterOnNormalizedSerial()
{
var (sut, handler) = BuildSut(Json("""{"value":[]}"""));
await sut.ValidateAsync(TenantId, "Microsoft Corporation", "Virtual Machine", Serial);
var request = Assert.Single(handler.Requests);
Assert.Equal(HttpMethod.Get, request.Method);
var url = request.RequestUri!.ToString();
Assert.Contains("/beta/deviceManagement/importedDeviceIdentities", url);
Assert.DoesNotContain("searchExistingIdentities", url);
// The stored value is normalized (dashless), so filtering with the raw dashed serial
// could never hit — the filter must use the normalized serial.
Assert.Contains($"contains(importedDeviceIdentifier,'{StoredSerial}')", Uri.UnescapeDataString(url));
}
[Fact]
public async Task ValidateAsync_storedNormalizedIdentity_matchesRawDeviceValues()
{
// THE field case: device reports "Microsoft Corporation" / "Virtual Machine" / dashed
// serial; Intune stores "MICROSOFTCORPORATION,VIRTUALMACHINE,<dashless>". Must match.
var (sut, _) = BuildSut(Json($$"""
{"value":[{"importedDeviceIdentityType":"manufacturerModelSerial","importedDeviceIdentifier":"{{StoredIdentifier}}"}]}
"""));
var result = await sut.ValidateAsync(TenantId, "Microsoft Corporation", "Virtual Machine", Serial);
Assert.True(result.IsValid);
Assert.False(result.IsTransient);
Assert.Equal(MmsIdentifier, result.Identifier);
}
[Fact]
public async Task ValidateAsync_rawStoredIdentity_alsoMatches()
{
// Entries created directly via Graph may be stored raw — both sides are normalized
// before comparison, so these match too.
var (sut, _) = BuildSut(Json($$"""
{"value":[{"importedDeviceIdentityType":"manufacturerModelSerial","importedDeviceIdentifier":"{{MmsIdentifier}}"}]}
"""));
var result = await sut.ValidateAsync(TenantId, "Microsoft Corporation", "Virtual Machine", Serial);
Assert.True(result.IsValid);
}
[Fact]
public async Task ValidateAsync_differentDevice_doesNotMatch()
{
var (sut, _) = BuildSut(Json("""
{"value":[{"importedDeviceIdentityType":"manufacturerModelSerial","importedDeviceIdentifier":"MICROSOFTCORPORATION,VIRTUALMACHINE,39260126318839370830655125"}]}
"""));
var result = await sut.ValidateAsync(TenantId, "Microsoft Corporation", "Virtual Machine", Serial);
Assert.False(result.IsValid);
Assert.False(result.IsTransient);
}
[Fact]
public async Task ValidateAsync_serialOnlyIdentity_doesNotAuthorize()
{
// Serial-type corporate identifiers are an Intune-side misconfiguration for Windows,
// deliberately not treated as authorization (decision 2026-08-17).
var (sut, _) = BuildSut(Json($$"""
{"value":[{"importedDeviceIdentityType":"serialNumber","importedDeviceIdentifier":"{{StoredSerial}}"}]}
"""));
var result = await sut.ValidateAsync(TenantId, "Microsoft Corporation", "Virtual Machine", Serial);
Assert.False(result.IsValid);
Assert.False(result.IsTransient);
}
[Fact]
public async Task ValidateAsync_followsNextLink_untilMatch()
{
var (sut, handler) = BuildSut(
Json("""{"value":[{"importedDeviceIdentityType":"manufacturerModelSerial","importedDeviceIdentifier":"DELL,XPS,OTHER1"}],"@odata.nextLink":"https://graph.microsoft.com/beta/deviceManagement/importedDeviceIdentities?$skiptoken=abc"}"""),
Json($$"""{"value":[{"importedDeviceIdentityType":"manufacturerModelSerial","importedDeviceIdentifier":"{{StoredIdentifier}}"}]}"""));
var result = await sut.ValidateAsync(TenantId, "Microsoft Corporation", "Virtual Machine", Serial);
Assert.True(result.IsValid);
Assert.Equal(2, handler.Requests.Count);
Assert.Contains("skiptoken", handler.Requests[1].RequestUri!.ToString());
}
[Fact]
public async Task ValidateAsync_noMatch_isDefinitive_andShowsBothForms()
{
var (sut, _) = BuildSut(Json("""{"value":[]}"""));
var result = await sut.ValidateAsync(TenantId, "Microsoft Corporation", "Virtual Machine", Serial);
Assert.False(result.IsValid);
Assert.False(result.IsTransient);
// Both the raw and the normalized form must appear so a rejection log is diagnosable
// against the portal list (which shows only the normalized form).
Assert.Contains(MmsIdentifier, result.ErrorMessage);
Assert.Contains(StoredIdentifier, result.ErrorMessage);
}
[Fact]
public async Task ValidateAsync_filterRejected_fallsBackToUnfilteredScan()
{
// Some Intune endpoints reject $filter with 400 — the validator must then page the
// unfiltered list instead of failing.
var (sut, handler) = BuildSut(
Json("""{"error":{"code":"BadRequest","message":"filter not supported"}}""", HttpStatusCode.BadRequest),
Json($$"""{"value":[{"importedDeviceIdentityType":"manufacturerModelSerial","importedDeviceIdentifier":"{{StoredIdentifier}}"}]}"""));
var result = await sut.ValidateAsync(TenantId, "Microsoft Corporation", "Virtual Machine", Serial);
Assert.True(result.IsValid);
Assert.Equal(2, handler.Requests.Count);
Assert.Contains("$filter", handler.Requests[0].RequestUri!.ToString());
Assert.DoesNotContain("$filter", handler.Requests[1].RequestUri!.ToString());
}
[Theory]
[InlineData("Microsoft Corporation", "MICROSOFTCORPORATION")]
[InlineData("Virtual Machine", "VIRTUALMACHINE")]
[InlineData("7801-5131-4473-3387-5637-4002-18", "78015131447333875637400218")]
[InlineData("ThinkPad T14 Gen 3", "THINKPADT14GEN3")]
[InlineData("Serial.With.Periods", "SERIALWITHPERIODS")]
public void NormalizeComponent_matchesIntunePortalNormalization(string raw, string expected)
{
Assert.Equal(expected, CorporateIdentifierValidator.NormalizeComponent(raw));
}
[Theory]
[InlineData(HttpStatusCode.Forbidden)]
[InlineData(HttpStatusCode.Unauthorized)]
public async Task ValidateAsync_graphPermissionDenied_isDefinitive_withActionableMessage(HttpStatusCode status)
{
var (sut, _) = BuildSut(Json("""{"error":{"code":"Forbidden","message":"Application is not authorized"}}""", status));
var result = await sut.ValidateAsync(TenantId, "Microsoft Corporation", "Virtual Machine", Serial);
Assert.False(result.IsValid);
Assert.False(result.IsTransient);
Assert.Contains("DeviceManagementServiceConfig.Read.All", result.ErrorMessage);
Assert.Contains("consent", result.ErrorMessage);
}
[Fact]
public async Task ValidateAsync_graphPermissionDenied_isCached_notRetried()
{
// Definitive permission failures are negative-cached so an agent retry storm does not
// hammer Graph while consent is being fixed; the in-call retry loop must not fire either.
var (sut, handler) = BuildSut(Json("""{"error":{"code":"Forbidden"}}""", HttpStatusCode.Forbidden));
await sut.ValidateAsync(TenantId, "Microsoft Corporation", "Virtual Machine", Serial);
var second = await sut.ValidateAsync(TenantId, "Microsoft Corporation", "Virtual Machine", Serial);
Assert.False(second.IsValid);
Assert.Single(handler.Requests);
}
[Fact]
public async Task ValidateAsync_graphServerError_isTransient()
{
var (sut, _) = BuildSut(Json("""{"error":{"code":"InternalServerError"}}""", HttpStatusCode.InternalServerError));
var result = await sut.ValidateAsync(TenantId, "Microsoft Corporation", "Virtual Machine", Serial);
Assert.False(result.IsValid);
Assert.True(result.IsTransient);
}
}
@@ -141,4 +141,28 @@ public class SecurityValidatorTests
Assert.False(SecurityValidator.TryGetIntuneDeviceIdFromCertSubject(
"CN=hostname, CN=07623d56-1e77-4948-bff5-5bdac8167560", out _));
}
// --- Device-validation error aggregation ---
// With several validators enabled, the 403 Details used to carry only the LAST failure
// (CloudPc), hiding why the Autopilot / CorporateIdentifier lookups missed — the exact
// trap during the first live WDP enrollment (2026-08-17).
[Fact]
public void CombineValidationErrors_joinsInChainOrder()
{
var combined = SecurityValidator.CombineValidationErrors("autopilot miss", "corp miss");
combined = SecurityValidator.CombineValidationErrors(combined, "cloudpc miss");
Assert.Equal("autopilot miss | corp miss | cloudpc miss", combined);
}
[Theory]
[InlineData(null, "only", "only")]
[InlineData("only", null, "only")]
[InlineData(null, null, null)]
[InlineData("", "only", "only")]
public void CombineValidationErrors_handlesMissingHalves(string? a, string? b, string? expected)
{
Assert.Equal(expected, SecurityValidator.CombineValidationErrors(a, b));
}
}
@@ -27,6 +27,8 @@ namespace AutopilotMonitor.Functions.Functions.Admin
/// NCRONTAB format: {second} {minute} {hour} {day} {month} {day-of-week}
/// Reduced from daily (2:00 UTC) to every 2 hours to catch stalled sessions faster.
/// MarkStalledSessionsAsTimedOutAsync is idempotent (terminal states not overwritten).
/// The stalled-session sweep additionally runs hourly at minute 30 via the dedicated
/// SessionSweepFunction interleave — this chain keeps it as its first step regardless.
/// </summary>
[Function("Maintenance")]
public async Task Run([TimerTrigger("0 0 */2 * * *")] object timer)
@@ -0,0 +1,65 @@
using System.Diagnostics;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Services;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Functions.Maintenance
{
/// <summary>
/// Hourly interleave for the stalled-session sweep
/// (<see cref="MaintenanceService.MarkStalledSessionsAsTimedOutAsync"/>). The 2h
/// <c>Maintenance</c> timer keeps running the sweep as its first step; this function adds an
/// offset tick at minute 30, so between the two the sweep effectively runs every hour and the
/// sweep-induced classification latency drops from ≤2h to ≤1h (a 5h-silent session is
/// terminalized after ~6h worst case instead of ~7h). Thresholds (2h silence → Stalled,
/// SessionTimeoutHours → terminal) are unchanged — only the sampling rate rises.
/// <para>
/// No lease/serialization on purpose (unlike <see cref="SessionDeletionMaintenanceFunction"/>):
/// the sweep is idempotent — terminal states are never overwritten, the silence guard protects
/// live sessions, and a doubled pass is a no-op — so overlapping with the 2h run is harmless.
/// Extraction pattern per the cadence-split precedent documented in
/// <c>MaintenanceService.RunAllAsync</c> (SessionDeletionMaintenanceFunction).
/// </para>
/// </summary>
public class SessionSweepFunction
{
// Minute 30 — deliberately offset from the 2h Maintenance timer (minute 0) so the two
// never fire together. const per repo convention (see IndexReconcileTimer.cs).
private const string Cron = "0 30 * * * *";
private readonly MaintenanceService _maintenanceService;
private readonly OpsEventService _opsEventService;
private readonly ILogger<SessionSweepFunction> _logger;
public SessionSweepFunction(
MaintenanceService maintenanceService,
OpsEventService opsEventService,
ILogger<SessionSweepFunction> logger)
{
_maintenanceService = maintenanceService;
_opsEventService = opsEventService;
_logger = logger;
}
[Function("SessionSweep")]
public async Task Run([TimerTrigger(Cron)] object timer)
{
_logger.LogInformation("SessionSweep timer trigger fired");
var sw = Stopwatch.StartNew();
var result = await _maintenanceService.MarkStalledSessionsAsTimedOutAsync();
sw.Stop();
if (result.Error is null)
{
await _opsEventService.RecordSessionSweepCompletedAsync(
result.StalledMarked, result.TimedOut, (int)sw.ElapsedMilliseconds);
}
else
{
await _opsEventService.RecordSessionSweepFailedAsync(result.Error);
}
}
}
}
@@ -13,7 +13,15 @@ namespace AutopilotMonitor.Functions.Security
{
/// <summary>
/// Validates devices against Intune Corporate Device Identifiers via Microsoft Graph beta API.
/// Uses the importedDeviceIdentities/searchExistingIdentities endpoint with manufacturerModelSerial type.
/// Reads /deviceManagement/importedDeviceIdentities (GET, DeviceManagementServiceConfig.Read.All
/// suffices) instead of the searchExistingIdentities action, which would require the heavy
/// ReadWrite.All scope the app deliberately does not hold (field case 2026-08-17: the action
/// 403'd in every tenant). Only the manufacturerModelSerial type counts — the sole
/// corporate-identifier type Windows supports; serial-only entries are an Intune-side admin
/// misconfiguration and must not authorize. Matching happens in Intune's NORMALIZED space
/// (uppercase, non-alphanumerics stripped per component — see <see cref="NormalizeComponent"/>),
/// because that is the form the portal stores and matches at enrollment. A Graph 401/403
/// means missing admin consent and is classified as a definitive failure, not transient.
/// Caches positive/negative lookups to reduce Graph traffic.
/// </summary>
public class CorporateIdentifierValidator
@@ -114,70 +122,83 @@ namespace AutopilotMonitor.Functions.Security
var graphClient = _httpClientFactory.CreateClient();
graphClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", tokenResult.AccessToken);
// POST https://graph.microsoft.com/beta/deviceManagement/importedDeviceIdentities/searchExistingIdentities
var identifier = $"{normalizedManufacturer},{normalizedModel},{normalizedSerial}";
var requestBody = new
{
importedDeviceIdentities = new[]
{
new
{
importedDeviceIdentityType = "manufacturerModelSerial",
importedDeviceIdentifier = identifier
}
}
};
var content = new StringContent(
JsonConvert.SerializeObject(requestBody),
Encoding.UTF8,
"application/json");
var graphUrl = Constants.GraphBaseUrl + "/beta/deviceManagement/importedDeviceIdentities/searchExistingIdentities";
var response = await graphClient.PostAsync(graphUrl, content);
var responseBody = await response.Content.ReadAsStringAsync();
if (!response.IsSuccessStatusCode)
// GET /beta/deviceManagement/importedDeviceIdentities — deliberately NOT the
// searchExistingIdentities action: the action requires the heavy
// DeviceManagementServiceConfig.ReadWrite.All scope, while listing works with
// Read.All, which the app already holds for windowsAutopilotDeviceIdentities.
// Mirrors AutopilotDeviceValidator: contains() for server-side narrowing (eq is
// unreliable on Intune endpoints), exact match client-side.
//
// Intune NORMALIZES corporate identifiers on upload: uppercased, every
// non-alphanumeric character stripped per component — "Microsoft Corporation,
// Virtual Machine,7801-5131-…" is stored as "MICROSOFTCORPORATION,
// VIRTUALMACHINE,78015131…" (field-verified 2026-08-17: the portal list shows
// the normalized form and the device matched + enrolled against it). All
// matching therefore happens in normalized space — including the contains()
// narrowing, which with the raw dashed serial would never hit the stored value.
var rawIdentifier = $"{normalizedManufacturer},{normalizedModel},{normalizedSerial}";
var normalizedIdentifier = BuildNormalizedIdentifier(normalizedManufacturer, normalizedModel, normalizedSerial);
// Normalized serial is alphanumeric-only, so no OData quote-escaping is needed.
var filter = Uri.EscapeDataString($"contains(importedDeviceIdentifier,'{NormalizeComponent(normalizedSerial)}')");
var filteredUrl = $"{Constants.GraphBaseUrl}/beta/deviceManagement/importedDeviceIdentities?$top=100&$filter={filter}";
var scan = await ScanPagesForIdentifierAsync(graphClient, filteredUrl, normalizedIdentifier, tenantId, attempt, maxPages: 5);
if (scan.Outcome == ScanOutcome.FilterRejected)
{
// Fallback: the endpoint rejected the filter (400) — page the unfiltered list
// and match client-side. 60 pages x 1000 covers the portal maximum of
// 10 CSV files x 5000 identifiers.
_logger.LogWarning(
"Corporate identifier validation Graph query failed for tenant {TenantId} (attempt {Attempt}). Status: {StatusCode}. Body: {ResponseBody}",
tenantId, attempt, (int)response.StatusCode, responseBody);
// Graph errors are transient — do NOT cache
return new CorporateIdentifierValidationResult
{
IsValid = false,
IsTransient = true,
ErrorMessage = $"Graph query failed with status {(int)response.StatusCode}"
};
"Corporate identifier validation: contains() filter rejected by Graph for tenant {TenantId}; falling back to unfiltered scan.",
tenantId);
var unfilteredUrl = $"{Constants.GraphBaseUrl}/beta/deviceManagement/importedDeviceIdentities?$top=1000";
scan = await ScanPagesForIdentifierAsync(graphClient, unfilteredUrl, normalizedIdentifier, tenantId, attempt, maxPages: 60);
}
var data = JsonConvert.DeserializeObject<JObject>(responseBody);
var identities = data?["value"] as JArray;
if (identities == null || identities.Count == 0)
switch (scan.Outcome)
{
// Definitive: device not found — cache negative result
return CacheAndReturn(cacheKey, new CorporateIdentifierValidationResult
{
IsValid = false,
ErrorMessage = $"Device '{identifier}' is not registered as a Corporate Identifier"
}, isPositive: false);
case ScanOutcome.Found:
_logger.LogInformation(
"Corporate identifier validation succeeded for tenant {TenantId}, session {SessionId}, identifier {Identifier} (normalized {NormalizedIdentifier})",
tenantId, sessionId ?? "<none>", rawIdentifier, normalizedIdentifier);
return CacheAndReturn(cacheKey, new CorporateIdentifierValidationResult
{
IsValid = true,
Identifier = rawIdentifier
}, isPositive: true);
case ScanOutcome.NotFound:
// Definitive: device not found — cache negative result
return CacheAndReturn(cacheKey, new CorporateIdentifierValidationResult
{
IsValid = false,
ErrorMessage = $"Device '{rawIdentifier}' (normalized '{normalizedIdentifier}') is not registered as a Corporate Identifier (manufacturerModelSerial)"
}, isPositive: false);
case ScanOutcome.PermissionDenied:
// 401/403 = missing application permission / admin consent. Retries can
// never heal that, so it is a DEFINITIVE failure — a transient
// classification would trap every agent in an endless 503 Retry-After
// loop (field case 2026-08-17). Cached like a negative lookup (5 min) so
// a fleet of agents does not hammer Graph while consent is being fixed.
return CacheAndReturn(cacheKey, new CorporateIdentifierValidationResult
{
IsValid = false,
ErrorMessage = "Corporate identifier lookup not permitted: the app registration lacks the "
+ "DeviceManagementServiceConfig.Read.All (or ReadWrite.All) Graph permission or admin "
+ "consent. Grant consent or disable ValidateCorporateIdentifier."
}, isPositive: false);
default:
// Transient (throttling, 5xx, page-cap exceeded) — do NOT cache
return new CorporateIdentifierValidationResult
{
IsValid = false,
IsTransient = true,
ErrorMessage = scan.Error ?? "Graph query failed"
};
}
var result = new CorporateIdentifierValidationResult
{
IsValid = true,
Identifier = identifier
};
_logger.LogInformation(
"Corporate identifier validation succeeded for tenant {TenantId}, session {SessionId}, identifier {Identifier}",
tenantId,
sessionId ?? "<none>",
identifier);
return CacheAndReturn(cacheKey, result, isPositive: true);
}
catch (Exception ex)
{
@@ -201,6 +222,116 @@ namespace AutopilotMonitor.Functions.Security
}
}
private enum ScanOutcome { Found, NotFound, PermissionDenied, FilterRejected, Transient }
private readonly record struct ScanResult(ScanOutcome Outcome, string? Error = null);
/// <summary>
/// Pages through an importedDeviceIdentities GET url (following @odata.nextLink) and
/// looks for a manufacturerModelSerial identity whose importedDeviceIdentifier equals
/// <paramref name="normalizedIdentifier"/> after normalizing the stored value the same
/// way (both sides pass through <see cref="NormalizeStoredIdentifier"/>, so raw legacy
/// entries match too). Serial-only identities never match: they are an Intune-side
/// misconfiguration for Windows, not authorization.
/// </summary>
private async Task<ScanResult> ScanPagesForIdentifierAsync(
HttpClient graphClient, string url, string normalizedIdentifier, string tenantId, int attempt, int maxPages)
{
for (var page = 0; page < maxPages && !string.IsNullOrEmpty(url); page++)
{
var response = await graphClient.GetAsync(url);
var responseBody = await response.Content.ReadAsStringAsync();
if (!response.IsSuccessStatusCode)
{
_logger.LogWarning(
"Corporate identifier validation Graph query failed for tenant {TenantId} (attempt {Attempt}, page {Page}). Status: {StatusCode}. Body: {ResponseBody}",
tenantId, attempt, page + 1, (int)response.StatusCode, responseBody);
return response.StatusCode switch
{
System.Net.HttpStatusCode.Unauthorized or System.Net.HttpStatusCode.Forbidden
=> new ScanResult(ScanOutcome.PermissionDenied),
// 400 on the first page = the $filter was rejected; mid-pagination a 400
// is unexpected and treated as transient like any other Graph error.
System.Net.HttpStatusCode.BadRequest when page == 0
=> new ScanResult(ScanOutcome.FilterRejected),
_ => new ScanResult(ScanOutcome.Transient, $"Graph query failed with status {(int)response.StatusCode}"),
};
}
var data = JsonConvert.DeserializeObject<JObject>(responseBody);
var identities = data?["value"] as JArray;
if (identities != null)
{
foreach (var identity in identities)
{
var type = identity?["importedDeviceIdentityType"]?.ToString();
var value = identity?["importedDeviceIdentifier"]?.ToString();
if (string.Equals(type, "manufacturerModelSerial", StringComparison.OrdinalIgnoreCase)
&& value != null
&& string.Equals(NormalizeStoredIdentifier(value), normalizedIdentifier, StringComparison.Ordinal))
{
return new ScanResult(ScanOutcome.Found);
}
}
}
url = data?["@odata.nextLink"]?.ToString() ?? string.Empty;
}
if (!string.IsNullOrEmpty(url))
{
// Page cap exceeded with more data pending — treating this as "not found" could
// cache a false negative, so it is transient instead.
_logger.LogWarning(
"Corporate identifier validation exceeded {MaxPages} pages for tenant {TenantId} without exhausting the list.",
maxPages, tenantId);
return new ScanResult(ScanOutcome.Transient, "Corporate identifier list scan exceeded the page budget");
}
return new ScanResult(ScanOutcome.NotFound);
}
/// <summary>
/// Normalizes one identifier component the way the Intune portal does on upload:
/// uppercase, every non-alphanumeric character removed ("Microsoft Corporation" →
/// "MICROSOFTCORPORATION", "7801-5131-…-18" → "78015131…18"). Field-verified 2026-08-17
/// against the portal's stored corporate identifier list.
/// </summary>
internal static string NormalizeComponent(string value)
{
var sb = new StringBuilder(value.Length);
foreach (var c in value)
{
if (char.IsLetterOrDigit(c))
sb.Append(char.ToUpperInvariant(c));
}
return sb.ToString();
}
internal static string BuildNormalizedIdentifier(string manufacturer, string model, string serial) =>
$"{NormalizeComponent(manufacturer)},{NormalizeComponent(model)},{NormalizeComponent(serial)}";
/// <summary>
/// Normalizes a stored importedDeviceIdentifier for comparison: same per-character rule
/// as <see cref="NormalizeComponent"/> but keeps the comma separators. Stored values are
/// already normalized by the portal; running them through again is a no-op there and
/// makes raw-form entries (e.g. created via Graph directly) match as well.
/// </summary>
internal static string NormalizeStoredIdentifier(string value)
{
var sb = new StringBuilder(value.Length);
foreach (var c in value)
{
if (c == ',')
sb.Append(c);
else if (char.IsLetterOrDigit(c))
sb.Append(char.ToUpperInvariant(c));
}
return sb.ToString();
}
private static string BuildCacheKey(string tenantId, string manufacturer, string model, string serialNumber)
{
return $"corporate-id-validation:{tenantId}:{manufacturer}:{model}:{serialNumber}";
@@ -362,8 +362,13 @@ namespace AutopilotMonitor.Functions.Security
}
else
{
deviceValidationError = corpResult.ErrorMessage;
deviceValidationTransient = corpResult.IsTransient;
// Accumulate instead of overwrite: with several validators enabled, only the
// LAST failure used to surface in the 403 Details / rejection log, hiding why
// the earlier (usually more relevant) lookups missed. Transient is sticky-OR:
// if ANY validator failed transiently the device might actually be authorized,
// so the caller must return 503 Retry-After rather than a definitive 403.
deviceValidationError = CombineValidationErrors(deviceValidationError, corpResult.ErrorMessage);
deviceValidationTransient |= corpResult.IsTransient;
}
}
@@ -384,8 +389,8 @@ namespace AutopilotMonitor.Functions.Security
}
else
{
deviceValidationError = cloudPcResult.ErrorMessage;
deviceValidationTransient = cloudPcResult.IsTransient;
deviceValidationError = CombineValidationErrors(deviceValidationError, cloudPcResult.ErrorMessage);
deviceValidationTransient |= cloudPcResult.IsTransient;
}
}
@@ -466,6 +471,18 @@ namespace AutopilotMonitor.Functions.Security
};
}
/// <summary>
/// Joins the per-validator failure messages so the 403 Details / rejection log carries
/// every miss in chain order (Autopilot | CorporateIdentifier | CloudPc), not just the
/// last validator's.
/// </summary>
internal static string? CombineValidationErrors(string? accumulated, string? next)
{
if (string.IsNullOrEmpty(next)) return accumulated;
if (string.IsNullOrEmpty(accumulated)) return next;
return accumulated + " | " + next;
}
/// <summary>
/// Extracts the Intune device id from an MDM client certificate subject. Certs issued by
/// the Microsoft Intune MDM Device CA carry the Intune managedDevice id as the CN
@@ -307,7 +307,16 @@ namespace AutopilotMonitor.Functions.Services
/// Both stages run in the same 2h maintenance pass so no new timers are introduced
/// (preserving Container App scale-to-zero).
/// </summary>
private async Task MarkStalledSessionsAsTimedOutAsync()
/// <summary>
/// Outcome of one stalled-session sweep pass. <see cref="Error"/> is null on success;
/// on a catastrophic failure (e.g. the tenant enumeration itself failed) it carries the
/// exception message and the counters are 0 — per-tenant failures are caught inside the
/// loop and never surface here. Consumed by the hourly <c>SessionSweep</c> timer to emit
/// its Completed/Failed OpsEvent; the 2h maintenance chain ignores the result (fail-soft).
/// </summary>
internal sealed record SessionSweepResult(int StalledMarked, int TimedOut, string? Error = null);
internal async Task<SessionSweepResult> MarkStalledSessionsAsTimedOutAsync()
{
_logger.LogInformation("Checking for stalled sessions...");
var stalledStart = Stopwatch.StartNew();
@@ -543,10 +552,12 @@ namespace AutopilotMonitor.Functions.Services
stalledStart.Stop();
_logger.LogInformation($"Stalled session check completed: {totalSessionsMarkedStalled} marked Stalled, {totalSessionsTimedOut} timed out in {stalledStart.ElapsedMilliseconds}ms");
return new SessionSweepResult(totalSessionsMarkedStalled, totalSessionsTimedOut);
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to check for stalled sessions");
return new SessionSweepResult(0, 0, ex.Message);
}
}
@@ -83,6 +83,22 @@ namespace AutopilotMonitor.Functions.Services
$"Maintenance took {durationMs}ms (> {thresholdMinutes}min soft threshold; host aborts at 60min) — triggered by {triggeredBy}",
null, triggeredBy, new { durationMs, thresholdMinutes });
/// <summary>
/// Heartbeat of the hourly <c>SessionSweep</c> timer (the :30 interleave of the stalled-session
/// sweep — the 2h Maintenance chain still runs the same sweep at minute 0). Emitted every tick,
/// counts included, so a dead timer is detectable and dashboards can separate the interleave
/// from full maintenance runs.
/// </summary>
public Task RecordSessionSweepCompletedAsync(int stalledMarked, int timedOut, int durationMs)
=> WriteAsync(OpsEventCategory.Maintenance, "SessionSweepCompleted", OpsEventSeverity.Info,
$"Hourly session sweep completed in {durationMs}ms — {stalledMarked} marked Stalled, {timedOut} terminalized",
null, "System.SessionSweep", new { stalledMarked, timedOut, durationMs });
public Task RecordSessionSweepFailedAsync(string error)
=> WriteAsync(OpsEventCategory.Maintenance, "SessionSweepFailed", OpsEventSeverity.Error,
$"Hourly session sweep failed: {error}",
null, "System.SessionSweep", new { error });
public Task RecordOpsEventCleanupAsync(int deletedCount, int retentionDays)
=> WriteAsync(OpsEventCategory.Maintenance, "OpsEventCleanup", OpsEventSeverity.Info,
$"Cleaned up {deletedCount} ops events older than {retentionDays} days",
@@ -25,6 +25,12 @@ const OPS_EVENT_TYPES: Record<string, string[]> = {
"MaintenanceLongRunning",
"OpsEventCleanup",
"SessionTimeouts",
// Hourly stalled-session sweep interleave (SessionSweepFunction, minute 30) — dual-register
// per memory feedback_ops_event_types_dual_register. Backend helpers
// RecordSessionSweep{Completed,Failed}Async. Completed is an hourly heartbeat (Info);
// Failed fires only when the sweep's tenant enumeration itself blows up (Error).
"SessionSweepCompleted",
"SessionSweepFailed",
// Cascade-delete maintenance (PR6) — dual-register per memory feedback_ops_event_types_dual_register.
// Dispatched by SessionDeletionMaintenanceFunction; backend helpers in OpsEventService.cs are
// RecordSessionDeletionMaintenance{LongRunning,LongRunningSevere,Failed,Completed,FanoutSkipped}Async +