diff --git a/docs/backend/silent-session-classification.md b/docs/backend/silent-session-classification.md index b2308bac..e4e35fd3 100644 --- a/docs/backend/silent-session-classification.md +++ b/docs/backend/silent-session-classification.md @@ -10,7 +10,7 @@ tags: - classification - whiteglove - self-deploying -timestamp: 2026-08-23T18:00:00+02:00 +timestamp: 2026-08-27T20:00:00+02:00 --- # Silent-Session Classification @@ -30,7 +30,10 @@ so a session gets the same verdict regardless of which path noticed the silence subcategory counts (incl. the 30s fallback-confirmed messages), `HasExplicitFailure`, `HasEnrollmentComplete` vs. `HasTerminalComplete` (the Part-1 `whiteglove_complete`), `HasAgentEmergencyBreak`, `DesktopArrived`, `HelloResolved`, `HelloPolicyDisabled`, -`SkipUserEsp`, RealmJoin detection/resolution. Session-row flags are passed alongside: +`SkipUserEsp`, RealmJoin detection/resolution, `HasAppInstallFailure` (any +`app_install_failed`), and `HasAgentMaxLifetimeTimeout` (either max-lifetime shape: +`enrollment_failed`/`agent_timeout` or `agent_shutting_down`/`max_lifetime` — the agent is +provably gone). Session-row flags are passed alongside: `IsPreProvisioned` + `ResumedAt` (WhiteGlove Part 2) and `IsSelfDeployingProfile` (registry-confirmed `CloudAssignedOobeConfig 0x20|0x40`, set at registration). @@ -44,7 +47,9 @@ subcategory counts (incl. the 30s fallback-confirmed messages), `HasExplicitFail | 2 | Account Setup all-succeeded, or `enrollment_complete`, or `whiteglove_complete` on a **non-resumed** session | Succeeded | | 3 | `HasAgentEmergencyBreak` | Incomplete immediately (agent self-destructed, nothing can arrive) | | 4 | `DesktopArrived` + (Hello terminal, or Hello disabled + SkipUserEsp) | Succeeded (user provably finished) | -| 5 | Device Setup done | AwaitingUser within grace, Incomplete past grace | +| 5 | Device Setup done + within grace + **no** max-lifetime watchdog observed | AwaitingUser | +| 5a | Device Setup done + past grace (or watchdog observed) + `DesktopArrived` + no `app_install_failed` | **Succeeded** — completed (assumed), `r5_assumed` | +| 5b | Device Setup done, otherwise past grace (or watchdog observed) | Incomplete | | 6 | otherwise | Incomplete | Grace = `ResolveGraceHours(SessionGraceHours, AbsoluteMaxSessionHours)` (~51h default), @@ -65,7 +70,12 @@ budget on a Wi-Fi link still associating), and rule 5 labeled finished kiosks Incomplete turned a working rollout into a red wall. Other self-deploying tenants completed normally in the same window, so this is a classification gap, not a profile-wide agent bug. Agent-side companion fix: `SessionRegistrationHelper` now waits up to 15s for a network link -and retries 6× (~62s backoff) instead of 5× (~30s). +and retries 6× (~62s backoff) instead of 5× (~30s). Measured 2026-08-27 (builds +2.0.1428/1429, both carrying the fix): the reconcile share did not move, and the backend saw +no registration attempt from any relaunch — no success, no 4xx, zero `register:superseded` +in the tenant — so the relaunch never reaches the network at all (device powered off after +ESP, or the relaunch never runs). The sweep reconcile is the permanent completion path for +this profile, not a stopgap. ## Callers (all pass both gate inputs) @@ -78,13 +88,33 @@ and retries 6× (~62s backoff) instead of 5× (~30s). | Agent stall probe (`session_stalled`) | `EventIngestProcessor.TryMapStallToWhiteGloveAwaitingUserAsync` | Only the WG gate (1b); a stall probe on a self-deploying device with Device Setup done means the agent is alive and something else blocks — stays Stalled. | | Admin retro-reconcile | `LegacyReclassificationService` | Modes `legacy_timeouts`, `pending_orphans`, `self_deploying_silent` (Incomplete/AwaitingUser/Stalled rows with the profile flag, heal-only to Succeeded, admin-marked rows skipped). `POST /api/maintenance/reclassify-legacy?mode=…&dryRun=false&tenantId=…&maxSessions=…`, dry-run by default. | -## Known gap (decided, not built) +## Why rule 5a exists (calibration read 2026-08-27) + +The verdict-calibration matrix showed r5-Incomplete devices re-enrolling at 2.6 % within +7 days — the succeeded background (`agent:complete_soft` sits at 2.8 %), not the ~29 % +failure level — consistently across tenants, and half the sampled sessions were healthy +desktops with a clean app record whose agent died before the Hello terminal (some with the +agent's own "AccountSetup treated as complete" synthesis as the literally last event). Past +the full grace, "the user phase is still running" is no longer a possible explanation, so +desktop + zero `app_install_failed` + no explicit failure now reconciles to Succeeded +("completed (assumed)") instead of a red Incomplete. Sessions without a desktop (user never +began sign-in: battery-drained, powered off, max-lifetime) and sessions with any failed app +stay Incomplete — "completed" must not overclaim. Rule 4 remains the IMMEDIATE reconcile +(desktop + Hello proof); 5a only fires where waiting is over. Desktop arrival alone is still +rejected as a live completion signal. + +## Closed gap: max-lifetime AwaitingUser (2026-08-27) After max-lifetime the agent writes `enrollment-complete.marker` and self-destructs, so an -AwaitingUser verdict from that path can never heal — it is a 51h countdown to Incomplete. -The emergency-break path (rule 3) already terminalizes immediately; max-lifetime still goes -through the grace window by design ("user simply absent"). Revisit if customers read -AwaitingUser as "still possible". +AwaitingUser verdict from that path could never heal — the calibration correction stream +confirmed it: all five observed `maxlife:r5_awaiting` episodes expired to Incomplete +unhealed, while live-agent `sweep:r5_awaiting` parks upgraded 7 of 26 to Succeeded. Rule 5 +therefore skips the grace once `HasAgentMaxLifetimeTimeout` is in the stream and decides the +terminal fork (5a/5b) immediately; the max-lifetime ingest appends its trigger event to the +read stream so the fact is never missed, and late straggler telemetry still heals a terminal +verdict via `TryLateTelemetryReconcileAsync`. WhiteGlove rule 1b deliberately keeps its +park — sealed/boxed devices do heal (the observed `maxlife:r1b_awaiting` episode resolved to +Succeeded). # Examples diff --git a/docs/backend/verdict-calibration.md b/docs/backend/verdict-calibration.md index 9355c822..788af584 100644 --- a/docs/backend/verdict-calibration.md +++ b/docs/backend/verdict-calibration.md @@ -9,7 +9,7 @@ tags: - classifier - calibration - regression-radar -timestamp: 2026-08-23T18:00:00+02:00 +timestamp: 2026-08-27T20:00:00+02:00 --- # Verdict Calibration (classifier thermometer) @@ -114,15 +114,18 @@ and is wiped with the tenant on offboarding. ## Drift radar — `VerdictCalibrationRadar` Same statistics as the [rule regression radar](rule-regression-radar.md) (7-day window vs the -prior 28 days, ≥ 5 window hits, ≥ 20 window sessions, lift ≥ 2, `MetricsMath. -RateIncreaseSeparated`), anchored on yesterday, over every partition with rows (each tenant -and `global`): +prior 28 days, ≥ 20 window sessions, lift ≥ 2, `MetricsMath.RateIncreaseSeparated`), with +stricter floors since the first-month tuning (2026-08-27): **≥ 10 window hits** (verdict +shares on single-digit counts are noise — a 3-hit episode survived as such), and a per-path +share regression additionally needs **≥ 5 baseline hits** — a path with no established +baseline is new (or renamed) vocabulary, not a regression. Anchored on yesterday, over every +partition with rows (each tenant and `global`): | Kind | Signal | Re-arm | | --- | --- | --- | -| `share_regression` | one **backend-decided** path's share of all sessions doubled (`sweep`/`maxlife`/`late`/`retro`/`rule`/`manual`/`ingest`; `agent:*` and `register:*` mirror customer workflow mix — first prod pass fired on `agent:whiteglove_pending` rollout weeks — and `legacy:*` is derived, all excluded) | share < 1.5× baseline, the path stops occurring, or the path is no longer eligible | -| `silence_share_regression` | the `sweep:*` + `maxlife:*` share doubled — the backend had to decide more often because the agent went silent (a liveness signal, not a classifier one) | same | -| `evidence_gap` | absolute: `r6` (pure fallthrough) decides ≥ 20 % of the window's classifier verdicts, ≥ 20 verdicts | < 15 % or < 20 verdicts | +| `share_regression` | one **backend-decided** path's share of all sessions doubled (`sweep`/`maxlife`/`late`/`retro`/`rule`/`manual`/`ingest`; `agent:*` and `register:*` mirror customer workflow mix — first prod pass fired on `agent:whiteglove_pending` rollout weeks — and `legacy:*` is derived, all excluded per-path) | share < 1.5× baseline, the path stops occurring, the path is no longer eligible, or the current numbers fall below the fire floors | +| `silence_share_regression` | the `sweep:*` + `maxlife:*` share doubled — the backend had to decide more often because the agent went silent (a liveness signal, not a classifier one). Rule-shaped `legacy:*` paths count into BOTH sums (window and baseline): they were the same backend decisions before the 2026-08-23 instrumentation, and without that continuity the group had a near-zero baseline by construction and fired a lift-124 rollout artifact | share < 1.5× baseline or hits stop | +| `evidence_gap` | absolute: `r6` (pure fallthrough) decides ≥ 20 % of the window's classifier verdicts (stamped rule paths plus rule-shaped `legacy:*`, same continuity), ≥ 20 verdicts | < 15 % or < 20 verdicts | Episodes are tracker rows `verdictcalibration|{kind}|{path}|{status}` (register = `AddEntity`, 409 = already burning, fail-closed; refresh carries `FirstNotifiedAt`; 30-day tracker @@ -140,6 +143,15 @@ Kill switch `VerdictCalibrationRadarDisabled`. re-enrollment within 7 days after Succeeded 8.9 % (lab background), after Failed 24.3 %, after Incomplete 10.6 % — Incomplete behaves like Succeeded, so rules 5/6 are probably too cautious; the matrix now shows this per rule instead of per status. +* **First real calibration read (2026-08-27, four days instrumented):** the per-rule and + per-tenant split resolved that hand analysis — `legacy:r5_incomplete` re-enrolled at 2.6 % + (the `agent:complete_soft` background), spread across tenants, while `legacy:r6`'s 29 % + was concentrated in one tenant that disciplinedly redoes failed enrollments (48.5 % there, + 6.7 % elsewhere — so r6 stays untouched, it correctly flags that tenant). Consequences + shipped the same day: rule 5a "completed (assumed)" (see + [Silent-Session Classification](silent-session-classification.md)), the max-lifetime grace + skip, and the radar floors + legacy group continuity above — the first production month + had produced nine standing alerts, every one an instrumentation-rollout artifact. * **Correction stream:** a session swept to `sweep:r5_incomplete`, later completed by the agent → row `agent:complete` with `PriorVerdictPath = sweep:r5_incomplete`; the aggregate increments `OverriddenByLateCompletion` on the `sweep:r5_incomplete / Incomplete` bucket. diff --git a/docs/log.md b/docs/log.md index 6f08dce0..d092b427 100644 --- a/docs/log.md +++ b/docs/log.md @@ -2,6 +2,8 @@ ## 2026-08-27 +* **Update**: `backend/silent-session-classification.md`, `backend/verdict-calibration.md` - the first real calibration read tuned what it measures. Rule 5 grew a past-grace fork 5a: Device Setup done + real-user desktop + zero `app_install_failed` reconciles to Succeeded "completed (assumed)" (`r5_assumed`) - r5-Incomplete devices re-enrolled at 2.6 %, the succeeded background, and the sampled desktop half had clean app records; no desktop or any failed app stays Incomplete. The max-lifetime watchdog (either event shape) now skips the AwaitingUser grace entirely - all five observed maxlife parks expired unhealed, and late telemetry heals a terminal verdict anyway - while WhiteGlove rule 1b keeps its park (its one episode healed). The drift radar got floors (≥ 10 window hits; per-path share regression needs ≥ 5 baseline hits, so new/renamed vocabulary cannot "regress") and legacy continuity: rule-shaped `legacy:*` paths count into the silence and classifier-verdict group sums on both sides of the 2026-08-23 cutover, which dissolves the lift-124 silence artifact and lets sub-floor episodes clear themselves. Backend only; the new path shows up in the matrix on its own. + * **Update**: `backend/tenant-activation-welcome-mail.md` - the reverse lookup. A delivery log names only the recipient, so an address had to lead back to its tenant: the tenant console's search now matches the notification and contact address as well as domain and tenant id, with the quoted exact mode unchanged. The addresses sit in `PreviewWhitelist`, not on the tenant config, so a new GA/Reader-only `GET /api/preview/notification-emails` returns them as one map (single cross-partition scan of `RowKey eq 'notification-email'`) instead of a point read per tenant; both sides join on the lowercased tenant id because the address rows carry the JWT `tid` while a config PartitionKey's casing is not guaranteed, and matching is case-insensitive because stored addresses keep the casing the user typed. The map is best-effort in the portal - a failed read costs only the search-by-address, never the tenant list - and the addresses stay out of the list rows (search key, not a column). Match rules moved into `tenantSearch.ts` with unit pins. Backend before web. diff --git a/src/Backend/AutopilotMonitor.Functions.Tests/EnrollmentTimeoutClassifierTests.cs b/src/Backend/AutopilotMonitor.Functions.Tests/EnrollmentTimeoutClassifierTests.cs index 5cfabc98..6c0d5f04 100644 --- a/src/Backend/AutopilotMonitor.Functions.Tests/EnrollmentTimeoutClassifierTests.cs +++ b/src/Backend/AutopilotMonitor.Functions.Tests/EnrollmentTimeoutClassifierTests.cs @@ -394,14 +394,91 @@ public class EnrollmentTimeoutClassifierTests } [Fact] - public void Classify_agent_timeout_failed_session_classifies_honestly() + public void Classify_agent_timeout_terminalizes_immediately_instead_of_parking() { - // WG Part-2 shape (session 1506ce9f): DeviceSetup 4/4, no user phase, watchdog fired at - // 6h — honest verdict is AwaitingUser (within grace), never Failed. + // The watchdog firing proves the agent is gone for good — an AwaitingUser park from this + // state is a countdown, not a wait: all 5 observed maxlife:r5_awaiting episodes expired + // unhealed (calibration read 2026-08-27). Still never Failed, and late straggler + // telemetry can heal the Incomplete (TryLateTelemetryReconcileAsync). var timeout = Evt("enrollment_failed", data: new Dictionary { ["failureType"] = "agent_timeout" }); - var (status, _) = Classify(new[] { Esp(DeviceSetup44), timeout }, hoursSinceStart: 6); - Assert.Equal(SessionStatus.AwaitingUser, status); + var (status, reason) = Classify(new[] { Esp(DeviceSetup44), timeout }, hoursSinceStart: 6); + Assert.Equal(SessionStatus.Incomplete, status); + Assert.Contains("max-lifetime watchdog", reason); + + // The V2 shutdown shape carries the same fact. + var shutdown = Evt("agent_shutting_down", + data: new Dictionary { ["reason"] = "max_lifetime" }); + var (viaShutdown, _) = Classify(new[] { Esp(DeviceSetup44), shutdown }, hoursSinceStart: 6); + Assert.Equal(SessionStatus.Incomplete, viaShutdown); + + // WhiteGlove Part 2 (rule 1b) is deliberately untouched: a sealed/powered-off device + // between technician and end user stays AwaitingUser — that park DOES heal + // (the observed maxlife:r1b_awaiting episode resolved to Succeeded). + var rollup = EnrollmentTimeoutClassifier.ExtractRollup(new[] { Esp(DeviceSetup44), timeout }); + var (wgStatus, _, wgRule) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession( + rollup, Start, Start.AddHours(6), 72, isPreProvisioned: true, resumedAt: Start.AddHours(1)); + Assert.Equal(SessionStatus.AwaitingUser, wgStatus); + Assert.Equal(ClassifierRules.R1bWhiteGloveAwaiting, wgRule); + } + + // -------- Rule 5a: completed (assumed) — calibration read 2026-08-27 -------- + + [Fact] + public void ExtractRollup_detects_app_failures_and_maxlife_timeout() + { + Assert.True(EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("app_install_failed") }).HasAppInstallFailure); + var timeout = Evt("enrollment_failed", data: new Dictionary { ["failureType"] = "agent_timeout" }); + Assert.True(EnrollmentTimeoutClassifier.ExtractRollup(new[] { timeout }).HasAgentMaxLifetimeTimeout); + var shutdown = Evt("agent_shutting_down", data: new Dictionary { ["reason"] = "max_lifetime" }); + Assert.True(EnrollmentTimeoutClassifier.ExtractRollup(new[] { shutdown }).HasAgentMaxLifetimeTimeout); + // Other shutdown reasons say nothing about max lifetime. + var ctrlC = Evt("agent_shutting_down", data: new Dictionary { ["reason"] = "ctrl_c" }); + Assert.False(EnrollmentTimeoutClassifier.ExtractRollup(new[] { ctrlC }).HasAgentMaxLifetimeTimeout); + // A real failure event does not raise the app flag. + Assert.False(EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("enrollment_failed") }).HasAppInstallFailure); + } + + [Fact] + public void Classify_desktop_without_hello_past_grace_is_assumed_complete() + { + // The healthy half of the r5 population (drill-down 2026-08-27): desktop reached, apps + // clean, agent died before the Hello terminal. Past the full grace "user phase still + // running" is no longer a possible explanation — completed (assumed), not a red Incomplete. + var (status, reason, rule) = ClassifyWithRule(new[] + { + Esp(DeviceSetup44), Esp(AccountSetup15), Evt("desktop_arrived"), + }, hoursSinceStart: 80, grace: 72); + Assert.Equal(SessionStatus.Succeeded, status); + Assert.Equal(ClassifierRules.R5DesktopAssumed, rule); + Assert.Contains("treating the enrollment as completed", reason); + Assert.Contains("Agent last reported", reason); // silence-transparency clause + } + + [Fact] + public void Classify_desktop_with_app_failure_past_grace_stays_Incomplete() + { + // "Completed" must not overclaim a session whose ESP payload provably failed + // (drill-down session b1edc6f8: desktop reached but .NET 3.5 install failed). + var (status, _) = Classify(new[] + { + Esp(DeviceSetup44), Evt("desktop_arrived"), Evt("app_install_failed"), + }, hoursSinceStart: 80, grace: 72); + Assert.Equal(SessionStatus.Incomplete, status); + } + + [Fact] + public void Classify_maxlife_with_desktop_and_clean_apps_is_assumed_complete_within_grace() + { + // Agent provably gone + desktop + clean apps: no reason to park — the assumption is as + // good now as it would be after the 51h grace. + var timeout = Evt("enrollment_failed", data: new Dictionary { ["failureType"] = "agent_timeout" }); + var (status, _, rule) = ClassifyWithRule(new[] + { + Esp(DeviceSetup44), Evt("desktop_arrived"), timeout, + }, hoursSinceStart: 6); + Assert.Equal(SessionStatus.Succeeded, status); + Assert.Equal(ClassifierRules.R5DesktopAssumed, rule); } [Fact] diff --git a/src/Backend/AutopilotMonitor.Functions.Tests/VerdictCalibrationRadarTests.cs b/src/Backend/AutopilotMonitor.Functions.Tests/VerdictCalibrationRadarTests.cs index f50e2e63..df5ecbf8 100644 --- a/src/Backend/AutopilotMonitor.Functions.Tests/VerdictCalibrationRadarTests.cs +++ b/src/Backend/AutopilotMonitor.Functions.Tests/VerdictCalibrationRadarTests.cs @@ -89,10 +89,79 @@ public class VerdictCalibrationRadarTests } [Fact] - public void Derived_legacy_paths_never_alert() + public void Derived_legacy_paths_never_alert_per_path_but_feed_the_group_sums() { var rows = Horizon(w => new[] { ("legacy:r6", "Incomplete", w ? 10 : 1), ("agent:complete", "Succeeded", w ? 10 : 19) }); - Assert.DoesNotContain(VerdictCalibrationRadar.Evaluate(rows, Target), f => f.Kind == VerdictCalibrationAlertKinds.ShareRegression); + var findings = VerdictCalibrationRadar.Evaluate(rows, Target); + Assert.DoesNotContain(findings, f => f.Kind == VerdictCalibrationAlertKinds.ShareRegression); + // Legacy continuity: the rule-shaped legacy subset counts into the silence group, so a + // real silence jump stays visible even when the history predates the instrumentation. + Assert.Contains(findings, f => f.Kind == VerdictCalibrationAlertKinds.SilenceShareRegression); + } + + [Fact] + public void Share_regression_requires_an_established_baseline() + { + // The 2026-08-23 instrumentation-cutover shape: the same decisions exist only under the + // legacy name in the baseline and only under the new name in the window. Zero baseline + // hits is new vocabulary, not a regression — and the legacy continuity keeps the silence + // group flat across the rename (production fired a lift-124 artifact without both). + var rows = Horizon(w => new[] + { + ("sweep:r5_incomplete", "Incomplete", w ? 4 : 0), + ("legacy:r5_incomplete", "Incomplete", w ? 0 : 4), + ("agent:complete", "Succeeded", 16), + }); + var findings = VerdictCalibrationRadar.Evaluate(rows, Target); + Assert.DoesNotContain(findings, f => f.Kind == VerdictCalibrationAlertKinds.ShareRegression); + Assert.DoesNotContain(findings, f => f.Kind == VerdictCalibrationAlertKinds.SilenceShareRegression); + } + + [Fact] + public void Share_regression_needs_ten_window_hits() + { + // 7 window hits at 5% vs an established 1.07% baseline — lift ~4.7, but verdict shares + // on single-digit counts are noise (a 3-hit episode survived as such, read 2026-08-27). + var rows = new List(); + for (var i = 34; i >= 0; i--) + { + var inWindow = i < 7; + rows.Add(new VerdictCalibrationDailyAggregate + { + TenantId = "t", Date = Target.AddDays(-i).ToString("yyyy-MM-dd"), SessionCount = 20, TerminalSessionCount = 20, + Buckets = + { + new VerdictCalibrationBucket { VerdictPath = "sweep:r6", Status = "Incomplete", Count = inWindow ? 1 : (i == 20 ? 6 : 0) }, + new VerdictCalibrationBucket { VerdictPath = "agent:complete", Status = "Succeeded", Count = inWindow ? 19 : 20 }, + }, + }); + } + Assert.DoesNotContain(VerdictCalibrationRadar.Evaluate(rows, Target), + f => f.Kind == VerdictCalibrationAlertKinds.ShareRegression); + } + + [Fact] + public void ReArm_clears_episodes_below_the_fire_floors() + { + // A zero-baseline episode from before the floors existed (rename artifact) clears itself + // even while its share is elevated… + var artifact = new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.ShareRegression, VerdictPath = "sweep:r5_incomplete", Status = "Incomplete" }; + var renamed = Horizon(w => new[] + { + ("sweep:r5_incomplete", "Incomplete", w ? 4 : 0), + ("legacy:r5_incomplete", "Incomplete", w ? 0 : 4), + ("agent:complete", "Succeeded", 16), + }); + Assert.True(VerdictCalibrationRadar.ShouldReArm(artifact, renamed, Target)); + + // …and so does one whose window has shrunk under the 10-hit floor. + var shrunk = new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.ShareRegression, VerdictPath = "maxlife:r5_awaiting", Status = "AwaitingUser" }; + var tiny = Horizon(w => new[] + { + ("maxlife:r5_awaiting", "AwaitingUser", 1), + ("agent:complete", "Succeeded", 19), + }); + Assert.True(VerdictCalibrationRadar.ShouldReArm(shrunk, tiny, Target)); } [Fact] diff --git a/src/Backend/AutopilotMonitor.Functions.Tests/VerdictPathTests.cs b/src/Backend/AutopilotMonitor.Functions.Tests/VerdictPathTests.cs index 15948912..100a83ca 100644 --- a/src/Backend/AutopilotMonitor.Functions.Tests/VerdictPathTests.cs +++ b/src/Backend/AutopilotMonitor.Functions.Tests/VerdictPathTests.cs @@ -60,6 +60,7 @@ public class VerdictPathTests Assert.Equal("sweep:r5_incomplete", path); Assert.Equal("sweep", VerdictPaths.Origin(path)); Assert.True(VerdictPaths.IsClassifierPath(path)); + Assert.True(VerdictPaths.IsClassifierPath("sweep:r5_assumed")); Assert.True(VerdictPaths.IsClassifierPath("maxlife:r1")); Assert.True(VerdictPaths.IsClassifierPath("late:r4")); Assert.True(VerdictPaths.IsClassifierPath("retro:r6")); @@ -107,6 +108,7 @@ public class VerdictPathTests Assert.Equal(ClassifierRules.R3EmergencyBreak, RuleOf(new[] { Esp(DeviceSetup44), Evt("agent_emergency_break") })); Assert.Equal(ClassifierRules.R4DesktopHello, RuleOf(new[] { Esp(DeviceSetup44), Evt("desktop_arrived"), Evt("hello_provisioning_completed") })); Assert.Equal(ClassifierRules.R5DeviceSetupAwaiting, RuleOf(new[] { Esp(DeviceSetup44), Esp(AccountSetup15) }, hours: 6, grace: 72)); + Assert.Equal(ClassifierRules.R5DesktopAssumed, RuleOf(new[] { Esp(DeviceSetup44), Esp(AccountSetup15), Evt("desktop_arrived") }, hours: 80, grace: 72)); Assert.Equal(ClassifierRules.R5DeviceSetupIncomplete, RuleOf(new[] { Esp(DeviceSetup44), Esp(AccountSetup15) }, hours: 80, grace: 72)); Assert.Equal(ClassifierRules.R6Fallthrough, RuleOf(Array.Empty())); } diff --git a/src/Backend/AutopilotMonitor.Functions/Helpers/VerdictCalibrationRadar.cs b/src/Backend/AutopilotMonitor.Functions/Helpers/VerdictCalibrationRadar.cs index 390d4009..4d67e9ba 100644 --- a/src/Backend/AutopilotMonitor.Functions/Helpers/VerdictCalibrationRadar.cs +++ b/src/Backend/AutopilotMonitor.Functions/Helpers/VerdictCalibrationRadar.cs @@ -38,8 +38,10 @@ public class VerdictCalibrationFinding /// evidence_gap — absolute gate: the pure fallthrough rule (r6) decides ≥20 % of the /// window's classifier verdicts. The evidence the other rules need is not arriving. /// -/// Derived (legacy:*) paths are excluded from all three — pre-instrumentation attribution -/// is too weak to alert on. Agent-declared paths (agent:*, register:*) are excluded +/// Derived (legacy:*) paths never alert per-path — pre-instrumentation attribution is too +/// weak for that — but their rule-shaped subset DOES feed the silence and classifier-verdict +/// group sums so the groups stay continuous across the 2026-08-23 instrumentation cutover +/// (see ). Agent-declared paths (agent:*, register:*) are excluded /// from the per-path share regression: their share tracks the customer's workflow mix (a /// WhiteGlove rollout week doubles agent:whiteglove_pending; a Continue-Anyway tenant /// lives on agent:complete_soft) — real signals, but not about the BACKEND classifier @@ -51,7 +53,21 @@ public static class VerdictCalibrationRadar { public const int WindowDays = RuleRegressionRadar.WindowDays; public const int BaselineDays = RuleRegressionRadar.BaselineDays; - public const int MinWindowHits = RuleRegressionRadar.MinWindowHitSessions; + /// + /// Minimum window hits before a share finding fires or keeps its episode. Deliberately + /// higher than the rule radar's 5: verdict shares swing on tiny counts, and the first + /// production month kept a 3-hit maxlife:r5_awaiting episode alive as pure noise + /// (calibration read 2026-08-27). + /// + public const int MinWindowHits = 10; + /// + /// Minimum baseline hits before a PER-PATH share regression is meaningful. A path with no + /// established baseline cannot regress — it is new vocabulary (or a renamed path), and + /// alerting on it just counts the rollout: the 2026-08-23 instrumentation cut over from + /// legacy:* to sweep:*/register:* names and fired seven zero-baseline artifacts. The group + /// kinds are exempt — their baselines carry the legacy continuity (see ). + /// + public const int MinBaselineHits = 5; public const int MinWindowSessions = RuleRegressionRadar.MinWindowSessions; public const double MinLift = RuleRegressionRadar.MinLift; public const double ReArmLiftFactor = RuleRegressionRadar.ReArmLiftFactor; @@ -135,7 +151,7 @@ public static class VerdictCalibrationRadar .ToList(); } - /// True when an ACTIVE alert may re-arm given the current horizon sums (or when its path is no longer eligible — episodes from an earlier, broader gate clear themselves). + /// True when an ACTIVE alert may re-arm given the current horizon sums (or when its path/numbers no longer pass the fire gates — episodes from an earlier, broader gate clear themselves). public static bool ShouldReArm(VerdictCalibrationAlert alert, IReadOnlyList tenantRows, DateTime targetDateUtc) { if (alert.Kind == VerdictCalibrationAlertKinds.ShareRegression && !IsShareRegressionEligible(alert.VerdictPath)) @@ -153,6 +169,9 @@ public static class VerdictCalibrationRadar default: horizon.Paths.TryGetValue((alert.VerdictPath, alert.Status), out var sums); + // Below the fire floors the episode is noise-retention, not an elevated share — + // clear it (it re-fires only by passing the full gates again). + if (sums.WindowHits < MinWindowHits || sums.BaselineHits < MinBaselineHits) return true; return ReArmByRate(sums.WindowHits, horizon.WindowSessions, sums.BaselineHits, horizon.BaselineSessions); } } @@ -189,6 +208,8 @@ public static class VerdictCalibrationRadar if (windowHits < MinWindowHits) return null; if (windowSessions < MinWindowSessions) return null; if (baselineSessions <= 0) return null; + // Per-path only: no established baseline → no regression claim (new/renamed vocabulary). + if (kind == VerdictCalibrationAlertKinds.ShareRegression && baselineHits < MinBaselineHits) return null; var windowRate = (double)windowHits / windowSessions; var baselineRate = (double)baselineHits / baselineSessions; @@ -212,6 +233,14 @@ public static class VerdictCalibrationRadar }; } + /// Derived classifier-rule path (legacy:r5_incomplete) — same detail shape as , legacy origin. + private static bool IsLegacyClassifierPath(string path, string origin) + { + if (origin != LegacyOrigin) return false; + var detail = path.Length > origin.Length + 1 ? path.Substring(origin.Length + 1) : string.Empty; + return detail.Length >= 2 && detail[0] == 'r' && char.IsDigit(detail[1]); + } + internal readonly record struct PathSums(int WindowHits, int BaselineHits); internal sealed class Horizon @@ -234,6 +263,16 @@ public static class VerdictCalibrationRadar /// question is "how often did the backend have to decide", not which rule did). Classifier /// verdicts = stamped rule paths (); r6 = their /// fallthrough subset. + /// Legacy continuity: derived rule-shaped paths (legacy:r5_incomplete, legacy:r6, …) + /// were the SAME backend decisions before the 2026-08-23 instrumentation — the derivation reads + /// the writers' own reason literals, which is strong enough for a group sum (though still too + /// weak for per-path alerting). Counting them symmetrically (window AND baseline) keeps the + /// silence and classifier-verdict groups continuous across the cutover; without this the groups + /// had a near-zero baseline by construction and fired a lift-124 artifact. Legacy rows only + /// exist for pre-instrumentation sessions, so the term self-deprecates as they age out. + /// (Non-rule legacy paths — legacy:superseded, legacy:wg_awaiting, + /// legacy:unknown — stay excluded: superseded is a registration write, unknown is + /// unattributable, and wg_awaiting is a negligible tail.) /// internal static Horizon Summarize(IReadOnlyList rows, DateTime targetDateUtc) { @@ -262,11 +301,12 @@ public static class VerdictCalibrationRadar : new PathSums(sums.WindowHits, sums.BaselineHits + b.Count); var origin = VerdictPaths.Origin(b.VerdictPath); - if (origin == VerdictPaths.OriginSweep || origin == VerdictPaths.OriginMaxLifetime) + var legacyRule = IsLegacyClassifierPath(b.VerdictPath, origin); + if (origin == VerdictPaths.OriginSweep || origin == VerdictPaths.OriginMaxLifetime || legacyRule) { if (inWindow) h.SilenceWindowHits += b.Count; else h.SilenceBaselineHits += b.Count; } - if (VerdictPaths.IsClassifierPath(b.VerdictPath)) + if (VerdictPaths.IsClassifierPath(b.VerdictPath) || legacyRule) { if (inWindow) h.ClassifierWindowVerdicts += b.Count; else h.ClassifierBaselineVerdicts += b.Count; if (b.VerdictPath.EndsWith(":" + ClassifierRules.R6Fallthrough, StringComparison.Ordinal)) diff --git a/src/Backend/AutopilotMonitor.Functions/Services/EnrollmentTimeoutClassifier.cs b/src/Backend/AutopilotMonitor.Functions/Services/EnrollmentTimeoutClassifier.cs index 0470ae89..fceb3357 100644 --- a/src/Backend/AutopilotMonitor.Functions/Services/EnrollmentTimeoutClassifier.cs +++ b/src/Backend/AutopilotMonitor.Functions/Services/EnrollmentTimeoutClassifier.cs @@ -99,7 +99,9 @@ namespace AutopilotMonitor.Functions.Services bool HelloPolicyDisabled, bool SkipUserEsp, bool RealmJoinDetected, - bool RealmJoinResolved); + bool RealmJoinResolved, + bool HasAppInstallFailure, + bool HasAgentMaxLifetimeTimeout); /// /// Walk a session's events and distill the ESP rollup. Tolerant of missing/empty input @@ -115,6 +117,7 @@ namespace AutopilotMonitor.Functions.Services int acctBestN = 0, acctBestM = 0; bool acctFallbackAll = false; bool hasFailure = false, hasComplete = false, hasEnrollmentComplete = false, hasEmergencyBreak = false; + bool hasAppInstallFailure = false, hasMaxLifetimeTimeout = false; bool desktopArrived = false, helloResolved = false; bool realmJoinDetected = false, realmJoinResolved = false; bool sawHelloPolicyEnabled = false, sawHelloPolicyDisabled = false; @@ -134,7 +137,19 @@ namespace AutopilotMonitor.Functions.Services // (misclassification audit 2026-07-16, tenant a53e67ec cluster). if (!Eq(TryGetDataString(evt, "failureType") ?? string.Empty, "agent_timeout")) hasFailure = true; + else + hasMaxLifetimeTimeout = true; } + // The V2 watchdog shape of the same fact: the agent shut itself down at its + // max-lifetime cap. Either shape proves the agent is gone for good — rule 5 uses + // that to skip the AwaitingUser grace (a wait nothing can ever end). + else if (Eq(type, "agent_shutting_down")) + { + if (Eq(TryGetDataString(evt, "reason") ?? string.Empty, "max_lifetime")) + hasMaxLifetimeTimeout = true; + } + else if (Eq(type, "app_install_failed")) + hasAppInstallFailure = true; else if (Eq(type, "enrollment_complete") || Eq(type, "whiteglove_complete")) { hasComplete = true; @@ -230,7 +245,9 @@ namespace AutopilotMonitor.Functions.Services HelloPolicyDisabled: sawHelloPolicyDisabled && !sawHelloPolicyEnabled, SkipUserEsp: sawSkipUserTrue && !sawSkipUserFalse, RealmJoinDetected: realmJoinDetected, - RealmJoinResolved: realmJoinResolved); + RealmJoinResolved: realmJoinResolved, + HasAppInstallFailure: hasAppInstallFailure, + HasAgentMaxLifetimeTimeout: hasMaxLifetimeTimeout); } /// @@ -418,7 +435,13 @@ namespace AutopilotMonitor.Functions.Services if (rollup.DeviceSetupAllSucceeded) { var elapsedHours = (nowUtc - startedAtUtc).TotalHours; - if (elapsedHours < graceHours) + // The grace only buys time for a LIVE agent to come back with real evidence. Once + // the agent's own max-lifetime watchdog has fired it has cleaned up and exited — + // parking such a session AwaitingUser is a countdown, not a wait (all 5 observed + // maxlife:r5_awaiting episodes expired unhealed; calibration read 2026-08-27), so + // it skips straight to the terminal fork. Nothing is lost by deciding now: late + // straggler telemetry still heals a terminal verdict (TryLateTelemetryReconcileAsync). + if (elapsedHours < graceHours && !rollup.HasAgentMaxLifetimeTimeout) { var acct = rollup.AccountSetupTotal > 0 ? $" (Account Setup {rollup.AccountSetupSucceededCount}/{rollup.AccountSetupTotal})" @@ -428,11 +451,32 @@ namespace AutopilotMonitor.Functions.Services ClassifierRules.R5DeviceSetupAwaiting); } + // 5a. Desktop reached + clean app record → completed (assumed). Unlike rule 4 this + // has no Hello terminal, so it never fires as an IMMEDIATE completion signal — + // only here, after the full grace (or with the agent provably gone), where + // "the user phase is still running" is no longer a possible explanation. + // Calibration justified the assumption (2026-08-27 read): r5-Incomplete devices + // re-enrolled at 2.6 % — the succeeded background, not the 29 % failure level — + // and the sampled half with a desktop had all tracked apps terminal with none + // failed. An app_install_failed anywhere disqualifies: "completed" must not + // overclaim a session whose ESP payload is provably not clean. + if (rollup.DesktopArrived && !rollup.HasAppInstallFailure) + return (SessionStatus.Succeeded, AppendReconcileTiming( + "Reconciled: Device Setup fully provisioned and a real-user desktop was observed — " + + "no completion, failure or app-install error before the agent went silent; treating the enrollment as completed", + startedAtUtc, nowUtc, lastEventAtUtc), ClassifierRules.R5DesktopAssumed); + // "0/0" would suggest a parsed rollup that never existed — when no Account Setup // rollup was ever observed, say so instead (misclassification audit 2026-07-16). var acctDetail = rollup.AccountSetupTotal > 0 ? $"last Account Setup {rollup.AccountSetupSucceededCount}/{rollup.AccountSetupTotal}" : "Account Setup progress never observed"; + // Inside the grace the only way here is the max-lifetime short-circuit — the + // "within {grace}h" wording would be false, so name the watchdog instead. + if (elapsedHours < graceHours) + return (SessionStatus.Incomplete, + $"Agent max-lifetime watchdog fired — agent gone without completion after Device Setup completed ({acctDetail})", + ClassifierRules.R5DeviceSetupIncomplete); return (SessionStatus.Incomplete, $"No completion signal within {graceHours}h grace after Device Setup completed ({acctDetail})", ClassifierRules.R5DeviceSetupIncomplete); diff --git a/src/Backend/AutopilotMonitor.Functions/Services/EventIngestProcessor.Classification.cs b/src/Backend/AutopilotMonitor.Functions/Services/EventIngestProcessor.Classification.cs index f7720e12..87de5896 100644 --- a/src/Backend/AutopilotMonitor.Functions/Services/EventIngestProcessor.Classification.cs +++ b/src/Backend/AutopilotMonitor.Functions/Services/EventIngestProcessor.Classification.cs @@ -285,6 +285,12 @@ namespace AutopilotMonitor.Functions.Services // Part-1 start (same anchor the maintenance sweep uses). var effectiveStart = session?.ResumedAt ?? session?.StartedAt ?? triggerEvent.Timestamp; + // The trigger event itself may not be persisted yet when this read runs — append it so + // the rollup always carries the max-lifetime fact (rule 5 skips the AwaitingUser grace + // on it: the agent is provably gone). The rollup folds booleans, so a stored duplicate + // is harmless. + sessionEvents.Add(triggerEvent); + var rollup = EnrollmentTimeoutClassifier.ExtractRollup(sessionEvents); var (targetStatus, reason, rule) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession( rollup, effectiveStart, now, graceHours, session?.LastEventAt ?? c.LatestEventTimestamp, diff --git a/src/Shared/AutopilotMonitor.Shared/Models/VerdictPaths.cs b/src/Shared/AutopilotMonitor.Shared/Models/VerdictPaths.cs index 2e260735..996007d2 100644 --- a/src/Shared/AutopilotMonitor.Shared/Models/VerdictPaths.cs +++ b/src/Shared/AutopilotMonitor.Shared/Models/VerdictPaths.cs @@ -97,6 +97,13 @@ namespace AutopilotMonitor.Shared.Models public const string R3EmergencyBreak = "r3"; public const string R4DesktopHello = "r4"; public const string R5DeviceSetupAwaiting = "r5_awaiting"; + /// + /// Rule-5 grace expired (or the agent is provably gone) with a real-user desktop observed and a + /// clean app record — completed (assumed). Split from after + /// the 2026-08-27 calibration read: r5-Incomplete devices re-enrolled at 2.6 % (the succeeded + /// background), and half the sampled sessions were healthy desktops whose agent died after the work. + /// + public const string R5DesktopAssumed = "r5_assumed"; public const string R5DeviceSetupIncomplete = "r5_incomplete"; public const string R6Fallthrough = "r6"; }