mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-24 04:32:32 +00:00
feat(classification,calibration): rule 5a completed-assumed, max-lifetime grace skip, drift-radar floors
First real calibration read (2026-08-27, four days instrumented) drove three data-backed changes: - Rule 5a: past grace (or with the agent provably gone), Device Setup done + real-user desktop + zero app_install_failed reconciles to Succeeded "completed (assumed)" (r5_assumed). r5-Incomplete devices re-enrolled at 2.6% - the succeeded background, not the ~29% failure level - and the sampled desktop half had clean app records. No desktop or any failed app stays Incomplete; rule 4 remains the immediate desktop+Hello reconcile. - Rule 5 skips the AwaitingUser grace once either max-lifetime watchdog shape (enrollment_failed/agent_timeout, agent_shutting_down/max_lifetime) is in the stream: all five observed maxlife:r5_awaiting parks expired unhealed while live-agent parks upgraded 7/26. The maxlife ingest appends its trigger event to the read stream so the fact is never missed; late telemetry still heals a terminal verdict. WhiteGlove rule 1b keeps its park (its one episode healed). - Drift radar: MinWindowHits raised to 10 (a 3-hit episode survived as noise), per-path share regression now needs >=5 baseline hits (new or renamed vocabulary cannot "regress"), and rule-shaped legacy:* paths count into the silence/classifier-verdict group sums on both sides of the 2026-08-23 instrumentation cutover - dissolving the lift-124 silence artifact. Episodes below the fire floors clear themselves. Backend only; the new path appears in the calibration matrix dynamically. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Oliver Kieselbach
@@ -10,7 +10,7 @@ tags:
|
||||
- classification
|
||||
- whiteglove
|
||||
- self-deploying
|
||||
timestamp: 2026-08-23T18:00:00+02:00
|
||||
timestamp: 2026-08-27T20:00:00+02:00
|
||||
---
|
||||
|
||||
# Silent-Session Classification
|
||||
@@ -30,7 +30,10 @@ so a session gets the same verdict regardless of which path noticed the silence
|
||||
subcategory counts (incl. the 30s fallback-confirmed messages), `HasExplicitFailure`,
|
||||
`HasEnrollmentComplete` vs. `HasTerminalComplete` (the Part-1 `whiteglove_complete`),
|
||||
`HasAgentEmergencyBreak`, `DesktopArrived`, `HelloResolved`, `HelloPolicyDisabled`,
|
||||
`SkipUserEsp`, RealmJoin detection/resolution. Session-row flags are passed alongside:
|
||||
`SkipUserEsp`, RealmJoin detection/resolution, `HasAppInstallFailure` (any
|
||||
`app_install_failed`), and `HasAgentMaxLifetimeTimeout` (either max-lifetime shape:
|
||||
`enrollment_failed`/`agent_timeout` or `agent_shutting_down`/`max_lifetime` — the agent is
|
||||
provably gone). Session-row flags are passed alongside:
|
||||
`IsPreProvisioned` + `ResumedAt` (WhiteGlove Part 2) and `IsSelfDeployingProfile`
|
||||
(registry-confirmed `CloudAssignedOobeConfig 0x20|0x40`, set at registration).
|
||||
|
||||
@@ -44,7 +47,9 @@ subcategory counts (incl. the 30s fallback-confirmed messages), `HasExplicitFail
|
||||
| 2 | Account Setup all-succeeded, or `enrollment_complete`, or `whiteglove_complete` on a **non-resumed** session | Succeeded |
|
||||
| 3 | `HasAgentEmergencyBreak` | Incomplete immediately (agent self-destructed, nothing can arrive) |
|
||||
| 4 | `DesktopArrived` + (Hello terminal, or Hello disabled + SkipUserEsp) | Succeeded (user provably finished) |
|
||||
| 5 | Device Setup done | AwaitingUser within grace, Incomplete past grace |
|
||||
| 5 | Device Setup done + within grace + **no** max-lifetime watchdog observed | AwaitingUser |
|
||||
| 5a | Device Setup done + past grace (or watchdog observed) + `DesktopArrived` + no `app_install_failed` | **Succeeded** — completed (assumed), `r5_assumed` |
|
||||
| 5b | Device Setup done, otherwise past grace (or watchdog observed) | Incomplete |
|
||||
| 6 | otherwise | Incomplete |
|
||||
|
||||
Grace = `ResolveGraceHours(SessionGraceHours, AbsoluteMaxSessionHours)` (~51h default),
|
||||
@@ -65,7 +70,12 @@ budget on a Wi-Fi link still associating), and rule 5 labeled finished kiosks
|
||||
Incomplete turned a working rollout into a red wall. Other self-deploying tenants completed
|
||||
normally in the same window, so this is a classification gap, not a profile-wide agent bug.
|
||||
Agent-side companion fix: `SessionRegistrationHelper` now waits up to 15s for a network link
|
||||
and retries 6× (~62s backoff) instead of 5× (~30s).
|
||||
and retries 6× (~62s backoff) instead of 5× (~30s). Measured 2026-08-27 (builds
|
||||
2.0.1428/1429, both carrying the fix): the reconcile share did not move, and the backend saw
|
||||
no registration attempt from any relaunch — no success, no 4xx, zero `register:superseded`
|
||||
in the tenant — so the relaunch never reaches the network at all (device powered off after
|
||||
ESP, or the relaunch never runs). The sweep reconcile is the permanent completion path for
|
||||
this profile, not a stopgap.
|
||||
|
||||
## Callers (all pass both gate inputs)
|
||||
|
||||
@@ -78,13 +88,33 @@ and retries 6× (~62s backoff) instead of 5× (~30s).
|
||||
| Agent stall probe (`session_stalled`) | `EventIngestProcessor.TryMapStallToWhiteGloveAwaitingUserAsync` | Only the WG gate (1b); a stall probe on a self-deploying device with Device Setup done means the agent is alive and something else blocks — stays Stalled. |
|
||||
| Admin retro-reconcile | `LegacyReclassificationService` | Modes `legacy_timeouts`, `pending_orphans`, `self_deploying_silent` (Incomplete/AwaitingUser/Stalled rows with the profile flag, heal-only to Succeeded, admin-marked rows skipped). `POST /api/maintenance/reclassify-legacy?mode=…&dryRun=false&tenantId=…&maxSessions=…`, dry-run by default. |
|
||||
|
||||
## Known gap (decided, not built)
|
||||
## Why rule 5a exists (calibration read 2026-08-27)
|
||||
|
||||
The verdict-calibration matrix showed r5-Incomplete devices re-enrolling at 2.6 % within
|
||||
7 days — the succeeded background (`agent:complete_soft` sits at 2.8 %), not the ~29 %
|
||||
failure level — consistently across tenants, and half the sampled sessions were healthy
|
||||
desktops with a clean app record whose agent died before the Hello terminal (some with the
|
||||
agent's own "AccountSetup treated as complete" synthesis as the literally last event). Past
|
||||
the full grace, "the user phase is still running" is no longer a possible explanation, so
|
||||
desktop + zero `app_install_failed` + no explicit failure now reconciles to Succeeded
|
||||
("completed (assumed)") instead of a red Incomplete. Sessions without a desktop (user never
|
||||
began sign-in: battery-drained, powered off, max-lifetime) and sessions with any failed app
|
||||
stay Incomplete — "completed" must not overclaim. Rule 4 remains the IMMEDIATE reconcile
|
||||
(desktop + Hello proof); 5a only fires where waiting is over. Desktop arrival alone is still
|
||||
rejected as a live completion signal.
|
||||
|
||||
## Closed gap: max-lifetime AwaitingUser (2026-08-27)
|
||||
|
||||
After max-lifetime the agent writes `enrollment-complete.marker` and self-destructs, so an
|
||||
AwaitingUser verdict from that path can never heal — it is a 51h countdown to Incomplete.
|
||||
The emergency-break path (rule 3) already terminalizes immediately; max-lifetime still goes
|
||||
through the grace window by design ("user simply absent"). Revisit if customers read
|
||||
AwaitingUser as "still possible".
|
||||
AwaitingUser verdict from that path could never heal — the calibration correction stream
|
||||
confirmed it: all five observed `maxlife:r5_awaiting` episodes expired to Incomplete
|
||||
unhealed, while live-agent `sweep:r5_awaiting` parks upgraded 7 of 26 to Succeeded. Rule 5
|
||||
therefore skips the grace once `HasAgentMaxLifetimeTimeout` is in the stream and decides the
|
||||
terminal fork (5a/5b) immediately; the max-lifetime ingest appends its trigger event to the
|
||||
read stream so the fact is never missed, and late straggler telemetry still heals a terminal
|
||||
verdict via `TryLateTelemetryReconcileAsync`. WhiteGlove rule 1b deliberately keeps its
|
||||
park — sealed/boxed devices do heal (the observed `maxlife:r1b_awaiting` episode resolved to
|
||||
Succeeded).
|
||||
|
||||
# Examples
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ tags:
|
||||
- classifier
|
||||
- calibration
|
||||
- regression-radar
|
||||
timestamp: 2026-08-23T18:00:00+02:00
|
||||
timestamp: 2026-08-27T20:00:00+02:00
|
||||
---
|
||||
|
||||
# Verdict Calibration (classifier thermometer)
|
||||
@@ -114,15 +114,18 @@ and is wiped with the tenant on offboarding.
|
||||
## Drift radar — `VerdictCalibrationRadar`
|
||||
|
||||
Same statistics as the [rule regression radar](rule-regression-radar.md) (7-day window vs the
|
||||
prior 28 days, ≥ 5 window hits, ≥ 20 window sessions, lift ≥ 2, `MetricsMath.
|
||||
RateIncreaseSeparated`), anchored on yesterday, over every partition with rows (each tenant
|
||||
and `global`):
|
||||
prior 28 days, ≥ 20 window sessions, lift ≥ 2, `MetricsMath.RateIncreaseSeparated`), with
|
||||
stricter floors since the first-month tuning (2026-08-27): **≥ 10 window hits** (verdict
|
||||
shares on single-digit counts are noise — a 3-hit episode survived as such), and a per-path
|
||||
share regression additionally needs **≥ 5 baseline hits** — a path with no established
|
||||
baseline is new (or renamed) vocabulary, not a regression. Anchored on yesterday, over every
|
||||
partition with rows (each tenant and `global`):
|
||||
|
||||
| Kind | Signal | Re-arm |
|
||||
| --- | --- | --- |
|
||||
| `share_regression` | one **backend-decided** path's share of all sessions doubled (`sweep`/`maxlife`/`late`/`retro`/`rule`/`manual`/`ingest`; `agent:*` and `register:*` mirror customer workflow mix — first prod pass fired on `agent:whiteglove_pending` rollout weeks — and `legacy:*` is derived, all excluded) | share < 1.5× baseline, the path stops occurring, or the path is no longer eligible |
|
||||
| `silence_share_regression` | the `sweep:*` + `maxlife:*` share doubled — the backend had to decide more often because the agent went silent (a liveness signal, not a classifier one) | same |
|
||||
| `evidence_gap` | absolute: `r6` (pure fallthrough) decides ≥ 20 % of the window's classifier verdicts, ≥ 20 verdicts | < 15 % or < 20 verdicts |
|
||||
| `share_regression` | one **backend-decided** path's share of all sessions doubled (`sweep`/`maxlife`/`late`/`retro`/`rule`/`manual`/`ingest`; `agent:*` and `register:*` mirror customer workflow mix — first prod pass fired on `agent:whiteglove_pending` rollout weeks — and `legacy:*` is derived, all excluded per-path) | share < 1.5× baseline, the path stops occurring, the path is no longer eligible, or the current numbers fall below the fire floors |
|
||||
| `silence_share_regression` | the `sweep:*` + `maxlife:*` share doubled — the backend had to decide more often because the agent went silent (a liveness signal, not a classifier one). Rule-shaped `legacy:*` paths count into BOTH sums (window and baseline): they were the same backend decisions before the 2026-08-23 instrumentation, and without that continuity the group had a near-zero baseline by construction and fired a lift-124 rollout artifact | share < 1.5× baseline or hits stop |
|
||||
| `evidence_gap` | absolute: `r6` (pure fallthrough) decides ≥ 20 % of the window's classifier verdicts (stamped rule paths plus rule-shaped `legacy:*`, same continuity), ≥ 20 verdicts | < 15 % or < 20 verdicts |
|
||||
|
||||
Episodes are tracker rows `verdictcalibration|{kind}|{path}|{status}` (register = `AddEntity`,
|
||||
409 = already burning, fail-closed; refresh carries `FirstNotifiedAt`; 30-day tracker
|
||||
@@ -140,6 +143,15 @@ Kill switch `VerdictCalibrationRadarDisabled`.
|
||||
re-enrollment within 7 days after Succeeded 8.9 % (lab background), after Failed 24.3 %,
|
||||
after Incomplete 10.6 % — Incomplete behaves like Succeeded, so rules 5/6 are probably too
|
||||
cautious; the matrix now shows this per rule instead of per status.
|
||||
* **First real calibration read (2026-08-27, four days instrumented):** the per-rule and
|
||||
per-tenant split resolved that hand analysis — `legacy:r5_incomplete` re-enrolled at 2.6 %
|
||||
(the `agent:complete_soft` background), spread across tenants, while `legacy:r6`'s 29 %
|
||||
was concentrated in one tenant that disciplinedly redoes failed enrollments (48.5 % there,
|
||||
6.7 % elsewhere — so r6 stays untouched, it correctly flags that tenant). Consequences
|
||||
shipped the same day: rule 5a "completed (assumed)" (see
|
||||
[Silent-Session Classification](silent-session-classification.md)), the max-lifetime grace
|
||||
skip, and the radar floors + legacy group continuity above — the first production month
|
||||
had produced nine standing alerts, every one an instrumentation-rollout artifact.
|
||||
* **Correction stream:** a session swept to `sweep:r5_incomplete`, later completed by the
|
||||
agent → row `agent:complete` with `PriorVerdictPath = sweep:r5_incomplete`; the aggregate
|
||||
increments `OverriddenByLateCompletion` on the `sweep:r5_incomplete / Incomplete` bucket.
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
## 2026-08-27
|
||||
|
||||
* **Update**: `backend/silent-session-classification.md`, `backend/verdict-calibration.md` - the first real calibration read tuned what it measures. Rule 5 grew a past-grace fork 5a: Device Setup done + real-user desktop + zero `app_install_failed` reconciles to Succeeded "completed (assumed)" (`r5_assumed`) - r5-Incomplete devices re-enrolled at 2.6 %, the succeeded background, and the sampled desktop half had clean app records; no desktop or any failed app stays Incomplete. The max-lifetime watchdog (either event shape) now skips the AwaitingUser grace entirely - all five observed maxlife parks expired unhealed, and late telemetry heals a terminal verdict anyway - while WhiteGlove rule 1b keeps its park (its one episode healed). The drift radar got floors (≥ 10 window hits; per-path share regression needs ≥ 5 baseline hits, so new/renamed vocabulary cannot "regress") and legacy continuity: rule-shaped `legacy:*` paths count into the silence and classifier-verdict group sums on both sides of the 2026-08-23 cutover, which dissolves the lift-124 silence artifact and lets sub-floor episodes clear themselves. Backend only; the new path shows up in the matrix on its own.
|
||||
|
||||
* **Update**: `backend/tenant-activation-welcome-mail.md` - the reverse lookup. A delivery log names only the recipient, so an address had to lead back to its tenant: the tenant console's search now matches the notification and contact address as well as domain and tenant id, with the quoted exact mode unchanged. The addresses sit in `PreviewWhitelist`, not on the tenant config, so a new GA/Reader-only `GET /api/preview/notification-emails` returns them as one map (single cross-partition scan of `RowKey eq 'notification-email'`) instead of a point read per tenant; both sides join on the lowercased tenant id because the address rows carry the JWT `tid` while a config PartitionKey's casing is not guaranteed, and matching is case-insensitive because stored addresses keep the casing the user typed. The map is best-effort in the portal - a failed read costs only the search-by-address, never the tenant list - and the addresses stay out of the list rows (search key, not a column). Match rules moved into `tenantSearch.ts` with unit pins. Backend before web.
|
||||
|
||||
|
||||
|
||||
@@ -394,14 +394,91 @@ public class EnrollmentTimeoutClassifierTests
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Classify_agent_timeout_failed_session_classifies_honestly()
|
||||
public void Classify_agent_timeout_terminalizes_immediately_instead_of_parking()
|
||||
{
|
||||
// WG Part-2 shape (session 1506ce9f): DeviceSetup 4/4, no user phase, watchdog fired at
|
||||
// 6h — honest verdict is AwaitingUser (within grace), never Failed.
|
||||
// The watchdog firing proves the agent is gone for good — an AwaitingUser park from this
|
||||
// state is a countdown, not a wait: all 5 observed maxlife:r5_awaiting episodes expired
|
||||
// unhealed (calibration read 2026-08-27). Still never Failed, and late straggler
|
||||
// telemetry can heal the Incomplete (TryLateTelemetryReconcileAsync).
|
||||
var timeout = Evt("enrollment_failed",
|
||||
data: new Dictionary<string, object> { ["failureType"] = "agent_timeout" });
|
||||
var (status, _) = Classify(new[] { Esp(DeviceSetup44), timeout }, hoursSinceStart: 6);
|
||||
Assert.Equal(SessionStatus.AwaitingUser, status);
|
||||
var (status, reason) = Classify(new[] { Esp(DeviceSetup44), timeout }, hoursSinceStart: 6);
|
||||
Assert.Equal(SessionStatus.Incomplete, status);
|
||||
Assert.Contains("max-lifetime watchdog", reason);
|
||||
|
||||
// The V2 shutdown shape carries the same fact.
|
||||
var shutdown = Evt("agent_shutting_down",
|
||||
data: new Dictionary<string, object> { ["reason"] = "max_lifetime" });
|
||||
var (viaShutdown, _) = Classify(new[] { Esp(DeviceSetup44), shutdown }, hoursSinceStart: 6);
|
||||
Assert.Equal(SessionStatus.Incomplete, viaShutdown);
|
||||
|
||||
// WhiteGlove Part 2 (rule 1b) is deliberately untouched: a sealed/powered-off device
|
||||
// between technician and end user stays AwaitingUser — that park DOES heal
|
||||
// (the observed maxlife:r1b_awaiting episode resolved to Succeeded).
|
||||
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(new[] { Esp(DeviceSetup44), timeout });
|
||||
var (wgStatus, _, wgRule) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
|
||||
rollup, Start, Start.AddHours(6), 72, isPreProvisioned: true, resumedAt: Start.AddHours(1));
|
||||
Assert.Equal(SessionStatus.AwaitingUser, wgStatus);
|
||||
Assert.Equal(ClassifierRules.R1bWhiteGloveAwaiting, wgRule);
|
||||
}
|
||||
|
||||
// -------- Rule 5a: completed (assumed) — calibration read 2026-08-27 --------
|
||||
|
||||
[Fact]
|
||||
public void ExtractRollup_detects_app_failures_and_maxlife_timeout()
|
||||
{
|
||||
Assert.True(EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("app_install_failed") }).HasAppInstallFailure);
|
||||
var timeout = Evt("enrollment_failed", data: new Dictionary<string, object> { ["failureType"] = "agent_timeout" });
|
||||
Assert.True(EnrollmentTimeoutClassifier.ExtractRollup(new[] { timeout }).HasAgentMaxLifetimeTimeout);
|
||||
var shutdown = Evt("agent_shutting_down", data: new Dictionary<string, object> { ["reason"] = "max_lifetime" });
|
||||
Assert.True(EnrollmentTimeoutClassifier.ExtractRollup(new[] { shutdown }).HasAgentMaxLifetimeTimeout);
|
||||
// Other shutdown reasons say nothing about max lifetime.
|
||||
var ctrlC = Evt("agent_shutting_down", data: new Dictionary<string, object> { ["reason"] = "ctrl_c" });
|
||||
Assert.False(EnrollmentTimeoutClassifier.ExtractRollup(new[] { ctrlC }).HasAgentMaxLifetimeTimeout);
|
||||
// A real failure event does not raise the app flag.
|
||||
Assert.False(EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("enrollment_failed") }).HasAppInstallFailure);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Classify_desktop_without_hello_past_grace_is_assumed_complete()
|
||||
{
|
||||
// The healthy half of the r5 population (drill-down 2026-08-27): desktop reached, apps
|
||||
// clean, agent died before the Hello terminal. Past the full grace "user phase still
|
||||
// running" is no longer a possible explanation — completed (assumed), not a red Incomplete.
|
||||
var (status, reason, rule) = ClassifyWithRule(new[]
|
||||
{
|
||||
Esp(DeviceSetup44), Esp(AccountSetup15), Evt("desktop_arrived"),
|
||||
}, hoursSinceStart: 80, grace: 72);
|
||||
Assert.Equal(SessionStatus.Succeeded, status);
|
||||
Assert.Equal(ClassifierRules.R5DesktopAssumed, rule);
|
||||
Assert.Contains("treating the enrollment as completed", reason);
|
||||
Assert.Contains("Agent last reported", reason); // silence-transparency clause
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Classify_desktop_with_app_failure_past_grace_stays_Incomplete()
|
||||
{
|
||||
// "Completed" must not overclaim a session whose ESP payload provably failed
|
||||
// (drill-down session b1edc6f8: desktop reached but .NET 3.5 install failed).
|
||||
var (status, _) = Classify(new[]
|
||||
{
|
||||
Esp(DeviceSetup44), Evt("desktop_arrived"), Evt("app_install_failed"),
|
||||
}, hoursSinceStart: 80, grace: 72);
|
||||
Assert.Equal(SessionStatus.Incomplete, status);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Classify_maxlife_with_desktop_and_clean_apps_is_assumed_complete_within_grace()
|
||||
{
|
||||
// Agent provably gone + desktop + clean apps: no reason to park — the assumption is as
|
||||
// good now as it would be after the 51h grace.
|
||||
var timeout = Evt("enrollment_failed", data: new Dictionary<string, object> { ["failureType"] = "agent_timeout" });
|
||||
var (status, _, rule) = ClassifyWithRule(new[]
|
||||
{
|
||||
Esp(DeviceSetup44), Evt("desktop_arrived"), timeout,
|
||||
}, hoursSinceStart: 6);
|
||||
Assert.Equal(SessionStatus.Succeeded, status);
|
||||
Assert.Equal(ClassifierRules.R5DesktopAssumed, rule);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
|
||||
@@ -89,10 +89,79 @@ public class VerdictCalibrationRadarTests
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Derived_legacy_paths_never_alert()
|
||||
public void Derived_legacy_paths_never_alert_per_path_but_feed_the_group_sums()
|
||||
{
|
||||
var rows = Horizon(w => new[] { ("legacy:r6", "Incomplete", w ? 10 : 1), ("agent:complete", "Succeeded", w ? 10 : 19) });
|
||||
Assert.DoesNotContain(VerdictCalibrationRadar.Evaluate(rows, Target), f => f.Kind == VerdictCalibrationAlertKinds.ShareRegression);
|
||||
var findings = VerdictCalibrationRadar.Evaluate(rows, Target);
|
||||
Assert.DoesNotContain(findings, f => f.Kind == VerdictCalibrationAlertKinds.ShareRegression);
|
||||
// Legacy continuity: the rule-shaped legacy subset counts into the silence group, so a
|
||||
// real silence jump stays visible even when the history predates the instrumentation.
|
||||
Assert.Contains(findings, f => f.Kind == VerdictCalibrationAlertKinds.SilenceShareRegression);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Share_regression_requires_an_established_baseline()
|
||||
{
|
||||
// The 2026-08-23 instrumentation-cutover shape: the same decisions exist only under the
|
||||
// legacy name in the baseline and only under the new name in the window. Zero baseline
|
||||
// hits is new vocabulary, not a regression — and the legacy continuity keeps the silence
|
||||
// group flat across the rename (production fired a lift-124 artifact without both).
|
||||
var rows = Horizon(w => new[]
|
||||
{
|
||||
("sweep:r5_incomplete", "Incomplete", w ? 4 : 0),
|
||||
("legacy:r5_incomplete", "Incomplete", w ? 0 : 4),
|
||||
("agent:complete", "Succeeded", 16),
|
||||
});
|
||||
var findings = VerdictCalibrationRadar.Evaluate(rows, Target);
|
||||
Assert.DoesNotContain(findings, f => f.Kind == VerdictCalibrationAlertKinds.ShareRegression);
|
||||
Assert.DoesNotContain(findings, f => f.Kind == VerdictCalibrationAlertKinds.SilenceShareRegression);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Share_regression_needs_ten_window_hits()
|
||||
{
|
||||
// 7 window hits at 5% vs an established 1.07% baseline — lift ~4.7, but verdict shares
|
||||
// on single-digit counts are noise (a 3-hit episode survived as such, read 2026-08-27).
|
||||
var rows = new List<VerdictCalibrationDailyAggregate>();
|
||||
for (var i = 34; i >= 0; i--)
|
||||
{
|
||||
var inWindow = i < 7;
|
||||
rows.Add(new VerdictCalibrationDailyAggregate
|
||||
{
|
||||
TenantId = "t", Date = Target.AddDays(-i).ToString("yyyy-MM-dd"), SessionCount = 20, TerminalSessionCount = 20,
|
||||
Buckets =
|
||||
{
|
||||
new VerdictCalibrationBucket { VerdictPath = "sweep:r6", Status = "Incomplete", Count = inWindow ? 1 : (i == 20 ? 6 : 0) },
|
||||
new VerdictCalibrationBucket { VerdictPath = "agent:complete", Status = "Succeeded", Count = inWindow ? 19 : 20 },
|
||||
},
|
||||
});
|
||||
}
|
||||
Assert.DoesNotContain(VerdictCalibrationRadar.Evaluate(rows, Target),
|
||||
f => f.Kind == VerdictCalibrationAlertKinds.ShareRegression);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ReArm_clears_episodes_below_the_fire_floors()
|
||||
{
|
||||
// A zero-baseline episode from before the floors existed (rename artifact) clears itself
|
||||
// even while its share is elevated…
|
||||
var artifact = new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.ShareRegression, VerdictPath = "sweep:r5_incomplete", Status = "Incomplete" };
|
||||
var renamed = Horizon(w => new[]
|
||||
{
|
||||
("sweep:r5_incomplete", "Incomplete", w ? 4 : 0),
|
||||
("legacy:r5_incomplete", "Incomplete", w ? 0 : 4),
|
||||
("agent:complete", "Succeeded", 16),
|
||||
});
|
||||
Assert.True(VerdictCalibrationRadar.ShouldReArm(artifact, renamed, Target));
|
||||
|
||||
// …and so does one whose window has shrunk under the 10-hit floor.
|
||||
var shrunk = new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.ShareRegression, VerdictPath = "maxlife:r5_awaiting", Status = "AwaitingUser" };
|
||||
var tiny = Horizon(w => new[]
|
||||
{
|
||||
("maxlife:r5_awaiting", "AwaitingUser", 1),
|
||||
("agent:complete", "Succeeded", 19),
|
||||
});
|
||||
Assert.True(VerdictCalibrationRadar.ShouldReArm(shrunk, tiny, Target));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
|
||||
@@ -60,6 +60,7 @@ public class VerdictPathTests
|
||||
Assert.Equal("sweep:r5_incomplete", path);
|
||||
Assert.Equal("sweep", VerdictPaths.Origin(path));
|
||||
Assert.True(VerdictPaths.IsClassifierPath(path));
|
||||
Assert.True(VerdictPaths.IsClassifierPath("sweep:r5_assumed"));
|
||||
Assert.True(VerdictPaths.IsClassifierPath("maxlife:r1"));
|
||||
Assert.True(VerdictPaths.IsClassifierPath("late:r4"));
|
||||
Assert.True(VerdictPaths.IsClassifierPath("retro:r6"));
|
||||
@@ -107,6 +108,7 @@ public class VerdictPathTests
|
||||
Assert.Equal(ClassifierRules.R3EmergencyBreak, RuleOf(new[] { Esp(DeviceSetup44), Evt("agent_emergency_break") }));
|
||||
Assert.Equal(ClassifierRules.R4DesktopHello, RuleOf(new[] { Esp(DeviceSetup44), Evt("desktop_arrived"), Evt("hello_provisioning_completed") }));
|
||||
Assert.Equal(ClassifierRules.R5DeviceSetupAwaiting, RuleOf(new[] { Esp(DeviceSetup44), Esp(AccountSetup15) }, hours: 6, grace: 72));
|
||||
Assert.Equal(ClassifierRules.R5DesktopAssumed, RuleOf(new[] { Esp(DeviceSetup44), Esp(AccountSetup15), Evt("desktop_arrived") }, hours: 80, grace: 72));
|
||||
Assert.Equal(ClassifierRules.R5DeviceSetupIncomplete, RuleOf(new[] { Esp(DeviceSetup44), Esp(AccountSetup15) }, hours: 80, grace: 72));
|
||||
Assert.Equal(ClassifierRules.R6Fallthrough, RuleOf(Array.Empty<EnrollmentEvent>()));
|
||||
}
|
||||
|
||||
@@ -38,8 +38,10 @@ public class VerdictCalibrationFinding
|
||||
/// <item><b>evidence_gap</b> — absolute gate: the pure fallthrough rule (r6) decides ≥20 % of the
|
||||
/// window's classifier verdicts. The evidence the other rules need is not arriving.</item>
|
||||
/// </list>
|
||||
/// Derived (<c>legacy:*</c>) paths are excluded from all three — pre-instrumentation attribution
|
||||
/// is too weak to alert on. Agent-declared paths (<c>agent:*</c>, <c>register:*</c>) are excluded
|
||||
/// Derived (<c>legacy:*</c>) paths never alert per-path — pre-instrumentation attribution is too
|
||||
/// weak for that — but their rule-shaped subset DOES feed the silence and classifier-verdict
|
||||
/// group sums so the groups stay continuous across the 2026-08-23 instrumentation cutover
|
||||
/// (see <see cref="Summarize"/>). Agent-declared paths (<c>agent:*</c>, <c>register:*</c>) are excluded
|
||||
/// from the per-path share regression: their share tracks the customer's workflow mix (a
|
||||
/// WhiteGlove rollout week doubles <c>agent:whiteglove_pending</c>; a Continue-Anyway tenant
|
||||
/// lives on <c>agent:complete_soft</c>) — real signals, but not about the BACKEND classifier
|
||||
@@ -51,7 +53,21 @@ public static class VerdictCalibrationRadar
|
||||
{
|
||||
public const int WindowDays = RuleRegressionRadar.WindowDays;
|
||||
public const int BaselineDays = RuleRegressionRadar.BaselineDays;
|
||||
public const int MinWindowHits = RuleRegressionRadar.MinWindowHitSessions;
|
||||
/// <summary>
|
||||
/// Minimum window hits before a share finding fires or keeps its episode. Deliberately
|
||||
/// higher than the rule radar's 5: verdict shares swing on tiny counts, and the first
|
||||
/// production month kept a 3-hit maxlife:r5_awaiting episode alive as pure noise
|
||||
/// (calibration read 2026-08-27).
|
||||
/// </summary>
|
||||
public const int MinWindowHits = 10;
|
||||
/// <summary>
|
||||
/// Minimum baseline hits before a PER-PATH share regression is meaningful. A path with no
|
||||
/// established baseline cannot regress — it is new vocabulary (or a renamed path), and
|
||||
/// alerting on it just counts the rollout: the 2026-08-23 instrumentation cut over from
|
||||
/// legacy:* to sweep:*/register:* names and fired seven zero-baseline artifacts. The group
|
||||
/// kinds are exempt — their baselines carry the legacy continuity (see <see cref="Summarize"/>).
|
||||
/// </summary>
|
||||
public const int MinBaselineHits = 5;
|
||||
public const int MinWindowSessions = RuleRegressionRadar.MinWindowSessions;
|
||||
public const double MinLift = RuleRegressionRadar.MinLift;
|
||||
public const double ReArmLiftFactor = RuleRegressionRadar.ReArmLiftFactor;
|
||||
@@ -135,7 +151,7 @@ public static class VerdictCalibrationRadar
|
||||
.ToList();
|
||||
}
|
||||
|
||||
/// <summary>True when an ACTIVE alert may re-arm given the current horizon sums (or when its path is no longer eligible — episodes from an earlier, broader gate clear themselves).</summary>
|
||||
/// <summary>True when an ACTIVE alert may re-arm given the current horizon sums (or when its path/numbers no longer pass the fire gates — episodes from an earlier, broader gate clear themselves).</summary>
|
||||
public static bool ShouldReArm(VerdictCalibrationAlert alert, IReadOnlyList<VerdictCalibrationDailyAggregate> tenantRows, DateTime targetDateUtc)
|
||||
{
|
||||
if (alert.Kind == VerdictCalibrationAlertKinds.ShareRegression && !IsShareRegressionEligible(alert.VerdictPath))
|
||||
@@ -153,6 +169,9 @@ public static class VerdictCalibrationRadar
|
||||
|
||||
default:
|
||||
horizon.Paths.TryGetValue((alert.VerdictPath, alert.Status), out var sums);
|
||||
// Below the fire floors the episode is noise-retention, not an elevated share —
|
||||
// clear it (it re-fires only by passing the full gates again).
|
||||
if (sums.WindowHits < MinWindowHits || sums.BaselineHits < MinBaselineHits) return true;
|
||||
return ReArmByRate(sums.WindowHits, horizon.WindowSessions, sums.BaselineHits, horizon.BaselineSessions);
|
||||
}
|
||||
}
|
||||
@@ -189,6 +208,8 @@ public static class VerdictCalibrationRadar
|
||||
if (windowHits < MinWindowHits) return null;
|
||||
if (windowSessions < MinWindowSessions) return null;
|
||||
if (baselineSessions <= 0) return null;
|
||||
// Per-path only: no established baseline → no regression claim (new/renamed vocabulary).
|
||||
if (kind == VerdictCalibrationAlertKinds.ShareRegression && baselineHits < MinBaselineHits) return null;
|
||||
|
||||
var windowRate = (double)windowHits / windowSessions;
|
||||
var baselineRate = (double)baselineHits / baselineSessions;
|
||||
@@ -212,6 +233,14 @@ public static class VerdictCalibrationRadar
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>Derived classifier-rule path (<c>legacy:r5_incomplete</c>) — same detail shape as <see cref="VerdictPaths.IsClassifierPath"/>, legacy origin.</summary>
|
||||
private static bool IsLegacyClassifierPath(string path, string origin)
|
||||
{
|
||||
if (origin != LegacyOrigin) return false;
|
||||
var detail = path.Length > origin.Length + 1 ? path.Substring(origin.Length + 1) : string.Empty;
|
||||
return detail.Length >= 2 && detail[0] == 'r' && char.IsDigit(detail[1]);
|
||||
}
|
||||
|
||||
internal readonly record struct PathSums(int WindowHits, int BaselineHits);
|
||||
|
||||
internal sealed class Horizon
|
||||
@@ -234,6 +263,16 @@ public static class VerdictCalibrationRadar
|
||||
/// question is "how often did the backend have to decide", not which rule did). Classifier
|
||||
/// verdicts = stamped rule paths (<see cref="VerdictPaths.IsClassifierPath"/>); r6 = their
|
||||
/// fallthrough subset.
|
||||
/// Legacy continuity: derived rule-shaped paths (<c>legacy:r5_incomplete</c>, <c>legacy:r6</c>, …)
|
||||
/// were the SAME backend decisions before the 2026-08-23 instrumentation — the derivation reads
|
||||
/// the writers' own reason literals, which is strong enough for a group sum (though still too
|
||||
/// weak for per-path alerting). Counting them symmetrically (window AND baseline) keeps the
|
||||
/// silence and classifier-verdict groups continuous across the cutover; without this the groups
|
||||
/// had a near-zero baseline by construction and fired a lift-124 artifact. Legacy rows only
|
||||
/// exist for pre-instrumentation sessions, so the term self-deprecates as they age out.
|
||||
/// (Non-rule legacy paths — <c>legacy:superseded</c>, <c>legacy:wg_awaiting</c>,
|
||||
/// <c>legacy:unknown</c> — stay excluded: superseded is a registration write, unknown is
|
||||
/// unattributable, and wg_awaiting is a negligible tail.)
|
||||
/// </summary>
|
||||
internal static Horizon Summarize(IReadOnlyList<VerdictCalibrationDailyAggregate> rows, DateTime targetDateUtc)
|
||||
{
|
||||
@@ -262,11 +301,12 @@ public static class VerdictCalibrationRadar
|
||||
: new PathSums(sums.WindowHits, sums.BaselineHits + b.Count);
|
||||
|
||||
var origin = VerdictPaths.Origin(b.VerdictPath);
|
||||
if (origin == VerdictPaths.OriginSweep || origin == VerdictPaths.OriginMaxLifetime)
|
||||
var legacyRule = IsLegacyClassifierPath(b.VerdictPath, origin);
|
||||
if (origin == VerdictPaths.OriginSweep || origin == VerdictPaths.OriginMaxLifetime || legacyRule)
|
||||
{
|
||||
if (inWindow) h.SilenceWindowHits += b.Count; else h.SilenceBaselineHits += b.Count;
|
||||
}
|
||||
if (VerdictPaths.IsClassifierPath(b.VerdictPath))
|
||||
if (VerdictPaths.IsClassifierPath(b.VerdictPath) || legacyRule)
|
||||
{
|
||||
if (inWindow) h.ClassifierWindowVerdicts += b.Count; else h.ClassifierBaselineVerdicts += b.Count;
|
||||
if (b.VerdictPath.EndsWith(":" + ClassifierRules.R6Fallthrough, StringComparison.Ordinal))
|
||||
|
||||
@@ -99,7 +99,9 @@ namespace AutopilotMonitor.Functions.Services
|
||||
bool HelloPolicyDisabled,
|
||||
bool SkipUserEsp,
|
||||
bool RealmJoinDetected,
|
||||
bool RealmJoinResolved);
|
||||
bool RealmJoinResolved,
|
||||
bool HasAppInstallFailure,
|
||||
bool HasAgentMaxLifetimeTimeout);
|
||||
|
||||
/// <summary>
|
||||
/// Walk a session's events and distill the ESP rollup. Tolerant of missing/empty input
|
||||
@@ -115,6 +117,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
int acctBestN = 0, acctBestM = 0;
|
||||
bool acctFallbackAll = false;
|
||||
bool hasFailure = false, hasComplete = false, hasEnrollmentComplete = false, hasEmergencyBreak = false;
|
||||
bool hasAppInstallFailure = false, hasMaxLifetimeTimeout = false;
|
||||
bool desktopArrived = false, helloResolved = false;
|
||||
bool realmJoinDetected = false, realmJoinResolved = false;
|
||||
bool sawHelloPolicyEnabled = false, sawHelloPolicyDisabled = false;
|
||||
@@ -134,7 +137,19 @@ namespace AutopilotMonitor.Functions.Services
|
||||
// (misclassification audit 2026-07-16, tenant a53e67ec cluster).
|
||||
if (!Eq(TryGetDataString(evt, "failureType") ?? string.Empty, "agent_timeout"))
|
||||
hasFailure = true;
|
||||
else
|
||||
hasMaxLifetimeTimeout = true;
|
||||
}
|
||||
// The V2 watchdog shape of the same fact: the agent shut itself down at its
|
||||
// max-lifetime cap. Either shape proves the agent is gone for good — rule 5 uses
|
||||
// that to skip the AwaitingUser grace (a wait nothing can ever end).
|
||||
else if (Eq(type, "agent_shutting_down"))
|
||||
{
|
||||
if (Eq(TryGetDataString(evt, "reason") ?? string.Empty, "max_lifetime"))
|
||||
hasMaxLifetimeTimeout = true;
|
||||
}
|
||||
else if (Eq(type, "app_install_failed"))
|
||||
hasAppInstallFailure = true;
|
||||
else if (Eq(type, "enrollment_complete") || Eq(type, "whiteglove_complete"))
|
||||
{
|
||||
hasComplete = true;
|
||||
@@ -230,7 +245,9 @@ namespace AutopilotMonitor.Functions.Services
|
||||
HelloPolicyDisabled: sawHelloPolicyDisabled && !sawHelloPolicyEnabled,
|
||||
SkipUserEsp: sawSkipUserTrue && !sawSkipUserFalse,
|
||||
RealmJoinDetected: realmJoinDetected,
|
||||
RealmJoinResolved: realmJoinResolved);
|
||||
RealmJoinResolved: realmJoinResolved,
|
||||
HasAppInstallFailure: hasAppInstallFailure,
|
||||
HasAgentMaxLifetimeTimeout: hasMaxLifetimeTimeout);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -418,7 +435,13 @@ namespace AutopilotMonitor.Functions.Services
|
||||
if (rollup.DeviceSetupAllSucceeded)
|
||||
{
|
||||
var elapsedHours = (nowUtc - startedAtUtc).TotalHours;
|
||||
if (elapsedHours < graceHours)
|
||||
// The grace only buys time for a LIVE agent to come back with real evidence. Once
|
||||
// the agent's own max-lifetime watchdog has fired it has cleaned up and exited —
|
||||
// parking such a session AwaitingUser is a countdown, not a wait (all 5 observed
|
||||
// maxlife:r5_awaiting episodes expired unhealed; calibration read 2026-08-27), so
|
||||
// it skips straight to the terminal fork. Nothing is lost by deciding now: late
|
||||
// straggler telemetry still heals a terminal verdict (TryLateTelemetryReconcileAsync).
|
||||
if (elapsedHours < graceHours && !rollup.HasAgentMaxLifetimeTimeout)
|
||||
{
|
||||
var acct = rollup.AccountSetupTotal > 0
|
||||
? $" (Account Setup {rollup.AccountSetupSucceededCount}/{rollup.AccountSetupTotal})"
|
||||
@@ -428,11 +451,32 @@ namespace AutopilotMonitor.Functions.Services
|
||||
ClassifierRules.R5DeviceSetupAwaiting);
|
||||
}
|
||||
|
||||
// 5a. Desktop reached + clean app record → completed (assumed). Unlike rule 4 this
|
||||
// has no Hello terminal, so it never fires as an IMMEDIATE completion signal —
|
||||
// only here, after the full grace (or with the agent provably gone), where
|
||||
// "the user phase is still running" is no longer a possible explanation.
|
||||
// Calibration justified the assumption (2026-08-27 read): r5-Incomplete devices
|
||||
// re-enrolled at 2.6 % — the succeeded background, not the 29 % failure level —
|
||||
// and the sampled half with a desktop had all tracked apps terminal with none
|
||||
// failed. An app_install_failed anywhere disqualifies: "completed" must not
|
||||
// overclaim a session whose ESP payload is provably not clean.
|
||||
if (rollup.DesktopArrived && !rollup.HasAppInstallFailure)
|
||||
return (SessionStatus.Succeeded, AppendReconcileTiming(
|
||||
"Reconciled: Device Setup fully provisioned and a real-user desktop was observed — " +
|
||||
"no completion, failure or app-install error before the agent went silent; treating the enrollment as completed",
|
||||
startedAtUtc, nowUtc, lastEventAtUtc), ClassifierRules.R5DesktopAssumed);
|
||||
|
||||
// "0/0" would suggest a parsed rollup that never existed — when no Account Setup
|
||||
// rollup was ever observed, say so instead (misclassification audit 2026-07-16).
|
||||
var acctDetail = rollup.AccountSetupTotal > 0
|
||||
? $"last Account Setup {rollup.AccountSetupSucceededCount}/{rollup.AccountSetupTotal}"
|
||||
: "Account Setup progress never observed";
|
||||
// Inside the grace the only way here is the max-lifetime short-circuit — the
|
||||
// "within {grace}h" wording would be false, so name the watchdog instead.
|
||||
if (elapsedHours < graceHours)
|
||||
return (SessionStatus.Incomplete,
|
||||
$"Agent max-lifetime watchdog fired — agent gone without completion after Device Setup completed ({acctDetail})",
|
||||
ClassifierRules.R5DeviceSetupIncomplete);
|
||||
return (SessionStatus.Incomplete,
|
||||
$"No completion signal within {graceHours}h grace after Device Setup completed ({acctDetail})",
|
||||
ClassifierRules.R5DeviceSetupIncomplete);
|
||||
|
||||
@@ -285,6 +285,12 @@ namespace AutopilotMonitor.Functions.Services
|
||||
// Part-1 start (same anchor the maintenance sweep uses).
|
||||
var effectiveStart = session?.ResumedAt ?? session?.StartedAt ?? triggerEvent.Timestamp;
|
||||
|
||||
// The trigger event itself may not be persisted yet when this read runs — append it so
|
||||
// the rollup always carries the max-lifetime fact (rule 5 skips the AwaitingUser grace
|
||||
// on it: the agent is provably gone). The rollup folds booleans, so a stored duplicate
|
||||
// is harmless.
|
||||
sessionEvents.Add(triggerEvent);
|
||||
|
||||
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(sessionEvents);
|
||||
var (targetStatus, reason, rule) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
|
||||
rollup, effectiveStart, now, graceHours, session?.LastEventAt ?? c.LatestEventTimestamp,
|
||||
|
||||
@@ -97,6 +97,13 @@ namespace AutopilotMonitor.Shared.Models
|
||||
public const string R3EmergencyBreak = "r3";
|
||||
public const string R4DesktopHello = "r4";
|
||||
public const string R5DeviceSetupAwaiting = "r5_awaiting";
|
||||
/// <summary>
|
||||
/// Rule-5 grace expired (or the agent is provably gone) with a real-user desktop observed and a
|
||||
/// clean app record — completed (assumed). Split from <see cref="R5DeviceSetupIncomplete"/> after
|
||||
/// the 2026-08-27 calibration read: r5-Incomplete devices re-enrolled at 2.6 % (the succeeded
|
||||
/// background), and half the sampled sessions were healthy desktops whose agent died after the work.
|
||||
/// </summary>
|
||||
public const string R5DesktopAssumed = "r5_assumed";
|
||||
public const string R5DeviceSetupIncomplete = "r5_incomplete";
|
||||
public const string R6Fallthrough = "r6";
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user