mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-24 12:42:38 +00:00
docs: move design docs out of docs/design; keep customer work-files git-ignored
Reorganize docs/ tree:
- docs/design/mcp-oauth-flow.md -> docs/mcp-oauth-flow.md (no customer data)
- enrollment-status-reclassification{,-plan}.md -> tasks/ (git-ignored work
files; they contained a customer domain that must not ship in the public repo)
- remove now-empty docs/design/ folder
Repoint the 22 source files whose comments referenced
docs/design/enrollment-status-reclassification.md to the new tasks/ path.
Comment/docstring text only; no functional code changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Oliver Kieselbach
@@ -1,140 +0,0 @@
|
||||
# Implementation Plan — Enrollment Status Reclassification
|
||||
|
||||
> Working copy lives in `tasks/todo.md` (gitignored); this is the tracked, buildable checklist.
|
||||
|
||||
Design note: [`docs/design/enrollment-status-reclassification.md`](./enrollment-status-reclassification.md)
|
||||
Branch: `claude/crcins-enrollment-failures-ac943m`
|
||||
|
||||
Goal: stop the 5h sweep from labelling silent-but-provisioned sessions `Failed`; add a
|
||||
third terminal state `Incomplete/Unknown`; reconcile late completions to `Succeeded`.
|
||||
Backend + stats only — no agent changes, no heartbeat.
|
||||
|
||||
---
|
||||
|
||||
## PR1 — Status model + ESP rollup extraction (foundation) ✅ done
|
||||
|
||||
- [x] Add `AwaitingUser` and `Incomplete` to `SessionStatus` enum
|
||||
(`SessionApiModels.cs`, appended — existing ordinals unchanged).
|
||||
- [x] Extend `FailureSnapshotBuilder` to embed the ESP rollup
|
||||
(`deviceSetupAllSucceeded` / `accountSetup{SucceededCount,Total,AllSucceeded}`),
|
||||
schema v2. Gated on the subcategory rollup, NOT `categorySucceeded`.
|
||||
- [x] Pure `EnrollmentTimeoutClassifier.ExtractRollup` + `ClassifyTimedOutSession`
|
||||
(new `EnrollmentTimeoutClassifier.cs`) implementing the decision table.
|
||||
- [x] Unit tests `EnrollmentTimeoutClassifierTests` (12) — rollup extraction (0/5, 1–4/5,
|
||||
5/5, fallback, missing) + every classification branch. **Full Functions.Tests: 2670 pass.**
|
||||
|
||||
## PR2 — Sweep uses the classifier + grace window ✅ done
|
||||
|
||||
- [x] `TenantConfiguration.SessionGraceHours` (default 72) + default-config plumbing.
|
||||
- [x] `MarkStalledSessionsAsTimedOutAsync` (Stage 2): replaced hard-coded `Failed` with
|
||||
`EnrollmentTimeoutClassifier.ClassifyTimedOutSession(...)`. Fast-path skips within-grace
|
||||
`AwaitingUser` sessions with NO event read (grace is time-based). Per-outcome counters +
|
||||
audit/ops only when the pass did something.
|
||||
- [x] Synthetic `session_timeout` event + analyze enqueue kept ONLY for `Failed`/`Incomplete`
|
||||
terminal transitions (not `AwaitingUser`/`Succeeded`).
|
||||
- [x] `GetStalledSessionsAsync` now includes `AwaitingUser` so grace→`Incomplete` fires later.
|
||||
`GetAgentSilentSessionsAsync` left InProgress-only (no regress).
|
||||
- [x] `UpdateSessionStatusAsync`: `Incomplete` treated as terminal (idempotency guard);
|
||||
`AwaitingUser` never regresses a terminal; `CompletedAt` stamped for `Incomplete` (no
|
||||
duration); reason + snapshot persisted for `Incomplete`/`AwaitingUser`.
|
||||
- [x] Decision logic + grace boundary covered by `EnrollmentTimeoutClassifierTests` (pure —
|
||||
the sweep itself has ~17 deps, same rationale as `MaintenanceServiceSessionTimeoutEventTests`).
|
||||
**Full Functions.Tests: 2670 pass.** End-to-end validated via the live forward-only watch.
|
||||
|
||||
## PR-EB — Agent emergency-break event (transparency: close the silent-48h blind spot)
|
||||
|
||||
The agent's 48h absolute session-age break (`Program.Guards.CheckSessionAgeEmergencyBreak`)
|
||||
writes only a LOCAL marker + cleans up + exits — silent to the backend. Make it visible in the
|
||||
timeline AND let the backend terminalize precisely instead of guessing with grace.
|
||||
|
||||
- [x] `Constants.EventTypes.AgentEmergencyBreak` + `AgentErrorType.SessionAgeEmergencyBreak`.
|
||||
- [x] Classifier consumes it: `ExtractRollup.HasAgentEmergencyBreak`; a session carrying the break
|
||||
classifies as `Incomplete` NOW (skips the AwaitingUser grace) unless it actually completed.
|
||||
Tested. **This is the load-bearing backend consumer.**
|
||||
- [x] **Part B (backend): materialize the timeline event.** `ReportAgentErrorFunction` now injects
|
||||
`ISessionRepository` and, on `SessionAgeEmergencyBreak`, writes an `agent_emergency_break`
|
||||
`EnrollmentEvent` (Sequence = max+1, idempotent, best-effort) into the stream. Pure
|
||||
`BuildAgentEmergencyBreakEvent` helper (Warning, non-terminal) + tests. Suite: 2685 pass.
|
||||
- [x] **Part C (agent, net48 — CI/local build only): best-effort emit.** Implemented via a decoupled
|
||||
callback so the guard stays pure and its unit tests are untouched:
|
||||
- `CheckSessionAgeEmergencyBreak` gains an optional `Action onBreakFired = null`, invoked once when
|
||||
the break is confirmed, BEFORE the marker/cleanup, fully swallowed.
|
||||
- New `EmergencyBreakReporter.TrySend` builds a throwaway auth bundle (`BackendClientFactory`) and
|
||||
fires a `SessionAgeEmergencyBreak` `AgentErrorReport` over the resilient emergency channel, bounded
|
||||
to a 5s wait. `AgentBootstrap` wires it (`GetAgentVersion` made `internal`).
|
||||
- Lost if the device is fully offline — acceptable (that case falls back to the grace window).
|
||||
⚠️ **Not compiled here** (agent targets net48; this container has only the .NET 8 SDK). The Shared
|
||||
`AgentErrorType.SessionAgeEmergencyBreak` value IS verified (it compiled in the Functions build).
|
||||
**@okieselbach to run the net48 agent build locally.**
|
||||
|
||||
## PR3 — Reconciliation of late completions ✅ done
|
||||
|
||||
- [x] `UpdateSessionStatusAsync` terminal-guard replaced by pure `IsTerminalTransitionAllowed`:
|
||||
a genuine completion (`Succeeded`) UPGRADES a prior `Failed`/`Incomplete`/`AwaitingUser`
|
||||
verdict (late reconcile); silent-terminal verdicts + `AwaitingUser` never overwrite a terminal.
|
||||
- [x] On `Succeeded`, clear stale `FailureReason` + `FailureSnapshotJson` (reconcile hygiene).
|
||||
- [x] Admin-marked terminal decisions are preserved — a late completion does NOT auto-override a
|
||||
hand-marked `Failed`/`Incomplete`.
|
||||
- [x] Ingest already routes late `enrollment_complete`/`gather` → `Succeeded` with no pre-terminal
|
||||
skip, so the relaxed guard makes reconcile work end-to-end (no ingest change needed).
|
||||
- [x] Tests `SessionStatusTransitionTests` (20, full matrix). Suite: 2705 pass (one pre-existing
|
||||
timing-flaky maintenance test, green on retry/isolation — unrelated).
|
||||
|
||||
## PR4 — Stats / metrics expose 3 states ✅ done (Fleet Health deferred to PR5)
|
||||
|
||||
- [x] `SessionStatusBuckets` (`MetricsMath.cs`): explicit `AwaitingUser` + `Incomplete` buckets
|
||||
(out of `Other`); buckets still reconcile to Total by construction.
|
||||
- [x] Metrics summary (`GetMetricsSummaryAsync`): surface `awaitingUser` + `incomplete`; **failure
|
||||
rate now `Failed / (Succeeded + Failed)`** — Incomplete + non-terminal excluded from the denominator.
|
||||
- [x] `SessionStats` (`SessionApiModels.cs`): add `IncompleteLastNDays`; dashboard `SuccessRatePct`
|
||||
was already terminal-only (`Succeeded/(Succeeded+Failed)`), so Incomplete is excluded there too.
|
||||
- [x] Daily aggregation (`ComputeUsageMetricsSnapshotAsync`) already uses `Succeeded+Failed` as the
|
||||
denominator — no change needed (Incomplete already excluded).
|
||||
- [x] Tests `SessionStatusBucketsTests` (bucketing + terminal-only failure-rate). Suite: 2705 pass.
|
||||
- [x] Fleet Health payload Incomplete count — done in PR5 (`FleetHealthStats.Incomplete` + slate card).
|
||||
Its SuccessRate stays "over all sessions" (documented), only the Incomplete count was added.
|
||||
|
||||
## PR5 — Web surface ✅ done (typecheck clean)
|
||||
|
||||
- [x] `utils/sessionStatus.ts`: `isTerminalStatus` includes `Incomplete`; docstring notes AwaitingUser non-terminal.
|
||||
- [x] `SessionStatusBadge.tsx`: `Incomplete` = neutral slate (clearly not red), `AwaitingUser` = sky/blue
|
||||
("still going"); ⏱️ affordance extended to Incomplete (the silence family).
|
||||
- [x] Fleet Health: `FleetHealthStats.Incomplete` (C# model + `MetricsMath.BuildFleetHealthPayload`) +
|
||||
web interface + a dedicated slate "Incomplete" stat card (`FleetStatCard` gained a `slate` color);
|
||||
grid widened to 5.
|
||||
- [x] Main dashboard KPI row left as-is (its Success Rate is already the honest terminal rate); the third
|
||||
state is surfaced via the badge everywhere, the Fleet Health card, and the metrics summary `incomplete`.
|
||||
- [x] Verified: `tsc --noEmit` clean; backend suite 2709 pass.
|
||||
|
||||
## PR6 — Backfill (optional, one-shot)
|
||||
|
||||
- [ ] Manual maintenance path to re-classify historical `Failed` 5h-timeouts using the same
|
||||
classifier, so existing stats (crcins.com + platform) heal retroactively.
|
||||
- [ ] Dry-run counters logged before any write; gated behind an explicit admin trigger.
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
- [ ] `dotnet test` green across Functions.Tests + Shared/DecisionCore tests.
|
||||
- [ ] Re-run the crcins.com window (via MCP `query_raw_events` / `search_sessions`) and confirm
|
||||
the "after" table in the design note within rounding (Failed ~122, Incomplete ~115,
|
||||
+25 reconciled, failure rate ~4–5 %).
|
||||
- [ ] Confirm no `AwaitingUser` session ever emits a `session_timeout`/analyze `Failed` artifact.
|
||||
|
||||
## Review (fill in after implementation)
|
||||
- _pending_
|
||||
|
||||
## Decisions (2026-07-08)
|
||||
- **Third state label:** `Incomplete` (enum member `SessionStatus.Incomplete`, badge "Incomplete").
|
||||
- **Scope:** forward-only first (PR1–PR5) to watch how the first live sessions classify.
|
||||
**PR6 backfill is deferred** — run it once forward-only looks good, to heal historical
|
||||
crcins.com + platform stats. Remembered follow-up.
|
||||
- **`SessionGraceHours`:** now **auto-derived** from the agent's absolute cap, not a magic 72h.
|
||||
`effectiveGrace = max(AbsoluteMaxSessionHours + 3h buffer, override)` = **51h** by default (48 + 3)
|
||||
(`EnrollmentTimeoutClassifier.ResolveGraceHours`). Rationale: the 48h agent emergency break is
|
||||
*silent* to the backend, so grace must be ≥ that cap and only slightly beyond it. `SessionGraceHours=0`
|
||||
means auto; an override can only raise the floor. `AbsoluteMaxSessionHours` mirrored into TenantConfiguration,
|
||||
but until the agent honors it (follow-up below) the derivation clamps the assumed cap to ≥ the real agent
|
||||
default (48), so a lower override can never drag the grace below the agent's actual runtime.
|
||||
- **Follow-up:** wire `TenantConfiguration.AbsoluteMaxSessionHours` down into `AgentConfigResponse` +
|
||||
RemoteConfigMerger so a tenant override reaches the agent too (today the agent still uses its own default 48).
|
||||
@@ -1,236 +0,0 @@
|
||||
# Enrollment Status Reclassification — Timeout ≠ Failure
|
||||
|
||||
**Status:** Proposal · **Owner:** Backend · **Trigger tenant:** crcins.com (`ca9e3c59-d8ec-4ee6-9544-05f62f85ac98`)
|
||||
|
||||
## Problem
|
||||
|
||||
The 5‑hour maintenance sweep marks every silent, non‑terminal session as **`Failed`**
|
||||
([`MaintenanceService.MarkStalledSessionsAsTimedOutAsync`](../../src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.cs) — Stage 2 hard‑codes `SessionStatus.Failed`). This conflates
|
||||
*"the agent stopped sending telemetry"* with *"the enrollment failed"* and produces a
|
||||
massively inflated failure rate.
|
||||
|
||||
For crcins.com this is dramatic, but the mechanism is platform‑wide.
|
||||
|
||||
## Evidence (crcins.com, 90‑day window)
|
||||
|
||||
Reported failure rate **57.2 %** (1321 / 2310). Measured over the full failed population
|
||||
(n = 1311) via `query_raw_events(eventType=esp_provisioning_status)` + `search_sessions`:
|
||||
|
||||
| Failure reason | Count | Share |
|
||||
|---|---:|---:|
|
||||
| **`Session timed out after 5 hours`** (no terminal event) | **1189** | **90.7 %** |
|
||||
| ESP `WhiteGlove_Failed` | 37 | 2.8 % |
|
||||
| Agent max‑lifetime | 35 | 2.7 % |
|
||||
| `esp_terminal_failure` | 27 | 2.1 % |
|
||||
| `Autopilot enrollment failed: Provisioning…` | 19 | 1.4 % |
|
||||
| `esp_exit_without_completion` | 4 | 0.3 % |
|
||||
|
||||
**Explicit, real failures ≈ 87 → true failure rate ≈ 3.8 % (max ~5.3 %), not 57.2 %.**
|
||||
|
||||
### What the "5h timeouts" actually were
|
||||
|
||||
Last observed ESP state across the 1189 timeouts (1077 with captured
|
||||
`esp_provisioning_status`, 90.6 % coverage):
|
||||
|
||||
| Last observed state | Count | Verdict |
|
||||
|---|---:|---|
|
||||
| **DeviceSetup 4/4 = fully provisioned (AADJ+MDM)** | **1067 / 1077 (99.1 %)** | device is enrolled |
|
||||
| AccountSetup `0/5` — user phase never observed | 1024 (95.1 %) | **AwaitingUser** — not a failure |
|
||||
| AccountSetup `1–4/5` — mid user phase, agent gone | 25 (2.3 %) | **AwaitingUser** |
|
||||
| AccountSetup `5/5` — complete | 25 (2.3 %) | **reconcile → Succeeded** |
|
||||
| no `esp_provisioning_status` at all | 112 (9.4 %) | **Incomplete/Unknown** |
|
||||
|
||||
Not a single 5h‑timeout carries an explicit `enrollment_failed` / `esp_failure` event.
|
||||
|
||||
### Two key data facts (drive the design)
|
||||
|
||||
1. **`categorySucceeded` is unreliable.** In every `esp_provisioning_status` payload —
|
||||
even at DeviceSetup 4/4 and AccountSetup 5/5 — Windows reports
|
||||
`categorySucceeded = "in_progress"`. The agent already compensates with a 30 s
|
||||
"all subcategories succeeded → treat complete" fallback for DeviceSetup
|
||||
([`ProvisioningStatusTracker` / `EspAndHelloTracker`](../../src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/)).
|
||||
→ **Gate on the subcategory rollup, never on `categorySucceeded`.**
|
||||
2. **Desktop arrival ≠ complete.** In User‑driven ESP the desktop appears while the
|
||||
Account/user phase is still running. `desktop_arrived` is a hint, not a terminal.
|
||||
The authoritative terminal is **AccountSetup subcategory rollup = all succeeded**
|
||||
(which precedes the agent's own `enrollment_complete`).
|
||||
|
||||
## Root cause
|
||||
|
||||
The agent captures exactly the right signal (`esp_provisioning_status` with per‑category
|
||||
subcategory rollups incl. AccountSetup), but:
|
||||
|
||||
- In ~95 % of timeouts the agent goes dark at the DeviceSetup→AccountSetup reboot,
|
||||
**before** the user logs in, so AccountSetup never progresses in telemetry (0/5). The
|
||||
user phase legitimately completes hours/days later (proven by session
|
||||
`cfa820da-…`: AccountSetup 5/5 + `enrollment_complete` the **next day**).
|
||||
- The backend's 5h sweep fires first, hard‑labels `Failed`, and **never reconciles** when
|
||||
the late completion arrives.
|
||||
|
||||
So the fix is **classification + reconciliation in the backend** — no new agent load,
|
||||
no heartbeat. The signals are already in the event stream.
|
||||
|
||||
## Design
|
||||
|
||||
### 1. Status model — a third terminal bucket
|
||||
|
||||
`SessionStatus` ([`SessionApiModels.cs:426`](../../src/Shared/AutopilotMonitor.Shared/Models/SessionApiModels.cs))
|
||||
today: `InProgress, Pending, Stalled, Succeeded, Failed, Unknown`.
|
||||
|
||||
Add:
|
||||
|
||||
- **`AwaitingUser`** (non‑terminal): DeviceSetup succeeded, user/account phase not yet
|
||||
complete, agent silent but within grace. Can heal to `Succeeded` or expire to `Incomplete`.
|
||||
- **`Incomplete`** (terminal, **non‑failure**): grace expired with no terminal evidence,
|
||||
or silence before DeviceSetup completed with no explicit failure. This is the
|
||||
"success/failed/**incomplete‑unknown**" third state for the stats. (Existing `Unknown`
|
||||
is folded into the Incomplete bucket for reporting.)
|
||||
|
||||
### 2. Classification at the sweep (deterministic, from data we already load)
|
||||
|
||||
The sweep already loads up to 1000 events and builds a snapshot
|
||||
([`FailureSnapshotBuilder`](../../src/Backend/AutopilotMonitor.Functions/Services/FailureSnapshotBuilder.cs)).
|
||||
Extend the snapshot to extract the **last DeviceSetup and AccountSetup subcategory rollups**
|
||||
from `esp_provisioning_status` events, then decide the target status instead of hard‑coding
|
||||
`Failed`:
|
||||
|
||||
```
|
||||
Given a non-terminal session past the inactivity window:
|
||||
|
||||
1. Explicit enrollment_failed / esp_failure present → Failed (real)
|
||||
2. AccountSetup rollup = ALL subcategories succeeded
|
||||
OR enrollment_complete / whiteglove_complete present → Succeeded (reconcile)
|
||||
3. agent_emergency_break present (agent provably gone) → Incomplete (skip grace)
|
||||
4. desktop_arrived AND ( positive Hello terminal
|
||||
(hello_provisioning_completed / hello_skipped) present
|
||||
OR hello policy disabled AND User ESP skipped ) → Succeeded (reconcile)
|
||||
5. DeviceSetup rollup = ALL subcategories succeeded (or 30s
|
||||
fallback) AND no failure event:
|
||||
a. within grace (now - StartedAt < SessionGraceHours) → AwaitingUser (non-terminal)
|
||||
b. grace expired → Incomplete (terminal, non-failure)
|
||||
6. else (silence before DeviceSetup complete, no failure) → Incomplete
|
||||
```
|
||||
|
||||
Step 4 (added 2026-07-10, trigger session `294ab5b4`): desktop arrival **plus** a positive
|
||||
Hello terminal is exactly the agent's own Classic completion AND‑gate — once both are in,
|
||||
the only thing that can still block the agent's `enrollment_complete` is the RealmJoin
|
||||
completion gate, which self‑releases via a 60‑min hard timeout. A session silent long
|
||||
enough to reach the sweep is past every agent‑side wait: the enrollment succeeded and only
|
||||
the completion report never left the device (shutdown / process kill / egress cut).
|
||||
Labeling it `AwaitingUser` is factually wrong — the user was provably there. The reason
|
||||
string distinguishes "RealmJoin deployment never reported completion" (RJ detected but
|
||||
unresolved) from the generic "agent went silent before reporting completion". The failure
|
||||
snapshot (schema v3) carries the evidence: `helloResolved`, `skipUserEsp`,
|
||||
`realmJoinDetected`, `realmJoinResolved`.
|
||||
|
||||
The Hello‑disabled branch mirrors the agent's Hello‑disabled fast‑path exactly: with
|
||||
`hello_policy_detected(helloEnabled=false)` **and** `esp_config_detected(skipUserStatusPage=true)`
|
||||
no Hello terminal can ever exist, and the agent completes on desktop arrival in that
|
||||
configuration. Hello disabled **without** SkipUserEsp does NOT qualify — the agent's strong
|
||||
post‑AccountSetup gate (session `08c99638`) requires the AccountSetup rollup there, which
|
||||
step 2 already covers. Note `esp_exiting` (Shell‑Core 62407) is deliberately NOT used as
|
||||
success evidence anywhere in this table: it is a page‑lifecycle event that fires with
|
||||
`errorCode=0` even while AccountSetup apps are unfinished (the very lesson of `08c99638`),
|
||||
unlike the Hello events, which are outcome terminals.
|
||||
|
||||
Rationale for each bucket sizing is in the Evidence table above.
|
||||
⚠️ Gate step 2/3 on the **subcategory rollup**, not `categorySucceeded`.
|
||||
|
||||
### 3. Grace window (backend owns the wait — zero client cost)
|
||||
|
||||
Two‑stage terminalization in the same sweep that already runs:
|
||||
|
||||
- At `SessionTimeoutHours` (5h): non‑terminal → **`AwaitingUser`** (was `Failed`).
|
||||
- At the grace window: `AwaitingUser` → **`Incomplete`** (unless already reconciled).
|
||||
|
||||
**The grace is anchored to the agent, not a magic number.** The agent has two brakes:
|
||||
a **6h per‑run** max‑lifetime (`AgentMaxLifetimeMinutes`, re‑armed each reboot) that emits an
|
||||
explicit `enrollment_failed(max_lifetime)` → the session is already `Failed` at ~6h; and a
|
||||
**48h absolute session‑age emergency break** (`AbsoluteMaxSessionHours`, cumulative across
|
||||
reboots, skipped while WhiteGlove‑paused) that is **silent to the backend** — it writes a local
|
||||
marker, cleans up and exits without any terminal event. So a session only stays *non‑terminal*
|
||||
past 48h precisely when the agent hit that silent break and is now provably gone.
|
||||
|
||||
Therefore the grace must be **≥ the agent's absolute cap** (never terminalize `Incomplete`
|
||||
while the agent could still legitimately be enrolling) and only slightly beyond it (once the
|
||||
cap has fired + last spooled telemetry has had time to land, silence = dead):
|
||||
|
||||
```
|
||||
effectiveGrace = max( AbsoluteMaxSessionHours + bufferHours , SessionGraceHours override )
|
||||
= max( 48 + 3 , 0 ) = 51h (defaults)
|
||||
```
|
||||
|
||||
The buffer is deliberately **small (3h)**, not a long straggler‑wait: a completion can only ever
|
||||
arrive ≤ the cap (the agent self‑destructs at the cap and sends nothing after), so the buffer only
|
||||
has to span one 2h maintenance‑sweep cycle + absorb minor clock skew. The reconcile path heals any
|
||||
prematurely‑set `Incomplete` back to `Succeeded`, so a tight buffer costs at most a brief flicker.
|
||||
|
||||
`SessionGraceHours` defaults to `0` (auto‑derive) and can only *raise* the grace above the
|
||||
floor, never below it (`EnrollmentTimeoutClassifier.ResolveGraceHours`). `AbsoluteMaxSessionHours`
|
||||
is mirrored into `TenantConfiguration` so the floor follows any tenant override of the agent cap.
|
||||
|
||||
**Agent emergency‑break signal (closes the blind spot).** The 48h break is otherwise silent to the
|
||||
backend. The agent best‑effort sends an emergency‑break report (resilient channel — may be lost if
|
||||
the device is fully offline); the backend materializes an `agent_emergency_break` timeline event
|
||||
from it and treats it as a definitive "agent is gone" signal → terminalize **now** by the ESP rollup
|
||||
(Account Setup all‑succeeded → `Succeeded`, else `Incomplete`), skipping the grace wait. With this
|
||||
signal the grace/buffer only governs the *truly silent* death (bluescreen / power‑cut with no
|
||||
network), where no event can ever be sent.
|
||||
|
||||
The waiting session is just a table row compared against a timestamp — no process, no
|
||||
telemetry, no agent change.
|
||||
|
||||
### 4. Reconciliation on late events
|
||||
|
||||
When a session in `Failed` / `Incomplete` / `AwaitingUser` later receives a genuine terminal
|
||||
signal (`enrollment_complete`, `whiteglove_complete`, or AccountSetup rollup = all
|
||||
succeeded), flip to **`Succeeded`**
|
||||
([`EventIngestProcessor.Classification.cs`](../../src/Backend/AutopilotMonitor.Functions/Services/EventIngestProcessor.Classification.cs)
|
||||
already routes `enrollment_complete` → `Succeeded`; the `UpdateSessionStatusAsync` guard must
|
||||
permit an already‑terminal → `Succeeded` upgrade for these reconcile signals only).
|
||||
|
||||
### 5. Stats / metrics — expose 3 states, exclude Incomplete from failure rate
|
||||
|
||||
- `SessionStatusBuckets` ([`MetricsMath.cs:317`](../../src/Backend/AutopilotMonitor.Functions/Helpers/MetricsMath.cs))
|
||||
gains explicit `AwaitingUser` and `Incomplete` buckets (they currently fall into `Other`).
|
||||
- `SessionStats` / Fleet Health: add `IncompleteLastNDays`; **failure rate denominator =
|
||||
`Succeeded + Failed` only** (Incomplete and AwaitingUser excluded). Headline becomes
|
||||
Success / Failed / Incomplete‑Unknown.
|
||||
- Same `Incomplete` bucket added to the other reporting surfaces that read session status:
|
||||
the live usage metrics (`UsageMetricsService.SessionMetrics`, both platform + tenant paths,
|
||||
feeding `get_usage_metrics` and the usage dashboards) and the persisted daily
|
||||
`UsageMetricsSnapshot` (`SessionsIncomplete`, stored + mapped in `TableStorageService.Metrics`).
|
||||
Their success rate already used the `Succeeded + Failed` denominator; only the count bucket
|
||||
was missing, so terminal Incomplete sessions were silently absent from the breakdown.
|
||||
- Web: `SessionStatus` mirror + badge + `isTerminalStatus`
|
||||
([`utils/sessionStatus.ts`](../../src/Web/autopilot-monitor-web/utils/sessionStatus.ts),
|
||||
[`SessionStatusBadge.tsx`](../../src/Web/autopilot-monitor-web/components/SessionStatusBadge.tsx)).
|
||||
|
||||
## Expected impact (crcins.com, from the measured buckets)
|
||||
|
||||
| | today | after |
|
||||
|---|---:|---:|
|
||||
| Succeeded | 599 | ~624 (+25 reconciled) |
|
||||
| Failed | 1311 | ~122 (explicit failures + max‑lifetime) |
|
||||
| AwaitingUser (new, transient) | – | ~1052 |
|
||||
| Incomplete/Unknown (new) | – | ~115 |
|
||||
| **Failure rate** | **57.2 %** | **~4–5 %** |
|
||||
|
||||
Platform‑wide the same reclassification de‑noises every tenant's failure rate.
|
||||
|
||||
## Non‑goals / explicitly rejected
|
||||
|
||||
- **Agent heartbeat** — rejected: continuous client load + zombie risk; the completion
|
||||
truth is already persisted (registry/ESP tracking survives reboots) and the wait belongs
|
||||
in the backend.
|
||||
- **`desktop_arrived` ALONE as completion** — rejected: fires while the user phase is still
|
||||
running. (`desktop_arrived` **+ a positive Hello terminal** is accepted since 2026-07-10 —
|
||||
that pair is the agent's own Classic completion AND‑gate, see classification step 4.)
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- No non‑terminal session is marked `Failed` by the sweep without an explicit failure event.
|
||||
- A session reaching AccountSetup all‑succeeded (or `enrollment_complete`) after a prior
|
||||
sweep ends as `Succeeded`.
|
||||
- `Failed` count only contains explicit‑failure sessions; failure‑rate excludes Incomplete.
|
||||
- Reprocessing the crcins.com window reproduces the "after" table within rounding.
|
||||
@@ -290,7 +290,7 @@ namespace AutopilotMonitor.Agent.V2
|
||||
|
||||
// Best-effort: tell the backend the agent is emergency-breaking so the otherwise-silent
|
||||
// 48h absolute cap is no longer a blind spot in the timeline
|
||||
// (docs/design/enrollment-status-reclassification.md). Fired BEFORE cleanup, while the
|
||||
// (tasks/enrollment-status-reclassification.md). Fired BEFORE cleanup, while the
|
||||
// session state and network are still intact. Never throws — a send failure (e.g. no
|
||||
// network) must not block the cleanup/exit that is the whole point of this guard.
|
||||
try { onBreakFired?.Invoke(); }
|
||||
|
||||
@@ -464,7 +464,7 @@ namespace AutopilotMonitor.Agent.V2
|
||||
}
|
||||
|
||||
// internal (not private) so AgentBootstrap can stamp the best-effort emergency-break report
|
||||
// with the running agent version (docs/design/enrollment-status-reclassification.md).
|
||||
// with the running agent version (tasks/enrollment-status-reclassification.md).
|
||||
internal static string GetAgentVersion()
|
||||
{
|
||||
try
|
||||
|
||||
@@ -85,7 +85,7 @@ namespace AutopilotMonitor.Agent.V2.Runtime
|
||||
agentConfig.AbsoluteMaxSessionHours, agentConfig.SelfDestructOnComplete,
|
||||
cleanupServiceFactory, logger, consoleMode,
|
||||
// Best-effort: surface the otherwise-silent 48h break to the backend before cleanup
|
||||
// (docs/design/enrollment-status-reclassification.md). Never blocks the exit.
|
||||
// (tasks/enrollment-status-reclassification.md). Never blocks the exit.
|
||||
onBreakFired: () => EmergencyBreakReporter.TrySend(agentConfig, Program.GetAgentVersion(), logger)))
|
||||
{
|
||||
logger.Info("Emergency break fired — agent exiting.");
|
||||
|
||||
@@ -14,7 +14,7 @@ namespace AutopilotMonitor.Agent.V2.Runtime
|
||||
/// (<c>ReportAgentErrorFunction</c>), closing the silent-48h blind spot and letting the timeout
|
||||
/// classifier terminalize the session instead of waiting out the grace. Fully swallowed — a send
|
||||
/// failure (no network, no cert) must never delay the cleanup/exit the break exists to perform.
|
||||
/// See docs/design/enrollment-status-reclassification.md.
|
||||
/// See tasks/enrollment-status-reclassification.md.
|
||||
/// </summary>
|
||||
internal static class EmergencyBreakReporter
|
||||
{
|
||||
|
||||
@@ -6,7 +6,7 @@ using AutopilotMonitor.Shared.Models;
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Tests for the timeout reclassification (docs/design/enrollment-status-reclassification.md).
|
||||
/// Tests for the timeout reclassification (tasks/enrollment-status-reclassification.md).
|
||||
/// The maintenance sweep must stop labelling every silent session Failed: a session whose
|
||||
/// Account Setup rollup reached all-succeeded reconciles to Succeeded; one that finished
|
||||
/// Device Setup but whose user phase never completed is AwaitingUser (within grace) then
|
||||
|
||||
@@ -8,7 +8,7 @@ namespace AutopilotMonitor.Functions.Tests;
|
||||
/// <summary>
|
||||
/// Pins <see cref="ReportAgentErrorFunction.BuildAgentEmergencyBreakEvent"/> — the backend-materialized
|
||||
/// timeline event synthesized from the agent's best-effort 48h emergency-break report
|
||||
/// (docs/design/enrollment-status-reclassification.md). The load-bearing bits are the Sequence assignment
|
||||
/// (tasks/enrollment-status-reclassification.md). The load-bearing bits are the Sequence assignment
|
||||
/// (must sort AFTER the session's last event) and that it is a Warning, non-terminal marker — the timeout
|
||||
/// classifier, not this event, decides the real verdict.
|
||||
/// </summary>
|
||||
|
||||
@@ -3,7 +3,7 @@ using AutopilotMonitor.Functions.Helpers;
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Pins the third-state stats bucketing (docs/design/enrollment-status-reclassification.md):
|
||||
/// Pins the third-state stats bucketing (tasks/enrollment-status-reclassification.md):
|
||||
/// AwaitingUser and Incomplete get their own buckets instead of hiding in Other, and the buckets
|
||||
/// always reconcile to Total by construction — so a session in either state can never be silently
|
||||
/// miscounted as a failure.
|
||||
|
||||
@@ -4,7 +4,7 @@ using AutopilotMonitor.Shared.Models;
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Pins the terminal/reconcile transition matrix (docs/design/enrollment-status-reclassification.md):
|
||||
/// Pins the terminal/reconcile transition matrix (tasks/enrollment-status-reclassification.md):
|
||||
/// a genuine completion may upgrade a prior Failed/Incomplete/AwaitingUser verdict (late reconcile),
|
||||
/// while the silent-terminal verdicts never overwrite an existing terminal. This is the pure core of
|
||||
/// PR3 — the reconcile that lets a device which completed after a sweep-timeout heal to Succeeded.
|
||||
|
||||
@@ -118,7 +118,7 @@ public class UsageMetricsServiceWindowTests
|
||||
[Fact]
|
||||
public async Task Incomplete_sessions_get_their_own_bucket_and_are_excluded_from_success_rate()
|
||||
{
|
||||
// The reclassification shape (docs/design/enrollment-status-reclassification.md): timed-out
|
||||
// The reclassification shape (tasks/enrollment-status-reclassification.md): timed-out
|
||||
// sessions become terminal Incomplete, not Failed. They must be counted in their own bucket
|
||||
// (previously invisible: in Total but no breakdown) and kept out of the failure-rate
|
||||
// denominator — Succeeded + Failed only. 1 succeeded, 1 failed, 8 incomplete -> 50%, not 10%.
|
||||
|
||||
@@ -170,7 +170,7 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
|
||||
});
|
||||
|
||||
// Materialize the agent's silent 48h emergency break as a timeline event so it shows in the
|
||||
// session and the timeout classifier can see it (docs/design/enrollment-status-reclassification.md).
|
||||
// session and the timeout classifier can see it (tasks/enrollment-status-reclassification.md).
|
||||
// Best-effort — a failure here must never turn the always-200 emergency channel into a retry loop.
|
||||
if (report.ErrorType == AgentErrorType.SessionAgeEmergencyBreak
|
||||
&& !string.IsNullOrEmpty(report.SessionId))
|
||||
|
||||
@@ -7,7 +7,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
{
|
||||
/// <summary>
|
||||
/// Pure classification of a session that the maintenance sweep is about to terminalize
|
||||
/// (docs/design/enrollment-status-reclassification.md). Instead of hard-coding every
|
||||
/// (tasks/enrollment-status-reclassification.md). Instead of hard-coding every
|
||||
/// silent session to <see cref="SessionStatus.Failed"/>, we read the ESP subcategory
|
||||
/// rollup the agent already emits and decide the honest target state.
|
||||
///
|
||||
|
||||
@@ -26,7 +26,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
{
|
||||
// v2 (2026-07-08): added the ESP DeviceSetup/AccountSetup subcategory rollup fields
|
||||
// (deviceSetupAllSucceeded / accountSetup*) that the timeout reclassification uses —
|
||||
// see docs/design/enrollment-status-reclassification.md.
|
||||
// see tasks/enrollment-status-reclassification.md.
|
||||
// v3 (2026-07-10): added helloResolved + realmJoinDetected/realmJoinResolved +
|
||||
// skipUserEsp — the evidence behind the "user completed setup" reconcile rule
|
||||
// (session 294ab5b4: user reached desktop + finished Hello, only the RealmJoin gate
|
||||
|
||||
@@ -367,7 +367,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
|
||||
// Timeout ≠ failure. Classify from the ESP subcategory rollup the agent already
|
||||
// emits instead of blindly failing every silent session
|
||||
// (docs/design/enrollment-status-reclassification.md): explicit failure → Failed;
|
||||
// (tasks/enrollment-status-reclassification.md): explicit failure → Failed;
|
||||
// Account Setup all-succeeded / enrollment_complete → Succeeded (reconcile);
|
||||
// desktop + positive Hello terminal observed → Succeeded (user provably finished
|
||||
// setup; only the completion report never left the device — session 294ab5b4);
|
||||
|
||||
@@ -564,7 +564,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
// At the 5h mark the session is reclassified rather than blindly failed — see
|
||||
// EnrollmentTimeoutClassifier. AwaitingUser is included too so it keeps being
|
||||
// re-evaluated each pass and graduates to Incomplete once SessionGraceHours elapses
|
||||
// (docs/design/enrollment-status-reclassification.md).
|
||||
// (tasks/enrollment-status-reclassification.md).
|
||||
// IsPreProvisioned ne true → WhiteGlove sessions are intentionally long-lived and
|
||||
// must never be timed out, even after months in storage.
|
||||
var filter = $"PartitionKey eq '{tenantId}' " +
|
||||
|
||||
@@ -1383,7 +1383,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
/// reflects when the session went silent rather than when the operator clicked the button.
|
||||
/// </param>
|
||||
/// <summary>
|
||||
/// Pure gate for the terminal/reconcile status rules (docs/design/enrollment-status-reclassification.md).
|
||||
/// Pure gate for the terminal/reconcile status rules (tasks/enrollment-status-reclassification.md).
|
||||
/// Returns whether a write to <paramref name="incoming"/> may proceed given the persisted
|
||||
/// <paramref name="existingStatus"/> string:
|
||||
/// <list type="bullet">
|
||||
@@ -1427,7 +1427,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
var entity = await tableClient.GetEntityAsync<TableEntity>(tenantId, sessionId);
|
||||
var session = entity.Value;
|
||||
|
||||
// Terminal / reconcile transition gate (docs/design/enrollment-status-reclassification.md):
|
||||
// Terminal / reconcile transition gate (tasks/enrollment-status-reclassification.md):
|
||||
// a genuine completion (Succeeded) may UPGRADE a prior Failed/Incomplete/AwaitingUser
|
||||
// verdict — a device that reached the desktop is enrolled regardless of an earlier timeout
|
||||
// guess — but the silent-terminal verdicts (Failed/Incomplete) and AwaitingUser never
|
||||
|
||||
@@ -306,7 +306,7 @@ namespace AutopilotMonitor.Shared
|
||||
public const string AgentStarted = "agent_started"; // Lifecycle anchor — fired Seq=1 at agent boot. PR1: replaces hardcoded string-literals at emit sites.
|
||||
public const string AgentShuttingDown = "agent_shutting_down"; // V2 single-rail plan §6.2 — terminate-hygiene acknowledgement emitted before CleanupService tears down
|
||||
// Backend-materialized from the agent's best-effort SessionAgeEmergencyBreak error report
|
||||
// (docs/design/enrollment-status-reclassification.md). The agent's 48h absolute session-age
|
||||
// (tasks/enrollment-status-reclassification.md). The agent's 48h absolute session-age
|
||||
// break (Program.Guards.CheckSessionAgeEmergencyBreak) is otherwise silent to the backend; this
|
||||
// event closes that blind spot in the timeline AND tells the timeout classifier the agent is
|
||||
// definitively gone, so the session is terminalized now (by ESP rollup) instead of waiting out grace.
|
||||
|
||||
@@ -178,7 +178,7 @@ namespace AutopilotMonitor.Shared.Models
|
||||
/// <summary>
|
||||
/// Session inactivity timeout in hours. Once an "InProgress" session is idle past this, the
|
||||
/// maintenance sweep reclassifies it out of "InProgress" (see
|
||||
/// docs/design/enrollment-status-reclassification.md): if Device Setup already finished it
|
||||
/// tasks/enrollment-status-reclassification.md): if Device Setup already finished it
|
||||
/// becomes AwaitingUser (non-terminal), otherwise it eventually settles as the terminal,
|
||||
/// non-failure Incomplete state once <see cref="SessionGraceHours"/> elapses — it is NOT
|
||||
/// counted as a failure. This prevents stalled sessions from running indefinitely and skewing statistics.
|
||||
@@ -189,7 +189,7 @@ namespace AutopilotMonitor.Shared.Models
|
||||
|
||||
/// <summary>
|
||||
/// Grace window in hours for a session that reached the inactivity timeout with Device Setup
|
||||
/// already provisioned but no completion signal yet (docs/design/enrollment-status-reclassification.md).
|
||||
/// already provisioned but no completion signal yet (tasks/enrollment-status-reclassification.md).
|
||||
/// At <see cref="SessionTimeoutHours"/> such a session becomes AwaitingUser (non-terminal) instead
|
||||
/// of Failed; only after this window elapses without a completion does it graduate to the terminal,
|
||||
/// non-failure Incomplete state.
|
||||
|
||||
@@ -35,7 +35,7 @@ namespace AutopilotMonitor.Shared.Models
|
||||
/// cleaning up and exiting. Best-effort report over the resilient emergency channel (may be
|
||||
/// lost if the device is fully offline) so the backend can materialize an
|
||||
/// <c>agent_emergency_break</c> timeline event and terminalize the session instead of waiting
|
||||
/// out the silence grace. See docs/design/enrollment-status-reclassification.md.
|
||||
/// out the silence grace. See tasks/enrollment-status-reclassification.md.
|
||||
/// </summary>
|
||||
SessionAgeEmergencyBreak = 4,
|
||||
}
|
||||
|
||||
@@ -83,7 +83,7 @@ namespace AutopilotMonitor.Shared.Models
|
||||
/// Terminal, non-failure sessions (timeout reclassification): the sweep classified them as
|
||||
/// Incomplete instead of Failed. Surfaced as its own count and deliberately excluded from
|
||||
/// <see cref="SuccessRate"/> (denominator = Succeeded + Failed only), mirroring SessionStats
|
||||
/// and FleetHealthStats. See docs/design/enrollment-status-reclassification.md §5.
|
||||
/// and FleetHealthStats. See tasks/enrollment-status-reclassification.md §5.
|
||||
/// </summary>
|
||||
public int Incomplete { get; set; }
|
||||
public double SuccessRate { get; set; }
|
||||
|
||||
@@ -56,7 +56,7 @@ namespace AutopilotMonitor.Shared.Models
|
||||
/// Terminal, non-failure sessions (timeout reclassification): classified Incomplete rather
|
||||
/// than Failed. Tracked separately and excluded from <see cref="SessionsSuccessRate"/>
|
||||
/// (denominator = Succeeded + Failed). Historical rows written before this field existed
|
||||
/// deserialize to 0. See docs/design/enrollment-status-reclassification.md §5.
|
||||
/// deserialize to 0. See tasks/enrollment-status-reclassification.md §5.
|
||||
/// </summary>
|
||||
public int SessionsIncomplete { get; set; }
|
||||
|
||||
|
||||
@@ -431,7 +431,7 @@ namespace AutopilotMonitor.Shared.Models
|
||||
Succeeded,
|
||||
Failed,
|
||||
Unknown,
|
||||
// Appended 2026-07-08 (docs/design/enrollment-status-reclassification.md). New members
|
||||
// Appended 2026-07-08 (tasks/enrollment-status-reclassification.md). New members
|
||||
// MUST stay appended so existing persisted ordinals never shift.
|
||||
AwaitingUser, // Device Setup (ESP DeviceSetup) fully succeeded but the user/Account-Setup
|
||||
// phase has not completed and the agent went silent within the grace window.
|
||||
@@ -464,7 +464,7 @@ namespace AutopilotMonitor.Shared.Models
|
||||
public int FailedLastNDays { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Terminal, non-failure sessions in the window (docs/design/enrollment-status-reclassification.md):
|
||||
/// Terminal, non-failure sessions in the window (tasks/enrollment-status-reclassification.md):
|
||||
/// the sweep saw no completion or explicit failure. Reported as the third headline bucket and
|
||||
/// deliberately excluded from <see cref="SuccessRatePct"/> (which is over Succeeded + Failed only).
|
||||
/// </summary>
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
//
|
||||
// AwaitingUser (Device Setup done, user/Account-Setup phase pending) and Incomplete
|
||||
// (terminal, non-failure — no completion or explicit failure was observed) come from the
|
||||
// timeout reclassification, see docs/design/enrollment-status-reclassification.md.
|
||||
// timeout reclassification, see tasks/enrollment-status-reclassification.md.
|
||||
|
||||
export function isTerminalStatus(status: string | null | undefined): boolean {
|
||||
return status === "Succeeded" || status === "Failed" || status === "Incomplete";
|
||||
|
||||
Reference in New Issue
Block a user