Fresh cross-tenant usage computes took 20-43s server-side but the web app's
default 30s fetch timeout aborted them client-side (HTTP 499), and every
retry stacked another concurrent full compute.
Backend:
- UsageMetricsService: cache 5→15min, single-flight per days-window (retries
join the in-flight compute), the four source scans run concurrently via
Task.WhenAll, statics converted to instance fields (singleton, testable)
- New column-projected queries for the usage compute only:
GetUsageWindowSessionsAsync ($select 13 consumed columns, drops
FailureSnapshotJson & co.) and GetAppInstallRefsAsync ($select
SessionId+AppName, lean SessionAppRef DTO); GetSessionsByDateRangeAsync
is untouched for its other consumers
- Projection equivalence pinned by UsageMetricsProjectionEquivalenceTests
(incl. the WhiteGlove Part-2 duration branch via IsPreProvisioned/
ResumedAt); single-flight/cache/failure-eviction pinned by
UsageMetricsServiceSingleFlightTests
Web (SectionPlatformUsage):
- 180s AbortSignal for this call (backend keeps computing and caches)
- Calculating card with elapsed time and a progress bar sized from the last
fresh-compute duration (localStorage, cache hits excluded, capped at 95%)
- Friendly timeout error explaining the retry will hit the server cache
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- IncrementPlatformStatAsync("TotalEnrollments") fired on EVERY
re-register (agent restarts, WG Part 2 resumes) — the nightly
recompute corrected it, but the public stat visibly overcounted in
between. Now gated on isFreshRegistration alongside the per-tenant
counter (one shared block)
- MCP get_usage_metrics: the tool passes the backend response through,
so sessions.totalAllTime flows automatically — extend the tool
description so callers know the field is cumulative since signup and
retention-independent; bump MCP server to 1.5.1
Backend tests 2876/2876, MCP tsc clean + vitest 393 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
"Total Sessions" on the usage-metrics page is window-bound (90d default)
and capped by retention. Add a retention-independent per-tenant counter
surfaced as "Total Enrollments (since signup)".
- TenantStats row in the PlatformStats table (PK=tenantId, RK="current";
the platform row's "global" PK cannot collide with a tenant GUID)
- Incremented at session registration, fire-and-forget, fresh
registrations only (agent restarts / WG Part 2 resumes re-register the
same session and must not inflate the counter)
- ETag CAS with retries instead of last-writer-wins: lost increments are
permanent (no recompute can recover pruned sessions), and rollout
waves register concurrently
- Nightly platform-stats recompute seeds/self-heals the counter to at
least the live session count (raise-only floor, analog TotalUsers clamp)
- Tenant usage metrics expose SessionMetrics.TotalAllTime =
max(counter, window total); web page shows a "Total Enrollments" card
plus clarifying window note ("Total Sessions" is not a lifetime total)
Backend tests 2876/2876, tsc clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reorganize docs/ tree:
- docs/design/mcp-oauth-flow.md -> docs/mcp-oauth-flow.md (no customer data)
- enrollment-status-reclassification{,-plan}.md -> tasks/ (git-ignored work
files; they contained a customer domain that must not ship in the public repo)
- remove now-empty docs/design/ folder
Repoint the 22 source files whose comments referenced
docs/design/enrollment-status-reclassification.md to the new tasks/ path.
Comment/docstring text only; no functional code changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sequence diagram and explanation of the dual-identity connect flow (AI-product
account vs. Entra work account), including the frequently-missed requirement
that the default browser must hold an active vendor session (claude.ai) for
the callback in step 6 to be accepted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Medium finding: the anonymous dynamic-client-registration allowlist accepted
any path on whole vendor domains plus wildcard subdomains, so an open redirect
(or controllable subdomain) on an allowed domain could exfiltrate authorization
codes — PKCE does not help when the attacker runs the flow with their own
verifier. Defaults are now the documented callback URIs only (claude.ai +
claude.com /api/mcp/auth_callback, chatgpt.com /connector_platform_oauth_redirect,
vscode.dev + insiders.vscode.dev /redirect); gemini.google.com dropped (Gemini
CLI uses loopback). MCP_ALLOWED_REDIRECT_HOSTS gains optional /path templates
(host-only entries still work). redirect_uris with userinfo or fragments are
rejected outright, loopback included; localhost stays allowed because Claude
Code and Gemini CLI register http://localhost callbacks.
Low finding: /oauth/token now sets Cache-Control: no-store + Pragma: no-cache
on every response; Entra error logging is reduced to status/error/error_codes/
correlation_id; all user-controlled log fields (client_name, redirect_uri) pass
through sanitizeForLog (control-char strip + length cap).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A periodic collector (e.g. StartupEnvironmentProbes power_state_check) can flush
one last event AFTER whiteglove_part1_complete, still from the draining Part-1
agent. computeWhiteGloveSplitSequence's pre-provisioning-only branch split at the
last cleanup marker, stranding such a straggler into a phantom "User Enrollment /
Resumed" block despite no whiteglove_resumed and no Part-2 agent_started.
Restore the invariant: no Part-2 boot evidence => no User Enrollment part. That
branch now splits at the highest sequence in the session so every trailing
straggler stays in Pre-Provisioning; the session renders the accurate "Awaiting
User Enrollment" placeholder instead. Adds a regression test from field session
1293f80e. Existing case-3 tests unaffected (marker was already the max sequence).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
IME writes its "[Win32App] .... Completed user session N ...." line at the
end of EVERY user-session processing pass, not when user-scope enforcement
is finished. Field sessions 14690fc2 and 6cb01530 (2026-07-10, WhiteGlove
Part 2 + mid-AccountSetup reboot) matched the line on the first post-reboot
pass with 13-18 required apps still pending; arm C of
ShouldTransitionToAwaitingHello took the signal as its fourth fact and the
session completed Succeeded five minutes later (HelloSafety timeout ->
FinalizingGrace) mid-install, spraying app_install_starved events.
Harden at the signal source: ImeLogTrackerAdapter.EmitUserSessionCompleted
now consults the new ImeLogTracker.GetPendingRequiredUserEspInstallApps()
probe (AccountSetup phase, Install intent, not terminal) and defers the
decision signal while the list is non-empty. The fire-once flag stays
unclaimed, so the completion line of a later pass re-evaluates and posts
once the apps have settled. Error counts as terminal (GRS owns retries) so
a permanently failed app cannot park the signal - and with it the
AdvisoryCompletion conjunction - into a 30-min-backstop false Failed.
Non-AccountSetup completions post as before (pre-anchor stamps stay
harmless via IsImeUserSessionGenuine).
Observability for the field watch: INFO log line per suppressed pass plus
a one-shot agent_trace (decision=ime_user_session_completion_deferred,
pending count + capped names) on the session timeline.
Session 04e8d52e
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Session 7443317c proved a WU-tracker blind spot: an OOBE quality update
installed (OsBuild 26200.8037 -> 26200.8655 across the mid-enrollment reboot)
yet the deployed WindowsUpdateTracker emitted ZERO events - the watcher armed
fine (no collector_degradation), the spool is durable, and two 60-min backfills
covered the install window, so the WindowsUpdateClient channel simply carried
none of the targeted EventIDs 19/20/43/44. The tenant has diagnostics upload
disabled, so the fix makes the feature self-evidencing instead of guessing IDs:
- OsBuildChangeHost: persists CurrentBuild.UBR (Registry64 view) to
AgentState/os-build-state.json and emits os_build_changed (Info, immediate)
when it differs across a restart - deterministic update corroboration that
cannot be lost to timing, watermarks, or unexpected servicing paths. First
run seeds silently; corrupt state re-seeds.
- WindowsUpdateTracker channel census: when the build provably changed AND zero
targeted WU events were emitted this run, emit a one-shot
windows_update_channel_census (Debug) with an unfiltered EventID histogram of
the WindowsUpdateClient AND UpdateOrchestrator channels over the lookback
window (capped at 5000 records, cap surfaced as censusTruncated). The
histogram arrives via normal event upload and tells us which IDs to add to
WindowsUpdateTargetedEventIds via remote config - no redeploy, no diag ZIP.
Gated by CollectorConfiguration.WindowsUpdateChannelCensusEnabled (default
on, model-default pattern, no ConfigVersion bump).
- No new hardcoded EventIDs (per the device-validation rule) - census data
decides what to enable remotely.
- ANALYZE-DEV-006 "OS Build Changed During Enrollment" (info, not KO):
interpolates {{previousBuild}}/{{currentBuild}}, counts the census as
corroboration. Catalogs: Constants.cs + MCP resource-catalog.ts + web
eventTypes.ts (drift-test enforced).
Tests: 6 OsBuildChangeHost (seed/same/changed/unreadable/corrupt/no-dir),
7 census (gating matrix, truncation surfaced, scanner-throw swallowed,
histogram format), 3 ANALYZE-DEV-006. Agent.V2.Core 1754/1754, Backend
2874/2874, MCP 379 (drift green), web tsc + vitest 367/367, sln 0 errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 9a107a19 reboot-cancel guard correctly stopped false
esp_exit_without_completion_evidence failures, but a pure cancel left the
post-AccountSetup dead-end zone with no resolution-capable deadline: session
7443317c (same tenant/shape as 1924092e, plus a mid-OOBE Windows-Update reboot
chain) tripped the session_parked_without_deadline liveness tripwire on a live
enrollment that later completed normally.
RebaseEspExitVariantAdvisoryWindowOnReboot now replaces the stale window with a
fresh 30-min one whose enforcement baselines (armSignalOrdinal /
armAppTerminalCount) anchor at the reboot signal, from both
HandleSystemRebootObservedV1 and HandleEspResumedV1:
- A live enrollment keeps disproving the dead-end at each fire via
HasEnforcementProgressSinceArming (re-arm, never fail).
- A true dead-end where the user rebooted inside the window now resolves one
window later (30-60 min) instead of idling to the 6-h max-lifetime watchdog -
the 9a107a19 trade-off gap is closed.
- The parked tripwire needs no change: a resolution-capable deadline is armed
throughout, so IsParkedWithoutDeadline stays false and the tripwire keeps its
"every fire is a bug" charter.
- The rebase emits a completion_waiting with the new resolutionDeadlineDueAtUtc
(trigger <Signal>:AdvisoryRebase), deliberately bypassing the fingerprint
dedupe like the EnforcementActive re-arm site - bounded to one per reboot.
Tests: rebase with reboot-anchored baselines (both call sites), advisory
variant exempt, no-window no-op, true-dead-end-after-reboot fails at the
rebased due-time, 7443317c double-reboot live-install replay, fire-once keeps
the rebased window, 1924092e full replay updated (stray fire re-arms on
progress), engine-driven parked-tripwire regression.
DecisionCore 445/445, Agent.V2.Core 1754/1754.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Session a4537c36: Windows closed the User-ESP page normally (Shell-Core 62407,
errorCode=0) while the AccountSetup registry never wrote categorySucceeded -
the strong gate (arm A) was unsatisfiable by construction and the session only
completed because the Hello policy happened to flip to enabled after a reboot.
ShouldTransitionToAwaitingHello gains arm C: AccountSetup entered + normal ESP
final exit (post-AccountSetup by INGEST ordinal - backdated CMTrace timestamps
are irrelevant, sequence is canonical) + genuine IME user-session completion
(defaultuser0 guard: at-or-after the AccountSetup anchor, extracted as shared
IsImeUserSessionGenuine and reused by the AdvisoryCompletion backstop) + DAD-
validated real-user desktop. ALL FOUR mandatory; Hello gating stays at the
call sites - arm C never bypasses an enabled-unresolved Hello.
Wiring per signal ordering: desktop last via the fast-path (desktopArrived-
InFlight), exit last via HandleEspExitingV1 (espFinalExitInFlight + deferred-
completion parity mirroring the caa6cf50 branch), IME last via new completion/
promote wiring in HandleImeUserSessionCompletedV1. completion_waiting stops
reporting account_setup_provisioning_complete once the evidence set holds.
AdvisoryCompletion stays as the 30-min backstop for incomplete evidence.
The 08c99638 regression guard (3-of-4, no IME fact -> parked) stays green
unmodified; the L5 backdated-exit test now asserts the eager completion.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Session a4537c36: the agent could not tell ESP-blocking apps from merely-required
IME assignments. New EspTrackingInfoProbe reads the ESP''s own tracking lists from
HKLM\...\Autopilot\EnrollmentStatusTracking\ESPTrackingInfo\Diagnostics (device
scope + per-SID user scope; categories ExpectedMSIAppPackages / Expected-
ModernAppPackages / Sidecar, deduped across timestamped status-write subkeys)
and attaches them to esp_config_detected: espTrackedWin32AppIds (+ user-scoped
subset), espTrackedMsiProductCodes, espTrackedModernAppPfns, plus uncapped
counts (lists capped at 50). Registry semantics per Get-AutopilotDiagnostics.ps1.
Sidecar GUIDs are normalized to lowercase dashed form so they string-match
AppPackageState.Id - prepared for the espBlocking annotation follow-up on
app_install_starved (deliberately not wired yet; collect field data first).
Fail-soft like EspSkipConfigurationProbe; keys fully omitted when the
Diagnostics key is absent. Decision-signal payload untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Session a4537c36 debug: 4 of 6 app_install_starved warnings were Uninstall-intent
system-app removals (Xbox, WMP, Ubuntu) - the ESP apps gate never blocks on
uninstalls and "never started installing" is meaningless for them. Filter at the
single chokepoint ImeLogTracker.GetStarvedUserEspApps so both emission paths
(live esp_exited + termination sweep) are covered.
The termination sweep also claimed "the app starved the enrollment completion"
on terminationOutcome=Succeeded - contradictory on a successful session. Now
Info severity + neutral wording (installs continue via IME post-ESP); Failed/
TimedOut keep the Warning verdict. Data fields unchanged.
AreUserEspAppsSettled deliberately untouched (completion behaviour stays
conservative until field data proves the ESP never waits on uninstalls).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 1ec8f4c6 dead-end window (guard-blocked post-AccountSetup esp_exiting arms a
30-min advisory_completion resolution deadline) misfired on the Device->Account
handoff exit whenever the IME logged an AccountSetup phase line pre-sign-in
(defaultuser0/AutoLogon frame). The recovered deadline then failed sessions
mid-AccountSetup while apps were actively installing:
enrollment_failed: esp_exit_without_completion_evidence on a live enrollment.
41 of the 65 affected sessions since 2026-06-15 show reboots or app activity
after the arming exit - false positives.
Two guards, esp-exit variant only (the advisory variant keeps its real-failure
un-defang semantics):
- Reboot cancel: SystemRebootObserved / EspResumed cancel the armed window -
an arming exit that predates a reboot was a pre-reboot page close, not the
"page closed, nothing happens" dead-end. A later genuine guard-blocked exit
re-arms a fresh window.
- Enforcement-progress re-arm: arming baselines (armSignalOrdinal,
armAppTerminalCount) ride on the deadline's FiresPayload (replay-deterministic,
snapshot-round-trips, no DecisionState schema change). At fire time, apps
reaching terminal states or an ESP user-phase re-assert since arming re-arms
the window instead of failing. Convergent: each re-arm demands new progress;
legacy deadlines without baselines resolve with pre-fix semantics.
11 new tests incl. a full session-1924092e replay (handoff exit -> reboot ->
active installs -> stray recovered-timer fire dead-ends -> Completed) and pins
for the true dead-end verdict, the advisory variant, and baseline serialization.
DecisionCore 435/435, Agent.V2.Core 1725/1725.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Multiple matched conditions often extract different fields from the same
event (e.g. enrollment_failed + enrollment_failed_reason), which rendered
duplicate identically-labeled chips all jumping to the same timeline event.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A delegated caller previously had no way to resolve its managed tenants
GUIDs to display names - the tool catalog registered list_tenants only
for platform scope (GA/Reader), and the server instructions listed the
managed set as bare GUIDs. "Which tenants can I access?" answered with
GUIDs, and "search sessions of tenant contoso.com" had no resolution
path, although the backend (config/all, GlobalReadOrDelegatedSubset)
already serves the bounded, secret-redacted managed subset.
- register list_tenants for delegated callers with role-aware wording;
backend bounds the response, delegatedTenantListView re-intersects as
defense-in-depth (web analog: delegatedScopedTenantList)
- surface the caller's home tenant (isHome: true), synthesized on the
first page with the UPN-domain label when not in the managed subset
- carry the caller UPN through CallerContext for that label
- point tenantIdDescription, enforceDelegatedTenant errors and the
delegated instructions line at list_tenants for name resolution
- bump MCP server to 1.5.0
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Web analog of the MCP fix (88a432dc): a delegated admin who is also a
member/operator of their own home tenant could not see or read it in the
web app — every delegated surface bound the tenant list strictly to
delegatedTenantIds (which never contains the home tenant) and routed all
reads over /global/* endpoints, whose delegated fan-out is bounded to
the managed set and would return empty for the home tenant.
Fix is web-only; the backend already authorizes home-tenant reads on the
JWT-bound member path (MemberRead) and stays authoritative:
- utils/homeTenantScope.ts: isHomeTenantTarget, upnDomain, and
delegatedScopedTenantList — managed subset plus a synthesized home
entry (UPN-suffix domain, isHome flag) when the caller has a home
member role (user.role). Returns empty while config/all is still
loading so the render-time seeding cannot discard a persisted managed
selection in favor of an early home-only list.
- useGlobalAdminScope / useAggregatedAdminScope: home tenant in the
selector list; new routeGlobal flag (= isGlobalAdmin EXCEPT
delegated+home -> member path), mirroring MCP pickGlobalOrTenantPath.
- Pages switched from isGlobalAdmin to routeGlobal for endpoint choice:
analyze-rules (rule stats), audit, fleet-health (hook param renamed,
also drives SignalR scope), apps tabs (installs/inventory/vulns),
geographic-performance (+ sessions drill via homeDrill carve-out).
isGlobalOverride-keyed pages (gather-rules, analyze-rules list,
usage-metrics, sla) and apps/[appName] were already correct.
- Dashboard: home tenant in the filter autocomplete; session list and
stats route a home-tenant filter to the member endpoints before the
delegated bound (boundTenantToDelegatedScope itself stays strict).
- Fleet: home-tenant card ("your tenant" badge); useFleetSummaries
fetches the home summary via the member stats endpoint.
- TenantInfo.isHome + selector label "— your tenant".
Verify: tsc clean; vitest 556 passed (+13 homeTenantScope cases).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Follow-up to 049b6e5b: tenants running with Windows Hello disabled AND
the User ESP skipped can never produce a Hello terminal event, so the
reconcile rule would never fire for them and silent post-desktop
sessions stayed mislabeled AwaitingUser -> Incomplete. Mirror the
agent's Hello-disabled fast-path (HandleDesktopArrivedV1: policy
disabled + SkipUserEsp + desktop => complete):
- ExtractRollup reads hello_policy_detected Data.helloEnabled and
esp_config_detected Data.skipUserStatusPage (contradicting
observations resolve pessimistically to enabled/required).
- Rule 4 accepts desktop + (Hello terminal OR policy-disabled +
SkipUserEsp). Hello disabled WITHOUT SkipUserEsp keeps waiting —
the agent's strong post-AccountSetup gate (session 08c99638) blocks
its fast-path there too, and rule 2 covers the rollup-complete case.
- esp_exiting stays rejected as success evidence (page-lifecycle event,
fires with errorCode=0 while apps are unfinished — lesson 08c99638).
- Snapshot v3 + Web block additionally carry skipUserEsp.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The timeout classifier only read the ESP registry rollup, so a session
where the user reached the desktop and completed Windows Hello — with
only the RealmJoin completion gate still pending when the agent went
silent — was labeled AwaitingUser (and would have graduated to
Incomplete at grace expiry). Desktop arrival + a positive Hello terminal
are the agent's own Classic completion AND-gate; the RJ gate
self-releases after 60 min, so such a session's enrollment succeeded and
only the completion report never left the device.
- EnrollmentTimeoutClassifier: rollup now carries DesktopArrived,
HelloResolved (hello_provisioning_completed/hello_skipped only) and
RealmJoinDetected/Resolved (realmjoin_resolved/realmjoin_timeout);
new rule 4 reconciles desktop+Hello sessions to Succeeded with an
RJ-aware reason. Explicit failure and agent_emergency_break still win.
- FailureSnapshotBuilder: schema v3 embeds the new evidence
(helloResolved, realmJoinDetected, realmJoinResolved).
- Web FailureSnapshotBlock: renders Hello completed + RealmJoin rows.
- Design doc: decision table updated; desktop_arrived ALONE stays
rejected as a completion signal.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Follow-up hardening to 57e4334b (session b9b92d89). The runtime hash-mismatch
trigger runs the update pipeline on a thread-pool task concurrent with the live
orchestrator; RestartAgent then called Environment.Exit directly, leaving the
ProcessExit handlers only the CLR's ~2-3s window — far too short for the full
drain (ingress -> orchestrator.Stop -> spool flush -> snapshot). That kill
window is what produced the duplicated SignalLog line in the field.
Coordination via static hook (same idiom as BackendExpectedSha256):
- AgentRuntimeHost wires SelfUpdater.RequestGracefulShutdown BEFORE
orchestrator.Start (no window where the SignalLog is open but a restart
would hard-exit) and clears it in the shutdown finally
- Hook emits agent_shutting_down(reason=self_update_restart) through the
cross-path idempotency gate and signals the host loop -> normal
TerminationPipeline drain, process exits on its own
- RestartAgent: graceful path waits bounded (15s hard-exit fallback, well
under the restart script's 30s Wait-Process cap so the new agent never
races the old one into the multi-instance guard); startup-phase updates
(hook unset, no orchestrator) keep the original immediate exit
- Hook invocation is fail-safe: a race with the host's teardown (disposed
shutdown event) degrades to the immediate-exit fallback, which is safe
again once the orchestrator has stopped
Suites green: Agent.V2.Core 1724 (5 new hook-contract tests), DecisionCore 423.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A runtime self-update restart (runtime_hash_mismatch, force) killed the agent
mid-append: the final agent_shutting_down line landed byte-identically twice
(FileStream 4096-buffer re-flushed by Dispose after a faulted flush). The next
start's ReducerReplay threw non-monotonic ordinal (27 <= 27) and the uncaught
exception turned one crash artifact into a PERMANENT startup crash-loop.
Three surgical fixes:
- SignalLogWriter.Append: bufferSize 1 -> unbuffered write; Dispose can never
re-flush the payload (at-most-once physical append)
- SignalLogWriter.ReadAll: skip byte-identical consecutive duplicate lines
(crash artifact, not tamper); non-identical duplicate ordinals stay visible
- RecoveryCoordinator.Recover: catch InvalidOperationException from replay ->
quarantine all reducer segments + snapshot, reseed from Initial state
(mirrors the existing log-head-corrupt path) instead of fatal exit
E2E-verified against the real customer signal-log.jsonl: 29 lines -> 28
distinct signals -> replay OK (LastAppliedOrdinal=27, zero signal loss).
Suites green: Agent.V2.Core 1719, DecisionCore 423.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remediation rows showed "8m" for scripts that ran seconds: durationSeconds was
measured HS-SCRIPT-START -> HS-NEW-RESULT, and IME writes that line only after
its batched service report. The early-signal (HS-COMPLIANCE) completion carried
no duration at all because it never read the cycle-start map.
Agent:
- HS-COMPLIANCE early-signal completions now PEEK _healthScriptStartTimes and
emit the actual run time (durationBasis=script_runtime); HS-NEW-RESULT keeps
consuming the entry and stamps its phases cycle_including_reporting_latency.
- durationBasis now comes from the emitting handler (ScriptExecutionState.
DurationBasis) instead of pauschal per scriptType.
- Platform scripts emit the live script_started signal too (gated on pending-
slot creation so the duplicate agentexecutor+ime start line stays one signal).
Web:
- Reducer normalizes durationBasis: runtime -> durationSeconds, cycle ->
reportedAfterSeconds (legacy events without basis infer from scriptType, so
old remediation cycle values are never displayed as run time again).
- Dedupe merges both timings across the early/late emissions instead of
dropping the runtime when the more complete HS-NEW-RESULT entry wins.
- Card run time = max of phase runtimes; "Reported after Xm" shown in the
expanded Details strip (per UX decision), never on the compact row.
Inert for new events until agent redeploy; web deploy alone already fixes the
misleading display of stored data.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An ESP subcategory failure can be retracted inside the 30s settle window
(user clicks "Try again" / IME re-sync flips failed -> inProgress). The
expiry handler fired the latched EspFailureDetectedEventArgs
unconditionally, terminating the agent on a failure Windows no longer
reported and losing the retry outcome (session c071e92b: recovery at
+12s, terminal esp_terminal_failure at +30s anyway).
- OnProvisioningFailureSettleExpired re-checks current registry-derived
state under _stateLock: recovered = categorySucceeded != false AND no
subcategory currently "failed". On recovery: emit new
esp_failure_settle_recovered event, suppress EspFailureDetected, clear
the fire-once gate (+resolved set) so a re-failure arms a FRESH settle
window and is then terminal.
- esp_failure_settle_started now names tracked apps not completed
(trackedAppsNotCompleted/-Count) on Apps-subcategory failures via a new
packageStatesProbe (ImeLogHost.AllKnownPackageStates through
DefaultComponentFactory -> EspAndHelloHost -> EspAndHelloTracker).
Covers the Store/WinGet blind spot (Company Portal pending with zero
IME events) that both app_install_starved (IME-phase + intent gates)
and esp_apps_failure_correlation (IsActive device apps only) miss.
- New event type registered in Constants.EventTypes + MCP resource
catalog (drift test green); 5 new settle-window tests, suite 1711/1711.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
netstandard2.0's string.IsNullOrEmpty lacks [NotNullWhen(false)], so the
compiler cannot prove the else branch is non-null. Null-forgiving assert.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Session 2bc884b6: ESP failed AccountSetup/Apps with 0x80073cf9 while all 8
tracked Win32/WinGet apps succeeded — the failing app was a MSIX/Store app
that installs outside the IME Win32 pipeline and is invisible to
ImeLogTracker. The session ended as a blackbox failure.
When an Apps-subcategory failure arms the 30s settle window, the tracker now
queues a one-shot scan of Microsoft-Windows-AppXDeploymentServer/Operational
(Critical+Error, session window, 200-record/10s cap), scores candidate
packages (exact-HRESULT match > AppX-family error > time proximity, recency
bonus), and emits esp_appx_failure_analysis with flat RuleEngine fields plus
a capped candidates[] detail list — before enrollment_failed lands in the
timeline. Negative results are emitted too (no_appx_candidates points back
at Win32). Fire-once per session; scan runs lock-free on the threadpool;
settle expiry never waits; failures degrade to collector_degraded.
- New: AppxDeploymentFailureScanner (one-shot reader, injectable seam),
AppxFailureAnalyzer (pure scoring/verdicts, incl. negative-decimal
HRESULT + PackageFullName extraction from message/properties)
- ProvisioningStatusTracker: optional scanner/dispatcher ctor seams,
MaybeStartAppxFailureScan at settle-arm time
- Rule ANALYZE-APP-014 (MSIX/Store app failure during ESP) keyed on
verdict=appx_candidate_identified; dist regenerated
- MCP resource-catalog + web HRESULT badge + gather-rules eventTypes
- Tests: 16 analyzer + 7 trigger; settle-window fixture now injects a
fake scanner + sync dispatcher (no real event log in tests)
Inert until agent redeploy (rule needs backend redeploy). Post-deploy
checks in tasks/todo.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The network_bandwidth_estimate event now carries the CONFIGURED Delivery
Optimization bandwidth caps, so analyses can distinguish line-limited
(no caps -> the p90 is a real capacity lower bound) from policy-limited
(caps present -> the measured rate reflects the throttle, the line is
likely faster).
Verified against MS docs (Policy CSP + DO reference): GPO and Intune/CSP
do NOT share a registry store - both are read from the Registry64 view:
- GPO (ADMX): HKLM\SOFTWARE\Policies\Microsoft\Windows\
DeliveryOptimization, DO-prefixed value names
- Intune/Policy CSP: HKLM\SOFTWARE\Microsoft\PolicyManager\current\
device\DeliveryOptimization, same value names
Semantics: absolute caps in KB/s, percentage caps 0-100; a value of 0
means dynamic/no limit (OS default) and is treated as not configured.
DODownloadMode 0 is a real setting (HTTP only) and is reported whenever
present, but is not a throttle. Policies are read FRESH at each emission
because Intune policies land during the enrollment.
Event fields (omitted when absent): doThrottleConfigured,
doThrottleSources (gpo|mdm|gpo+mdm), doPolicy{Gpo,Mdm}{MaxForegroundKBps,
MaxBackgroundKBps,PctMaxForeground,PctMaxBackground,DownloadMode}; the
message gains a "[DO throttled by policy]" marker. Not covered (v1):
DOSetHoursToLimit* business-hours string policies.
+6 DoThrottlePolicyReaderTests (two-store split, 0-semantics,
string-DWORD tolerance, fail-soft); agent suite 1674/1674 green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The estimator accumulator previously lived in process memory only - a
mid-DeviceSetup reboot (e.g. a Win32 app forcing a restart) silently
discarded every rate sample collected so far. One enrollment commonly
spans several reboots.
- BandwidthStatePersistence (bandwidth-state.json in the state dir):
mirrors the OfficeInstallStatePersistence pattern - atomic write via
temp+rename, fail-soft load (missing/corrupt/foreign-session -> fresh
start, never throws). SessionId-validated on load.
- BandwidthEstimator.ExportState/ImportState: sanitizes NaN/Infinity/
negative samples, caps list sizes; baselines deliberately not
persisted (a reboot gap always exceeds the per-file delta window).
- Save points are event-driven, never per-poll: the dormant transition
(downloads just finished - exactly when app-forced reboots happen),
after the interim emission, and at collector stop.
- The InterimEmitted guard rides in the same file: device_setup_end
now fires once per SESSION, not once per process. Persisted AFTER
the emission (StartupEventGate MarkEmitted ordering) so a crash
between emit and save re-emits instead of suppressing forever.
- Wiring: DefaultComponentFactory passes the state directory through
DeliveryOptimizationHost; the collector param is optional (null =
in-memory only).
+13 tests (estimator roundtrip/merge/sanitizing, persistence roundtrip/
session-mismatch/corruption, collector-level resume incl. guard
survival); agent suite 1668/1668 green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New one-shot network_bandwidth_estimate event derived passively from the
Delivery Optimization byte counters the DeliveryOptimizationCollector
already polls every 3s - zero synthetic traffic, zero extra load, no
config change.
- BandwidthEstimator: per-FileId delta windows (a DO job that drops off
the list and returns is averaged over its own absence window), p90/max
reduction, WAN (HTTP + internet peers) vs LAN (LAN/group/link-local
peers + Connected Cache) split so peer-fed downloads cannot inflate
the WAN figure; bucket (<10/10-50/50-100/100-250/250+) + confidence.
- Emitted at most twice per session (snapshotTrigger): interim snapshot
on first AccountSetup sighting (device_setup_end - survives sessions
that starve in the account phase) and authoritative final at collector
stop (collector_stop, lands before diagnostics_collecting).
- All DO jobs feed it (IME apps, Office CDN, WU/Store/Defender).
- Registered in Constants.EventTypes, MCP resource-catalog
(network_events) and web gather-rules autocomplete.
- Drive-by: add missing agent_emergency_break to the MCP catalog
(pre-existing drift, event-types-drift test was red).
13 new estimator unit tests; agent suite 1655/1655, MCP drift 6/6,
web tsc + eventTypes vitest green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase 1 - named notification channels (1..N per tenant):
- NotificationChannel model (provider/url/headers/enabled + per-event
opt-ins incl. single NotifyOnSlaEvents); stored as JSON column
NotificationChannelsJson, cap 10, fail-soft ParseList
- No data migration: GetNotificationChannels() synthesizes one channel
from the legacy single-webhook fields under stable id "legacy";
web materializes on first save and clears legacy fields
- All dispatch sites (ingest success/failure/white-glove/ESP, SLA breach/
resolved/consecutive, enrollment-start, hardware-rejected, test) route
via SendToChannelsAsync with per-channel toggle filtering
- Per-channel secret redaction (structure stays visible to readers) with
restore-by-id; save-time validation (dup ids, provider, https, headers)
- Settings UI: channel card list with add/remove, event pills, per-channel
test button ({channelId} body, legacy-compatible without body)
Phase 2 - analyze-rule channel notifications:
- NotifyDefault (definition, false for all built-ins, test-guarded) +
Notify/NotifyChannelIds tenant override in RuleStates (two-tier mirror
of MarkSessionAsFailed); custom rules fold into the tenant-owned row
- Dispatch in AnalyzeOnEnrollmentEndHandler only: outcome.Results is
new-findings-only (engine dedup) so no re-notification; manual
"Analyze Now" reanalyze never dispatches
- BuildRuleFiredAlert (eventType analyze_rule_fired): device, serial,
rule, severity, category, confidence, explanation + session deep link
- Rule card: notify toggle + channel target pills, bell header badge
2858/2858 Functions tests (19 new), web tsc + vitest clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each model row now deep-links to /dashboard?status=Failed&search=<model>,
carrying the selected tenant for scoped global admins. The dashboard reads
?search= and ?status= from the URL (status validated against the known
buckets) and seeds the filter state via useDashboardFilters.
The search term stays fully client-side (never sent to the backend), the
status param is whitelist-validated, and the tenant param remains bounded
by the existing delegated-scope + backend enforcement, so no new trust
boundary is introduced.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 2026-07-08 timeout reclassification (Incomplete = terminal, non-failure)
migrated SessionStats, FleetHealthStats and SessionStatusBuckets, but two
reporting surfaces were overlooked: the live UsageMetricsService.SessionMetrics
(both platform + tenant paths, feeding get_usage_metrics and the usage
dashboards) and the persisted daily UsageMetricsSnapshot. Terminal Incomplete
sessions were counted in Total but in no breakdown bucket, so
Succeeded+Failed+InProgress no longer reconciled to Total and operators lost
sight of those devices.
- SessionMetrics + UsageMetricsSnapshot gain an Incomplete / SessionsIncomplete
count, mirroring the migrated peers (Incomplete only, not AwaitingUser).
- Snapshot column stored + mapped in TableStorageService.Metrics; legacy rows
deserialize to 0 (no seed bump needed).
- Success rate unchanged everywhere: denominator was already Succeeded+Failed,
correctly excluding Incomplete/AwaitingUser. SLA paths verified already-correct
(BuildSnapshot + breach evaluator receive terminal-filtered sessions).
- Web: Incomplete card added to both usage views (slate, matches the badge).
- MCP: query_raw_sessions status enum learns AwaitingUser/Incomplete so callers
can filter by the new states.
- Test: UsageMetricsServiceWindowTests pins the Incomplete bucket + 50% rate on
the crcins 1/1/8 shape.
Backend build clean, full Functions suite 2825 pass, web + MCP tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses four review rounds on the enrollment-status reclassification
(timeout != failure; new AwaitingUser/Incomplete states).
Backend correctness:
- UpdateSessionStatusAsync ETag force-fallback now mirrors the normal path:
uses IsTerminalTransitionAllowed + admin-marked guard, stamps CompletedAt
for Incomplete, and persists FailureReason/FailureSnapshotJson for
Incomplete/AwaitingUser (not just Failed) so a contended write cannot
regress a terminal or drop the classification reason/snapshot.
- StoreSessionAsync CAS loop re-reads the terminal tuple in the fresh-read
window and preserves a just-landed terminal (Succeeded/Failed/Incomplete)
verbatim, so re-registration can no longer revert it via the Replace.
- SessionGraceHours + AbsoluteMaxSessionHours now roundtrip through the
tenant-config Store/Map pair (overrides were silently lost).
- ResolveGraceHours clamps the assumed agent cap to >= the real runtime
default (48h) until the agent honors the override, so a low override can
never terminalize a still-running session early.
- Incomplete now funnels through the terminal counter-reconcile seam, and
the maintenance sweep re-reconciles after the synthetic session_timeout
event (also fixes the pre-existing off-by-one on the Failed path).
Web:
- SessionInfoCard delegates its status pill to the shared SessionStatusBadge
(removes a divergent local map; picks up AwaitingUser/Incomplete).
- SessionTable search color map + status filter chips learn the new states.
- useSessionEvents uses isTerminalStatus() instead of a hard-coded
Succeeded/Failed check (no needless refetch on Incomplete).
- DataManagementSection copy no longer says "Failed - Timed Out".
Docs/comments:
- Plan doc + TenantConfiguration comments corrected (+3h/51h buffer, and the
timeout now reclassifies to AwaitingUser/Incomplete, not "Failed").
Tests: +ResolveGraceHours clamp case, +grace/abs-cap config roundtrip +
legacy-row defaults. Full backend suite green (2712), web tsc green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Windows Update activity during Autopilot OOBE/ESP is a blind spot no other tool
(Intune console included) surfaces well — a cumulative update installing mid-OOBE
can break enrollment (r/Intune KB5095189). This makes it visible with secured
evidence.
Agent (net48):
- WindowsUpdateTracker + WindowsUpdateWatcherHost: watch
Microsoft-Windows-WindowsUpdateClient/Operational (EventIDs 19/20/43/44 =
success/failure/install-started/download-started), startup backfill (OOBE updates
run before the agent starts), RecordId watermark dedup (restart boundary vs
intra-run seen-set so a high-id live event can't hide older backfill events),
HRESULT->symbol decode. Emits windows_update_succeeded/_failed/_started with the
real event Timestamp. Default-on config knobs; uncertain EventIDs configurable
without redeploy.
- RegistryCollector: emitOnlyIfExists (suppress positive-assertion events when the
key is absent) + Registry64 base view by default (bitness-independent, matches
TenantIdResolver) with optional registryView override.
Rules:
- ANALYZE-DEV-004 (high: WU failed during enrollment, high-insight not KO) +
ANALYZE-DEV-005 (info: WU installed). reboot_pending corroboration matches
event_data exists==true.
- GATHER-DEVICE-005 -> windows_update_reboot_pending (emitOnlyIfExists),
GATHER-DEVICE-004 -> windows_update_history (Get-HotFix, opt-in).
- 5 EventTypes synced across Constants + MCP resource-catalog + web eventTypes.
Self-maintaining rule lifecycle (analyze + gather parity):
- Gather rules gain the sunset/orphan-GC that only analyze had
(ProcessSunsetGatherRuleAsync + runtime filter + seed/reseed/github paths).
- RuleProvenance field (embedded/github, null=embedded back-compat): the embedded
catalog only sunsets its own rules; GitHub-reseeded rules ahead of the deployed
binary (new ids AND content updates to existing ids) are exempt and reclaimed to
embedded once the binary catches up. No orphans, no operator knowledge required.
Verified: full solution builds; backend suite 2770 pass, agent suite 1638 pass;
rules AJV-valid + combined.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make the agent's silent 48h absolute session-age break visible: on break, before
cleanup/exit, fire a SessionAgeEmergencyBreak AgentErrorReport over the resilient
emergency channel; the backend (Part B) materializes it as an agent_emergency_break
timeline event and the timeout classifier terminalizes the session instead of
waiting out grace.
Decoupled so the guard stays pure and its unit tests are untouched:
- CheckSessionAgeEmergencyBreak gains optional Action onBreakFired = null, invoked
once when the break is confirmed, before marker/cleanup, fully swallowed.
- New EmergencyBreakReporter.TrySend builds a throwaway auth bundle
(BackendClientFactory) and sends, bounded to a 5s wait. AgentBootstrap wires it;
GetAgentVersion made internal.
NOTE: agent targets net48 and was NOT compiled in this environment (only the .NET 8
SDK is available) — to be built/verified locally. The Shared AgentErrorType value is
verified (compiled in the backend build).
- SessionStatusBuckets: explicit AwaitingUser + Incomplete buckets (out of Other),
still reconciling to Total.
- Metrics summary: expose awaitingUser + incomplete; failure rate is now
Failed/(Succeeded+Failed) — Incomplete and non-terminal states are out of the
denominator (crcins drops from failed/total to the real terminal rate).
- SessionStats.IncompleteLastNDays; dashboard SuccessRatePct was already terminal-only.
- Daily aggregation already used Succeeded+Failed denominator — unchanged.
- Tests: SessionStatusBucketsTests. Suite: 2705 pass. Fleet Health Incomplete surfacing
deferred to PR5 (web) with its documented over-all-sessions semantics.
A device that reaches the desktop is enrolled regardless of an earlier sweep
timeout guess, so a genuine completion must be able to heal a prior verdict.
- Replace the terminal idempotency guard with pure IsTerminalTransitionAllowed:
Succeeded UPGRADES any non-Succeeded state (Failed/Incomplete/AwaitingUser →
Succeeded); Failed/Incomplete and AwaitingUser never overwrite an existing
terminal. Ingest already calls UpdateSessionStatusAsync(Succeeded) on a late
enrollment_complete without a pre-terminal skip, so the relaxed guard wires the
reconcile end-to-end.
- Clear stale FailureReason + FailureSnapshotJson on Succeeded (reconcile hygiene).
- Preserve admin-marked terminals: a late completion never auto-overrides a
hand-marked Failed/Incomplete.
- Tests: SessionStatusTransitionTests (full matrix, 20). Suite: 2705 pass.
ReportAgentErrorFunction now injects ISessionRepository and, on a
SessionAgeEmergencyBreak report, writes an agent_emergency_break EnrollmentEvent
into the session stream (Sequence one past the last event, idempotent, best-effort
— never turns the always-200 emergency channel into a retry loop). This is what
makes the agent's otherwise-silent 48h break visible in the timeline and lets the
timeout classifier see it. Pure BuildAgentEmergencyBreakEvent helper (Warning,
non-terminal) + tests. Functions.Tests: 2685 pass.
Groundwork for making the agent's silent 48h emergency break transparent
(docs/design/enrollment-status-reclassification.md):
- Constants.EventTypes.AgentEmergencyBreak + AgentErrorType.SessionAgeEmergencyBreak.
- EnrollmentTimeoutClassifier: ExtractRollup.HasAgentEmergencyBreak; classify a
session carrying the break as Incomplete NOW (agent is gone) instead of holding it
as AwaitingUser through the grace — a real completion still wins. Tests added (22 total).
Next: agent best-effort emit at the break + backend materializes the timeline event.
Oliver Kieselbach