Compare commits

...

17 Commits

Author SHA1 Message Date
Oliver Kieselbach d43cfe2bb3 updated last modified page 2026-08-20 09:16:17 +02:00
Oliver Kieselbach 78cc038426 fix(agent): record real bytes-down when AutomaticDecompression strips Content-Length
The pipeline's AutomaticDecompression (GZip|Deflate) removes the response
Content-Length header on virtually every backend JSON response, so
net_total_bytes_down in agent_metrics_snapshot read 0 on healthy sessions.
When the header is absent, buffer the content (LoadIntoBufferAsync, 16 MB cap)
and record the DECOMPRESSED payload size — HttpClient buffers responses anyway
(default ResponseContentRead), so no extra memory cost. Fail-soft: a buffering
error keeps 0 and leaves the failure for the caller to observe.
AutomaticDecompression itself stays untouched (pinned by
MtlsHttpClientFactoryTests).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-20 01:25:41 +02:00
Oliver Kieselbach ff8b847dc0 fix(backend): fold do* fields from download_progress — DO data survives IME 1.104
IME 1.104 removed the [DO TEL] log line, leaving the agent's DO collector as
the only do_telemetry source — and its gate only fires when a 3s poll itself
observes the completed download. Fast, interrupted or between-poll-completed
downloads never emit one, so DoDownloadMode stayed -1 and DoAggregator dropped
the whole row: DO bytes read as "practically null" on 1.104 fleets even though
download_progress events carried the full do* set on every poll.

- Shared ApplyDoFields for both writers: download_progress folds byte counters
  and file size via Math.Max (monotonic, replay-idempotent) and sets the mode
  only when >= 0; do_telemetry stays authoritative (last-write-wins) but never
  regresses DoDownloadMode to the -1 unset sentinel.
- DownloadBytes now prefers actually-transferred bytes over doFileSize (the
  file size used to inflate the transfer measure).

Intended effect after deploy: DO aggregates count previously-invisible
incomplete downloads, so app counts/bytes rise. No backfill of historic rows.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-20 01:25:28 +02:00
Oliver Kieselbach aab7ae12ff feat(mcp): get_agent_efficiency_metrics tool + honest truncated + full platform pass-through
- New GA-only tool get_agent_efficiency_metrics: near-pass-through of the
  server-side per-version aggregation (120s timeout, default 500 sessions,
  optional tenantId). Added to GA_FULL + PLATFORM_ONLY catalog snapshots.
- platformWindowEcho decides truncated on the backend's sessionsScanned
  (sessions actually covered) and only falls back to the has-snapshots count
  for old backends — the previous comparison read false even when the cap was
  hit exactly, whenever some scanned sessions carried no snapshots.
- get_platform_metrics: 90s timeout (backend fans out one storage query per
  session); spool block, crashRate, deliveryLatency, computedAt,
  computeDurationMs and fromCache are no longer dropped on the MCP side;
  description clarifies totalBytesDown is the agent's own HTTP traffic.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-20 01:25:14 +02:00
Oliver Kieselbach 935f0e8c71 feat(backend): agent-efficiency aggregate + honest scan count + filtered metrics fetch
- New GET /api/global/metrics/agent-efficiency (GlobalReadOrAdmin, ?tenantId=):
  per-agent-version percentiles (CPU, working set, private bytes, threads,
  handles, spool, API latency/requests), crash rates per version, top offender
  sessions per dimension, 5-min cache keyed (days, limit, tenant). First
  backend consumption of agent_thread_count/agent_handle_count.
- PlatformAgentMetricsResponse.SessionsScanned: sessions the scan actually
  covered — the honest basis for the caller's truncated flag (Sessions.Count
  only counts sessions WITH snapshots).
- Per-session fetch no longer drains the full Events partition: new
  GetSessionEventsByTypesAsync (multi-type OData filter + column projection +
  real maxResults enforcement); the 20 delivery-latency sample sessions keep
  the full fetch so latency deltas stay computed over all events.
- Shared AgentMetricsAggregation (crash tally, payload getters, bounded
  fan-out, metrics event types) so the two services cannot drift.
- GetAllSessionsPageAsync gains an optional select projection (default null,
  existing callers unchanged), pinned by an equivalence test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-20 01:24:59 +02:00
Oliver Kieselbach 541dc32ffd feat(web): PDF export for the app install report + usage telemetry
Adds an "Export PDF" button to the app detail page that renders the full
90/30/7-day install report (KPIs, both charts, failure codes, device model
correlation, version breakdown, detection-rule/regression callouts) as a
branded A4 PDF, drawn data-driven with jsPDF — no DOM capture, so the output
is light-palette and theme-independent. jsPDF stays out of the route bundle
via on-click dynamic import (verified: separate async chunk).

Sample messages and the affected-sessions list are deliberately excluded:
the PDF is meant to be emailed outside the org.

Tracks app_report_pdf_exported in Application Insights (windowDays, scope,
durationMs, pageCount, success/errorName — chunk-load failures included).

Generic chrome (header/footer/tables/KPI cells/callouts) lives in
utils/pdf/pdfPrimitives.ts for reuse by future report exports.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-20 00:57:39 +02:00
Oliver Kieselbach 022cc0354b fix(engine): let the observed user-ESP skip stand in for the exit evidence
The five stuck sits-d Cloud PCs carried SkipUserStatusPage=True — verified
across three agent processes (10+ registry reads, True every time) and six
hours of live Shell-Core watching with zero 62407: Windows never rendered a
user ESP page, so no exit and no AccountSetup provisioning registry can ever
exist on this flow. Arm B of ShouldTransitionToAwaitingHello was open from
the bootstrap EspConfigDetected onward, but every evaluation site hangs off
a signal the flow cannot produce, and the AdvisoryCompletion backstop arms
exclusively from an EspExiting signal — an open gate nobody knocks on, six
hours to the max-lifetime watchdog. The earlier lost-exit-through-reboot
diagnosis is retired by the same evidence.

HandleImeUserSessionCompletedV1's completion attempt now accepts the
observed skip in place of IsPostAccountSetupFinalExit (anchor + genuine IME
user-session completion + real-user desktop stay mandatory; Device
Preparation excluded). The knock deliberately lives on the IME edge, not on
desktop arrival: Fix 10's AccountSetup bounce-back would undo a desktop-side
promotion two signals later, while the IME phase line always precedes the
user-session-complete line in log order, so a bounced promotion is
re-knocked by the very next re-emission — restart re-parse included.

Docs and comments no longer claim an ESP-exit recovery that does not exist
(stray duplicate summary in ShellCoreTracker, ImeLogHost restore-callback
comment); the OKF doc is reframed on the evidence and names the residual
SkipUser=false exit-in-downtime gap honestly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-20 00:40:23 +02:00
Oliver Kieselbach 3aa8b95676 fix(agent): narrow the Shell-Core replay to the Hello-wizard rail
Round-four review, confirmed — and the cause sits a layer below where I had been
fixing. The IsBackfill return added last round only guarded the coordinator's own
completion logic. Upstream of it, ShellCoreTracker still raised
FinalizingSetupPhaseTriggered, the coordinator still forwarded EspExited to the
adapter before that return, and the adapter discards the args entirely
(OnEspExited(...) => EmitEspExiting()). So a replayed exit still arrived at the
reducer as an ordinary, unmarked EspExiting.

There it is decisive: HandleEspExitingV1 passes espFinalExitInFlight: true for
every arriving exit, and IsPostAccountSetupFinalExit orders exits by ingest
ordinal rather than timestamp — so a historic exit replayed today reads as
post-AccountSetup by construction. With state restored from the snapshot
(AccountSetup entered, a genuine IME user session, desktop arrived) arm C of
ShouldTransitionToAwaitingHello opens on the Device→Account handoff. The reducer
is itself a completion gate and cannot tell the two apart, so the fix has to be
upstream.

The replay is therefore narrowed to what is provably safe: the Hello-wizard start
(62404). That fact is conservative — it vetoes a premature "Hello is disabled"
skip and can never by itself complete a session, so replaying it can only make the
agent wait longer. It is also the observation session 772fe502 wrote the replay
for. A 62407 is no longer raised at all, neither exit nor failure: the exit
because Windows writes the identical description for the intermediate and the
final one and nothing downstream can order it, the failure because re-injecting it
as fresh can fail a session that recovered on retry (ANALYZE-ESP-006). Skipped
records are reported once as an agent_trace with counts and bounds — silently
dropping evidence is how this class of bug survives.

The cross-layer proof is now a pair, as the review asked:
ClassicEspExitingOnRestoredStateTests pins that the reducer opens on any arriving
exit (including one with an hour-old source timestamp), and
ShellCoreTrackerReplayScopeTests pins that the replay never produces one.

Dead machinery removed with it: EspExitedEventArgs.IsBackfill, allowEspExit, the
newest-exit selection, the IsBackfill return. The method is now called
BackfillRecentHelloWizardStart.

One pre-existing integration test drove EspExited through HandleBackfillRecord
because the replay is easier to call than the event-log watcher, and its class doc
claimed that was the production path — it never was, the replay had no caller when
that test was written. Switched to the live path and the doc corrected.

Agent V2.Core 2361, DecisionCore 538, Backend 4287, all green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 19:05:14 +02:00
Oliver Kieselbach ec945a93d2 fix(agent): bar replayed ESP exits from the completion gate entirely
Round-three review, confirmed. The previous round stopped a replayed exit from
arming the DEFERRED re-check but left the immediate synthesis attempt right below
it untouched. My justification for that — "pre-existing behaviour" — was simply
wrong: the immediate attempt on a replayed exit never existed, because the
backfill had no caller. I introduced it, and by moving the replay behind the IME
state restore I made the dangerous state reachable by design: if the newest 62407
in the window is only the intermediate DeviceSetup→AccountSetup exit while the
restored app states are already terminal, that attempt opens the strong
AccountSetup gate before the final exit has even happened. A test was pinning the
faulty assumption as expected behaviour.

A replayed exit now returns before touching completion at all — no deferred edge,
no immediate attempt. Nothing available to it can establish its position in time:
Shell-Core writes the identical description for both exits, and every probe that
could order it reads state as it is now rather than at the event's time. The
replay keeps the job it was written for in session 772fe502 — feeding EspExiting,
FinalizingSetup and the Hello wizard to the reducer for the window in which no
agent process was observing. Completion is the exclusive business of exits this
agent watched live.

With the synthesis attempt gone, the replay no longer needs restored app states,
so it moves back into EspAndHelloHost.Start(); that also keeps this host's startup
independent of the IME host's. ImeLogHost.onStateRestored goes back to doing just
the re-check, which is still needed for a live exit observed before the restore
finishes.

The test asserting the faulty assumption is inverted, plus coverage that the
replay is not neutered (EspExiting still forwards) and that a live exit following
a replayed one completes normally.

Agent V2.Core 2364, DecisionCore 533, Backend 4287, all green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 18:25:22 +02:00
Oliver Kieselbach b36cf8ff36 fix(agent): order the ESP-exit replay by evidence, size its window by measurement
Round-two review, both findings confirmed against the code.

A replayed exit was being confirmed as post-AccountSetup by reading the registry
at replay time. That read reflects NOW, not the event's time, so an agent that was
down across the DeviceSetup→AccountSetup transition would confirm the stale
intermediate exit and, once the apps settled, synthesise a completion while the
AccountSetup page was still up. Windows offers no way to tell the two apart from
the event itself — Shell-Core writes the identical description
CommercialOOBE_ESPProgress_Page_Exiting for both. So the deferred edge is now
armed by LIVE exits only (EspExitedEventArgs.IsBackfill): only for those is the
continuous-observation argument valid. A replayed exit keeps the single immediate
synthesis attempt it always had.

For that attempt to be worth anything it must see real data, so the replay moved
out of EspAndHelloHost.Start() into ReplayEspExitBackfill(), invoked from
ImeLogHost.onStateRestored — after LoadState() has restored the package states.
That is what makes "the agent came back long after the ESP finished" recoverable.

The five-minute window for non-reboot restarts was unsound: the scheduled task
carries a BootTrigger only, with no restart-on-failure, so after an
exception_crash the agent does not return until the next boot — hours later, long
after Windows wrote the final exit. The window is now measured rather than
guessed: back to the last moment the previous run is known to have been alive,
taking the wider of the snapshot mtime (rewritten every decision step, available
for every exit type) and LastBootUtc. The five-minute default is the floor,
ClampLookbackMinutes the ceiling, and a future timestamp from clock skew is
ignored instead of producing a negative window.

Also switched the backfill tests to the real Shell-Core wording — the invented
string matched the regex but hid the fact that both exits look identical.

9 new tests. Agent V2.Core 2362, DecisionCore 533, Backend 4287, all green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 18:06:14 +02:00
Oliver Kieselbach c6ee7808f1 fix(agent): scope the ESP-exit replay to restart recovery only
Follow-up to the review hardening: the replay ran on EVERY agent start with a
5-minute window, first run included. On a first run there is no gap to recover —
no earlier agent process existed — and a 62407 already sitting in the Shell-Core
log belongs to a phase transition this agent was never meant to observe. Replaying
it pushes an EspExiting into the reducer that the pre-fix agent never saw, i.e. a
happy-path behaviour change for no gain.

ResolveEspExitBackfillLookbackMinutes now keys the window to the previous exit
type: first_run (or unknown) disables the replay entirely, clean/hard_kill/
exception_crash keep the 5-minute default because the process is back within
seconds, and reboot_kill widens to "since LastBootUtc" because that gap spans the
forced restart until the post-reboot logon. The result is still clamped to
BackfillLookbackMaxMinutes, so policy and backstop together cannot build an
unbounded event-log query.

With this the happy path is byte-identical to pre-fix behaviour: on a first run
nothing replays, no edge is remembered, and the deferred re-check returns on a
volatile bool read.

No other backfill is affected — the Hello, MDM-reboot, Windows-Update and
ModernDeployment trackers each own a private backfill invoked from their own
Start(), none of which this change touches.

13 policy tests including clock skew across the reboot, a missing boot timestamp
and the clamp backstop. Agent V2.Core 2353, DecisionCore 533, Backend 4287.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 17:37:40 +02:00
Oliver Kieselbach b61d7023bc fix(agent): harden post-reboot completion recovery after review
Three real regressions, all confirmed against the code before touching it — two of
them introduced by the recovery itself.

The remembered exit edge was unqualified. Shell-Core raises 62407 at every ESP
phase transition, so OnEspExited also latched the DeviceSetup→AccountSetup one;
the deferred re-check would then open the strong AccountSetup gate as soon as the
last user app settled, while the AccountSetup page was still up and its other
subcategories still running. IsConfirmedPostAccountSetupExit() now demands
positive evidence (AccountSetup activity observed, or SkipUser == true) and is
deliberately stricter than the existing IsIntermediateDeviceEspExit() forward
guard: an unknown SkipUser counts as not confirmed. Erring strict costs the
pre-fix stall; erring loose costs a premature Succeeded. The immediate on-edge
synthesis keeps its previous behaviour verbatim.

The replay picked the oldest exit, not the final one. The reader walks
oldest-first and the exit branch is fire-once, so the widened downtime window —
which routinely holds both the intermediate and the final exit — handed over the
intermediate one and the fire-once guard swallowed the exit that matters.
ReplayBackfillRecords buffers the batch and lets only the newest exit-matching
record through; ESP failures and the Hello-wizard rail keep replaying
chronologically.

The replay ran before the IME state restore. ImeLogTracker.Start() restores the
persisted package states via LoadState() without raising OnAppStateChanged, and
ImeLogHost starts after EspAndHelloHost, so the replay evaluated an empty app
list — leaving the sharpest case unfixed: apps already terminal before the
reboot, only the final exit lost to the downtime. ImeLogHost gained an
onStateRestored callback wired to the re-check, and the host order is now pinned
by DefaultComponentFactoryOrderingTests so a reshuffle fails loudly.

11 new tests. Agent V2.Core 2340, DecisionCore 533, Backend 4287, all green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 17:27:29 +02:00
Oliver Kieselbach e1b03de5b3 fix(agent): recover ESP completion after a forced mid-ESP reboot
Seven sits-d Cloud PCs enrolled the same day with identical configuration
(~138 required apps at the user-ESP gate, DeviceGuard/DmaGuard device-assigned).
The two that finished AccountSetup before the coalesced reboot landed succeeded;
the five the reboot killed in Stage=EspAccountSetup never derived a completion
again and ended Incomplete on healthy devices. Two defects behind that:

ShellCoreTracker.BackfillRecentEspExitEvents() has existed since session
772fe502 but no caller ever invoked it, so the Shell-Core watcher only saw
records written after Start(). Every agent restart silently dropped an ESP exit
(62407) or Hello-wizard start (62404) from its downtime. EspAndHelloHost.Start()
now calls it, and the fixed 5-minute window is sized to the actual downtime when
the previous run died to a reboot (LastBootUtc, clamped to 360 min).

The user-apps-settled synthesis ran only on the EspExited edge. When the ESP page
exits with apps still in flight the single attempt misses and nothing re-checks —
8110e262 and a89aac2d both reached 138/138 apps, 0 failed, and still never
completed. The exit edge is now recorded and the synthesis re-evaluated from
ImeLogTracker.OnAppStateChanged. No new completion path: the gate conditions are
unchanged, this only gives the existing conservative check a second look. The
fire-once claim moves to Interlocked (edge and re-check run on different threads)
and the re-check does not re-emit app_install_starved.

Rules: ANALYZE-ESP-005 gains on_event:session_stalled so a hung session produces
a finding at the 60-minute stall instead of staying finding-less for hours
(verified against 8110e262 — fired, confidence 95). New ANALYZE-APP-016 turns the
already-emitted app_install_starved events into a customer finding and, unlike
ANALYZE-APP-015, fires even on a succeeded session: 3b9291aa was recorded
Succeeded with 99 required apps that never started installing.

ANALYZE-APP-015 and ANALYZE-ESP-001 were considered for the stall trigger and
deliberately left alone — APP-015 is the documented worked example of the
terminal-precondition trap, and ESP-001's phase_duration gate on a still-open
phase is unverified mid-run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 17:00:59 +02:00
Oliver Kieselbach 25bac34acb fix(ci): upload raw bytes, not a stringified array
The first CI run overwrote all four customer-facing scripts on the download
host and the legacy mirror with their own byte values as decimal text
(Install-AutopilotMonitor.ps1 went from 27331 to 95679 bytes).

PowerShell unrolls an array returned from a function into the output stream, so
the caller receives Object[] rather than byte[]. Invoke-RestMethod serialises an
Object[] body as space-separated decimals. Every hash check stayed green because
[byte[]] parameter coercion repairs the type everywhere except on the wire.
build-agent.yml's Send-Blob was never affected: it read the bytes inside the
function. Proven with a local HttpListener: the old shape puts 50 bytes starting
54,48,32,51,53 ("60 35 13 1...") on the wire where 17 bytes starting 60,35,13,10
("<#\r\n.SYN") are expected.

Fixes:
- ',' on both byte-array returns, plus an explicit [byte[]] at the upload site.
- Read each blob back from storage right after its PUT and abort before the
  mirror and the remaining files. Verifying only at the end left every blob
  corrupted.
- The version-bump guard tolerates an unparseable published copy with a warning
  instead of throwing: that is the state a republish exists to repair, and a
  guard that blocks the repair is worse than no guard.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 15:31:06 +02:00
Oliver Kieselbach c0be389a95 ci(scripts): version-bump guard compares behaviour, not bytes
Byte comparison made the guard fire on comment fixes, typos in comments and
reflow -- changes no consumer of the version can act on, so forcing a bump for
them is friction without a payoff.

The guard now compares a code fingerprint: the PowerShell token stream with
comments, newlines and line continuations removed. A behaviour change under an
unchanged $ScriptVersion still fails; comment and formatting changes publish
with a warning. That is also the rule the source file's own comment on
$ScriptVersion already states ("bump on meaningful changes").

Tokenising rather than regex-stripping keeps a '#' inside a string literal from
being mistaken for a comment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 15:15:48 +02:00
Oliver Kieselbach 91f9d656ae ci(scripts): publish customer scripts on push, decoupled from the agent release
The scripts behind download.autopilotmonitor.com/agent/ rotated only on a
publish_as_stable run of build-agent.yml, so a bootstrap fix sat unpublished
until the next agent cutover -- while the WDP bootstrap MSI downloads the
published copy on every enrolling device.

Publish-BootstrapScripts.ps1 becomes the single owner of the chain: dev render
with anchor check, $ScriptVersion bump guard, uploads (eu + fail-soft legacy
mirror, no-cache), the two version oracles, and verification through the alias.
publish-scripts.yml calls it on every push to main touching the sources; the
publish_as_stable branch of build-agent.yml calls the same code, so a release
still lands a consistent set.

Blobs are published in a deterministic CRLF/UTF-8-without-BOM form. Git
normalises line endings on checkout, so raw working-copy bytes would differ per
environment and make every byte comparison -- guard and verify alike -- depend
on where the workflow ran. The form reproduces the currently published bytes
exactly.

The Pester (PS 5.1) and ASCII gates move into the reusable
bootstrap-script-gates.yml and now block the upload; pester.yml keeps the
pull-request view and drops its push trigger. AdminConfiguration
.LatestBootstrapV2ScriptVersion gets exactly one writer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 15:09:33 +02:00
Oliver Kieselbach b1e7077d0a fix(agent): spool pending-bytes drift guard + WDP backstop fallback test
Codex findings on d8c84f41/a369ff99: (1) MarkUploaded now recomputes
_pendingBytes from the pending list when accounting drifts negative or
stays non-zero on an empty list — a silent negative would disarm the
spool-pressure trigger. (2) New test covers the DevicePrepCompletion
handler''s AwaitingDesktop fallback (stale/replayed timer without a
desktop fact): synthetic Hello timeout, park, and completion on the
later real desktop arrival. Third finding (HashingCappedReadStream
maxBytes:0) refuted — the 0=uncapped contract is implemented in the
same class and exercised by the heal-path tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 12:59:56 +02:00
67 changed files with 6325 additions and 368 deletions
@@ -0,0 +1,367 @@
<#
.SYNOPSIS
Publishes the customer-facing PowerShell scripts behind download.autopilotmonitor.com/agent/.
.DESCRIPTION
Single owner of the script-publishing chain. Called from two places:
* publish-scripts.yml -- on every push to main that touches the sources,
so a script fix reaches customers without waiting
for an agent release.
* build-agent.yml -- on a publish_as_stable run, so a release always
lands a consistent set (ZIP + manifest + scripts).
Both callers run the same code. The scripts on the download alias are the live
copy the WDP bootstrap MSI (Invoke-BootstrapDownload.ps1) downloads on every
enrolling device, so "current" is a correctness property, not cosmetics.
Steps, in order:
1. Parse $ScriptVersion from the bootstrap source.
2. Render the -Dev variant by literal substitution of the two URL/manifest
defaults. A missing anchor is a hard failure -- a silently un-substituted
dev script would point the dev fleet at the stable agent.
3. Version-bump guard: if the published bootstrap differs from the source
but carries the same $ScriptVersion, abort. Otherwise the docs badge,
version.json.bootstrapVersion and the portal's "your script is outdated"
hint would all keep asserting a version that is no longer what ships.
4. Upload every script blob with Cache-Control: no-cache (they rotate in
place), mirrored fail-soft to the legacy account.
5. Reconcile the two version oracles: version.json.bootstrapVersion
(read-modify-write under If-Match, so a concurrent agent release cannot
lose its agent fields) and AdminConfiguration.LatestBootstrapV2ScriptVersion.
6. Verify through the alias -- re-download each blob and compare SHA-256.
.PARAMETER SasToken
Container SAS for the eu storage account with write permission. Leading '?' tolerated.
.PARAMETER LegacySasToken
Container SAS for the legacy account. Optional -- a mirror failure warns, never fails.
.PARAMETER TableSasToken
Table SAS for AdminConfiguration. Optional -- omitted or 'XXX' skips the oracle update.
.PARAMETER RepoRoot
Repository root. Defaults to the parent of .github/scripts.
.PARAMETER DryRun
Run every check and render, upload nothing.
.EXAMPLE
./Publish-BootstrapScripts.ps1 -DryRun
.EXAMPLE
./Publish-BootstrapScripts.ps1 -SasToken $env:SAS -TableSasToken $env:TABLE_SAS
#>
[CmdletBinding()]
param(
[string]$SasToken,
[string]$LegacySasToken,
[string]$TableSasToken,
[string]$RepoRoot,
[switch]$DryRun
)
$ErrorActionPreference = 'Stop'
if (-not $RepoRoot) {
$RepoRoot = (Resolve-Path (Join-Path $PSScriptRoot '../..')).Path
}
if (-not $DryRun -and [string]::IsNullOrWhiteSpace($SasToken)) {
throw 'SasToken is required unless -DryRun is used.'
}
$ContainerUrl = 'https://autopilotmonitoreu.blob.core.windows.net/agent'
$LegacyContainerUrl = 'https://autopilotmonitor.blob.core.windows.net/agent'
$AliasUrl = 'https://download.autopilotmonitor.com/agent'
$TableUrl = "https://autopilotmonitoreu.table.core.windows.net/AdminConfiguration(PartitionKey='GlobalConfig',RowKey='config')"
$BootstrapSource = Join-Path $RepoRoot 'scripts/Bootstrap/Install-AutopilotMonitor.ps1'
$BootstrapBlob = 'Install-AutopilotMonitor.ps1'
$DevBlob = 'Install-AutopilotMonitor-Dev.ps1'
$ScriptContentType = 'text/plain; charset=utf-8'
function Get-HttpStatus {
param($ErrorRecord)
if ($ErrorRecord.Exception.PSObject.Properties['Response'] -and $ErrorRecord.Exception.Response) {
return [int]$ErrorRecord.Exception.Response.StatusCode
}
return 0
}
function Get-Sha256 {
param([byte[]]$Bytes)
$sha = [System.Security.Cryptography.SHA256]::Create()
try { return [BitConverter]::ToString($sha.ComputeHash($Bytes)).Replace('-', '').ToLowerInvariant() }
finally { $sha.Dispose() }
}
# Raw bytes of a published blob, or $null when it does not exist yet.
function Get-PublishedBytes {
param([string]$Url)
try {
$resp = Invoke-WebRequest -Uri $Url -Method Get -UseBasicParsing
} catch {
if ((Get-HttpStatus $_) -eq 404) { return $null }
throw
}
# Comma operator: PowerShell unrolls an array returned from a function into the output
# stream and reassembles it as Object[], which would silently defeat every [byte[]] use
# downstream. ',' wraps it so the byte[] survives the return.
if ($resp.Content -is [byte[]]) { return ,$resp.Content }
return ,[System.Text.Encoding]::UTF8.GetBytes([string]$resp.Content)
}
# Deterministic publish form: CRLF, UTF-8 without BOM. Git normalises line endings on
# checkout (a Windows runner yields CRLF, a core.autocrlf=false clone yields LF), so
# publishing the raw working-copy bytes would emit different blobs for identical source
# and make every byte comparison -- guard and verify alike -- environment-dependent.
# CRLF is also what PS 5.1 on the device expects and what is published today.
function Get-PublishBytes {
param([string]$Text)
$normalised = $Text.Replace("`r`n", "`n").Replace("`n", "`r`n")
# ',' keeps this a byte[]: without it PowerShell unrolls the array into the output stream
# and the caller receives Object[]. Invoke-RestMethod then serialises an Object[] body as
# space-separated decimals instead of raw bytes -- a silent corruption of the upload.
return ,(New-Object System.Text.UTF8Encoding $false).GetBytes($normalised)
}
function Get-BootstrapScriptVersion {
param([string]$Content, [string]$Origin, [switch]$Optional)
if ($Content -match '\$ScriptVersion\s*=\s*"([\d\.\-a-zA-Z]+)"') { return $Matches[1] }
# Unreadable SOURCE is fatal. An unreadable PUBLISHED copy must not be: that is exactly
# the state a republish is meant to repair, and a guard that blocks the repair is worse
# than no guard.
if ($Optional) { return $null }
throw "Could not parse ScriptVersion from $Origin"
}
# What the script actually DOES, with comments and layout removed: the PowerShell token
# stream minus comments, newlines and line continuations. Lets the version-bump guard tell
# a behaviour change (must be versioned -- consumers act on the version) apart from a
# comment fix or reflow (nobody needs to know). Tokenising rather than regex-stripping
# keeps '#' inside strings from being mistaken for a comment.
function Get-CodeFingerprint {
param([string]$Text, [string]$Origin)
$tokens = $null
$errors = $null
[System.Management.Automation.Language.Parser]::ParseInput($Text, [ref]$tokens, [ref]$errors) | Out-Null
if ($errors -and $errors.Count -gt 0) {
throw "Parse error in ${Origin}: $($errors[0])"
}
$ignored = @('Comment', 'NewLine', 'LineContinuation', 'EndOfInput')
return (($tokens | Where-Object { $ignored -notcontains $_.Kind.ToString() } | ForEach-Object { $_.Text }) -join "`n")
}
# ------------------------------------------------------------------ 1. source
$sourceContent = Get-Content $BootstrapSource -Raw
$sourceBytes = Get-PublishBytes $sourceContent
$scriptVersion = Get-BootstrapScriptVersion -Content $sourceContent -Origin $BootstrapSource
Write-Host "Bootstrap script version: $scriptVersion"
# ------------------------------------------------------------------ 2. dev render
$stableUrlLiteral = '$AgentDownloadUrl = "https://download.autopilotmonitor.com/agent/AutopilotMonitor-Agent.zip"'
$devUrlLiteral = '$AgentDownloadUrl = "https://download.autopilotmonitor.com/agent/AutopilotMonitor-Agent-dev.zip"'
$stableManLiteral = '$VersionJsonName = "version.json"'
$devManLiteral = '$VersionJsonName = "version-dev.json"'
if ($sourceContent.IndexOf($stableUrlLiteral) -lt 0 -or $sourceContent.IndexOf($stableManLiteral) -lt 0) {
throw "Bootstrap dev-render: anchor literal missing in $BootstrapSource"
}
$devContent = $sourceContent.Replace($stableUrlLiteral, $devUrlLiteral).Replace($stableManLiteral, $devManLiteral)
Write-Host "Rendered $DevBlob (dev agent URL + version-dev.json)"
# ------------------------------------------------------------------ 3. bump guard
$publishedBytes = Get-PublishedBytes "$AliasUrl/$BootstrapBlob"
if ($null -eq $publishedBytes) {
Write-Host 'No published bootstrap yet -- version-bump guard skipped (first publish)'
} elseif ((Get-Sha256 $publishedBytes) -eq (Get-Sha256 $sourceBytes)) {
Write-Host 'Published bootstrap is already byte-identical to the source'
} else {
$publishedText = [System.Text.Encoding]::UTF8.GetString($publishedBytes)
$publishedVersion = Get-BootstrapScriptVersion -Content $publishedText -Origin "$AliasUrl/$BootstrapBlob" -Optional
if ($null -eq $publishedVersion) {
Write-Host "::warning::The published $BootstrapBlob is not a readable bootstrap script (no ScriptVersion found). Republishing over it."
} elseif ($publishedVersion -ne $scriptVersion) {
Write-Host "Version bump: $publishedVersion -> $scriptVersion"
} elseif ((Get-CodeFingerprint -Text $publishedText -Origin "$AliasUrl/$BootstrapBlob") -eq
(Get-CodeFingerprint -Text $sourceContent -Origin $BootstrapSource)) {
# Comments, typos in comments, reflow -- nothing a consumer of the version can act on.
Write-Host "::warning::$BootstrapBlob changed in comments or formatting only; publishing under the unchanged version $scriptVersion."
} else {
throw ("Version-bump guard: $BootstrapBlob changed behaviour but ScriptVersion is still $scriptVersion. " +
"Bump it in $BootstrapSource -- the docs badge, version.json.bootstrapVersion and the " +
'portal outdated-script hint all read that value, and customers decide from it whether ' +
'to re-upload their Intune copy. Comment-only and formatting changes do not need a bump.')
}
}
# ------------------------------------------------------------------ 4. upload
# Sources beyond the bootstrap pair, read straight from the repo.
$extraSources = @(
@{ Path = 'scripts/Bootstrap/Test-ShouldBootstrapAgent.ps1'; BlobName = 'Test-ShouldBootstrapAgent.ps1' }
# Customer-side Graph add-on grant script -- docs and admin UI link it directly.
@{ Path = 'scripts/CustomerSetup/Grant-AutopilotMonitorAddOn.ps1'; BlobName = 'Grant-AutopilotMonitorAddOn.ps1' }
)
# Uninstall-AutopilotMonitor.ps1 is deliberately NOT published: nothing links it, and
# an unauthenticated uninstall script on the public download host is not a feature.
$publishSet = [System.Collections.Generic.List[object]]::new()
$publishSet.Add([pscustomobject]@{ BlobName = $BootstrapBlob; Bytes = $sourceBytes })
$publishSet.Add([pscustomobject]@{ BlobName = $DevBlob; Bytes = (Get-PublishBytes $devContent) })
foreach ($extra in $extraSources) {
$extraPath = Join-Path $RepoRoot $extra.Path
if (-not (Test-Path $extraPath)) {
throw "Publish source missing: $extraPath"
}
$publishSet.Add([pscustomobject]@{ BlobName = $extra.BlobName; Bytes = (Get-PublishBytes (Get-Content $extraPath -Raw)) })
}
$legacySas = if ($LegacySasToken) { $LegacySasToken.TrimStart('?') } else { '' }
$writeSas = if ($SasToken) { $SasToken.TrimStart('?') } else { '' }
foreach ($item in $publishSet) {
# Explicit type, not a convenience: an Object[] body is uploaded as space-separated
# decimals rather than raw bytes, and every hash check still passes because [byte[]]
# parameter coercion repairs it everywhere EXCEPT the wire.
[byte[]]$bytes = $item.Bytes
$item | Add-Member -NotePropertyName Sha256 -NotePropertyValue (Get-Sha256 $bytes) -Force
if ($DryRun) {
# Doubles as a drift check: says per blob whether the alias already serves this source.
$served = Get-PublishedBytes "$AliasUrl/$($item.BlobName)"
$state = if ($null -eq $served) { 'MISSING on the alias' }
elseif ((Get-Sha256 $served) -eq $item.Sha256) { 'already current' }
else { 'STALE on the alias' }
Write-Host " [dry-run] $($item.BlobName) ($($bytes.Length) bytes, sha256 $($item.Sha256)) -- $state"
continue
}
# no-cache on every blob: they all rotate in place and the download alias fronts them
# with Front Door. Route caching is disabled there, but a stale script paired with a
# fresh manifest would fail the bootstrap SHA check -- this keeps that class of bug
# impossible even if caching is ever re-enabled.
$headers = @{ 'x-ms-blob-type' = 'BlockBlob'; 'Content-Type' = $ScriptContentType; 'x-ms-blob-cache-control' = 'no-cache' }
Invoke-RestMethod -Uri "$ContainerUrl/$($item.BlobName)?$writeSas" -Method Put -Headers $headers -Body $bytes | Out-Null
# Read straight back from the blob (authoritative, no CDN in the way) before touching the
# mirror or the next file. The alias verification at the end would catch a bad body too,
# but only after every blob was already overwritten -- these are live customer downloads.
$written = Get-PublishedBytes "$ContainerUrl/$($item.BlobName)"
if ($null -eq $written -or (Get-Sha256 $written) -ne $item.Sha256) {
$writtenLength = if ($null -eq $written) { 'missing' } else { "$($written.Length) bytes" }
throw ("Upload of $($item.BlobName) did not round-trip: sent $($bytes.Length) bytes, " +
"blob now holds $writtenLength. Publishing stopped before the mirror and the " +
'remaining files, so no further blob was touched.')
}
Write-Host " uploaded $($item.BlobName) ($($bytes.Length) bytes, round-trip verified)"
if ($legacySas) {
try {
Invoke-RestMethod -Uri "$LegacyContainerUrl/$($item.BlobName)?$legacySas" -Method Put -Headers $headers -Body $bytes | Out-Null
Write-Host " mirrored $($item.BlobName) to legacy storage"
} catch {
Write-Host "::warning::legacy mirror failed for $($item.BlobName) -- $($_.Exception.Message)"
}
}
}
# ------------------------------------------------------------------ 5. version oracles
# version.json is the agent manifest; only bootstrapVersion belongs to us. Read from the
# blob (not the alias) for an authoritative ETag, write back under If-Match so a concurrent
# agent release cannot lose its version/sha256 fields.
if (-not $DryRun) {
try {
$manifestResp = Invoke-WebRequest -Uri "$ContainerUrl/version.json" -Method Get -UseBasicParsing
$manifestRaw = if ($manifestResp.Content -is [byte[]]) {
[System.Text.Encoding]::UTF8.GetString($manifestResp.Content)
} else { [string]$manifestResp.Content }
$manifest = $manifestRaw | ConvertFrom-Json
if ($manifest.bootstrapVersion -eq $scriptVersion) {
Write-Host "version.json already reports bootstrapVersion $scriptVersion"
} else {
$etag = '"' + ([string]($manifestResp.Headers['ETag'] | Select-Object -First 1)).Trim('"') + '"'
$manifest | Add-Member -NotePropertyName 'bootstrapVersion' -NotePropertyValue $scriptVersion -Force
$body = [System.Text.Encoding]::UTF8.GetBytes(($manifest | ConvertTo-Json -Compress))
$manifestHeaders = @{
'x-ms-blob-type' = 'BlockBlob'
'Content-Type' = 'application/json'
'x-ms-blob-cache-control' = 'no-cache'
'If-Match' = $etag
}
Invoke-RestMethod -Uri "$ContainerUrl/version.json?$writeSas" -Method Put -Headers $manifestHeaders -Body $body | Out-Null
Write-Host "version.json bootstrapVersion -> $scriptVersion (agent fields untouched)"
if ($legacySas) {
try {
$legacyHeaders = @{ 'x-ms-blob-type' = 'BlockBlob'; 'Content-Type' = 'application/json'; 'x-ms-blob-cache-control' = 'no-cache' }
Invoke-RestMethod -Uri "$LegacyContainerUrl/version.json?$legacySas" -Method Put -Headers $legacyHeaders -Body $body | Out-Null
} catch {
Write-Host "::warning::legacy mirror failed for version.json -- $($_.Exception.Message)"
}
}
}
} catch {
if ((Get-HttpStatus $_) -eq 412) {
throw 'version.json changed while publishing (concurrent agent release). Re-run this workflow.'
}
throw
}
}
$tableSas = if ($TableSasToken) { $TableSasToken.TrimStart('?') } else { '' }
if ($DryRun) {
Write-Host " [dry-run] would set AdminConfiguration.LatestBootstrapV2ScriptVersion = $scriptVersion"
} elseif ([string]::IsNullOrEmpty($tableSas) -or $tableSas -eq 'XXX') {
Write-Host 'SKIPPED: AdminConfiguration update -- set AZURE_TABLE_ADMIN_SAS_TOKEN to enable the portal version oracle'
} else {
$tableHeaders = @{
'Content-Type' = 'application/json'
'Accept' = 'application/json;odata=nometadata'
'If-Match' = '*'
}
$tableBody = @{ LatestBootstrapV2ScriptVersion = $scriptVersion } | ConvertTo-Json
Invoke-RestMethod -Uri "$TableUrl`?$tableSas" -Method Merge -Headers $tableHeaders -Body $tableBody | Out-Null
Write-Host "AdminConfiguration.LatestBootstrapV2ScriptVersion = $scriptVersion"
}
# ------------------------------------------------------------------ 6. verify via alias
if (-not $DryRun) {
$pending = [System.Collections.ArrayList]::new()
$publishSet | ForEach-Object { [void]$pending.Add($_) }
for ($attempt = 1; $attempt -le 6 -and $pending.Count -gt 0; $attempt++) {
if ($attempt -gt 1) { Start-Sleep -Seconds 20 }
foreach ($item in @($pending)) {
$servedBytes = Get-PublishedBytes "$AliasUrl/$($item.BlobName)"
if ($null -ne $servedBytes -and (Get-Sha256 $servedBytes) -eq $item.Sha256) {
Write-Host " verified $($item.BlobName) via $AliasUrl"
$pending.Remove($item)
}
}
if ($pending.Count -gt 0) {
Write-Host " attempt $attempt : $($pending.Count) blob(s) not current on the alias yet"
}
}
if ($pending.Count -gt 0) {
$names = ($pending | ForEach-Object { $_.BlobName }) -join ', '
throw ("Alias still serves stale content for: $names. The blobs were written, so this is a Front Door " +
'cache issue -- purge with: az afd endpoint purge --resource-group rg-autopilotmonitor-prd-gwc ' +
"--profile-name autopilotmonitor-fd --endpoint-name apm-download --content-paths '/agent/*'")
}
}
if ($env:GITHUB_STEP_SUMMARY) {
$mode = if ($DryRun) { 'DRY RUN -- nothing uploaded' } else { 'published + verified via the download alias' }
$summary = @(
"### Bootstrap scripts: $mode",
'',
"Bootstrap script version: **$scriptVersion**",
''
) + ($publishSet | ForEach-Object { "- $($_.BlobName) -- sha256 $($_.Sha256)" })
($summary -join [Environment]::NewLine) | Out-File -FilePath $env:GITHUB_STEP_SUMMARY -Append -Encoding utf8
}
Write-Host 'Done.'
@@ -0,0 +1,63 @@
name: Bootstrap script gates
# Quality gate for the customer-facing PowerShell scripts, factored out so it exists
# once and both callers run the identical checks:
# * pester.yml -- pull requests (reports, does not block the merge button)
# * publish-scripts.yml -- pushes to main, where it BLOCKS the upload to the
# download alias. Devices execute these scripts as SYSTEM
# during enrollment, so a red suite must never ship.
on:
workflow_call:
permissions:
contents: read
jobs:
gates:
name: Bootstrap guard tests (PS 5.1)
runs-on: windows-latest
steps:
- uses: actions/checkout@v7
- name: Install Pester 5+
shell: powershell
run: |
Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force -Scope CurrentUser | Out-Null
Install-Module Pester -Scope CurrentUser -Force -SkipPublisherCheck -MinimumVersion 5.5.0
# Windows PowerShell 5.1 on purpose: that is the runtime IME uses to execute
# platform scripts, so guard/relax logic is verified without resetting a device.
- name: Run Pester
shell: powershell
run: |
Import-Module Pester -MinimumVersion 5.5.0
$c = New-PesterConfiguration
$c.Run.Path = 'scripts/Bootstrap'
$c.Run.Exit = $true
$c.Output.Verbosity = 'Detailed'
Invoke-Pester -Configuration $c
- name: Guard scripts stay pure ASCII
shell: powershell
run: |
$failed = $false
Get-ChildItem 'scripts/Bootstrap' -Filter '*.ps1' | ForEach-Object {
$bytes = [System.IO.File]::ReadAllBytes($_.FullName)
for ($i = 0; $i -lt $bytes.Length; $i++) {
if ($bytes[$i] -gt 127) {
Write-Host "::error file=scripts/Bootstrap/$($_.Name)::Non-ASCII byte 0x$('{0:X2}' -f $bytes[$i]) at offset $i. PS 5.1 (IME) reads BOM-less files as ANSI and corrupts multi-byte chars."
$failed = $true
break
}
}
}
if ($failed) { exit 1 }
Write-Host 'All bootstrap scripts are pure ASCII.'
# The publish chain renders the -Dev variant by literal substitution and refuses to
# ship when an anchor moved. Running it dry here surfaces that (and any drift against
# the alias) on the PR instead of on the push that would have published.
- name: Publish dry run (dev-render anchors + version-bump guard + drift)
shell: pwsh
run: ./.github/scripts/Publish-BootstrapScripts.ps1 -DryRun
+22 -37
View File
@@ -33,6 +33,13 @@ permissions:
id-token: write # keyless Sigstore signing (artifact attestation)
attestations: write # store the provenance attestation
# Shared with publish-scripts.yml: both write version.json (this workflow the agent fields,
# the script publisher bootstrapVersion under If-Match). Serialising them keeps a script push
# from colliding with a release. Never cancel -- a half-published set is worse than a queue.
concurrency:
group: agent-blob-publish
cancel-in-progress: false
env:
LINE_MAJOR: 2
AGENT_PROJECT: src/Agent/AutopilotMonitor.Agent.V2/AutopilotMonitor.Agent.V2.csproj
@@ -46,8 +53,6 @@ env:
VERSIONED_ZIP: AutopilotMonitor-Agent-v2.zip
STABLE_ZIP: AutopilotMonitor-Agent.zip
STABLE_MANIFEST: version.json
STABLE_BOOTSTRAP: Install-AutopilotMonitor.ps1
STABLE_BOOTSTRAP_DEV: Install-AutopilotMonitor-Dev.ps1
jobs:
build:
@@ -181,6 +186,8 @@ jobs:
# commit/buildUtc are the same two fields the backend and MCP manifests carry —
# additive, and the bootstrap reads this file by field name, so older installers
# ignore them. sha256/bootstrapVersion stay: those are the customer contract.
# bootstrapVersion is seeded here so the published manifest is never briefly
# missing it; Publish-BootstrapScripts.ps1 owns and re-asserts the field afterwards.
$manifest | Add-Member -NotePropertyName "sha256" -NotePropertyValue $sha256Hash -Force
$manifest | Add-Member -NotePropertyName "commit" -NotePropertyValue '${{ github.sha }}'.Substring(0, 7) -Force
$manifest | Add-Member -NotePropertyName "buildUtc" -NotePropertyValue ([DateTime]::UtcNow.ToString('o')) -Force
@@ -190,29 +197,12 @@ jobs:
$manifest | ConvertTo-Json -Compress | Set-Content $env:MANIFEST_NAME -Encoding UTF8
Write-Host "$env:MANIFEST_NAME content: $(Get-Content $env:MANIFEST_NAME)"
# Bootstrap = single generic source. Stable variant = source 1:1.
# Dev variant = literal-string substitution of the two URL/manifest defaults.
# Both are uploaded only on -PublishAsStable runs (versioned namespace doesn't carry bootstrap).
if ('${{ inputs.publish_as_stable }}' -eq 'true') {
$stableUrl = '$AgentDownloadUrl = "https://download.autopilotmonitor.com/agent/AutopilotMonitor-Agent.zip"'
$devUrl = '$AgentDownloadUrl = "https://download.autopilotmonitor.com/agent/AutopilotMonitor-Agent-dev.zip"'
$stableMan = '$VersionJsonName = "version.json"'
$devMan = '$VersionJsonName = "version-dev.json"'
Copy-Item $env:BOOTSTRAP_SOURCE $env:STABLE_BOOTSTRAP -Force
$content = Get-Content $env:BOOTSTRAP_SOURCE -Raw
if ($content.IndexOf($stableUrl) -lt 0 -or $content.IndexOf($stableMan) -lt 0) {
Write-Error "Bootstrap dev-render: anchor literal missing in $env:BOOTSTRAP_SOURCE"
exit 1
}
$devContent = $content.Replace($stableUrl, $devUrl).Replace($stableMan, $devMan)
[System.IO.File]::WriteAllText("$env:STABLE_BOOTSTRAP_DEV", $devContent, (New-Object System.Text.UTF8Encoding $false))
}
# The customer-facing scripts (stable + dev render, test script, grant script) are
# published by .github/scripts/Publish-BootstrapScripts.ps1 -- see the upload step.
"versioned_zip_release=$versionedZip" >> $env:GITHUB_OUTPUT
"sha256=$sha256Hash" >> $env:GITHUB_OUTPUT
"exe_sha256=$exeSha256Hash" >> $env:GITHUB_OUTPUT
"bootstrap_version=$bootstrapVersion" >> $env:GITHUB_OUTPUT
- name: Upload to Azure Blob Storage
if: inputs.upload_blob
@@ -264,22 +254,21 @@ jobs:
Write-Host "PUBLISH AS STABLE -- writing V2 into stable namespace (customer-facing)" -ForegroundColor Magenta
Send-Blob -Local $env:VERSIONED_ZIP -BlobName $env:STABLE_ZIP -ContentType "application/zip"
Send-Blob -Local $env:MANIFEST_NAME -BlobName $env:STABLE_MANIFEST -ContentType "application/json"
Send-Blob -Local $env:STABLE_BOOTSTRAP -BlobName $env:STABLE_BOOTSTRAP -ContentType "text/plain; charset=utf-8"
Send-Blob -Local $env:STABLE_BOOTSTRAP_DEV -BlobName $env:STABLE_BOOTSTRAP_DEV -ContentType "text/plain; charset=utf-8"
if (Test-Path "scripts/Bootstrap/Test-ShouldBootstrapAgent.ps1") {
Send-Blob -Local "scripts/Bootstrap/Test-ShouldBootstrapAgent.ps1" -BlobName "Test-ShouldBootstrapAgent.ps1" -ContentType "text/plain; charset=utf-8"
}
# Customer-side Graph add-on grant script -- docs and admin UI link
# https://download.autopilotmonitor.com/agent/Grant-AutopilotMonitorAddOn.ps1
if (Test-Path "scripts/CustomerSetup/Grant-AutopilotMonitorAddOn.ps1") {
Send-Blob -Local "scripts/CustomerSetup/Grant-AutopilotMonitorAddOn.ps1" -BlobName "Grant-AutopilotMonitorAddOn.ps1" -ContentType "text/plain; charset=utf-8"
}
# Customer-facing scripts: same publisher publish-scripts.yml runs on every push
# to main, so a release lands a set that is consistent with what is already live
# (dev render, version-bump guard, bootstrapVersion oracles, alias verification).
# Runs AFTER version.json so its bootstrapVersion read sees this build's manifest.
./.github/scripts/Publish-BootstrapScripts.ps1 `
-SasToken '${{ secrets.AZURE_BLOB_AGENT_SAS_TOKEN }}' `
-LegacySasToken '${{ secrets.AZURE_BLOB_AGENT_LEGACY_SAS_TOKEN }}' `
-TableSasToken '${{ secrets.AZURE_TABLE_ADMIN_SAS_TOKEN }}'
}
# Update AdminConfiguration LatestAgentV2* -- ONLY on stable cutover. The hash-oracle
# must never point at a build that is not actually in the stable namespace.
# LatestBootstrapV2ScriptVersion is NOT set here: it belongs to the script publisher
# above, which also owns it on script-only pushes. One writer per field.
$tableSasToken = "${{ secrets.AZURE_TABLE_ADMIN_SAS_TOKEN }}".TrimStart('?')
if ('${{ inputs.publish_as_stable }}' -ne 'true') {
Write-Host "SKIPPED: AdminConfiguration update (versioned-only run -- stable namespace unchanged)"
@@ -289,22 +278,18 @@ jobs:
$sha256 = "${{ steps.artifact.outputs.sha256 }}"
$exeSha256 = "${{ steps.artifact.outputs.exe_sha256 }}"
$agentVersion = "${{ steps.version.outputs.version }}"
$bootstrapVersion = "${{ steps.artifact.outputs.bootstrap_version }}"
$tableUrl = "https://autopilotmonitoreu.table.core.windows.net/AdminConfiguration(PartitionKey='GlobalConfig',RowKey='config')?$tableSasToken"
$tableBody = @{
LatestAgentV2Version = $agentVersion
LatestAgentV2Sha256 = $sha256
LatestAgentV2ExeSha256 = $exeSha256
}
if ($bootstrapVersion) {
$tableBody["LatestBootstrapV2ScriptVersion"] = $bootstrapVersion
}
Invoke-RestMethod -Uri $tableUrl -Method Merge -Headers @{
"Content-Type" = "application/json"
"Accept" = "application/json;odata=nometadata"
"If-Match" = "*"
} -Body ($tableBody | ConvertTo-Json)
Write-Host "AdminConfiguration updated: V2 version=$agentVersion, sha256=$sha256, exeSha256=$exeSha256, bootstrapVersion=$bootstrapVersion"
Write-Host "AdminConfiguration updated: V2 version=$agentVersion, sha256=$sha256, exeSha256=$exeSha256"
}
# Sigstore keyless provenance: binds the ZIP digest to this workflow run (repo, ref,
+13 -45
View File
@@ -1,16 +1,20 @@
name: Pester (Bootstrap scripts)
# Runs the Pester suite for the Intune bootstrap scripts under Windows PowerShell 5.1
# (the runtime IME uses to execute platform scripts) so guard/relax logic changes are
# verified without resetting a device. Like ci.yml it reports rather than blocks.
# Pull-request view of the bootstrap script gates. Like ci.yml it reports rather than
# blocks the merge button.
#
# There is deliberately no push trigger: publish-scripts.yml runs the very same
# reusable gate workflow on every push to main and blocks the upload to the download
# alias on it. A push trigger here would only duplicate that run, with no consequence.
on:
push:
branches: [main]
paths: ['scripts/Bootstrap/**']
pull_request:
branches: [main]
paths: ['scripts/Bootstrap/**']
paths:
- 'scripts/Bootstrap/**'
- 'scripts/CustomerSetup/**'
- '.github/scripts/Publish-BootstrapScripts.ps1'
- '.github/workflows/bootstrap-script-gates.yml'
workflow_dispatch:
permissions:
@@ -21,41 +25,5 @@ concurrency:
cancel-in-progress: true
jobs:
pester:
name: Bootstrap guard tests (PS 5.1)
runs-on: windows-latest
steps:
- uses: actions/checkout@v7
- name: Install Pester 5+
shell: powershell
run: |
Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force -Scope CurrentUser | Out-Null
Install-Module Pester -Scope CurrentUser -Force -SkipPublisherCheck -MinimumVersion 5.5.0
- name: Run Pester
shell: powershell
run: |
Import-Module Pester -MinimumVersion 5.5.0
$c = New-PesterConfiguration
$c.Run.Path = 'scripts/Bootstrap'
$c.Run.Exit = $true
$c.Output.Verbosity = 'Detailed'
Invoke-Pester -Configuration $c
- name: Guard scripts stay pure ASCII
shell: powershell
run: |
$failed = $false
Get-ChildItem 'scripts/Bootstrap' -Filter '*.ps1' | ForEach-Object {
$bytes = [System.IO.File]::ReadAllBytes($_.FullName)
for ($i = 0; $i -lt $bytes.Length; $i++) {
if ($bytes[$i] -gt 127) {
Write-Host "::error file=scripts/Bootstrap/$($_.Name)::Non-ASCII byte 0x$('{0:X2}' -f $bytes[$i]) at offset $i. PS 5.1 (IME) reads BOM-less files as ANSI and corrupts multi-byte chars."
$failed = $true
break
}
}
}
if ($failed) { exit 1 }
Write-Host 'All bootstrap scripts are pure ASCII.'
gates:
uses: ./.github/workflows/bootstrap-script-gates.yml
+56
View File
@@ -0,0 +1,56 @@
name: Publish customer scripts
# Keeps the scripts behind download.autopilotmonitor.com/agent/ current WITHOUT waiting
# for an agent release. Before this workflow existed, the script blobs rotated only on a
# publish_as_stable run of build-agent.yml, so a bootstrap fix sat unpublished until the
# next agent cutover -- while the WDP bootstrap MSI downloads the published copy on every
# enrolling device.
#
# Same publish code as the release path (.github/scripts/Publish-BootstrapScripts.ps1);
# this workflow only supplies a second trigger for it.
on:
push:
branches: [main]
paths:
- 'scripts/Bootstrap/**'
- 'scripts/CustomerSetup/**'
- '.github/scripts/Publish-BootstrapScripts.ps1'
workflow_dispatch:
inputs:
dry_run:
description: 'Run every check and report drift, upload nothing'
required: true
type: boolean
default: false
permissions:
contents: read
# Shared with build-agent.yml: a script publish and an agent release both touch
# version.json, and the bootstrapVersion read-modify-write must not interleave with the
# release writing its agent fields. Never cancel -- a half-published set is worse than a queue.
concurrency:
group: agent-blob-publish
cancel-in-progress: false
jobs:
gates:
uses: ./.github/workflows/bootstrap-script-gates.yml
publish:
name: Publish to the download alias
needs: gates
runs-on: windows-latest
steps:
- uses: actions/checkout@v7
- name: Publish bootstrap + customer scripts
shell: pwsh
run: |
$dryRun = '${{ inputs.dry_run }}' -eq 'true'
./.github/scripts/Publish-BootstrapScripts.ps1 `
-SasToken '${{ secrets.AZURE_BLOB_AGENT_SAS_TOKEN }}' `
-LegacySasToken '${{ secrets.AZURE_BLOB_AGENT_LEGACY_SAS_TOKEN }}' `
-TableSasToken '${{ secrets.AZURE_TABLE_ADMIN_SAS_TOKEN }}' `
-DryRun:$dryRun
@@ -0,0 +1,200 @@
---
type: Concept
title: ESP Completion Starvation — the sits-d Cloud-PC complex
description: Why five Windows 365 Cloud PCs finished enrolling but never completed in the product — a SkipUser=true flow produces none of the signals every completion-gate evaluation site hangs off — plus the two adjacent defects fixed alongside (edge-only user-apps-settled synthesis, orphaned Hello-wizard restart replay) and what a Shell-Core replay deliberately does not do.
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals
tags:
- agent
- esp
- completion
- skip-user
- reboot
- recovery
timestamp: 2026-08-20T00:00:00+02:00
---
# ESP Completion Starvation — the sits-d Cloud-PC complex
Seven Windows 365 Cloud PCs, same tenant, first sign-in the same morning. Two sessions
succeeded within minutes; five sat at `Completion is waiting on: hello_resolution` from
second five onward and ran into the six-hour max-lifetime watchdog — verdict `Incomplete`
on devices whose users were working normally the whole time.
The first diagnosis blamed the mid-ESP reboot (DeviceGuard/DmaGuard, see
[MDM Reboot Coalescing](mdm-reboot-coalescing.md)): the agent is killed in
`Stage=EspAccountSetup` and, so the theory went, loses the Shell-Core ESP exit to the
downtime. The evidence disproved it. `8110e262` ran **six hours uninterrupted** after its
last reboot with the Shell-Core watcher armed — and Windows never wrote a 62407 in all
that time, because on these machines **the user ESP page never existed**.
# Schema
## The actual defect — a satisfied gate nobody evaluates
The five stuck machines carried `SkipUserStatusPage=True` in the enrollment `FirstSync`
key (read 10+ times across three agent processes, `True` on every read — not a stale
single read). Windows honours it: no user ESP page, hence
* no Shell-Core 62407 (`esp_exiting`) — the page that would exit does not exist,
* no `AccountSetupCategory` provisioning registry — nothing writes categories for a page
that never renders,
* desktop within seconds of sign-in.
The reducer is *built* for this flow: **arm B** of `ShouldTransitionToAwaitingHello`
(`SkipUserEsp == true`) declares the gate open, and the missing-prerequisites bookkeeping
already exempts the AccountSetup gate — which is why the sessions reported only
`hello_resolution` as missing. But the gate is a predicate, not a process. Something has
to evaluate it, and every evaluation site hangs off a signal a skip-user flow structurally
cannot produce:
| Evaluation site | Carrier signal | On SkipUser=true |
|---|---|---|
| `HandleEspExitingV1` | Shell-Core 62407 | never fires |
| `HandleEspPhaseChangedV1(FinalizingSetup)` | Shell-Core 62407 via coordinator | never fires |
| `HandleDesktopArrivedV1` fast-path | requires Hello policy **disabled** | Hello was enabled |
| `HandleImeUserSessionCompletedV1` arm-C attempt | requires the recorded final exit | unsatisfiable |
No knock, no promotion, no `HelloSafety` — and the `AdvisoryCompletion` backstop is armed
exclusively from an `EspExiting` signal, so it never armed either. The session idles to
the watchdog with an open gate.
## The fix — the observed skip is the exit evidence
`HandleImeUserSessionCompletedV1`'s completion attempt now accepts the observed skip in
place of the final exit (`skipUserExitEquivalent`): on a flow where no page exists, no
exit can ever say more than the `FirstSync` read already did. Everything else stays
mandatory — AccountSetup anchor, genuine (at-or-after-anchor) IME user-session completion,
DAD-validated real-user desktop — which makes this knock **stricter** than the existing
Shell-Core-carried arm-B promotion (pinned in `ClassicAwaitingHelloGuardTests`, which
requires no IME or desktop evidence at all). Hello semantics ride the existing rails:
enabled-or-unknown promotes to `AwaitingHello` with `HelloSafety` armed (on Cloud PCs
reached via RDP the Hello wizard never appears, so the synthetic timeout resolves the
wait), disabled completes directly with the synthetic `Skipped` outcome.
The knock deliberately lives on the IME user-session edge, not on desktop arrival:
* **Ordering-robust against Fix 10.** An `EspPhaseChanged(AccountSetup)` arriving on
`AwaitingHello` bounces the stage back and cancels `HelloSafety` (the premature-promotion
guard). The IME AccountSetup phase line lands *after* the desktop on this flow, so a
desktop-side knock would be undone two signals later. The IME phase line always precedes
the user-session-complete line in log order, so a bounced promotion is re-knocked by the
very next re-emission — including the restart re-parse.
* **Right semantics.** The IME user session completes when the required-app processing is
done; a desktop-side knock would mark the session `Succeeded` mid-installation.
* Device Preparation is excluded — WDP keeps its Hello+Desktop conjunction and the
`DevicePrepCompletion` backstop.
`ClassicSkipUserEspUserSessionCompletionTests` replays the real `8110e262` signal ordering
end to end (bootstrap `EspConfigDetected` → Hello enabled → desktop → AccountSetup anchor →
IME completion → HelloSafety → `enrollment_complete`), including the Fix-10
bounce-then-re-knock cycle, and pins the negative space (no skip observed / no desktop /
ghost pre-anchor IME completion / WDP).
Why five of seven identically administered Cloud PCs carried `SkipUserStatusPage=True` is
a tenant-side question (ESP profile assignment state at each machine's provisioning
moment); the value did not change during any observed session, and machine age does not
cleanly explain the split. The agent's job is to complete the flow Windows actually ran —
which it now does.
## Adjacent defect — the synthesis was edge-triggered
This one is real but belongs to the `SkipUser=false` flow (the two sessions that
succeeded). The user-apps-settled synthesis
(`MaybeSynthesizeAccountSetupCompleteFromSettledUserApps`, session `caa6cf50`) ran **only**
from `OnEspExited`. On a tenant with ~138 required apps the ESP page exits while apps are
still in flight: the single attempt misses and nothing ever re-checks, even when every
tracked app reaches a terminal state minutes later.
The ESP exit is an **edge** (happens once); settled user-ESP apps are a **level** (reached
later). `EspAndHelloTracker` records the edge in `_espExitObserved` and exposes
`ReevaluateUserAppsSettledSynthesis()`, chained into `ImeLogTracker.OnAppStateChanged` and
into the post-restore `onStateRestored` callback (a live exit observed before the IME
state restore completes would otherwise never get a second look). Gate conditions are
unchanged — no new completion path, only a second opportunity for the existing one. Only a
**live, confirmed post-AccountSetup** exit may be remembered
(`IsConfirmedPostAccountSetupExit`, stricter than the `IsIntermediateDeviceEspExit`
forward guard: unknown counts as not confirmed).
## Adjacent defect — the restart replay was never wired up
`ShellCoreTracker`'s backfill had existed since session `772fe502` with **no caller**; the
Shell-Core `EventLogWatcher` only delivers records written after `Start()`.
`EspAndHelloHost.Start()` now calls it as `BackfillRecentHelloWizardStart` — which is
exactly what it does and all it does:
| Record | Replayed | Why |
|---|---|---|
| 62404 Hello-wizard start | **yes** | Conservative: vetoes a premature "Hello is disabled" skip, can never complete a session by itself. The observation `772fe502` was about. |
| 62407 ESP exit | **no** | Cannot be placed in time — see below. |
| 62407 ESP failure | **no** | Re-injecting a historic failure as fresh can fail a session that recovered on retry (`ANALYZE-ESP-006`). |
A replayed exit is unusable, and every candidate ordering mechanism fails independently:
1. Windows writes the **identical** description `CommercialOOBE_ESPProgress_Page_Exiting`
for the intermediate DeviceSetup→AccountSetup transition and the final exit.
2. Every post-hoc ordering probe (AccountSetup registry, settled-apps) reads state as it
is **now**, not as it was at the event's time.
3. The reducer orders exits by **ingest ordinal**, not timestamp
(`IsPostAccountSetupFinalExit`) — a historic exit replayed today gets a fresher ordinal
than reality and reads as post-AccountSetup by construction.
4. `HandleEspExitingV1` passes `espFinalExitInFlight: true` for every arriving exit; with
restored state, arm C would open on a historic intermediate exit.
`ClassicEspExitingOnRestoredStateTests` pins that the reducer *does* open on any arriving
exit; `ShellCoreTrackerReplayScopeTests` pins that the replay never produces one. Skipped
62407 records are counted and reported once as an `agent_trace`
(`reason: replayed_62407_not_orderable`) — silently dropping evidence is how this class of
bug survives.
**Residual gap, named honestly:** a `SkipUser=false` session whose final exit falls
exactly into an agent downtime *and* whose AccountSetup category never resolves
(`1ec8f4c6` shape) still has no completion carrier after the restart. The normal case is
covered — `ProvisioningStatusTracker` reads the current registry at startup, so a
category that resolved during the downtime fires arm A. The residual shape is rare,
surfaced by `ANALYZE-ESP-005` via the `session_stalled` interim trigger, and deliberately
not patched with a replay that cannot be ordered.
### Sizing the replay window
A constant is provably wrong: the scheduled task has a `BootTrigger` only
(`Program.InstallMode.BuildScheduledTaskXml`) — after an `exception_crash` the agent does
not return until the next boot. `ResolveEspExitBackfillLookbackMinutes` reaches back to
the last moment the previous run is known to have been alive (max of `snapshot.json`
mtime and `LastBootUtc` for kill-type exits), floored at 5 minutes, capped at 360, off on
`first_run`, and immune to cross-reboot clock skew (a future timestamp is ignored).
# Examples
Tenant `sits-d.cloud`, 2026-08-19, seven Windows 365 Cloud PCs, first sign-ins the same
morning. The discriminator is a single registry value:
| Session | SkipUser | `esp_exiting` | AccountSetup registry | Reboots | Outcome |
|---|---|---|---|---|---|
| `08fc6bda` | False | yes | yes | 0 | Succeeded (10 min) |
| `3b9291aa` | False | yes | yes | 0 | Succeeded (2.1 h) |
| `8110e262` | True | — | — | 2 | Incomplete (6 h watchdog) |
| `a89aac2d` | True | — | — | 1 | Incomplete |
| `e7ba63c9` | True | — | — | 1 | Incomplete |
| `cb4a485a` | True | — | — | 2 | Incomplete |
| `3d6278fb` | True | — | — | 1 | Incomplete |
The reboot correlation was a red herring: `8110e262` ran six hours live after its final
reboot and would have seen every signal. `8110e262` and `a89aac2d` both reached
`138/138 completed, 0 failed` and a genuine IME user-session completion — with the fix,
all five complete a few minutes after their IME user session settles.
`08fc6bda` exercises the edge-level fix on the same day: ESP page exited normally while
`categorySucceeded` never confirmed — completed via the user-apps-settled synthesis.
`3b9291aa` shows the independent starvation visibility gap: 99 `app_install_starved`
events on a session recorded as `Succeeded` — surfaced from now on by `ANALYZE-APP-016`.
# Citations
* `src/Shared/AutopilotMonitor.DecisionCore/Engine/DecisionEngine.Classic.cs` — `HandleImeUserSessionCompletedV1`, `skipUserExitEquivalent`
* `src/Shared/AutopilotMonitor.DecisionCore/Engine/DecisionEngine.Shared.cs` — `ShouldTransitionToAwaitingHello` (arm B)
* `src/Agent/AutopilotMonitor.DecisionCore.Tests/ClassicSkipUserEspUserSessionCompletionTests.cs` — end-to-end replay of the 8110e262 ordering
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/ShellCoreTracker.cs` — `BackfillRecentHelloWizardStart`, `ReplayBackfillRecords`
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/EspAndHelloTracker.cs` — `_espExitObserved`, `ReevaluateUserAppsSettledSynthesis`
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/DefaultComponentFactory.cs` — lookback sizing, `OnAppStateChanged` chaining
* [MDM Reboot Coalescing](mdm-reboot-coalescing.md) — the policy attribution for the reboots themselves
* [Decision Engine](decision-engine.md) — the AccountSetup gate and completion arms
+2
View File
@@ -13,6 +13,7 @@ https://docs.autopilotmonitor.com (separate repository).
* [Architecture Guide](architecture.md) - High-level architecture overview and solution layout for contributors.
* [Version Contract](versioning.md) - One version shape for agent, backend and MCP: the ETag-CAS counter blob that mints build numbers, and the manifests published only after a deploy is verified live.
* [Customer Script Publishing](script-publishing.md) - How the customer-facing PowerShell scripts reach download.autopilotmonitor.com/agent/, why publishing is decoupled from agent releases, and the guards keeping the published copy, the version badge and the portal's version oracle in agreement.
* [URL Registry](url-registry.md) - Every well-known host lives in one registry file per component (Constants.cs / utils/config.ts / src/config.ts), enforced by guard tests; includes the agent-download migration to the download alias and its deploy-order constraint.
* [Drift Guards](drift-guards.md) - The 2026-08 fragility-audit follow-up contracts: SignalR message-name catalog (typed on both ends), the inverted-tick RowKey codec, tombstone table tagging, the shrink-only OkAsync baseline, per-section config PATCH with the section field map, and the web's scoped-endpoint builders + sidebar visibility matrix.
@@ -23,6 +24,7 @@ https://docs.autopilotmonitor.com (separate repository).
* [Agent Endpoint Migration](agent/endpoint-migration.md) - How the backend re-homes agents to a new API base URL via the config channel (backend move or per-tenant region move): live-fetch-only, allowlist-validated on both sides, one hop, kill wins.
* [Gather Rule Debug Log](agent/gather-rule-debug-log.md) - EnableGatherRuleDebugLog (ConfigVersion 36): the per-tenant local trace file that explains every gather-rule evaluation decision — scope skips, on_change suppression, empty collector results, logparser position/parse-mode outcomes — for rules that produce no timeline events.
* [MDM Reboot Coalescing](agent/mdm-reboot-coalescing.md) - Attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: aggregated DM-Enterprise 2800 observations (neutral, flush-time watermark) + the ANALYZE-ESP-005 advisory gated on an actually-observed reboot.
* [ESP Completion Starvation — the sits-d Cloud-PC complex](agent/post-reboot-completion-recovery.md) - Why a SkipUser=true flow (no user ESP page → no Shell-Core 62407, no AccountSetup registry) left the completion gate satisfied but never evaluated — five Cloud PCs idled six hours at "waiting on: hello_resolution" — and the fix: the observed skip stands in for the final-exit evidence in the IME user-session completion attempt. Plus the two adjacent defects fixed alongside: the edge-only user-apps-settled synthesis (now level-triggered) and the never-wired Shell-Core restart replay (now Hello-wizard-only, with the reasoning why a replayed 62407 must never reach the reducer).
* [Secure Boot CA-2023 Detection](agent/secureboot-ca2023-detection.md) - Why ANALYZE-SEC-001 v3 reads the UEFI db/KEK variables directly instead of trusting the SecureBoot\Servicing registry, and how the one-sided uefiCa2023FirmwareConfirmed marker plus a not_exists precondition give absence-tolerant suppression across old and new agents.
* [Continue-Anyway Observation Mode](agent/continue-anyway-observation.md) - EnableEspContinueAnywayObservation (ConfigVersion 37, operator-set): Device-phase ESP terminal failures on Continue-Anyway profiles get a bounded 60-min observation instead of an immediate hard fail; the DAD-validated real-user desktop + Hello gate detect the true end, the session completes Succeeded + EspSoftFailure (amber "with issues" badge), expiry un-defangs to the original esp_terminal_failure.
* [Registry App-State Second Pillar](agent/registry-app-state-pillar.md) - IME's authoritative per-app state read straight from the registry (Win32Apps EnforcementStateMessage, EspTrackingWin32Apps, StatusServiceReports) via a snapshot-and-diff watcher; registry/log reconciliation doubles as the built-in IME log-pattern drift alarm.
+12
View File
@@ -1,5 +1,17 @@
# Log
## 2026-08-20
* **Update**: `agent/post-reboot-completion-recovery.md` — retitled to "ESP Completion Starvation — the sits-d Cloud-PC complex" and reframed on the evidence: the reboot was a red herring. The five stuck Cloud PCs carried `SkipUserStatusPage=True` (read 10+ times across three agent processes, `True` every time) and Windows honoured it — no user ESP page, hence no Shell-Core 62407 and no AccountSetup provisioning registry, verified over a six-hour uninterrupted post-reboot run with the watcher armed. Arm B of `ShouldTransitionToAwaitingHello` was open from the bootstrap onward, but every evaluation site hangs off a signal a skip-user flow cannot produce, and the `AdvisoryCompletion` backstop arms exclusively from an `EspExiting` signal — an open gate nobody knocks on. Fix: `HandleImeUserSessionCompletedV1` accepts the observed skip in place of the final-exit evidence (`skipUserExitEquivalent`; anchor + genuine IME completion + real-user desktop stay mandatory, WDP excluded). The knock deliberately lives on the IME edge, not desktop arrival — ordering-robust against Fix 10's AccountSetup bounce-back and semantically "user-phase work finished" rather than mid-install. `ClassicSkipUserEspUserSessionCompletionTests` replays the real 8110e262 ordering end to end. Docs and comments no longer claim an ESP-exit recovery that does not exist (stray duplicate summary in `ShellCoreTracker`, `ImeLogHost` restore-callback comment); the residual `SkipUser=false` exit-in-downtime gap is named honestly and surfaced by `ANALYZE-ESP-005` via `session_stalled` instead of being patched with an unorderable replay.
## 2026-08-19
* **New**: `agent/post-reboot-completion-recovery.md` — a forced mid-ESP reboot (DeviceGuard/DmaGuard) used to cost the session its AccountSetup completion permanently. Two defects: `ShellCoreTracker.BackfillRecentEspExitEvents()` existed but had no caller, so every agent restart dropped an ESP exit (62407) written while it was down; and the user-apps-settled synthesis was edge-triggered on `EspExited` only, so apps settling afterwards never re-opened the gate. `EspAndHelloHost.Start()` now backfills with a lookback sized to the downtime (`previousExit=reboot_kill` → since `LastBootUtc`, clamped to 360 min), and `ReevaluateUserAppsSettledSynthesis()` is chained into `ImeLogTracker.OnAppStateChanged`. No new completion path — the gate conditions are unchanged. Evidence: sits-d Cloud PCs, five `Incomplete` sessions on healthy devices, two of them at `138/138 apps, 0 failed`. Hardened over two review rounds: the remembered edge must be a **live** exit that is **confirmed** post-AccountSetup (a replay carries no evidence of its own position — Shell-Core writes the same `CommercialOOBE_ESPProgress_Page_Exiting` for the intermediate and the final exit, and the registry check reads *now*, not the event's time); the replay hands over the **newest** exit rather than the oldest; the replay is narrowed to the Hello-wizard start (62404) alone — a conservative fact that can only make the agent wait longer. A replayed ESP exit is never re-raised at all: Windows writes the identical description for the intermediate and the final exit, the reducer orders exits by ingest ordinal (fresher than reality for a replay) and `HandleEspExitingV1` passes `espFinalExitInFlight: true` for every arriving exit, so arm C of `ShouldTransitionToAwaitingHello` would open on a historic handoff once the restored state carries AccountSetupEntered + IME user session + desktop. A replayed ESP failure is likewise dropped (it could fail a session that recovered). Skipped records are reported as an `agent_trace`. The window is **measured** from the previous run's last known liveness (snapshot mtime, widened by `LastBootUtc`) instead of a constant — the scheduled task has a BootTrigger only and no restart-on-failure, so a crashed agent returns hours later and five minutes would lose the exit every time. On `first_run` the replay stays off, leaving the happy path untouched.
* **Update**: `rules/analyze-rule-triggers.md` — `on_event:session_stalled` documented as an established interim trigger for the never-terminal stuck session (one emission per session, no throttle needed). `ANALYZE-ESP-005` carries it: both its conditions are monotonic and it has no terminal precondition, and it fires at confidence 95 on the sits-d stalled stream. `ANALYZE-APP-015` deliberately does NOT — it is the worked example of the terminal-precondition trap (lint 1) — and `ANALYZE-ESP-001` does not either, because its `phase_duration` gate on a still-open phase is unverified mid-run. New `ANALYZE-APP-016` ("Required Apps Never Started — ESP Apps Gate Starved") turns the existing `app_install_starved` events into a customer-facing finding and, unlike `ANALYZE-APP-015`, fires even when the session succeeded — session `3b9291aa` was recorded `Succeeded` with 99 required apps that never started installing.
* **New**: `script-publishing.md` — the customer script publishing chain. `.github/scripts/Publish-BootstrapScripts.ps1` becomes the single owner (dev render, version-bump guard, uploads, version oracles, alias verification), called both by the new `publish-scripts.yml` (push to `main` on `scripts/Bootstrap/**` and `scripts/CustomerSetup/**`) and by the `publish_as_stable` branch of `build-agent.yml`. Script publishing no longer waits for an agent release. Blobs are published in a deterministic CRLF/UTF-8-no-BOM form so byte comparisons do not depend on the checkout; the Pester + ASCII gates move into the reusable `bootstrap-script-gates.yml` and block the upload; `AdminConfiguration.LatestBootstrapV2ScriptVersion` now has exactly one writer. The version-bump guard compares a code fingerprint (PowerShell token stream without comments, newlines and line continuations), so only behaviour changes require a `$ScriptVersion` bump — comment and formatting changes publish with a warning.
## 2026-08-18
* **New**: `agent/wdp-scenario-gates.md` — WDP scenario gates (afee7ae0 audit): deterministic ExecutionContext marker as the only gate key (fail-toward-Classic), suppressed Autopilot/ESP-only emissions and hosts (autopilot_profile*/ZTD probe, esp_config unknown null-gate, EspPolicyProviderStallHost, ProvisioningStatusTracker, ModernDeployment noise IDs 100/417/418/1005 + error-backfill skip, starved sweep), and the WDP completion model (High-confidence v2 seed via `enrollmentTypeDeterministic`, desktop-based arm D, 30-min `device_prep_completion` backstop, parked-tripwire WDP branch).
+4
View File
@@ -101,6 +101,10 @@ Interim evaluation changes what a rule's conditions mean mid-flight. Three new l
- WG battery rule (sits-d): `evaluateOn: ["enrollment_end", "whiteglove_sealed"]` — technician gets the channel alert at the bench; Part-2 terminal pass finalizes or resolves the finding.
- Hybrid-affinity stuck (esa): ANALYZE-ID-004 gets `evaluateOn: ["enrollment_end", "on_event:hybrid_login_pending"]` — the second overdue-login event (threshold 65) produces a live finding on the still-InProgress session. ANALYZE-APP-015 stays `enrollment_end`-only until its conditions are hardened (lint 1).
- Stuck after a forced mid-ESP reboot (sits-d Cloud PCs, 2026-08-19): `on_event:session_stalled` is the established trigger for the never-terminal stuck session. The agent emits `session_stalled` exactly once per session (60-minute stall probe), so the trigger is rare by construction and needs no throttle. Five Cloud PCs sat at `InProgress` with **zero findings** for hours — a mid-ESP reboot had killed the agent in `Stage=EspAccountSetup` — before the max-lifetime watchdog eventually terminalised them as `Incomplete`. ANALYZE-ESP-005 carries the trigger and fires on that stream at confidence 95 (verified against session `8110e262`): both its conditions (`mdm_policy_reboot_required.firstRebootUri` exists, `system_reboot_detected` count ≥ 1) are **monotonic** and it has no terminal preconditions, so lint 1 does not apply. ANALYZE-APP-016 is interim-safe for the same reason (`app_install_starved` count ≥ 1 only ever grows).
Two rules were considered and deliberately **not** given the trigger: ANALYZE-APP-015 is the worked example of lint 1 above and stays `enrollment_end`-only; ANALYZE-ESP-001 gates on `phase_duration` of a phase that is still open mid-run, and that evaluation is not pinned by a test — an unverified trigger is not worth a possible false positive on every slow DeviceSetup.
# Product Decisions (settled 2026-08-14)
1. Notify-on-resolve (a "finding healed" message)? **No** (v1) — resolution is visible in UI/MCP.
+137
View File
@@ -0,0 +1,137 @@
---
type: Concept
title: Customer Script Publishing — download.autopilotmonitor.com/agent/
description: How the customer-facing PowerShell scripts reach the download alias, why publishing is decoupled from agent releases, and the guards that keep the published copy, the version badge and the portal's version oracle in agreement.
resource: /.github/scripts/Publish-BootstrapScripts.ps1
tags:
- build
- ci
- release
- bootstrap
- download-alias
timestamp: 2026-08-19T00:00:00+02:00
---
# Overview
Four PowerShell scripts are served to customers from `https://download.autopilotmonitor.com/agent/`:
| Blob | Source | Consumed by |
| --- | --- | --- |
| `Install-AutopilotMonitor.ps1` | `scripts/Bootstrap/Install-AutopilotMonitor.ps1` | Intune platform script (customer uploads a copy); downloaded live on every device by the WDP bootstrap MSI |
| `Install-AutopilotMonitor-Dev.ps1` | rendered from the same source | dev fleet |
| `Test-ShouldBootstrapAgent.ps1` | `scripts/Bootstrap/Test-ShouldBootstrapAgent.ps1` | customer dry run via `irm ... \| iex` |
| `Grant-AutopilotMonitorAddOn.ps1` | `scripts/CustomerSetup/Grant-AutopilotMonitorAddOn.ps1` | optional Graph permission grant |
`Uninstall-AutopilotMonitor.ps1` is deliberately not published — nothing links it, and an
unauthenticated uninstall script on the public download host is not a feature.
The alias is a Front Door route in front of the `agent` container; see
[url-registry.md](url-registry.md) for the host registry and the migration to the alias.
# Publishing chain
`.github/scripts/Publish-BootstrapScripts.ps1` is the single owner. Two workflows call it
with identical arguments:
* **`publish-scripts.yml`** — on every push to `main` touching `scripts/Bootstrap/**`,
`scripts/CustomerSetup/**` or the publisher itself, plus `workflow_dispatch` with a
`dry_run` input.
* **`build-agent.yml`** — inside the `publish_as_stable` branch, after `version.json` is
uploaded, so a release lands a set consistent with what is already live.
Before this split, the script blobs rotated **only** on a stable agent release. A bootstrap
fix therefore sat unpublished until the next agent cutover — while the WDP MSI downloads the
published copy on every enrolling device. Script publishing is now decoupled from the agent
release cadence; the release path still publishes, so it can never fall behind either.
Both workflows share the concurrency group `agent-blob-publish` (`cancel-in-progress: false`).
Both write `version.json`, and the publisher's `bootstrapVersion` read-modify-write must not
interleave with a release writing the agent fields.
# Publisher steps
1. **Parse** `$ScriptVersion` from the bootstrap source.
2. **Dev render** — literal substitution of the two `$AgentDownloadUrl` / `$VersionJsonName`
defaults. A missing anchor literal is a hard failure: a silently un-substituted dev script
would point the dev fleet at the stable agent.
3. **Version-bump guard** — if the published bootstrap changed *behaviour* under an unchanged
`$ScriptVersion`, the run aborts. Comment and formatting changes publish with a warning.
See [Version oracles](#version-oracles).
4. **Upload** every blob with `Cache-Control: no-cache` (they all rotate in place), mirrored
fail-soft to the legacy storage account.
5. **Reconcile the version oracles.**
6. **Verify through the alias** — re-download each blob and compare SHA-256, six attempts
20 s apart. A persistent mismatch means blobs were written but Front Door serves stale
content, and the error names the `az afd endpoint purge` command.
## Deterministic publish form
Every blob is published as **CRLF, UTF-8 without BOM**, normalized by the publisher rather
than taken from the working copy. Git normalizes line endings on checkout — a Windows runner
yields CRLF, a `core.autocrlf=false` clone yields LF — so publishing raw working-copy bytes
would emit different blobs for identical source and make every byte comparison, guard and
verification alike, depend on where the workflow ran. CRLF is also what PS 5.1 expects on the
device and what has always been published.
The ASCII-only rule for `scripts/Bootstrap/*.ps1` (enforced by the gates below) is unrelated
but adjacent: PS 5.1 reads BOM-less files as ANSI and corrupts multi-byte characters.
## Version oracles
Three places state the bootstrap script version, and all three are derived from
`$ScriptVersion` in the source:
* `version.json.bootstrapVersion` — read by the docs badge. Written by the publisher as a
read-modify-write under `If-Match`, so a concurrent agent release keeps its own fields.
`build-agent.yml` seeds the field when it builds the manifest so the published manifest is
never briefly missing it; the publisher owns and re-asserts it afterwards.
* `AdminConfiguration.LatestBootstrapV2ScriptVersion` — read by the portal to tell a customer
their uploaded copy is outdated. **Only** the publisher writes it; `build-agent.yml` writes
the `LatestAgentV2*` fields and nothing else. One writer per field.
* The `Bootstrap script version: v<x>` marker the script logs, parsed out of captured platform
script stdout by `utils/bootstrapVersion.ts`.
The version-bump guard exists because those three would otherwise keep asserting a version
that no longer matches what customers download: a behaviour change under an unchanged version
is invisible to every consumer, and customers decide from that version whether to re-upload
their Intune copy.
"Changed" means changed **behaviour**, matching what the source file's own comment on
`$ScriptVersion` already asks for ("bump on meaningful changes"). The guard compares a code
fingerprint — the PowerShell token stream with comments, newlines and line continuations
removed — so comment fixes, typo corrections in comments and reflow publish with a
`::warning::` instead of failing. Tokenising rather than regex-stripping keeps a `#` inside a
string literal from being mistaken for a comment.
# Gates
`.github/workflows/bootstrap-script-gates.yml` is a `workflow_call` workflow holding the
checks once:
* Pester suite under **Windows PowerShell 5.1** — the runtime IME uses for platform scripts.
* ASCII-only scan over `scripts/Bootstrap/*.ps1`.
* `Publish-BootstrapScripts.ps1 -DryRun` — exercises the dev-render anchors and the
version-bump guard, and reports per blob whether the alias is current, stale or missing.
Two callers: `pester.yml` on pull requests (reports, does not block the merge button), and
`publish-scripts.yml` on pushes to `main`, where the gates job is a `needs:` dependency of the
publish job and therefore **blocks** the upload. `pester.yml` deliberately has no push trigger
— it would duplicate the gate run with no consequence.
# Operating it
* Dry run and drift check, no credentials needed for the read paths:
`pwsh .github/scripts/Publish-BootstrapScripts.ps1 -DryRun`
* Republish without a code change: `workflow_dispatch` on **Publish customer scripts** with
`dry_run=false`.
* Front Door serving stale content after a successful upload:
`az afd endpoint purge --resource-group rg-autopilotmonitor-prd-gwc --profile-name autopilotmonitor-fd --endpoint-name apm-download --content-paths '/agent/*'`
# Citations
* `.github/scripts/Publish-BootstrapScripts.ps1`
* `.github/workflows/publish-scripts.yml`, `.github/workflows/bootstrap-script-gates.yml`, `.github/workflows/pester.yml`
* `.github/workflows/build-agent.yml` (the `publish_as_stable` branch of the upload step)
* `src/Agent/AutopilotMonitor.BootstrapMsi/Invoke-BootstrapDownload.ps1`
* `src/Web/autopilot-monitor-web/utils/bootstrapVersion.ts`
+113
View File
@@ -0,0 +1,113 @@
{
"$schema": "../schema/analyze-rule.schema.json",
"ruleId": "ANALYZE-APP-016",
"title": "Required Apps Never Started — ESP Apps Gate Starved",
"description": "One or more required user-ESP apps never started installing while the ESP AccountSetup apps gate waited on them — the agent observed the ESP page exit with those apps still in a non-terminal, never-started state. Unlike ANALYZE-APP-015 (which reports that completion was DEFERRED because of a pending app, and stays silent once the session reaches a terminal verdict) this rule reports the delivery gap itself and fires regardless of the final outcome: an enrollment that Windows called done while required apps never ran is a successful enrollment with missing software, and the operator must see it.",
"severity": "warning",
"category": "apps",
"version": "1.0.0",
"author": "Autopilot Monitor",
"enabled": true,
"evaluateOn": [
"enrollment_end",
"whiteglove_sealed",
"on_event:session_stalled"
],
"trigger": "single",
"baseConfidence": 75,
"confidenceThreshold": 70,
"conditions": [
{
"signal": "starved_apps_present",
"source": "event_count",
"eventType": "app_install_starved",
"operator": "count_gte",
"value": "1",
"required": true
},
{
"signal": "many_starved_apps",
"source": "event_count",
"eventType": "app_install_starved",
"operator": "count_gte",
"value": "5",
"required": false
},
{
"signal": "appName",
"source": "event_type",
"eventType": "app_install_starved",
"dataField": "appName",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "state",
"source": "event_type",
"eventType": "app_install_starved",
"dataField": "state",
"operator": "exists",
"value": "",
"required": false
}
],
"confidenceFactors": [
{
"signal": "many_starved_apps",
"condition": "exists",
"weight": 15
},
{
"signal": "appName",
"condition": "exists",
"weight": 10
}
],
"explanation": "At least one **required** app assigned to the user ESP never started installing. The Intune Management Extension never moved it out of its initial state (`{{state}}`) — it did not download, did not install, and did not fail — while the ESP AccountSetup apps gate was waiting on it. First app observed: `{{appName}}`.\n\nEvery starved app has its own `app_install_starved` timeline event carrying `appId`, `appName` and `state` — open the timeline and filter on that event type for the complete list.\n\nThis is a **delivery** problem, not an install error, and it is invisible in the Intune console: the app simply never appears. Typical causes:\n\n- far more required apps targeted at the ESP than the gate can service in the available window,\n- requirement rules that are never met on this device (OS / architecture / registry / script checks),\n- an unsatisfied dependency or supersedence chain,\n- an app that needs a context unavailable during provisioning (e.g. a VPN client waiting on a portal sign-in),\n- a mid-ESP reboot that cut the gate short — check whether the mid-ESP reboot advisory also fired for this session.\n\nNote that this finding is independent of the session verdict. A session can be marked **Succeeded** and still be missing every app listed here.",
"remediation": [
{
"title": "Shrink the ESP blocking set",
"steps": [
"Open the Enrollment Status Page profile and review the blocking app list — every required app there must finish before the user reaches the desktop.",
"Move apps that do not have to be present at first logon from *required* to *available*, or remove them from the ESP's blocking apps selection.",
"Prefer a small, deliberately chosen blocking set (security agent, VPN, management tooling) over the full application catalogue."
]
},
{
"title": "Unblock the individual apps",
"steps": [
"Collect the app names from the `app_install_starved` timeline events in this session.",
"For each app, review its requirement rules and dependencies in Intune — IntuneManagementExtension.log records every requirement evaluation.",
"Verify the assignment actually targets the signed-in user (user-targeted apps need an established Entra user affinity; check the hybrid user-affinity finding if it also fired).",
"Re-run an enrollment and confirm the apps now reach a terminal state before the ESP exits."
]
},
{
"title": "Rule out a truncated gate",
"steps": [
"Check whether a mid-ESP reboot was observed for this session — a forced restart during AccountSetup cuts the apps gate short and can starve everything still queued.",
"If so, reassign the reboot-forcing device policies to user groups so the restart lands after AccountSetup."
]
}
],
"relatedDocs": [
{
"title": "Microsoft Learn: Enrollment Status Page — blocking apps",
"url": "https://learn.microsoft.com/en-us/intune/intune-service/enrollment/windows-enrollment-status"
},
{
"title": "Microsoft Learn: Win32 app requirement rules",
"url": "https://learn.microsoft.com/en-us/intune/intune-service/apps/apps-win32-add"
}
],
"markSessionAsFailedDefault": false,
"tags": [
"apps",
"esp",
"starvation",
"assignment",
"blocking",
"delivery-gap"
]
}
+3 -2
View File
@@ -5,12 +5,13 @@
"description": "Attributes an observed enrollment reboot to device-assigned MDM policies that Windows flagged as reboot-required. Fires ONLY when both facts are present in the session: at least one actually-observed reboot (system_reboot_detected) AND reboot-required policy flags (mdm_policy_reboot_required). Policy flags alone — e.g. from a user-policy sync during AccountSetup — force no reboot and stay advisory-silent.",
"severity": "warning",
"category": "esp",
"version": "1.2.0",
"version": "1.3.0",
"author": "Autopilot Monitor",
"enabled": true,
"evaluateOn": [
"enrollment_end",
"whiteglove_sealed"
"whiteglove_sealed",
"on_event:session_stalled"
],
"trigger": "single",
"baseConfidence": 70,
+115 -2
View File
@@ -831,6 +831,118 @@
"pending"
]
},
{
"ruleId": "ANALYZE-APP-016",
"title": "Required Apps Never Started — ESP Apps Gate Starved",
"description": "One or more required user-ESP apps never started installing while the ESP AccountSetup apps gate waited on them — the agent observed the ESP page exit with those apps still in a non-terminal, never-started state. Unlike ANALYZE-APP-015 (which reports that completion was DEFERRED because of a pending app, and stays silent once the session reaches a terminal verdict) this rule reports the delivery gap itself and fires regardless of the final outcome: an enrollment that Windows called done while required apps never ran is a successful enrollment with missing software, and the operator must see it.",
"severity": "warning",
"category": "apps",
"version": "1.0.0",
"author": "Autopilot Monitor",
"enabled": true,
"evaluateOn": [
"enrollment_end",
"whiteglove_sealed",
"on_event:session_stalled"
],
"trigger": "single",
"baseConfidence": 75,
"confidenceThreshold": 70,
"conditions": [
{
"signal": "starved_apps_present",
"source": "event_count",
"eventType": "app_install_starved",
"operator": "count_gte",
"value": "1",
"required": true
},
{
"signal": "many_starved_apps",
"source": "event_count",
"eventType": "app_install_starved",
"operator": "count_gte",
"value": "5",
"required": false
},
{
"signal": "appName",
"source": "event_type",
"eventType": "app_install_starved",
"dataField": "appName",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "state",
"source": "event_type",
"eventType": "app_install_starved",
"dataField": "state",
"operator": "exists",
"value": "",
"required": false
}
],
"confidenceFactors": [
{
"signal": "many_starved_apps",
"condition": "exists",
"weight": 15
},
{
"signal": "appName",
"condition": "exists",
"weight": 10
}
],
"explanation": "At least one **required** app assigned to the user ESP never started installing. The Intune Management Extension never moved it out of its initial state (`{{state}}`) — it did not download, did not install, and did not fail — while the ESP AccountSetup apps gate was waiting on it. First app observed: `{{appName}}`.\n\nEvery starved app has its own `app_install_starved` timeline event carrying `appId`, `appName` and `state` — open the timeline and filter on that event type for the complete list.\n\nThis is a **delivery** problem, not an install error, and it is invisible in the Intune console: the app simply never appears. Typical causes:\n\n- far more required apps targeted at the ESP than the gate can service in the available window,\n- requirement rules that are never met on this device (OS / architecture / registry / script checks),\n- an unsatisfied dependency or supersedence chain,\n- an app that needs a context unavailable during provisioning (e.g. a VPN client waiting on a portal sign-in),\n- a mid-ESP reboot that cut the gate short — check whether the mid-ESP reboot advisory also fired for this session.\n\nNote that this finding is independent of the session verdict. A session can be marked **Succeeded** and still be missing every app listed here.",
"remediation": [
{
"title": "Shrink the ESP blocking set",
"steps": [
"Open the Enrollment Status Page profile and review the blocking app list — every required app there must finish before the user reaches the desktop.",
"Move apps that do not have to be present at first logon from *required* to *available*, or remove them from the ESP's blocking apps selection.",
"Prefer a small, deliberately chosen blocking set (security agent, VPN, management tooling) over the full application catalogue."
]
},
{
"title": "Unblock the individual apps",
"steps": [
"Collect the app names from the `app_install_starved` timeline events in this session.",
"For each app, review its requirement rules and dependencies in Intune — IntuneManagementExtension.log records every requirement evaluation.",
"Verify the assignment actually targets the signed-in user (user-targeted apps need an established Entra user affinity; check the hybrid user-affinity finding if it also fired).",
"Re-run an enrollment and confirm the apps now reach a terminal state before the ESP exits."
]
},
{
"title": "Rule out a truncated gate",
"steps": [
"Check whether a mid-ESP reboot was observed for this session — a forced restart during AccountSetup cuts the apps gate short and can starve everything still queued.",
"If so, reassign the reboot-forcing device policies to user groups so the restart lands after AccountSetup."
]
}
],
"relatedDocs": [
{
"title": "Microsoft Learn: Enrollment Status Page — blocking apps",
"url": "https://learn.microsoft.com/en-us/intune/intune-service/enrollment/windows-enrollment-status"
},
{
"title": "Microsoft Learn: Win32 app requirement rules",
"url": "https://learn.microsoft.com/en-us/intune/intune-service/apps/apps-win32-add"
}
],
"markSessionAsFailedDefault": false,
"tags": [
"apps",
"esp",
"starvation",
"assignment",
"blocking",
"delivery-gap"
]
},
{
"ruleId": "ANALYZE-CORR-003",
"title": "Proxy Configuration Causing Download Failure",
@@ -2069,12 +2181,13 @@
"description": "Attributes an observed enrollment reboot to device-assigned MDM policies that Windows flagged as reboot-required. Fires ONLY when both facts are present in the session: at least one actually-observed reboot (system_reboot_detected) AND reboot-required policy flags (mdm_policy_reboot_required). Policy flags alone — e.g. from a user-policy sync during AccountSetup — force no reboot and stay advisory-silent.",
"severity": "warning",
"category": "esp",
"version": "1.2.0",
"version": "1.3.0",
"author": "Autopilot Monitor",
"enabled": true,
"evaluateOn": [
"enrollment_end",
"whiteglove_sealed"
"whiteglove_sealed",
"on_event:session_stalled"
],
"trigger": "single",
"baseConfidence": 70,
@@ -17,14 +17,22 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Integration
{
/// <summary>
/// Plan §EspExiting Adapter — wire-through verification of the production EspExiting path:
/// <c>ShellCoreTracker.HandleBackfillRecord</c> → <c>EspExited</c> event → inner re-raise on
/// <c>EspAndHelloTracker.EspExited</c> → <c>EspAndHelloTrackerAdapter</c> → ingress → engine
/// reducer. This is the path the live V2-Host uses; without the coordinator forward the
/// EspExiting signal would never reach the engine in production.
/// <c>ShellCoreTracker.ProcessEvent</c> (live Shell-Core 62407) → <c>EspExited</c> event →
/// inner re-raise on <c>EspAndHelloTracker.EspExited</c> → <c>EspAndHelloTrackerAdapter</c> →
/// ingress → engine reducer. Without the coordinator forward the EspExiting signal would
/// never reach the engine in production.
/// <para>
/// The first test isolates the tracker's <c>EspExited</c> contract (source-event timestamp
/// preserved through backfill); the coordinator tests drive the full production chain through
/// to the <c>HelloSafety</c> deadline when AccountSetup is already entered.
/// preserved rather than collapsed to wall-clock-now); the coordinator tests drive the full
/// production chain through to the <c>HelloSafety</c> deadline when AccountSetup is already
/// entered.
/// </para>
/// <para>
/// It used to drive that first test through <c>HandleBackfillRecord</c> because the replay is
/// easier to call than the event-log watcher — but the replay no longer raises
/// <c>EspExited</c> at all (a replayed 62407 cannot be placed in time; see
/// <c>ShellCoreTrackerReplayScopeTests</c>), and it never was the production path: the replay
/// had no caller when this test was written.
/// </para>
/// </summary>
public sealed class ShellCoreToEngineEspExitingTests
@@ -34,12 +42,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Integration
"CommercialOOBE_ESPProgress_Page_Exiting"; // matches EspExitingPattern in ShellCoreTracker
[Fact]
public void Tracker_backfill_record_raises_EspExited_with_source_timestamp()
public void Tracker_live_record_raises_EspExited_with_source_timestamp()
{
// Tracker-level contract: HandleBackfillRecord raises EspExited carrying the source
// event time (not Clock.UtcNow). The EspExited→EspExiting signal mapping and SourceOrigin
// are covered by the coordinator tests below; here we isolate the backfill→event link
// that the production coordinator depends on.
// Tracker-level contract: the live 62407 path raises EspExited carrying the source
// event time (record.TimeCreated, not Clock.UtcNow). The EspExited→EspExiting signal
// mapping and SourceOrigin are covered by the coordinator tests below; here we isolate
// the event-log→event link that the production coordinator depends on.
using var tmp = new TempDirectory();
var logger = new AgentLogger(tmp.Path, AgentLogLevel.Info);
var clock = new VirtualClock(ClockNow);
@@ -57,7 +65,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Integration
// Source time IS in the past relative to the clock — verify the tracker carries the
// historical timestamp on the event rather than collapsing to Clock.UtcNow.
var sourceTime = ClockNow.AddMinutes(-2);
tracker.HandleBackfillRecord(ShellCoreTracker.EventId_ShellCore_WebAppEvent, EspExitingDescription, sourceTime);
tracker.ProcessEvent(
eventId: ShellCoreTracker.EventId_ShellCore_WebAppEvent,
description: EspExitingDescription,
timestamp: sourceTime,
providerName: "Microsoft-Windows-Shell-Core",
isBackfill: false);
Assert.NotNull(captured);
Assert.Equal(sourceTime, captured!.OccurredAtUtc);
@@ -170,5 +170,53 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
Assert.Empty(f.StarvedPosts());
Assert.Empty(coordinator.StarvedAppsReported);
}
[Fact]
public void Reevaluate_does_not_re_emit_starved_warnings()
{
// sits-d Cloud-PC fix (2026-08-19): the synthesis re-check now runs on EVERY IME
// app-state transition. It must not turn the one-shot starvation warning into a
// per-transition stream — the re-check path suppresses the emission entirely
// (the per-appId dedupe would swallow the duplicates, but walking the probe
// hundreds of times on the IME hot path is waste, not safety).
using var f = new Fixture();
using var coordinator = f.BuildCoordinator(
() => false,
() => new[] { StarvedApp("app-1", "Contoso Backgrounds") });
using var adapter = new EspAndHelloTrackerAdapter(coordinator, f.Ingress, f.Clock);
coordinator.TriggerEspExitedForTest(Fixed);
var afterExit = f.StarvedPosts().Count;
Assert.Equal(1, afterExit);
for (var i = 0; i < 10; i++) coordinator.ReevaluateUserAppsSettledSynthesis();
Assert.Equal(afterExit, f.StarvedPosts().Count);
Assert.Single(coordinator.StarvedAppsReported);
}
[Fact]
public void Starved_apps_that_settle_later_still_complete_the_enrollment()
{
// The real sits-d shape: the ESP exits with apps still starved (warning emitted),
// the apps then all settle, and the re-check must open the gate — the warning
// already on the timeline is history, not a permanent veto.
using var f = new Fixture();
var settled = false;
using var coordinator = f.BuildCoordinator(
() => settled,
() => new[] { StarvedApp("app-1", "Contoso Backgrounds") });
using var adapter = new EspAndHelloTrackerAdapter(coordinator, f.Ingress, f.Clock);
coordinator.TriggerEspExitedForTest(Fixed);
Assert.Single(f.StarvedPosts());
Assert.DoesNotContain(f.Ingress.Posted, p => p.Kind == DecisionSignalKind.AccountSetupProvisioningComplete);
settled = true;
coordinator.ReevaluateUserAppsSettledSynthesis();
Assert.Equal(1, f.Ingress.Posted.Count(p => p.Kind == DecisionSignalKind.AccountSetupProvisioningComplete));
Assert.Single(f.StarvedPosts());
}
}
}
@@ -35,14 +35,25 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
public Fixture() { Logger = new AgentLogger(Tmp.Path, AgentLogLevel.Debug); }
public EspAndHelloTracker BuildCoordinator(Func<bool> settledProbe)
=> BuildCoordinator(settledProbe, accountSetupActivityProbe: () => true, skipUserEsp: false);
/// <summary>
/// Codex review P1 (2026-08-19): the remembered exit edge is gated on positive evidence
/// that the exit is post-AccountSetup, so these two probes decide whether the deferred
/// re-check may fire at all.
/// </summary>
public EspAndHelloTracker BuildCoordinator(
Func<bool> settledProbe,
Func<bool> accountSetupActivityProbe,
bool? skipUserEsp)
{
return new EspAndHelloTracker(
sessionId: "S1",
tenantId: "T1",
post: new InformationalEventPost(TrackerPostSink, Clock),
logger: Logger,
skipConfigProbe: () => ((bool?)false, (bool?)false),
accountSetupActivityProbe: () => true,
skipConfigProbe: () => (skipUserEsp, (bool?)false),
accountSetupActivityProbe: accountSetupActivityProbe,
userEspAppsSettledProbe: settledProbe);
}
@@ -159,5 +170,174 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
var posted = Assert.Single(f.Ingress.Posted);
Assert.Equal(DecisionSignalKind.EspExiting, posted.Kind);
}
// ------------------------------------------------------------------------------
// sits-d Cloud-PC fix (2026-08-19) — the synthesis is no longer edge-only.
// Sessions 8110e262 / a89aac2d: the ESP page exited while 138 required user-ESP apps
// were still in flight, they all reached a terminal state (0 failed) minutes later,
// and because nothing re-checked, the enrollment never completed.
// ------------------------------------------------------------------------------
[Fact]
public void Reevaluate_after_apps_settle_synthesizes_without_a_second_exit()
{
using var f = new Fixture();
var settled = false;
using var coordinator = f.BuildCoordinator(() => settled);
using var adapter = new EspAndHelloTrackerAdapter(coordinator, f.Ingress, f.Clock);
coordinator.TriggerEspExitedForTest(Fixed);
Assert.DoesNotContain(f.Ingress.Posted, p => p.Kind == DecisionSignalKind.AccountSetupProvisioningComplete);
// The apps settle later; only the IME app-state callback fires — there is NO second
// ESP exit on a real device, which is exactly why the edge-only version starved.
settled = true;
coordinator.ReevaluateUserAppsSettledSynthesis();
Assert.Equal(1, f.Ingress.Posted.Count(p => p.Kind == DecisionSignalKind.AccountSetupProvisioningComplete));
Assert.True(coordinator.UserAppsSettledSynthesisFiredForTest);
var info = Assert.Single(f.TrackerPostSink.Posted);
var data = Assert.IsAssignableFrom<System.Collections.Generic.IReadOnlyDictionary<string, object>>(info.TypedPayload);
Assert.Equal("esp_exited_user_apps_settled_category_unresolved", data["fallbackReason"]);
}
[Fact]
public void Reevaluate_without_an_observed_esp_exit_never_synthesizes()
{
// The gate needs BOTH facts. Settled apps alone must never open it — otherwise every
// quiet moment during AccountSetup would look like completion.
using var f = new Fixture();
using var coordinator = f.BuildCoordinator(() => true);
using var adapter = new EspAndHelloTrackerAdapter(coordinator, f.Ingress, f.Clock);
coordinator.ReevaluateUserAppsSettledSynthesis();
coordinator.ReevaluateUserAppsSettledSynthesis();
Assert.Empty(f.Ingress.Posted);
Assert.Empty(f.TrackerPostSink.Posted);
Assert.False(coordinator.UserAppsSettledSynthesisFiredForTest);
}
[Fact]
public void Reevaluate_is_idempotent_after_the_synthesis_fired()
{
// The IME callback fires per app-state transition — on these tenants that is 138+
// times. The synthesis must stay fire-once.
using var f = new Fixture();
using var coordinator = f.BuildCoordinator(() => true);
using var adapter = new EspAndHelloTrackerAdapter(coordinator, f.Ingress, f.Clock);
coordinator.TriggerEspExitedForTest(Fixed);
for (var i = 0; i < 20; i++) coordinator.ReevaluateUserAppsSettledSynthesis();
Assert.Equal(1, f.Ingress.Posted.Count(p => p.Kind == DecisionSignalKind.AccountSetupProvisioningComplete));
Assert.Single(f.TrackerPostSink.Posted);
}
// ------------------------------------------------------------------------------
// Codex review P1 (2026-08-19): Shell-Core raises 62407 at EVERY ESP phase transition.
// Remembering the intermediate DeviceSetup→AccountSetup exit would let the deferred
// re-check open the strong AccountSetup gate as soon as the last user app settles —
// while the AccountSetup page is still up and its other subcategories still running.
// ------------------------------------------------------------------------------
[Fact]
public void Intermediate_device_exit_is_not_remembered_as_the_edge()
{
using var f = new Fixture();
var settled = false;
// Classic user-driven enrollment (SkipUser=false) that has NOT reached AccountSetup:
// this 62407 is the DeviceSetup→AccountSetup transition, not the final exit.
using var coordinator = f.BuildCoordinator(
() => settled, accountSetupActivityProbe: () => false, skipUserEsp: false);
using var adapter = new EspAndHelloTrackerAdapter(coordinator, f.Ingress, f.Clock);
coordinator.TriggerEspExitedForTest(Fixed);
// Apps settle afterwards — the level is reached, but the edge was never a valid one.
settled = true;
for (var i = 0; i < 10; i++) coordinator.ReevaluateUserAppsSettledSynthesis();
Assert.DoesNotContain(f.Ingress.Posted, p => p.Kind == DecisionSignalKind.AccountSetupProvisioningComplete);
Assert.False(coordinator.UserAppsSettledSynthesisFiredForTest);
Assert.Empty(f.TrackerPostSink.Posted);
}
[Fact]
public void Unknown_skip_user_is_treated_as_unconfirmed()
{
// Erring strict costs at worst today's stall; erring loose costs a premature Succeeded.
using var f = new Fixture();
var settled = false;
using var coordinator = f.BuildCoordinator(
() => settled, accountSetupActivityProbe: () => false, skipUserEsp: null);
using var adapter = new EspAndHelloTrackerAdapter(coordinator, f.Ingress, f.Clock);
coordinator.TriggerEspExitedForTest(Fixed);
settled = true;
coordinator.ReevaluateUserAppsSettledSynthesis();
Assert.False(coordinator.UserAppsSettledSynthesisFiredForTest);
}
[Fact]
public void Skip_user_profile_confirms_the_device_exit_as_final()
{
// SkipUser=true means there is no user ESP at all — the Device-ESP exit IS the final
// one and no second exit is coming, so the edge is legitimately remembered.
using var f = new Fixture();
var settled = false;
using var coordinator = f.BuildCoordinator(
() => settled, accountSetupActivityProbe: () => false, skipUserEsp: true);
using var adapter = new EspAndHelloTrackerAdapter(coordinator, f.Ingress, f.Clock);
coordinator.TriggerEspExitedForTest(Fixed);
settled = true;
coordinator.ReevaluateUserAppsSettledSynthesis();
Assert.Equal(1, f.Ingress.Posted.Count(p => p.Kind == DecisionSignalKind.AccountSetupProvisioningComplete));
}
[Fact]
public void Account_setup_activity_confirms_the_edge_for_the_deferred_recheck()
{
// The real reboot shape: AccountSetup activity is visible in the registry, the final
// exit lands, the apps settle a moment later.
using var f = new Fixture();
var settled = false;
using var coordinator = f.BuildCoordinator(
() => settled, accountSetupActivityProbe: () => true, skipUserEsp: false);
using var adapter = new EspAndHelloTrackerAdapter(coordinator, f.Ingress, f.Clock);
coordinator.TriggerEspExitedForTest(Fixed);
Assert.DoesNotContain(f.Ingress.Posted, p => p.Kind == DecisionSignalKind.AccountSetupProvisioningComplete);
settled = true;
coordinator.ReevaluateUserAppsSettledSynthesis();
Assert.Equal(1, f.Ingress.Posted.Count(p => p.Kind == DecisionSignalKind.AccountSetupProvisioningComplete));
}
// NOTE: the "replayed exit" cases that used to live here are gone by construction —
// ShellCoreTracker never re-raises a 62407 from the replay any more, so no backfilled exit
// can reach this handler at all. That contract is pinned one layer up, in
// ShellCoreTrackerReplayScopeTests, together with the reducer-side proof of why it matters
// (ClassicEspExitingOnRestoredStateTests).
[Fact]
public void Reevaluate_stays_silent_while_apps_are_still_unsettled()
{
using var f = new Fixture();
using var coordinator = f.BuildCoordinator(() => false);
using var adapter = new EspAndHelloTrackerAdapter(coordinator, f.Ingress, f.Clock);
coordinator.TriggerEspExitedForTest(Fixed);
for (var i = 0; i < 5; i++) coordinator.ReevaluateUserAppsSettledSynthesis();
var posted = Assert.Single(f.Ingress.Posted);
Assert.Equal(DecisionSignalKind.EspExiting, posted.Kind);
Assert.False(coordinator.UserAppsSettledSynthesisFiredForTest);
}
}
}
@@ -0,0 +1,322 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
{
/// <summary>
/// Scope of the Shell-Core replay: it recovers the Hello-wizard start and NOTHING else.
/// <para>
/// The replay had existed since session 772fe502 with no caller. Wiring it up (2026-08-19)
/// activated three rails at once, and only one of them is safe to replay:
/// </para>
/// <list type="bullet">
/// <item><b>62404 Hello-wizard start — replayed.</b> A conservative fact: it vetoes a
/// premature "Hello is disabled" skip and can never by itself complete a session, so
/// replaying it can only make the agent wait longer, never finish early.</item>
/// <item><b>62407 ESP exit — never replayed.</b> Windows writes the identical description
/// for the intermediate DeviceSetup-to-AccountSetup transition and the final exit, so the
/// record carries no evidence of its own position; the reducer orders exits by ingest
/// ordinal, which for a replay is fresher than reality; and arm C of
/// ShouldTransitionToAwaitingHello opens on any arriving exit once the restored state
/// carries AccountSetupEntered + a genuine IME user session + desktop arrival. See
/// ClassicEspExitingOnRestoredStateTests for the reducer half of that proof.</item>
/// <item><b>62407 ESP failure — never replayed.</b> Re-injecting a historic failure as
/// fresh can fail a session that recovered (ANALYZE-ESP-006).</item>
/// </list>
/// </summary>
public sealed class ShellCoreTrackerReplayScopeTests
{
private static readonly DateTime ClockNow = new DateTime(2026, 8, 19, 8, 43, 16, DateTimeKind.Utc);
// The REAL Shell-Core wording. Byte-identical for the intermediate DeviceSetup-to-
// AccountSetup transition and for the final post-AccountSetup exit — that identity is the
// whole reason a replayed exit cannot be classified.
private const string EspExitDescription = "CommercialOOBE_ESPProgress_Page_Exiting";
private const string EspFailureDescription = "CommercialOOBE_ESPProgress_Failure";
private const string AadHelloDescription = "CloudExperienceHost web app activity: AADHello";
private const string OtherWebAppDescription = "CloudExperienceHost web app activity: SomethingElse";
private sealed class Fixture : IDisposable
{
public TempDirectory Tmp { get; } = new TempDirectory();
public FakeSignalIngressSink PostSink { get; } = new FakeSignalIngressSink();
public VirtualClock Clock { get; } = new VirtualClock(ClockNow);
public ShellCoreTracker Build() => new ShellCoreTracker(
sessionId: "S1",
tenantId: "T1",
post: new InformationalEventPost(PostSink, Clock),
logger: new AgentLogger(Tmp.Path, AgentLogLevel.Info),
helloTracker: null);
public List<FakeSignalIngressSink.PostedSignal> AgentTraces() =>
PostSink.Posted.Where(p =>
p.Payload != null
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
&& et == "agent_trace").ToList();
public void Dispose() => Tmp.Dispose();
}
private static (int, string, DateTime) Record(int id, string description, DateTime at)
=> (id, description, at);
// ------------------------------------------------------------------ the wizard rail
[Fact]
public void The_hello_wizard_start_is_replayed_with_its_original_source_time()
{
// Session 772fe502: an agent restarted while the user sits inside the Hello wizard
// would otherwise never learn the wizard ran. This is the observation the replay
// exists for.
using var f = new Fixture();
using var tracker = f.Build();
var order = new List<string>();
HelloWizardStartedEventArgs? captured = null;
tracker.HelloWizardStarted += (_, args) => { order.Add("wizard"); captured = args; };
tracker.FinalizingSetupPhaseTriggered += (_, reason) => order.Add($"finalizing:{reason}");
var duringDowntime = ClockNow.AddMinutes(-37);
tracker.ReplayBackfillRecords(new List<(int, string, DateTime)>
{
Record(ShellCoreTracker.EventId_ShellCore_WebAppStarted, AadHelloDescription, duringDowntime),
});
Assert.Equal(new[] { "wizard", "finalizing:hello_wizard_started" }, order);
Assert.NotNull(captured);
// The historical time must survive the replay — collapsing to now would date the
// wizard after the restart and skew every duration derived from it.
Assert.Equal(duringDowntime, captured!.OccurredAtUtc);
}
[Fact]
public void The_wizard_replay_is_single_shot_across_repeated_records()
{
using var f = new Fixture();
using var tracker = f.Build();
var starts = 0;
tracker.HelloWizardStarted += (_, __) => starts++;
tracker.ReplayBackfillRecords(new List<(int, string, DateTime)>
{
Record(ShellCoreTracker.EventId_ShellCore_WebAppStarted, AadHelloDescription, ClockNow.AddMinutes(-30)),
Record(ShellCoreTracker.EventId_ShellCore_WebAppStarted, AadHelloDescription, ClockNow.AddMinutes(-20)),
});
Assert.Equal(1, starts);
}
[Fact]
public void A_non_hello_web_app_start_is_ignored()
{
using var f = new Fixture();
using var tracker = f.Build();
var starts = 0;
tracker.HelloWizardStarted += (_, __) => starts++;
tracker.ReplayBackfillRecords(new List<(int, string, DateTime)>
{
Record(ShellCoreTracker.EventId_ShellCore_WebAppStarted, OtherWebAppDescription, ClockNow.AddMinutes(-5)),
});
Assert.Equal(0, starts);
Assert.Empty(f.AgentTraces());
}
// ------------------------------------------------------------------ the exit rail
[Fact]
public void An_intermediate_only_window_raises_nothing_that_can_reach_the_reducer()
{
// THE regression this whole review chain is about: restart, and the only 62407 in the
// window is the historic DeviceSetup-to-AccountSetup handoff. Nothing may leave the
// tracker on a rail the decision engine listens to.
using var f = new Fixture();
using var tracker = f.Build();
var espExits = 0;
var finalizing = new List<string>();
var failures = 0;
tracker.EspExited += (_, __) => espExits++;
tracker.FinalizingSetupPhaseTriggered += (_, reason) => finalizing.Add(reason);
tracker.EspFailureDetected += (_, __) => failures++;
tracker.ReplayBackfillRecords(new List<(int, string, DateTime)>
{
Record(ShellCoreTracker.EventId_ShellCore_WebAppEvent, EspExitDescription, ClockNow.AddMinutes(-52)),
});
Assert.Equal(0, espExits);
Assert.Empty(finalizing);
Assert.Equal(0, failures);
}
[Fact]
public void Even_several_exits_in_the_window_stay_out_of_the_decision_stream()
{
// "Newest exit wins" was an earlier attempt at this problem. It does not help: if the
// final exit has not happened yet, the newest record in the window is still the
// intermediate one. Picking a record is the wrong lever — none of them is usable.
using var f = new Fixture();
using var tracker = f.Build();
var espExits = 0;
tracker.EspExited += (_, __) => espExits++;
tracker.ReplayBackfillRecords(new List<(int, string, DateTime)>
{
Record(ShellCoreTracker.EventId_ShellCore_WebAppEvent, EspExitDescription, ClockNow.AddMinutes(-52)),
Record(ShellCoreTracker.EventId_ShellCore_WebAppEvent, EspExitDescription, ClockNow.AddMinutes(-9)),
});
Assert.Equal(0, espExits);
}
[Fact]
public void A_historic_esp_failure_is_never_re_injected()
{
// A session that failed and then recovered on retry (ANALYZE-ESP-006) must not be
// failed again by its own history.
using var f = new Fixture();
using var tracker = f.Build();
var failures = new List<string>();
tracker.EspFailureDetected += (_, failureType) => failures.Add(failureType);
tracker.ReplayBackfillRecords(new List<(int, string, DateTime)>
{
Record(ShellCoreTracker.EventId_ShellCore_WebAppEvent, EspFailureDescription, ClockNow.AddMinutes(-40)),
});
Assert.Empty(failures);
}
// ------------------------------------------------------------------ visibility
[Fact]
public void Skipped_records_are_reported_once_as_an_agent_trace()
{
// Dropping evidence silently is how this class of bug survives. The gap has to stay
// visible to whoever debugs the session later — as an informational event, which is
// decision-neutral by construction.
using var f = new Fixture();
using var tracker = f.Build();
tracker.ReplayBackfillRecords(new List<(int, string, DateTime)>
{
Record(ShellCoreTracker.EventId_ShellCore_WebAppEvent, EspExitDescription, ClockNow.AddMinutes(-52)),
Record(ShellCoreTracker.EventId_ShellCore_WebAppEvent, EspFailureDescription, ClockNow.AddMinutes(-40)),
Record(ShellCoreTracker.EventId_ShellCore_WebAppEvent, EspExitDescription, ClockNow.AddMinutes(-9)),
});
var trace = Assert.Single(f.AgentTraces());
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(trace.TypedPayload);
Assert.Equal(2, data["skippedEspExits"]);
Assert.Equal(1, data["skippedEspFailures"]);
Assert.Equal("replayed_62407_not_orderable", data["reason"]);
Assert.Equal(ClockNow.AddMinutes(-52).ToString("o"), data["oldestSkippedUtc"]);
Assert.Equal(ClockNow.AddMinutes(-9).ToString("o"), data["newestSkippedUtc"]);
}
[Fact]
public void A_wizard_only_window_reports_nothing_as_skipped()
{
using var f = new Fixture();
using var tracker = f.Build();
tracker.ReplayBackfillRecords(new List<(int, string, DateTime)>
{
Record(ShellCoreTracker.EventId_ShellCore_WebAppStarted, AadHelloDescription, ClockNow.AddMinutes(-30)),
});
Assert.Empty(f.AgentTraces());
}
[Fact]
public void An_empty_batch_is_a_no_op()
{
using var f = new Fixture();
using var tracker = f.Build();
var raised = 0;
tracker.EspExited += (_, __) => raised++;
tracker.HelloWizardStarted += (_, __) => raised++;
tracker.ReplayBackfillRecords(new List<(int, string, DateTime)>());
Assert.Equal(0, raised);
Assert.Empty(f.AgentTraces());
}
// ------------------------------------------------------------------ live path untouched
[Fact]
public void The_live_path_still_raises_the_exit_normally()
{
// Narrowing the REPLAY must not narrow live observation — that is the agent's primary
// signal source and the one the completion recovery actually runs on.
using var f = new Fixture();
using var tracker = f.Build();
var exits = new List<DateTime>();
tracker.EspExited += (_, args) => exits.Add(args.OccurredAtUtc);
tracker.ProcessEvent(
eventId: ShellCoreTracker.EventId_ShellCore_WebAppEvent,
description: EspExitDescription,
timestamp: ClockNow,
providerName: "Microsoft-Windows-Shell-Core",
isBackfill: false);
Assert.Equal(ClockNow, Assert.Single(exits));
}
[Fact]
public void The_live_path_still_raises_esp_failures()
{
using var f = new Fixture();
using var tracker = f.Build();
var failures = new List<string>();
tracker.EspFailureDetected += (_, failureType) => failures.Add(failureType);
tracker.ProcessEvent(
eventId: ShellCoreTracker.EventId_ShellCore_WebAppEvent,
description: EspFailureDescription,
timestamp: ClockNow,
providerName: "Microsoft-Windows-Shell-Core",
isBackfill: false);
Assert.Single(failures);
}
// ------------------------------------------------------------------ window clamp
[Theory]
[InlineData(0, 1)]
[InlineData(-42, 1)]
[InlineData(5, 5)]
[InlineData(ShellCoreTracker.BackfillLookbackMaxMinutes, ShellCoreTracker.BackfillLookbackMaxMinutes)]
[InlineData(ShellCoreTracker.BackfillLookbackMaxMinutes + 1, ShellCoreTracker.BackfillLookbackMaxMinutes)]
[InlineData(int.MaxValue, ShellCoreTracker.BackfillLookbackMaxMinutes)]
public void Lookback_is_clamped_to_a_sane_window(int requested, int expected)
{
// int.MaxValue matters: the query multiplies minutes by 60_000, so an unclamped value
// would overflow into a negative timediff and silently match nothing.
Assert.Equal(expected, ShellCoreTracker.ClampLookbackMinutes(requested));
}
}
}
@@ -3,6 +3,7 @@ using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Configuration;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Shared.Models;
@@ -117,6 +118,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
hosts,
"AadJoinHost wires its onRealUserJoined callback to the already-constructed DesktopArrivalHost.");
// Codex review P2 (2026-08-19): EspAndHelloHost.Start() replays the Shell-Core
// ESP exit, but ImeLogTracker only restores its persisted app states in its own
// Start() — which happens later. Pin the order that makes ImeLogHost's
// onStateRestored nudge necessary, so a future reshuffle either keeps the nudge
// meaningful or fails here loudly.
AssertBefore<EspAndHelloHost, ImeLogHost>(
hosts,
"the ESP-exit replay runs before the IME app-state restore, which is why " +
"ImeLogHost re-triggers the user-apps-settled synthesis after LoadState.");
// Closure-latch sanity (factory lines ~139/297): the EspAndHelloHost probes close
// over the ImeLogHost reference, which is assigned during the same factory call —
// the host must therefore exist in the returned list.
@@ -130,5 +141,40 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
}
}
}
[Fact]
public void Ime_app_state_callback_is_chained_into_the_user_apps_settled_recheck()
{
// sits-d Cloud-PC fix (2026-08-19): the AccountSetup synthesis is re-evaluated on every
// terminal IME app transition, which only works while the factory keeps this chain
// wired. It is five lines that a refactor can drop silently and whose absence costs a
// whole class of sessions their completion — so pin it here.
var surfaces = CreateSurfaces();
var hosts = surfaces.Hosts;
try
{
var imeHost = hosts.OfType<ImeLogHost>().Single();
var handler = imeHost.Tracker.OnAppStateChanged;
Assert.NotNull(handler);
// Invoking it must reach the re-check without throwing. The re-check is a no-op
// here (no ESP exit observed), which is the point — it must stay cheap and silent
// on the IME hot path until both facts are in.
var pkg = new AppPackageState("app-1", listPos: 0);
var ex = Record.Exception(() => handler!(
pkg,
AppInstallationState.Installing,
AppInstallationState.Installed));
Assert.Null(ex);
}
finally
{
foreach (var host in hosts)
{
try { host.Dispose(); } catch { /* construction-only test — best-effort cleanup */ }
}
}
}
}
}
@@ -0,0 +1,147 @@
#nullable enable
using System;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
{
/// <summary>
/// The Shell-Core ESP-exit replay is RESTART RECOVERY: it re-reads observations the agent could
/// not make because no agent process was running. The window is therefore measured against the
/// previous run's last known liveness, not guessed from a constant.
/// <para>
/// Why a constant is not enough (Codex review P2): the scheduled task carries a BootTrigger
/// only, with no restart-on-failure. A crashed agent does not come back until the NEXT BOOT —
/// possibly hours later, long after Windows wrote the final ESP exit. Five minutes would lose
/// it every time.
/// </para>
/// </summary>
public sealed class EspExitBackfillLookbackPolicyTests
{
private static readonly DateTime Now = new DateTime(2026, 8, 19, 8, 43, 16, DateTimeKind.Utc);
private const int Default = ShellCoreTracker.BackfillLookbackMinutes;
private static int Resolve(string? exitType, DateTime? lastAlive = null, DateTime? bootUtc = null)
=> DefaultComponentFactory.ResolveEspExitBackfillLookbackMinutes(exitType, bootUtc, lastAlive, Now);
// ---- no previous run => nothing to recover ---------------------------------------
[Theory]
[InlineData("first_run")]
[InlineData("FIRST_RUN")]
[InlineData("")]
[InlineData(null)]
public void No_previous_run_means_no_replay(string? exitType)
{
Assert.Equal(0, Resolve(exitType));
}
[Fact]
public void First_run_stays_off_even_with_timestamps_available()
{
// The exit type decides, not the presence of a timestamp.
Assert.Equal(0, Resolve("first_run", lastAlive: Now.AddHours(-3), bootUtc: Now.AddHours(-3)));
}
// ---- the default is a floor, never a ceiling --------------------------------------
[Theory]
[InlineData("clean")]
[InlineData("hard_kill")]
[InlineData("exception_crash")]
[InlineData("reboot_kill")]
public void Without_any_timestamp_the_default_applies(string exitType)
{
Assert.Equal(Default, Resolve(exitType));
}
[Fact]
public void A_restart_that_returned_within_seconds_keeps_the_default()
{
Assert.Equal(Default, Resolve("clean", lastAlive: Now.AddSeconds(-20)));
}
// ---- the crash case P2 is about ---------------------------------------------------
[Fact]
public void A_crash_recovered_only_at_the_next_boot_gets_the_whole_gap()
{
// BootTrigger-only task: the agent was gone for three hours while the ESP finished.
Assert.Equal(181, Resolve("exception_crash", lastAlive: Now.AddHours(-3)));
}
[Fact]
public void Snapshot_mtime_carries_the_window_when_no_boot_time_exists()
{
// LastBootUtc is only populated for hard_kill / reboot_kill — exception_crash and
// clean have nothing else to go on.
Assert.Equal(46, Resolve("clean", lastAlive: Now.AddMinutes(-45)));
}
// ---- reboot: both inputs available, wider wins ------------------------------------
[Fact]
public void The_wider_of_the_two_inputs_wins()
{
// Snapshot last written 40 min ago, boot 12 min ago — the gap started at the snapshot.
Assert.Equal(41, Resolve("reboot_kill", lastAlive: Now.AddMinutes(-40), bootUtc: Now.AddMinutes(-12)));
}
[Fact]
public void Boot_time_alone_still_covers_a_reboot_without_a_snapshot()
{
Assert.Equal(41, Resolve("reboot_kill", bootUtc: Now.AddMinutes(-40)));
}
// ---- robustness -------------------------------------------------------------------
[Fact]
public void A_timestamp_in_the_future_is_ignored()
{
// Clock skew across a reboot must never produce a negative window.
Assert.Equal(Default, Resolve("reboot_kill", lastAlive: Now.AddMinutes(5), bootUtc: Now.AddMinutes(9)));
}
[Fact]
public void A_future_timestamp_does_not_suppress_a_valid_one()
{
Assert.Equal(31, Resolve("reboot_kill", lastAlive: Now.AddMinutes(9), bootUtc: Now.AddMinutes(-30)));
}
[Fact]
public void A_very_long_gap_is_clamped_by_the_tracker()
{
// The policy may propose more than the cap; ClampLookbackMinutes is the backstop, so
// the two together can never produce an unbounded event-log query.
var proposed = Resolve("exception_crash", lastAlive: Now.AddHours(-20));
Assert.True(proposed > ShellCoreTracker.BackfillLookbackMaxMinutes);
Assert.Equal(ShellCoreTracker.BackfillLookbackMaxMinutes,
ShellCoreTracker.ClampLookbackMinutes(proposed));
}
// ---- the snapshot probe itself ----------------------------------------------------
[Fact]
public void Missing_snapshot_reads_as_null_not_as_an_exception()
{
using var tmp = new Harness.TempDirectory();
Assert.Null(DefaultComponentFactory.ReadPreviousRunLastAliveUtc(tmp.Path));
Assert.Null(DefaultComponentFactory.ReadPreviousRunLastAliveUtc(string.Empty));
}
[Fact]
public void An_existing_snapshot_reports_its_write_time()
{
using var tmp = new Harness.TempDirectory();
var path = System.IO.Path.Combine(tmp.Path, "snapshot.json");
System.IO.File.WriteAllText(path, "{}");
var read = DefaultComponentFactory.ReadPreviousRunLastAliveUtc(tmp.Path);
Assert.NotNull(read);
Assert.Equal(DateTimeKind.Utc, read!.Value.Kind);
Assert.True((DateTime.UtcNow - read.Value).Duration() < TimeSpan.FromMinutes(5));
}
}
}
@@ -0,0 +1,72 @@
#nullable enable
using System;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
{
/// <summary>
/// Codex review P2 (2026-08-19) — <see cref="ImeLogHost"/> starts AFTER <c>EspAndHelloHost</c>,
/// and <c>ImeLogTracker.Start()</c> restores the persisted package states silently (LoadState
/// raises no <c>OnAppStateChanged</c>). On an agent restart the Shell-Core ESP-exit replay
/// therefore runs against an empty app list. Without the post-restore nudge the exact reboot
/// case the completion recovery exists for — apps already terminal BEFORE the reboot, only the
/// final exit lost to the downtime — would still never complete.
/// </summary>
public sealed class ImeLogHostStateRestoredCallbackTests : IDisposable
{
private readonly TempDirectory _tmp = new TempDirectory();
public void Dispose() => _tmp.Dispose();
private ImeLogHost CreateHost(Action? onStateRestored)
=> new ImeLogHost(
sessionId: "s",
tenantId: "t",
logger: new AgentLogger(_tmp.Path, AgentLogLevel.Info),
ingress: new FakeSignalIngressSink(),
clock: new VirtualClock(new DateTime(2026, 8, 19, 8, 43, 16, DateTimeKind.Utc)),
imeLogPathOverride: _tmp.Path,
imeMatchLogPath: null,
imePatterns: null,
stateDirectory: _tmp.Path,
whiteGloveSealingPatternIds: null,
onStateRestored: onStateRestored);
[Fact]
public void Start_invokes_the_state_restored_callback_exactly_once()
{
var calls = 0;
using var host = CreateHost(() => calls++);
host.Start();
try { Assert.Equal(1, calls); }
finally { host.Stop(); }
}
[Fact]
public void A_throwing_callback_does_not_break_host_start()
{
// Fail-soft like every other co-wired callback in the orchestration layer: a broken
// re-check must never take the IME log host — the session's primary signal source —
// down with it.
using var host = CreateHost(() => throw new InvalidOperationException("recheck boom"));
var ex = Record.Exception(() => host.Start());
try { Assert.Null(ex); }
finally { host.Stop(); }
}
[Fact]
public void No_callback_is_a_valid_configuration()
{
using var host = CreateHost(null);
var ex = Record.Exception(() => host.Start());
try { Assert.Null(ex); }
finally { host.Stop(); }
}
}
}
@@ -144,6 +144,77 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Transport
Assert.Equal(1 + 2 + 3, snap.TotalBytesUp);
}
[Fact]
public async Task Captures_bytes_down_via_buffering_when_content_length_is_missing()
{
// AutomaticDecompression strips Content-Length on the backend's gzipped JSON —
// the pre-fix handler recorded 0 for every such (healthy) response. A non-seekable
// stream body reproduces the headerless shape.
var metrics = new NetworkMetrics();
var inner = new RecordingHttpMessageHandler();
var payload = Encoding.UTF8.GetBytes("{\"ok\":true}"); // 11 bytes
inner.QueueResponse(() => new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new StreamContent(new NonSeekableReadStream(payload)),
});
using var client = BuildClient(metrics, inner);
using var resp = await client.SendAsync(NewPost(""));
var snap = metrics.GetSnapshot();
Assert.Equal(payload.Length, snap.TotalBytesDown);
// Buffering must not consume the body away from the caller.
Assert.Equal("{\"ok\":true}", await resp.Content.ReadAsStringAsync());
}
[Fact]
public async Task Bytes_down_fails_soft_to_zero_when_buffering_throws()
{
var metrics = new NetworkMetrics();
var inner = new RecordingHttpMessageHandler();
inner.QueueResponse(() => new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new ThrowingContent(),
});
using var client = BuildClient(metrics, inner);
// ResponseHeadersRead keeps HttpClient's own post-pipeline buffering out of the
// picture — the test isolates the handler's fallback catch (which must swallow
// the serialization failure and record 0, never throw).
using var resp = await client.SendAsync(NewPost(""), HttpCompletionOption.ResponseHeadersRead);
var snap = metrics.GetSnapshot();
Assert.Equal(1, snap.RequestCount);
Assert.Equal(0, snap.TotalBytesDown);
}
/// <summary>Read-only, non-seekable stream — CanSeek=false keeps StreamContent from
/// computing a Content-Length, mirroring a decompression-wrapped response body.</summary>
private sealed class NonSeekableReadStream : System.IO.Stream
{
private readonly System.IO.MemoryStream _inner;
public NonSeekableReadStream(byte[] data) { _inner = new System.IO.MemoryStream(data); }
public override bool CanRead => true;
public override bool CanSeek => false;
public override bool CanWrite => false;
public override long Length => throw new NotSupportedException();
public override long Position { get => throw new NotSupportedException(); set => throw new NotSupportedException(); }
public override void Flush() { }
public override int Read(byte[] buffer, int offset, int count) => _inner.Read(buffer, offset, count);
public override long Seek(long offset, System.IO.SeekOrigin origin) => throw new NotSupportedException();
public override void SetLength(long value) => throw new NotSupportedException();
public override void Write(byte[] buffer, int offset, int count) => throw new NotSupportedException();
protected override void Dispose(bool disposing) { if (disposing) _inner.Dispose(); base.Dispose(disposing); }
}
/// <summary>Content whose serialization always fails — drives the fallback's catch path.</summary>
private sealed class ThrowingContent : HttpContent
{
protected override Task SerializeToStreamAsync(System.IO.Stream stream, System.Net.TransportContext? context)
=> throw new System.IO.IOException("connection dropped mid-body");
protected override bool TryComputeLength(out long length) { length = 0; return false; }
}
[Fact]
public void Constructor_rejects_null_metrics()
{
@@ -40,6 +40,17 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Transport
return this;
}
/// <summary>
/// Queues an arbitrary response — for shapes <see cref="QueueStatus"/> can't build,
/// e.g. content without a Content-Length header (what AutomaticDecompression leaves
/// behind on real compressed responses).
/// </summary>
public RecordingHttpMessageHandler QueueResponse(Func<HttpResponseMessage> factory)
{
_script.Enqueue(_ => factory());
return this;
}
protected override async Task<HttpResponseMessage> SendAsync(
HttpRequestMessage request,
CancellationToken cancellationToken)
@@ -1,5 +1,6 @@
using System;
using System.Collections.Generic;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Shared;
@@ -46,7 +47,24 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
// Session caa6cf50 gate-starvation fix (2026-06-11) — fire-once guard for the
// user-ESP-apps-settled AccountSetup synthesis (see MaybeSynthesizeAccountSetupComplete).
private bool _userAppsSettledSynthesisFired;
// Interlocked rather than a plain bool: since the sits-d Cloud-PC fix the synthesis is
// ALSO re-evaluated from the IME app-state-change thread, so two threads can reach the
// guard concurrently (Shell-Core watcher thread + IME log thread).
private int _userAppsSettledSynthesisFired;
// sits-d Cloud-PC fix (2026-08-19) — set once a CONFIRMED post-AccountSetup Shell-Core
// exit has been observed (live OR backfill). The synthesis needs two facts: the ESP exit
// is an EDGE that happens once, the settled user-ESP apps are a LEVEL that may only be
// reached minutes later. Recording the edge lets ReevaluateUserAppsSettledSynthesis
// re-check the level afterwards instead of losing the completion for the rest of the
// session.
//
// Codex review P1 (2026-08-19): "confirmed" is load-bearing. Shell-Core raises 62407 at
// EVERY ESP phase transition, DeviceSetup→AccountSetup included. Remembering that
// intermediate exit would let the deferred re-check open the strong AccountSetup gate the
// moment the last user app settles — while the AccountSetup page is still up and its other
// subcategories are still running. See IsConfirmedPostAccountSetupExit.
private volatile bool _espExitObserved;
// Liveness plan PR3 — one-shot-per-appId dedupe for app_install_starved emissions.
// Written from the Shell-Core watcher thread (esp_exited path), read at termination by
@@ -314,11 +332,40 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
/// </summary>
public void ResetForEspResumption() => _helloTracker?.ResetForEspResumption();
/// <summary>
/// Backfills recent ESP exit and failure events from Shell-Core log on startup.
/// Secondary recovery mechanism when state persistence is unavailable.
/// Replays the recent Shell-Core Hello-wizard start with an explicit lookback window.
/// Called by <c>EspAndHelloHost</c> right after <see cref="Start"/> — see the host for why
/// the window is sized to the previous run's downtime, and
/// <see cref="ShellCoreTracker.ReplayBackfillRecords"/> for why the ESP exit is not part
/// of the replay.
/// </summary>
public void BackfillRecentEspExitEvents() => _shellCoreTracker?.BackfillRecentEspExitEvents();
public void BackfillRecentHelloWizardStart(int lookbackMinutes)
=> _shellCoreTracker?.BackfillRecentHelloWizardStart(lookbackMinutes);
/// <summary>
/// sits-d Cloud-PC fix (2026-08-19) — re-check the user-apps-settled AccountSetup
/// synthesis after the ESP exit has already been observed. Wired to the IME tracker's
/// app-state-change callback in <c>DefaultComponentFactory</c>.
/// <para>
/// Before this, the synthesis ran ONLY on the <c>EspExited</c> edge. When the ESP page
/// exits while user-ESP apps are still in flight (the normal case on a tenant with a
/// large required-app set), the single attempt misses and nothing re-checks — even once
/// every tracked app has reached a terminal state minutes later. Sessions 8110e262 and
/// a89aac2d reached 138/138 apps with 0 failed and still never completed.
/// </para>
/// <para>
/// Adds NO new completion path: the gate conditions inside the synthesis are unchanged
/// (real ESP exit + every required user-ESP app terminal + zero failures). This only
/// grants the existing, deliberately conservative check a second look.
/// </para>
/// </summary>
public void ReevaluateUserAppsSettledSynthesis()
{
if (!_espExitObserved) return;
if (Volatile.Read(ref _userAppsSettledSynthesisFired) != 0) return;
MaybeSynthesizeAccountSetupCompleteFromSettledUserApps(emitStarvedOnMiss: false);
}
// =====================================================================
// Lifecycle
@@ -536,6 +583,40 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
return skipUser == false;
}
/// <summary>
/// Codex review P1 (2026-08-19) — may THIS <c>esp_exiting</c> be remembered as the edge the
/// deferred user-apps-settled re-check builds on?
/// <para>
/// Deliberately STRICTER than <c>!IsIntermediateDeviceEspExit()</c>: that helper forwards
/// on an unknown SkipUser (forwarding a signal early is cheap), whereas remembering the
/// wrong edge here can open a completion gate minutes later with no second chance to take
/// it back. So this one demands positive evidence and treats "unknown" as "not confirmed":
/// </para>
/// <list type="bullet">
/// <item>the provisioning tracker has seen AccountSetup activity — the page that just
/// tore down is demonstrably the AccountSetup one; or</item>
/// <item>SkipUser is explicitly <c>true</c> — the profile has no user ESP at all, so the
/// Device-ESP exit IS the final one and no second exit is coming.</item>
/// </list>
/// <para>
/// Erring strict costs at worst today's behaviour (the session stalls as it did before the
/// fix); erring loose costs a premature Succeeded, which is unrecoverable.
/// </para>
/// </summary>
private bool IsConfirmedPostAccountSetupExit()
{
bool accountSetupSeen;
try { accountSetupSeen = _accountSetupActivityProbe(); }
catch (Exception ex)
{
_logger.Debug($"EspAndHelloTracker: account-setup-activity probe threw: {ex.Message}");
accountSetupSeen = false;
}
if (accountSetupSeen) return true;
return GetSkipUserEspCached() == true;
}
private bool? GetSkipUserEspCached()
{
lock (_skipConfigLock)
@@ -675,6 +756,17 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
try { EspExited?.Invoke(this, args); }
catch (Exception ex) { _logger.Error("Error forwarding EspExited", ex); }
// Every exit reaching this handler is one the agent observed LIVE: the Shell-Core
// replay never re-raises 62407 (see ShellCoreTracker.ReplayBackfillRecords — a
// replayed exit cannot be placed in time and is therefore kept out of the decision
// stream entirely). That is what makes the AccountSetup read below a valid
// ordering fact: the agent was continuously observing up to this instant.
//
// Record the edge BEFORE the synthesis attempt so a later app-state change can
// re-run it even when the apps are not settled at this instant.
if (IsConfirmedPostAccountSetupExit())
_espExitObserved = true;
// Session caa6cf50 gate-starvation fix: a Shell-Core normal exit while IME's
// user-ESP app tracking is fully settled is alternative evidence that
// AccountSetup completed. Raised AFTER the EspExited forward so the reducer
@@ -704,9 +796,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
/// conservative stall behaviour.
/// </para>
/// </summary>
private void MaybeSynthesizeAccountSetupCompleteFromSettledUserApps()
private void MaybeSynthesizeAccountSetupCompleteFromSettledUserApps(bool emitStarvedOnMiss = true)
{
if (_userAppsSettledSynthesisFired) return;
if (Volatile.Read(ref _userAppsSettledSynthesisFired) != 0) return;
bool settled;
try { settled = _userEspAppsSettledProbe(); }
@@ -722,11 +814,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
// operator with an anonymous "session hangs in AccountSetup". One-shot per
// appId; apps that are alive (Downloading/Installing) or failed are excluded
// by the probe itself.
EmitStarvedUserEspApps(trigger: "esp_exited_user_apps_not_settled");
// Only on the ESP-exit edge: the re-evaluation path passes false so a repeated
// miss cannot re-walk the probe on every single app-state change (the per-appId
// dedupe would swallow the events anyway — this keeps the work off the hot path).
if (emitStarvedOnMiss)
EmitStarvedUserEspApps(trigger: "esp_exited_user_apps_not_settled");
return;
}
_userAppsSettledSynthesisFired = true;
// Claim the single synthesis slot; a concurrent caller that loses the race returns.
if (Interlocked.CompareExchange(ref _userAppsSettledSynthesisFired, 1, 0) != 0) return;
_logger.Warning(
"EspAndHelloTracker: ESP exited normally with all tracked user-ESP apps terminal (0 failed) " +
"but AccountSetupCategory.Status never confirmed categorySucceeded — treating AccountSetup as complete " +
@@ -820,6 +917,6 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
// Test seam for the user-apps-settled synthesis — drives OnEspExited with the live
// handler signature, mirroring TriggerEspExitedForTest's contract.
internal bool UserAppsSettledSynthesisFiredForTest => _userAppsSettledSynthesisFired;
internal bool UserAppsSettledSynthesisFiredForTest => Volatile.Read(ref _userAppsSettledSynthesisFired) != 0;
}
}
@@ -34,6 +34,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
internal const int EventId_ShellCore_WebAppEvent = 62407;
internal const int BackfillLookbackMinutes = 5;
/// <summary>
/// Upper bound for the caller-supplied backfill lookback. Matches the agent's max
/// lifetime (360 min) — a window wider than the agent can ever live would only widen
/// the blast radius of a stale Shell-Core record without recovering anything the
/// current enrollment could still act on.
/// </summary>
internal const int BackfillLookbackMaxMinutes = 360;
private static readonly HashSet<int> TrackedShellCoreEventIds = new HashSet<int>
{
EventId_ShellCore_WebAppStarted,
@@ -350,22 +358,39 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
// Backfill (public — called by coordinator)
// =====================================================================
/// <summary>Clamps a caller-supplied lookback into [1, <see cref="BackfillLookbackMaxMinutes"/>].</summary>
internal static int ClampLookbackMinutes(int lookbackMinutes)
{
if (lookbackMinutes < 1) return 1;
if (lookbackMinutes > BackfillLookbackMaxMinutes) return BackfillLookbackMaxMinutes;
return lookbackMinutes;
}
public void BackfillRecentHelloWizardStart() => BackfillRecentHelloWizardStart(BackfillLookbackMinutes);
/// <summary>
/// Backfills recent ESP exit, ESP failure and Hello-wizard-start events from the
/// Shell-Core log on startup. Secondary recovery mechanism when state persistence is
/// unavailable — an agent restart while the user sits inside the Hello wizard would
/// otherwise lose the wizard-start observation (session 772fe502).
/// Same recovery with a caller-chosen lookback, so a restart can reach back over its whole
/// downtime (after a mid-ESP reboot the agent is gone from the forced restart until the
/// post-reboot logon relaunches it; after a crash, until the next boot — the scheduled task
/// has a BootTrigger only). Clamped to [1, <see cref="BackfillLookbackMaxMinutes"/>].
/// <para>
/// The 62407 records in the window are read but deliberately NOT replayed — see
/// <see cref="ReplayBackfillRecords"/>. They are counted and reported once as an
/// <c>agent_trace</c> so the gap stays visible in the timeline without becoming
/// decision-relevant.
/// </para>
/// </summary>
public void BackfillRecentEspExitEvents()
public void BackfillRecentHelloWizardStart(int lookbackMinutes)
{
try
{
var lookbackMs = BackfillLookbackMinutes * 60 * 1000;
var lookbackMs = ClampLookbackMinutes(lookbackMinutes) * 60 * 1000;
var query = new EventLogQuery(
ShellCoreEventLogChannel,
PathType.LogName,
$"*[System[(EventID=62404 or EventID=62407) and TimeCreated[timediff(@SystemTime) <= {lookbackMs}]]]");
var records = new List<(int Id, string Description, DateTime OccurredAtUtc)>();
using (var reader = new EventLogReader(query))
{
for (EventRecord record = reader.ReadEvent(); record != null; record = reader.ReadEvent())
@@ -377,20 +402,135 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
// (EspAndHelloTrackerAdapter) can stamp signals with the source time
// rather than collapsing to wall-clock-now.
var timestamp = (record.TimeCreated ?? DateTime.UtcNow).ToUniversalTime();
HandleBackfillRecord(record.Id, description, timestamp);
records.Add((record.Id, description, timestamp));
}
}
}
ReplayBackfillRecords(records);
}
catch (Exception ex)
{
_logger.Warning($"ESP exit/failure event backfill failed: {ex.Message}");
_logger.Warning($"Shell-Core replay failed: {ex.Message}");
}
}
/// <summary>
/// Internal backfill record handler — extracted for testability and to keep the
/// backfill loop free of direct event-processing logic. The
/// Replays a chronological batch of Shell-Core records. <b>Only the Hello-wizard start
/// (62404) is replayed.</b> ESP exits and ESP failures (62407) are counted and reported,
/// never re-raised.
/// <para>
/// Why the exit is excluded — the reasoning is the whole point of this method, so it lives
/// here rather than in a commit message:
/// </para>
/// <list type="number">
/// <item>Windows writes the IDENTICAL description
/// (<c>CommercialOOBE_ESPProgress_Page_Exiting</c>) for the intermediate
/// DeviceSetup→AccountSetup transition and for the final post-AccountSetup exit, so a
/// replayed record carries no evidence of its own position.</item>
/// <item>Everything that could order it after the fact — the AccountSetup registry
/// probe, the settled-apps probe — reads state as it is NOW, not as it was at the
/// event's time.</item>
/// <item>The reducer orders exits by INGEST ORDINAL, not by timestamp
/// (<c>IsPostAccountSetupFinalExit</c>). A replayed historic exit is assigned a fresher
/// ordinal than reality, so it looks post-AccountSetup by construction.</item>
/// <item><c>HandleEspExitingV1</c> passes <c>espFinalExitInFlight: true</c> for every
/// arriving exit. With restored state (AccountSetupEntered + a genuine IME user
/// session + desktop arrived) arm C of <c>ShouldTransitionToAwaitingHello</c> then
/// opens on a historic intermediate exit — a completion built on a fact that never
/// happened.</item>
/// </list>
/// <para>
/// There is therefore no honest way to classify a replayed exit, so it must not enter the
/// decision stream at all. The same applies to a replayed ESP FAILURE, for the opposite
/// reason: re-injecting a historic failure as fresh can fail a session that recovered
/// (see ANALYZE-ESP-006, "ESP Failure Recovered After User Retry").
/// </para>
/// <para>
/// The Hello-wizard start is different in kind and is the observation this replay was
/// written for (session 772fe502): it is a CONSERVATIVE fact. It vetoes a premature
/// "Hello is disabled" skip and can never by itself complete a session, so replaying it
/// can only make the agent wait longer, never finish early.
/// </para>
/// </summary>
internal void ReplayBackfillRecords(
IReadOnlyList<(int Id, string Description, DateTime OccurredAtUtc)> records)
{
if (records == null || records.Count == 0) return;
var skippedExits = 0;
var skippedFailures = 0;
DateTime? oldestSkipped = null;
DateTime? newestSkipped = null;
foreach (var record in records)
{
var description = record.Description ?? string.Empty;
if (record.Id == EventId_ShellCore_WebAppStarted)
{
HandleBackfillRecord(record.Id, description, record.OccurredAtUtc);
continue;
}
var isFailure = HasEspFailurePattern(description);
var isExit = !isFailure && EspExitingPattern.IsMatch(description);
if (!isFailure && !isExit) continue;
if (isFailure) skippedFailures++; else skippedExits++;
if (oldestSkipped == null || record.OccurredAtUtc < oldestSkipped.Value)
oldestSkipped = record.OccurredAtUtc;
if (newestSkipped == null || record.OccurredAtUtc > newestSkipped.Value)
newestSkipped = record.OccurredAtUtc;
}
if (skippedExits > 0 || skippedFailures > 0)
EmitSkippedShellCoreRecords(skippedExits, skippedFailures, oldestSkipped, newestSkipped);
}
/// <summary>
/// One informational <c>agent_trace</c> naming the 62407 records the replay deliberately
/// did not re-raise. Decision-neutral by construction (informational events are exempt
/// from the dispatch guard) — its only job is to keep the blind window visible to whoever
/// debugs a session later, instead of the replay silently dropping evidence.
/// </summary>
private void EmitSkippedShellCoreRecords(
int skippedExits, int skippedFailures, DateTime? oldestUtc, DateTime? newestUtc)
{
try
{
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
EventType = Constants.EventTypes.AgentTrace,
Severity = EventSeverity.Info,
Source = "ShellCoreTracker",
Phase = EnrollmentPhase.Unknown,
Message =
$"Shell-Core replay skipped {skippedExits} ESP exit(s) and {skippedFailures} ESP failure(s) " +
"from the window in which no agent process was running — a replayed 62407 cannot be placed " +
"in time and must not reach the decision engine.",
Data = new Dictionary<string, object>
{
{ "skippedEspExits", skippedExits },
{ "skippedEspFailures", skippedFailures },
{ "oldestSkippedUtc", oldestUtc?.ToString("o") ?? string.Empty },
{ "newestSkippedUtc", newestUtc?.ToString("o") ?? string.Empty },
{ "reason", "replayed_62407_not_orderable" },
},
});
}
catch (Exception ex)
{
_logger.Debug($"ShellCoreTracker: skipped-records trace emit failed: {ex.Message}");
}
}
/// <summary>
/// Internal backfill record handler — extracted for testability and to keep the replay
/// loop free of direct event-processing logic. Handles the Hello-wizard start ONLY; see
/// <see cref="ReplayBackfillRecords"/> for why 62407 is never replayed. The
/// <paramref name="occurredAtUtc"/> is the original Shell-Core event time
/// (<c>record.TimeCreated</c>); subscribers read it via
/// <see cref="LastEventOccurredAtUtc"/> during their synchronous event handler.
@@ -427,42 +567,6 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
return;
}
if (EspExitingPattern.IsMatch(description))
{
bool shouldNotify = false;
lock (_stateLock)
{
if (!_espExitDetected)
{
_espExitDetected = true;
shouldNotify = true;
}
}
if (shouldNotify)
{
_helloTracker?.NotifyEspExited();
_logger.Info($"Backfill: ESP exit event found in recent Shell-Core logs (originalAt={occurredAtUtc:o})");
LastEventOccurredAtUtc = occurredAtUtc;
try
{
try { FinalizingSetupPhaseTriggered?.Invoke(this, "esp_exiting"); }
catch (Exception ex) { _logger.Error("Backfill: FinalizingSetupPhaseTriggered handler failed", ex); }
try { EspExited?.Invoke(this, new EspExitedEventArgs(occurredAtUtc)); }
catch (Exception ex) { _logger.Error("Backfill: EspExited handler failed", ex); }
}
finally { LastEventOccurredAtUtc = null; }
}
}
if (HasEspFailurePattern(description))
{
var failureType = ExtractEspFailureType(description);
_logger.Info($"Backfill: ESP failure event found in recent Shell-Core logs: {failureType} (originalAt={occurredAtUtc:o})");
LastEventOccurredAtUtc = occurredAtUtc;
try { EspFailureDetected?.Invoke(this, failureType); }
catch (Exception ex) { _logger.Error($"Backfill: EspFailureDetected handler failed for '{failureType}'", ex); }
finally { LastEventOccurredAtUtc = null; }
}
}
// =====================================================================
@@ -21,12 +21,20 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
/// <remarks>
/// <b>bytes-up</b> is read from <see cref="System.Net.Http.Headers.HttpContentHeaders.ContentLength"/>
/// on the request content (set eagerly by <c>ByteArrayContent</c>/<c>StringContent</c>).
/// <b>bytes-down</b> is read from the response Content-Length header — when
/// <c>AutomaticDecompression</c> is enabled the framework may strip this, so the value
/// is best-effort (degrades to 0). The request count and failure count are always exact.
/// <b>bytes-down</b> prefers the response Content-Length header; the pipeline's
/// <c>AutomaticDecompression</c> strips that header on compressed responses (i.e. on
/// virtually every backend JSON response), so when it is absent the content is buffered
/// and the DECOMPRESSED payload size is recorded instead — before the fallback the
/// counter read 0 on healthy sessions. <c>HttpClient</c> buffers responses anyway
/// (default <c>HttpCompletionOption.ResponseContentRead</c>), so this adds no real
/// memory cost. The request count and failure count are always exact.
/// </remarks>
internal sealed class NetworkMetricsRecordingHandler : DelegatingHandler
{
// Buffering cap for the bytes-down fallback. Backend responses are small JSON
// (config, ingest acks); anything larger is counted as 0 rather than buffered.
private const long MaxCountedResponseBytes = 16 * 1024 * 1024;
private readonly NetworkMetrics _metrics;
public NetworkMetricsRecordingHandler(NetworkMetrics metrics)
@@ -54,6 +62,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
{
response = await base.SendAsync(request, cancellationToken).ConfigureAwait(false);
bytesDown = response?.Content?.Headers?.ContentLength ?? 0L;
if (bytesDown == 0 && response?.Content != null)
{
// AutomaticDecompression stripped Content-Length (the norm for the
// backend's gzipped JSON). Buffer the content — after buffering the
// header reflects the decompressed length. Fail-soft: a buffering
// error (too large, connection dropped mid-body) keeps 0 and leaves
// the response for the caller to observe the failure itself.
try
{
await response.Content.LoadIntoBufferAsync(MaxCountedResponseBytes).ConfigureAwait(false);
bytesDown = response.Content.Headers.ContentLength ?? 0L;
}
catch
{
// keep bytesDown = 0
}
}
if (response != null && !response.IsSuccessStatusCode)
{
failed = true;
@@ -74,13 +74,37 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
private readonly string _stateDirectory;
private readonly Persistence.StartupEventGate? _startupEventGate;
/// <summary>
/// How the previous agent run in this enrollment ended (<c>Program.PreviousExitSummary.ExitType</c>:
/// <c>first_run</c> | <c>clean</c> | <c>exception_crash</c> | <c>hard_kill</c> |
/// <c>reboot_kill</c>). Decides whether the Shell-Core ESP-exit replay runs at all — see
/// <see cref="ResolveEspExitBackfillLookbackMinutes"/>.
/// </summary>
private readonly string? _previousExitType;
/// <summary>
/// OS boot time when the previous agent run was killed by a reboot (<c>reboot_kill</c>),
/// otherwise null. One of the two inputs that size the Shell-Core ESP-exit backfill.
/// </summary>
private readonly DateTime? _previousBootUtc;
/// <summary>
/// Last moment the previous agent run is known to have been alive — the mtime of its
/// reducer snapshot, which is rewritten on every decision step. Independent of HOW that
/// run ended, which makes it the more broadly usable of the two window inputs.
/// </summary>
private readonly DateTime? _previousRunLastAliveUtc;
public DefaultComponentFactory(
AgentConfiguration agentConfig,
AgentConfigResponse remoteConfig,
NetworkMetrics? networkMetrics,
string agentVersion,
string stateDirectory,
Persistence.StartupEventGate? startupEventGate = null)
Persistence.StartupEventGate? startupEventGate = null,
DateTime? previousBootUtc = null,
string? previousExitType = null,
DateTime? previousRunLastAliveUtc = null)
{
_agentConfig = agentConfig ?? throw new ArgumentNullException(nameof(agentConfig));
_remoteConfig = remoteConfig ?? throw new ArgumentNullException(nameof(remoteConfig));
@@ -88,6 +112,88 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
_agentVersion = string.IsNullOrEmpty(agentVersion) ? "unknown" : agentVersion;
_stateDirectory = stateDirectory ?? throw new ArgumentNullException(nameof(stateDirectory));
_startupEventGate = startupEventGate;
_previousBootUtc = previousBootUtc;
_previousExitType = previousExitType;
_previousRunLastAliveUtc = previousRunLastAliveUtc;
}
/// <summary>
/// Lookback in minutes for the Shell-Core ESP-exit replay; <c>0</c> means "do not replay".
/// <para>
/// The replay is a RESTART-RECOVERY mechanism: it re-reads observations the agent could not
/// make because no agent process was running. That only ever applies when a previous run
/// existed. On <c>first_run</c> — the happy path — it stays off, so the agent behaves
/// exactly as it did before the recovery was added.
/// </para>
/// <para>
/// Otherwise the window is measured rather than guessed: it reaches back to the last moment
/// the previous run is known to have been alive. Two independent inputs feed it and the
/// wider one wins, because either can be missing:
/// </para>
/// <list type="bullet">
/// <item><b>Snapshot mtime</b> — the reducer snapshot is rewritten on every decision
/// step, so its last-write time is "when we last knew what was happening", regardless
/// of how that run ended. This is what makes <c>exception_crash</c> safe: the scheduled
/// task carries a BootTrigger only, with no restart-on-failure, so a crashed agent does
/// not come back until the next boot — possibly hours later, long after Windows wrote
/// the final ESP exit. A fixed five minutes would lose it every time.</item>
/// <item><b>Last boot</b> — populated for <c>hard_kill</c> / <c>reboot_kill</c>; covers
/// the case where the snapshot is missing or was never written.</item>
/// </list>
/// <para>
/// The default stays the floor (a restart that returns in seconds still gets its five
/// minutes) and <c>ShellCoreTracker.ClampLookbackMinutes</c> is the ceiling, so policy and
/// backstop together can never build an unbounded event-log query.
/// </para>
/// </summary>
internal static int ResolveEspExitBackfillLookbackMinutes(
string? previousExitType,
DateTime? previousBootUtc,
DateTime? previousRunLastAliveUtc,
DateTime utcNow)
{
if (string.IsNullOrEmpty(previousExitType)
|| string.Equals(previousExitType, "first_run", StringComparison.OrdinalIgnoreCase))
{
return 0;
}
var lookback = ShellCoreTracker.BackfillLookbackMinutes;
foreach (var since in new[] { previousRunLastAliveUtc, previousBootUtc })
{
if (!since.HasValue) continue;
var elapsed = utcNow - since.Value;
if (elapsed <= TimeSpan.Zero) continue; // clock skew — never build a negative window
var minutes = (int)Math.Ceiling(elapsed.TotalMinutes) + 1;
if (minutes > lookback) lookback = minutes;
}
return lookback;
}
/// <summary>
/// Last-write time of the previous run's reducer snapshot, or null when it does not exist
/// (first run) or cannot be read. Best-effort by design — a missing timestamp only means
/// the replay window falls back to the other inputs.
/// </summary>
internal static DateTime? ReadPreviousRunLastAliveUtc(string stateDirectory)
{
try
{
if (string.IsNullOrEmpty(stateDirectory)) return null;
var snapshotPath = Path.Combine(
stateDirectory, Persistence.DeathRattlePrelude.SnapshotFileName);
return File.Exists(snapshotPath)
? (DateTime?)File.GetLastWriteTimeUtc(snapshotPath)
: null;
}
catch
{
return null;
}
}
public CollectorSurfaces CreateCollectorHosts(
@@ -150,6 +256,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
// on esp_exiting events long after Start.
ImeLogHost? imeLogHostRef = null;
// sits-d Cloud-PC fix (2026-08-19), narrowed after review: the Shell-Core ESP-exit
// replay is restart recovery, so it stays OFF on a first run and the happy path keeps
// its pre-fix behaviour exactly. See ResolveEspExitBackfillLookbackMinutes.
var espExitBackfillLookbackMinutes = ResolveEspExitBackfillLookbackMinutes(
_previousExitType, _previousBootUtc, _previousRunLastAliveUtc, DateTime.UtcNow);
if (espExitBackfillLookbackMinutes > 0)
{
logger.Info(
$"DefaultComponentFactory: previous run ended as '{_previousExitType}' — Shell-Core " +
$"ESP-exit replay enabled with a {espExitBackfillLookbackMinutes} min lookback");
}
var espAndHelloHost = new EspAndHelloHost(
sessionId: sessionId,
tenantId: tenantId,
@@ -174,7 +292,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
// covers Store/WinGet apps invisible to the starved/correlation paths.
packageStatesProbe: () =>
imeLogHostRef?.AllKnownPackageStates
?? Array.Empty<Monitoring.Enrollment.Ime.AppPackageState>());
?? Array.Empty<Monitoring.Enrollment.Ime.AppPackageState>(),
espExitBackfillLookbackMinutes: espExitBackfillLookbackMinutes);
hosts.Add(espAndHelloHost);
// Hybrid User-Driven completion-gap fix (2026-05-01): the AadJoinHost notifies
@@ -305,10 +424,41 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
stateDirectory: _stateDirectory,
whiteGloveSealingPatternIds: whiteGloveSealingPatternIds,
simulationMode: simulationMode,
simulationSpeedFactor: _agentConfig.ReplaySpeedFactor);
simulationSpeedFactor: _agentConfig.ReplaySpeedFactor,
// Codex review P2 (2026-08-19): the IME tracker restores its persisted app states
// inside Start(), silently (LoadState raises no OnAppStateChanged). This host starts
// AFTER EspAndHelloHost, so the Shell-Core ESP-exit replay has already run against
// an empty app list. Nudge the synthesis once the restored level is in place —
// otherwise the exact reboot case (apps terminal BEFORE the reboot, only the final
// exit lost to the downtime) still never completes.
onStateRestored: () => espAndHelloHost.ReevaluateUserAppsSettledSynthesis());
hosts.Add(imeLogHost);
imeLogHostRef = imeLogHost;
// sits-d Cloud-PC fix (2026-08-19): the user-apps-settled AccountSetup synthesis used
// to run ONLY on the EspExited edge. On a tenant with a large required-app set the ESP
// page exits while apps are still in flight, the single attempt misses, and nothing
// ever re-checks — sessions 8110e262 / a89aac2d reached 138/138 apps with 0 failed and
// still never completed. Chain into the tracker's app-state callback (same
// preserve-previous pattern DeliveryOptimizationHost uses) so every terminal app
// transition gives the synthesis another look. Cheap: the re-check returns immediately
// unless the ESP exit was already observed and the synthesis has not fired yet.
var previousAppStateChanged = imeLogHost.Tracker.OnAppStateChanged;
imeLogHost.Tracker.OnAppStateChanged = (pkg, oldState, newState) =>
{
try { previousAppStateChanged?.Invoke(pkg, oldState, newState); }
catch (Exception ex)
{
logger.Warning($"DefaultComponentFactory: previous OnAppStateChanged handler threw: {ex.Message}");
}
try { espAndHelloHost.ReevaluateUserAppsSettledSynthesis(); }
catch (Exception ex)
{
logger.Warning($"DefaultComponentFactory: user-apps-settled re-check threw: {ex.Message}");
}
};
if (collectors.StallProbeEnabled)
{
hosts.Add(new StallProbeHost(
@@ -14,6 +14,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
private readonly EspAndHelloTracker _tracker;
private readonly EspAndHelloTrackerAdapter _adapter;
private readonly int _espExitBackfillLookbackMinutes;
private int _disposed;
/// <summary>
@@ -73,8 +74,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
bool isDevicePreparation = false,
Func<bool>? userEspAppsSettledProbe = null,
Func<System.Collections.Generic.IReadOnlyList<Monitoring.Enrollment.Ime.AppPackageState>>? starvedUserEspAppsProbe = null,
Func<System.Collections.Generic.IReadOnlyList<Monitoring.Enrollment.Ime.AppPackageState>>? packageStatesProbe = null)
Func<System.Collections.Generic.IReadOnlyList<Monitoring.Enrollment.Ime.AppPackageState>>? packageStatesProbe = null,
int espExitBackfillLookbackMinutes = ShellCoreTracker.BackfillLookbackMinutes)
{
_espExitBackfillLookbackMinutes = espExitBackfillLookbackMinutes;
if (ingress == null) throw new ArgumentNullException(nameof(ingress));
if (clock == null) throw new ArgumentNullException(nameof(clock));
var post = new InformationalEventPost(ingress, clock);
@@ -100,9 +103,56 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
_adapter = new EspAndHelloTrackerAdapter(_tracker, ingress, clock);
}
public void Start() => _tracker.Start();
/// <summary>
/// Starts the tracker and replays the recent Shell-Core Hello-wizard start — restart
/// recovery for an observation no agent process was around to make.
/// <para>
/// The backfill has existed since session 772fe502 but was never wired to a caller, so
/// the Shell-Core watcher only ever saw events written AFTER <see cref="Start"/>. Any
/// agent restart therefore lost an ESP exit (62407) or Hello-wizard start (62404) that
/// happened while the agent was down — and after a mid-ESP reboot that is exactly the
/// window the completion signal lands in (sits-d Cloud PCs, 2026-08-19: five sessions
/// hung in AccountSetup until the server-side timeout while the devices were fine).
/// </para>
/// <para>
/// A lookback of zero disables the replay entirely. That is the FIRST run of an enrollment:
/// there is no gap to recover because no earlier agent process existed, and any 62407
/// already in the log belongs to a phase transition that happened before this agent was
/// ever meant to observe anything. Replaying it would push an <c>EspExiting</c> into the
/// reducer that the pre-fix agent never saw — a happy-path behaviour change for no gain.
/// The caller (<c>DefaultComponentFactory</c>) owns that policy.
/// </para>
/// </summary>
/// <para>
/// Ordering is deliberate: Start first, then replay. A record written between the two is
/// observed twice, which the wizard rail already tolerates (single-shot replay guard, plus
/// the HelloTracker once-guard and the adapter's dedup flag downstream). The reverse order
/// could drop a record entirely.
/// </para>
/// <para>
/// The replay covers the Hello-wizard start only — a conservative fact that can make the
/// agent wait longer but never finish early. ESP exits are deliberately excluded because a
/// replayed 62407 cannot be placed in time; see
/// <c>ShellCoreTracker.ReplayBackfillRecords</c>.
/// </para>
/// </summary>
public void Start()
{
_tracker.Start();
if (_espExitBackfillLookbackMinutes > 0)
_tracker.BackfillRecentHelloWizardStart(_espExitBackfillLookbackMinutes);
}
public void Stop() => _tracker.Stop();
/// <summary>
/// Re-checks the user-apps-settled AccountSetup synthesis — see
/// <see cref="EspAndHelloTracker.ReevaluateUserAppsSettledSynthesis"/>. Driven by the IME
/// tracker's app-state-change callback (wired in <c>DefaultComponentFactory</c>).
/// </summary>
public void ReevaluateUserAppsSettledSynthesis() => _tracker.ReevaluateUserAppsSettledSynthesis();
private void OnTrackerWhiteGloveCompleted(object sender, EventArgs e)
=> WhiteGloveCompleted?.Invoke(this, e);
@@ -18,6 +18,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
private readonly ImeLogTracker _tracker;
private readonly ImeProcessWatcher _processWatcher;
private readonly Action? _onStateRestored;
private readonly ImeLogTrackerAdapter _adapter;
private readonly AgentLogger _logger;
private int _disposed;
@@ -96,9 +97,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
string stateDirectory,
IReadOnlyCollection<string>? whiteGloveSealingPatternIds,
bool simulationMode = false,
double simulationSpeedFactor = 50)
double simulationSpeedFactor = 50,
Action? onStateRestored = null)
{
_logger = logger;
_onStateRestored = onStateRestored;
var logFolder = string.IsNullOrEmpty(imeLogPathOverride) ? DefaultImeLogFolder : imeLogPathOverride!;
var expandedMatchLogPath = string.IsNullOrEmpty(imeMatchLogPath)
? null
@@ -128,10 +131,24 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
_processWatcher = new ImeProcessWatcher(sessionId, tenantId, processWatcherPost, logger);
}
/// <summary>
/// Codex review P2 (2026-08-19) — <see cref="ImeLogTracker.Start"/> restores the persisted
/// package states (LoadState) and does NOT raise <c>OnAppStateChanged</c> for them. This
/// host is started AFTER <c>EspAndHelloHost</c>, so a LIVE ESP exit observed before this
/// restore completes would evaluate the user-apps-settled synthesis against an empty app
/// list — and without a nudge here nothing would ever re-check the restored level. The
/// <c>onStateRestored</c> callback fires once, after the restore, so the deferred
/// synthesis gets its look. (It deliberately covers ONLY live exits: a 62407 that fired
/// while no agent was running is never replayed — see
/// <c>ShellCoreTracker.ReplayBackfillRecords</c> for why it cannot be placed in time.)
/// </summary>
public void Start()
{
_tracker.Start();
_processWatcher.Start();
try { _onStateRestored?.Invoke(); }
catch (Exception ex) { _logger.Warning($"ImeLogHost: onStateRestored callback threw: {ex.Message}"); }
}
public void Stop()
@@ -258,6 +258,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Transport.Telemetry
_pendingBytes -= _pending.First.Value.ByteLength;
_pending.RemoveFirst();
}
// _pendingBytes drives the spool-pressure trigger: negative drift would
// silently disarm it, a stale positive on an empty list would pin it on.
// The list is authoritative — on drift, recompute from it and log loudly.
if (_pendingBytes < 0 || (_pending.Count == 0 && _pendingBytes != 0))
{
long recomputed = 0;
foreach (var entry in _pending) recomputed += entry.ByteLength;
_logger?.Warning(
$"TelemetrySpool: pending-bytes accounting drifted to {_pendingBytes} with {_pending.Count} pending items after MarkUploaded({upToItemIdInclusive}) — recomputed to {recomputed}.");
_pendingBytes = recomputed;
}
}
}
@@ -79,13 +79,27 @@ namespace AutopilotMonitor.Agent.V2.Runtime
// write to the same state file; per-enrollment lifecycle like the other state files.
var startupEventGate = new AutopilotMonitor.Agent.V2.Core.Persistence.StartupEventGate(stateSubdir, logger);
// sits-d Cloud-PC fix (2026-08-19): a reboot_kill means the agent was down across the
// forced mid-ESP restart. Hand the boot timestamp to the factory so the Shell-Core
// ESP-exit backfill can reach back over the whole downtime instead of 5 minutes.
var previousBootUtc =
string.Equals(previousExit?.ExitType, "reboot_kill", StringComparison.OrdinalIgnoreCase)
? previousExit?.LastBootUtc
: null;
var componentFactory = new DefaultComponentFactory(
agentConfig: agentConfig,
remoteConfig: remoteConfig,
networkMetrics: auth.NetworkMetrics,
agentVersion: agentVersion,
stateDirectory: stateSubdir,
startupEventGate: startupEventGate);
startupEventGate: startupEventGate,
previousBootUtc: previousBootUtc,
previousExitType: previousExit?.ExitType,
// Measured, not guessed: the reducer snapshot is rewritten on every decision step,
// so its mtime is the last moment the previous run is known to have been alive —
// and unlike LastBootUtc it exists for every exit type, exception_crash included.
previousRunLastAliveUtc: DefaultComponentFactory.ReadPreviousRunLastAliveUtc(stateSubdir));
var whiteGloveSealingPatternIds = (System.Collections.Generic.IReadOnlyCollection<string>)remoteConfig.WhiteGloveSealingPatternIds
?? Array.Empty<string>();
@@ -0,0 +1,141 @@
using System;
using System.Collections.Generic;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.DecisionCore.State;
using Xunit;
namespace AutopilotMonitor.DecisionCore.Tests
{
/// <summary>
/// The reducer half of the "a replayed 62407 must never reach the decision engine" contract
/// (2026-08-19). The agent half — ShellCoreTracker never re-raising a replayed exit — lives in
/// <c>ShellCoreTrackerReplayScopeTests</c>; this file proves WHY that upstream restriction is
/// the only place the problem can be solved.
/// <para>
/// The reducer is itself a completion gate, and by design it cannot tell a replayed exit from
/// a live one:
/// </para>
/// <list type="bullet">
/// <item><c>HandleEspExitingV1</c> passes <c>espFinalExitInFlight: true</c> for EVERY
/// arriving exit — the signal carries no provenance.</item>
/// <item><c>IsPostAccountSetupFinalExit</c> orders exits by INGEST ORDINAL rather than by
/// timestamp (deliberately: replayed CMTrace lines carry backdated source times). A
/// historic exit replayed today is assigned a fresher ordinal than reality, so it reads as
/// post-AccountSetup by construction.</item>
/// </list>
/// <para>
/// After a restart the state is restored from the snapshot, so AccountSetupEntered, the IME
/// user-session completion and the desktop arrival are all already present. Arm C of
/// <c>ShouldTransitionToAwaitingHello</c> then needs exactly one more fact: an exit. These
/// tests pin that it opens — which is correct behaviour for a live exit and would be a
/// premature completion for a replayed one. Hence the upstream ban.
/// </para>
/// </summary>
public sealed class ClassicEspExitingOnRestoredStateTests
{
private static readonly DateTime Fixed = new DateTime(2026, 8, 19, 8, 43, 16, DateTimeKind.Utc);
[Fact]
public void Arm_C_opens_on_any_arriving_exit_once_restored_state_carries_the_other_three_facts()
{
// This is the danger, stated as a test: with restored facts in place, a single
// EspExiting signal is all that separates EspAccountSetup from AwaitingHello. The
// reducer has no way to ask "was that exit real, or did we dig it out of the event
// log?" — so the answer has to be that a replayed exit is never turned into a signal.
var engine = new DecisionEngine();
var state = BuildRestoredState(
accountSetupEntered: true,
imeUserSessionCompleted: true,
desktopArrived: true);
var step = engine.Reduce(state, MakeEspExitingSignal(ordinal: 10));
Assert.Equal(SessionStage.AwaitingHello, step.NewState.Stage);
}
[Theory]
// Each row drops exactly one of arm C's mandatory facts — the transition must not happen.
[InlineData(false, true, true)]
[InlineData(true, false, true)]
[InlineData(true, true, false)]
public void Arm_C_stays_shut_when_any_of_its_mandatory_facts_is_missing(
bool accountSetupEntered, bool imeUserSessionCompleted, bool desktopArrived)
{
var engine = new DecisionEngine();
var state = BuildRestoredState(accountSetupEntered, imeUserSessionCompleted, desktopArrived);
var step = engine.Reduce(state, MakeEspExitingSignal(ordinal: 10));
Assert.NotEqual(SessionStage.AwaitingHello, step.NewState.Stage);
}
[Fact]
public void A_historic_source_timestamp_does_not_hold_the_gate_shut()
{
// The exit's OccurredAtUtc is an hour older than the AccountSetup anchor — exactly the
// shape a replayed record has. It makes no difference: the ordering that arm C uses is
// the ingest ordinal, and the in-flight flag bypasses even that. Timestamps are not a
// defence, which is why the fix cannot live in the reducer.
var engine = new DecisionEngine();
var state = BuildRestoredState(
accountSetupEntered: true,
imeUserSessionCompleted: true,
desktopArrived: true);
var step = engine.Reduce(
state,
MakeEspExitingSignal(ordinal: 10, occurredAtUtc: Fixed.AddHours(-1)));
Assert.Equal(SessionStage.AwaitingHello, step.NewState.Stage);
}
// ------------------------------------------------------------------------------
private static DecisionState BuildRestoredState(
bool accountSetupEntered,
bool imeUserSessionCompleted,
bool desktopArrived)
{
var anchor = Fixed.AddMinutes(-90);
var builder = DecisionState.CreateInitial("s", "t", Fixed.AddDays(-1))
.ToBuilder()
.WithStage(SessionStage.EspAccountSetup)
.WithStepIndex(452) // the restored StepIndex the sits-d sessions showed
.WithLastAppliedSignalOrdinal(9);
if (accountSetupEntered)
builder.AccountSetupEnteredUtc = new SignalFact<DateTime>(anchor, 1);
// IsImeUserSessionGenuine requires the completion to be at-or-after the AccountSetup
// anchor (the defaultuser0-ghost guard).
if (imeUserSessionCompleted)
builder.ImeUserSessionCompletedUtc = new SignalFact<DateTime>(anchor.AddMinutes(5), 2);
if (desktopArrived)
builder.DesktopArrivedUtc = new SignalFact<DateTime>(anchor.AddMinutes(6), 3);
return builder.Build();
}
private static DecisionSignal MakeEspExitingSignal(long ordinal, DateTime? occurredAtUtc = null) =>
new DecisionSignal(
sessionSignalOrdinal: ordinal,
sessionTraceOrdinal: ordinal,
kind: DecisionSignalKind.EspExiting,
kindSchemaVersion: 1,
occurredAtUtc: occurredAtUtc ?? Fixed,
sourceOrigin: "EspAndHelloTracker",
evidence: new Evidence(
kind: EvidenceKind.Derived,
identifier: "esp-hello-detector-v1",
summary: "ESP exiting (coordinator-forwarded Shell-Core 62407)",
derivationInputs: new Dictionary<string, string>(StringComparer.Ordinal)
{
["subSource"] = "ShellCoreTracker",
["eventSource"] = "Microsoft-Windows-Shell-Core",
["eventId"] = "62407",
}));
}
}
@@ -0,0 +1,286 @@
using System;
using System.Collections.Generic;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.DecisionCore.State;
using Xunit;
namespace AutopilotMonitor.DecisionCore.Tests
{
/// <summary>
/// sits-d Cloud-PC fix (2026-08-20, sessions 8110e262 / a89aac2d / e7ba63c9 / cb4a485a /
/// 3d6278fb). On a SkipUser=true flow Windows never renders the user ESP page: Shell-Core
/// 62407 does not fire and the AccountSetup provisioning categories are never written —
/// verified live (Shell-Core watcher armed across three agent runs, zero 62407 in six
/// hours; SkipUserStatusPage read 10+ times across all runs, True every time). Arm B of
/// <c>ShouldTransitionToAwaitingHello</c> was therefore open from the bootstrap
/// EspConfigDetected onward, but every site that evaluates the gate hangs off a signal the
/// flow structurally cannot produce, so five sessions sat at "waiting on: hello_resolution"
/// until the max-lifetime watchdog.
/// <para>
/// The fix lets the observed skip stand in for the final-exit evidence inside
/// <c>HandleImeUserSessionCompletedV1</c>'s completion attempt. These tests replay the real
/// 8110e262 signal ordering end to end (including the Fix-10 AccountSetup bounce-back a
/// restart re-emission triggers) and pin the negative space: no skip observed, no desktop,
/// ghost IME completion, and Device Preparation flows are all unchanged.
/// </para>
/// </summary>
public sealed class ClassicSkipUserEspUserSessionCompletionTests
{
// 8110e262 run 1: agent_started 08:30:08Z.
private static readonly DateTime T0 = new DateTime(2026, 8, 19, 8, 30, 8, DateTimeKind.Utc);
[Fact]
public void Sits_d_signal_ordering_completes_via_hello_safety()
{
var engine = new DecisionEngine();
var state = DecisionState.CreateInitial(
"8110e262-da57-4282-9290-bb0c96d05614", "5ca2b350", T0);
state = engine.Reduce(state, MakeSignal(0, DecisionSignalKind.SessionStarted, T0, null)).NewState;
// Bootstrap EspConfigDetected — SkipUserStatusPage=True, read from FirstSync 2 s in.
state = engine.Reduce(state, MakeEspConfigSignal(1, skipUser: "true")).NewState;
Assert.True(state.ScenarioObservations.SkipUserEsp!.Value);
// Hello policy enabled (device-scoped CSP) — the reason the desktop fast-path
// could never take this flow.
state = engine.Reduce(state, MakeSignal(2, DecisionSignalKind.HelloPolicyDetected, T0.AddSeconds(1),
new Dictionary<string, string> { [SignalPayloadKeys.HelloEnabled] = "true" })).NewState;
// Desktop 5 s after agent start (the user ESP page does not exist — the user is
// simply logged on). Pins that desktop arrival alone does NOT promote: the knock
// deliberately lives on the IME user-session edge, because a desktop-side promotion
// would be undone by Fix 10's AccountSetup bounce-back two signals later (the IME
// AccountSetup phase line lands AFTER the desktop on this flow).
state = engine.Reduce(state, MakeSignal(3, DecisionSignalKind.DesktopArrived, T0.AddSeconds(5), null)).NewState;
Assert.NotEqual(SessionStage.AwaitingHello, state.Stage);
Assert.DoesNotContain(state.Deadlines, d => d.Name == DeadlineNames.HelloSafety);
// IME's AccountSetup phase line (source ts 08:29:56 — before agent start; the IME
// logs the phase for its user-session app processing even though no page is shown).
state = engine.Reduce(state, MakeSignal(4, DecisionSignalKind.EspPhaseChanged, T0.AddSeconds(-12),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState;
Assert.Equal(SessionStage.EspAccountSetup, state.Stage);
// 08:36:59 — IME user session completed (138/138 apps, 0 failed). THE knock:
// skip-user stands in for the exit that cannot exist; Hello enabled + unresolved
// promotes to AwaitingHello with HelloSafety armed.
var promoted = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.ImeUserSessionCompleted,
T0.AddMinutes(6).AddSeconds(51), null));
Assert.Equal(SessionStage.AwaitingHello, promoted.NewState.Stage);
var helloSafety = Assert.Single(promoted.NewState.Deadlines, d => d.Name == DeadlineNames.HelloSafety);
Assert.Contains(promoted.Effects,
e => e.Kind == DecisionEffectKind.ScheduleDeadline && e.Deadline?.Name == DeadlineNames.HelloSafety);
// The Hello wizard never appears over RDP — HelloSafety resolves synthetically.
var afterSafety = engine.Reduce(promoted.NewState,
MakeSignal(6, DecisionSignalKind.DeadlineFired, helloSafety.DueAtUtc,
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.HelloSafety }));
Assert.Equal(SessionStage.Finalizing, afterSafety.NewState.Stage);
Assert.Equal("Timeout", afterSafety.NewState.HelloOutcome!.Value);
var completed = engine.Reduce(afterSafety.NewState,
MakeSignal(7, DecisionSignalKind.DeadlineFired, helloSafety.DueAtUtc.AddSeconds(5),
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.FinalizingGrace }));
Assert.Equal(SessionStage.Completed, completed.NewState.Stage);
Assert.Equal(SessionOutcome.EnrollmentComplete, completed.NewState.Outcome);
Assert.Contains(completed.Effects,
e => e.Kind == DecisionEffectKind.EmitEventTimelineEntry
&& e.Parameters != null
&& e.Parameters.TryGetValue("eventType", out var et) && et == "enrollment_complete");
}
[Fact]
public void Fix10_bounceback_is_healed_by_the_ime_reemission()
{
// Restart shape (8110e262 run 3): the promoted AwaitingHello gets bounced back by a
// re-emitted AccountSetup phase line (Fix 10 cancels HelloSafety, deliberately — it
// guards the premature-promotion case). The IME phase line always precedes the
// user-session-complete line in log order, so the re-emitted completion re-knocks.
var engine = new DecisionEngine();
var state = ProgressToPromotedAwaitingHello(engine);
var bounced = engine.Reduce(state, MakeSignal(10, DecisionSignalKind.EspPhaseChanged,
T0.AddMinutes(18).AddSeconds(10),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" }));
Assert.Equal(SessionStage.EspAccountSetup, bounced.NewState.Stage);
Assert.DoesNotContain(bounced.NewState.Deadlines, d => d.Name == DeadlineNames.HelloSafety);
var reknocked = engine.Reduce(bounced.NewState,
MakeSignal(11, DecisionSignalKind.ImeUserSessionCompleted, T0.AddMinutes(6).AddSeconds(51), null));
Assert.Equal(SessionStage.AwaitingHello, reknocked.NewState.Stage);
Assert.Single(reknocked.NewState.Deadlines, d => d.Name == DeadlineNames.HelloSafety);
}
[Theory]
// No EspConfigDetected at all — SkipUserEsp unknown.
[InlineData(null)]
// Explicit SkipUser=false (the two sits-d machines that succeeded had this shape and
// a REAL page: exit + registry evidence. Without the exit the gate must stay shut.)
[InlineData("false")]
public void No_promotion_without_an_observed_skip(string? skipUser)
{
var engine = new DecisionEngine();
var state = DecisionState.CreateInitial("s", "t", T0);
state = engine.Reduce(state, MakeSignal(0, DecisionSignalKind.SessionStarted, T0, null)).NewState;
if (skipUser != null)
{
state = engine.Reduce(state, MakeEspConfigSignal(1, skipUser)).NewState;
}
state = engine.Reduce(state, MakeSignal(2, DecisionSignalKind.HelloPolicyDetected, T0.AddSeconds(1),
new Dictionary<string, string> { [SignalPayloadKeys.HelloEnabled] = "true" })).NewState;
state = engine.Reduce(state, MakeSignal(3, DecisionSignalKind.DesktopArrived, T0.AddSeconds(5), null)).NewState;
state = engine.Reduce(state, MakeSignal(4, DecisionSignalKind.EspPhaseChanged, T0.AddSeconds(10),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState;
var step = engine.Reduce(state,
MakeSignal(5, DecisionSignalKind.ImeUserSessionCompleted, T0.AddMinutes(7), null));
Assert.Equal(SessionStage.EspAccountSetup, step.NewState.Stage);
Assert.DoesNotContain(step.NewState.Deadlines, d => d.Name == DeadlineNames.HelloSafety);
}
[Fact]
public void No_promotion_without_the_desktop()
{
var engine = new DecisionEngine();
var state = DecisionState.CreateInitial("s", "t", T0);
state = engine.Reduce(state, MakeSignal(0, DecisionSignalKind.SessionStarted, T0, null)).NewState;
state = engine.Reduce(state, MakeEspConfigSignal(1, skipUser: "true")).NewState;
state = engine.Reduce(state, MakeSignal(2, DecisionSignalKind.EspPhaseChanged, T0.AddSeconds(10),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState;
var step = engine.Reduce(state,
MakeSignal(3, DecisionSignalKind.ImeUserSessionCompleted, T0.AddMinutes(7), null));
Assert.Equal(SessionStage.EspAccountSetup, step.NewState.Stage);
Assert.DoesNotContain(step.NewState.Deadlines, d => d.Name == DeadlineNames.HelloSafety);
}
[Fact]
public void A_ghost_pre_anchor_ime_completion_does_not_knock()
{
// defaultuser0-ghost guard stays load-bearing: an IME completion whose source time
// precedes the AccountSetup anchor is the OOBE/technician frame, not the user.
var engine = new DecisionEngine();
var state = DecisionState.CreateInitial("s", "t", T0);
state = engine.Reduce(state, MakeSignal(0, DecisionSignalKind.SessionStarted, T0, null)).NewState;
state = engine.Reduce(state, MakeEspConfigSignal(1, skipUser: "true")).NewState;
state = engine.Reduce(state, MakeSignal(2, DecisionSignalKind.DesktopArrived, T0.AddSeconds(5), null)).NewState;
state = engine.Reduce(state, MakeSignal(3, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(2),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState;
var step = engine.Reduce(state,
MakeSignal(4, DecisionSignalKind.ImeUserSessionCompleted, T0.AddMinutes(1), null));
Assert.Equal(SessionStage.EspAccountSetup, step.NewState.Stage);
Assert.DoesNotContain(step.NewState.Deadlines, d => d.Name == DeadlineNames.HelloSafety);
}
[Fact]
public void Device_preparation_flows_keep_their_own_rails()
{
// WDP has no ESP at all — completion rides the Hello+Desktop conjunction and the
// DevicePrepCompletion backstop (afee7ae0). Even a (spurious) SkipUser=true read
// must not route WDP through the classic skip-user knock.
var engine = new DecisionEngine();
var seed = DecisionState.CreateInitial("s", "t", T0)
.ToBuilder()
.WithStage(SessionStage.SessionStarted)
.WithStepIndex(5)
.WithLastAppliedSignalOrdinal(4)
.WithScenarioProfile(EnrollmentScenarioProfile.Empty.With(
mode: EnrollmentMode.DevicePreparation,
confidence: ProfileConfidence.High,
evidenceOrdinal: 1,
reason: "test"));
seed.ScenarioObservations = EnrollmentScenarioObservations.Empty
.WithSkipUserEsp(true, 1);
seed.AccountSetupEnteredUtc = new SignalFact<DateTime>(T0.AddMinutes(1), 2);
seed.DesktopArrivedUtc = new SignalFact<DateTime>(T0.AddMinutes(2), 3);
var state = seed.Build();
var step = engine.Reduce(state,
MakeSignal(5, DecisionSignalKind.ImeUserSessionCompleted, T0.AddMinutes(3), null));
Assert.DoesNotContain(step.NewState.Deadlines, d => d.Name == DeadlineNames.HelloSafety);
Assert.NotEqual(SessionStage.AwaitingHello, step.NewState.Stage);
}
[Fact]
public void Hello_disabled_skip_user_completes_directly_from_the_knock()
{
// HelloSatisfiedForCompletion holds (policy disabled, no wizard observed) — the
// knock completes through Finalizing with the synthetic Skipped outcome instead of
// parking 5 minutes in AwaitingHello.
var engine = new DecisionEngine();
var state = DecisionState.CreateInitial("s", "t", T0);
state = engine.Reduce(state, MakeSignal(0, DecisionSignalKind.SessionStarted, T0, null)).NewState;
state = engine.Reduce(state, MakeEspConfigSignal(1, skipUser: "true")).NewState;
state = engine.Reduce(state, MakeSignal(2, DecisionSignalKind.HelloPolicyDetected, T0.AddSeconds(1),
new Dictionary<string, string> { [SignalPayloadKeys.HelloEnabled] = "false" })).NewState;
state = engine.Reduce(state, MakeSignal(3, DecisionSignalKind.EspPhaseChanged, T0.AddSeconds(10),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState;
// Desktop arrives; the hello-disabled fast-path already promotes/completes via the
// gate's arm B on this ordering, so the knock is not even needed here — assert the
// flow terminates either way (belt for orderings where desktop precedes the config).
state = engine.Reduce(state, MakeSignal(4, DecisionSignalKind.DesktopArrived, T0.AddMinutes(5), null)).NewState;
if (state.Stage != SessionStage.Finalizing && !state.Stage.IsTerminal())
{
var step = engine.Reduce(state,
MakeSignal(5, DecisionSignalKind.ImeUserSessionCompleted, T0.AddMinutes(7), null));
Assert.Equal(SessionStage.Finalizing, step.NewState.Stage);
Assert.Equal("Skipped", step.NewState.HelloOutcome!.Value);
}
else
{
Assert.Equal(SessionStage.Finalizing, state.Stage);
}
}
// ------------------------------------------------------------------------------
private static DecisionState ProgressToPromotedAwaitingHello(DecisionEngine engine)
{
var state = DecisionState.CreateInitial("s", "t", T0);
state = engine.Reduce(state, MakeSignal(0, DecisionSignalKind.SessionStarted, T0, null)).NewState;
state = engine.Reduce(state, MakeEspConfigSignal(1, skipUser: "true")).NewState;
state = engine.Reduce(state, MakeSignal(2, DecisionSignalKind.HelloPolicyDetected, T0.AddSeconds(1),
new Dictionary<string, string> { [SignalPayloadKeys.HelloEnabled] = "true" })).NewState;
state = engine.Reduce(state, MakeSignal(3, DecisionSignalKind.DesktopArrived, T0.AddSeconds(5), null)).NewState;
state = engine.Reduce(state, MakeSignal(4, DecisionSignalKind.EspPhaseChanged, T0.AddSeconds(-12),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState;
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.ImeUserSessionCompleted,
T0.AddMinutes(6).AddSeconds(51), null)).NewState;
Assert.Equal(SessionStage.AwaitingHello, state.Stage);
return state;
}
private static DecisionSignal MakeEspConfigSignal(long ordinal, string skipUser) =>
MakeSignal(ordinal, DecisionSignalKind.EspConfigDetected, T0.AddSeconds(ordinal),
new Dictionary<string, string>
{
[SignalPayloadKeys.SkipUserEsp] = skipUser,
[SignalPayloadKeys.SkipDeviceEsp] = "false",
});
private static DecisionSignal MakeSignal(
long ordinal,
DecisionSignalKind kind,
DateTime occurredAtUtc,
IReadOnlyDictionary<string, string>? payload)
{
return new DecisionSignal(
sessionSignalOrdinal: ordinal,
sessionTraceOrdinal: ordinal,
kind: kind,
kindSchemaVersion: 1,
occurredAtUtc: occurredAtUtc,
sourceOrigin: "test",
evidence: new Evidence(EvidenceKind.Synthetic, $"test-{kind}-{ordinal}", $"synthetic {kind}"),
payload: payload);
}
}
}
@@ -192,6 +192,33 @@ namespace AutopilotMonitor.DecisionCore.Tests
Assert.Equal(SessionStage.Completed, completed.NewState.Stage);
}
[Fact]
public void BackstopFired_WithoutDesktop_ParksInAwaitingDesktopUntilDesktopCompletes()
{
// The arming site (HandleDesktopArrivedV1) guarantees desktop-first, so this
// fires only on a stale/replayed timer — Dispatch routes DeadlineFired purely on
// the payload name, no armed-state check. The fallback must keep the handler
// total: synthetic Hello timeout recorded, session parks in AwaitingDesktop, and
// a later real desktop arrival still completes it.
var engine = new DecisionEngine();
var state = SeedWdpSession(engine);
var fired = engine.Reduce(state, DeadlineFired(20, T0.AddMinutes(39), DeadlineNames.DevicePrepCompletion));
Assert.Equal(SessionStage.AwaitingDesktop, fired.NewState.Stage);
Assert.NotNull(fired.NewState.HelloResolvedUtc);
Assert.Equal("Timeout", fired.NewState.HelloOutcome!.Value);
Assert.DoesNotContain(fired.NewState.Deadlines, d => d.Name == DeadlineNames.DevicePrepCompletion);
// Hello is satisfied (synthetically) — only the desktop is still missing.
var waiting = Assert.Single(CompletionWaitingEffects(fired));
Assert.Equal("desktop_arrival", waiting.Parameters!["missingPrerequisites"]);
var completed = engine.Reduce(fired.NewState,
MakeSignal(21, DecisionSignalKind.DesktopArrived, T0.AddMinutes(45)));
Assert.Equal(SessionStage.Finalizing, completed.NewState.Stage);
}
[Fact]
public void HelloResolved_BeforeBackstopFires_CompletesNormally()
{
@@ -0,0 +1,235 @@
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using AutopilotMonitor.Shared.Pagination;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// AgentEfficiencyMetricsService: per-agent-version bucketing, nearest-rank percentiles,
/// crash tally per bucket, top-offender selection and the scanned/with-snapshots split.
/// Distinct `days` values per test keep the static per-(days,limit,tenant) cache from
/// cross-talking between tests.
/// </summary>
public class AgentEfficiencyMetricsServiceTests
{
private const string Tenant = "11111111-1111-1111-1111-111111111111";
private static (AgentEfficiencyMetricsService Service, Mock<ISessionRepository> Repo) CreateService()
{
var repo = new Mock<ISessionRepository>();
repo
.Setup(r => r.GetAllSessionsPageAsync(It.IsAny<string?>(), It.IsAny<int?>(), It.IsAny<int>(), It.IsAny<string?>(), It.IsAny<IReadOnlyCollection<string>?>(), It.IsAny<IEnumerable<string>?>()))
.ReturnsAsync(new RawPage<SessionSummary>(new List<SessionSummary>(), null));
repo
.Setup(r => r.GetSessionEventsByTypesAsync(It.IsAny<string>(), It.IsAny<string>(), It.IsAny<IReadOnlyCollection<string>>(), It.IsAny<IEnumerable<string>?>(), It.IsAny<int>()))
.ReturnsAsync(new List<EnrollmentEvent>());
var service = new AgentEfficiencyMetricsService(repo.Object, NullLogger<AgentEfficiencyMetricsService>.Instance);
return (service, repo);
}
private static SessionSummary MakeSession(int i, string agentVersion, double? avgLatency = null, int? requestCount = null) => new SessionSummary
{
SessionId = $"00000000-0000-0000-0000-{i:D12}",
TenantId = Tenant,
AgentVersion = agentVersion,
DeviceName = $"PC-{i}",
StartedAt = DateTime.UtcNow.AddHours(-i),
Status = SessionStatus.Succeeded,
AvgApiLatencyMs = avgLatency,
ApiRequestCount = requestCount
};
private static EnrollmentEvent Snapshot(double cpu, double ws, double handles = 0, double threads = 0, double spool = 0, long sequence = 1) => new EnrollmentEvent
{
EventType = "agent_metrics_snapshot",
Sequence = sequence,
Data = new Dictionary<string, object>
{
["agent_cpu_percent"] = cpu,
["agent_working_set_mb"] = ws,
["agent_handle_count"] = handles,
["agent_thread_count"] = threads,
["spool_pending_item_count"] = spool
}
};
private static EnrollmentEvent AgentStarted(string exitType, string? exception = null)
{
var data = new Dictionary<string, object> { ["previousExitType"] = exitType };
if (exception != null) data["previousCrashException"] = exception;
return new EnrollmentEvent { EventType = "agent_started", Data = data };
}
private static void SetupSessions(Mock<ISessionRepository> repo, List<SessionSummary> sessions)
{
repo
.Setup(r => r.GetAllSessionsPageAsync(It.IsAny<string?>(), It.IsAny<int?>(), It.IsAny<int>(), It.IsAny<string?>(), It.IsAny<IReadOnlyCollection<string>?>(), It.IsAny<IEnumerable<string>?>()))
.ReturnsAsync(new RawPage<SessionSummary>(sessions, null));
}
private static void SetupEvents(Mock<ISessionRepository> repo, string sessionId, params EnrollmentEvent[] events)
{
repo
.Setup(r => r.GetSessionEventsByTypesAsync(It.IsAny<string>(), sessionId, It.IsAny<IReadOnlyCollection<string>>(), It.IsAny<IEnumerable<string>?>(), It.IsAny<int>()))
.ReturnsAsync(events.ToList());
}
[Fact]
public async Task Compute_buckets_by_agent_version_with_nearest_rank_percentiles()
{
var (service, repo) = CreateService();
var sessions = new List<SessionSummary>
{
MakeSession(1, "2.0.1400"), MakeSession(2, "2.0.1400"),
MakeSession(3, "2.0.1400"), MakeSession(4, "2.0.1400"),
MakeSession(5, "2.0.1402")
};
SetupSessions(repo, sessions);
// Per-session max working set for the 1400 bucket: 100, 200, 300, 400.
SetupEvents(repo, sessions[0].SessionId, Snapshot(cpu: 1, ws: 50), Snapshot(cpu: 3, ws: 100, sequence: 2));
SetupEvents(repo, sessions[1].SessionId, Snapshot(cpu: 2, ws: 200));
SetupEvents(repo, sessions[2].SessionId, Snapshot(cpu: 4, ws: 300));
SetupEvents(repo, sessions[3].SessionId, Snapshot(cpu: 8, ws: 400));
SetupEvents(repo, sessions[4].SessionId, Snapshot(cpu: 5, ws: 500));
var result = await service.ComputeAsync(days: 41, limit: 100);
Assert.Equal(5, result.SessionsScanned);
Assert.Equal(5, result.SessionsWithSnapshots);
Assert.Equal(2, result.ByVersion.Count);
var v1400 = result.ByVersion.Single(b => b.AgentVersion == "2.0.1400");
Assert.Equal(4, v1400.SessionsScanned);
Assert.Equal(4, v1400.SessionsWithSnapshots);
// Nearest-rank over sorted [100, 200, 300, 400]: p50 → 2nd (200), p95 → 4th (400).
Assert.Equal(200, v1400.MaxWorkingSetMb!.P50);
Assert.Equal(400, v1400.MaxWorkingSetMb.P95);
Assert.Equal(400, v1400.MaxWorkingSetMb.Max);
Assert.Equal(4, v1400.MaxWorkingSetMb.SampleCount);
// Session 1's max CPU is 3 (max over its two snapshots), not the last value.
Assert.Equal(8, v1400.MaxCpuPercent!.Max);
var v1402 = result.ByVersion.Single(b => b.AgentVersion == "2.0.1402");
Assert.Equal(1, v1402.SessionsScanned);
Assert.Equal(500, v1402.MaxWorkingSetMb!.Max);
Assert.NotNull(result.Overall);
Assert.Null(result.Overall!.AgentVersion);
Assert.Equal(5, result.Overall.SessionsScanned);
Assert.Equal(500, result.Overall.MaxWorkingSetMb!.Max);
}
[Fact]
public async Task Compute_tallies_crash_rate_per_version_bucket()
{
var (service, repo) = CreateService();
var sessions = new List<SessionSummary> { MakeSession(1, "2.0.1400"), MakeSession(2, "2.0.1402") };
SetupSessions(repo, sessions);
SetupEvents(repo, sessions[0].SessionId,
AgentStarted("first_run"),
AgentStarted("exception_crash", "System.IO.IOException"),
AgentStarted("clean"));
SetupEvents(repo, sessions[1].SessionId, AgentStarted("first_run"), AgentStarted("clean"));
var result = await service.ComputeAsync(days: 42, limit: 100);
var v1400 = result.ByVersion.Single(b => b.AgentVersion == "2.0.1400");
Assert.Equal(3, v1400.CrashRate!.TotalStarts);
Assert.Equal(1, v1400.CrashRate.ExceptionCrashes);
Assert.Equal("System.IO.IOException", Assert.Single(v1400.CrashRate.TopExceptions).ExceptionType);
// 2 non-first-run starts, 1 crash → 50%.
Assert.Equal(50, v1400.CrashRate.CrashRatePercent);
var v1402 = result.ByVersion.Single(b => b.AgentVersion == "2.0.1402");
Assert.Equal(0, v1402.CrashRate!.ExceptionCrashes);
Assert.Equal(0, v1402.CrashRate.CrashRatePercent);
}
[Fact]
public async Task Compute_selects_top_offenders_per_dimension_descending()
{
var (service, repo) = CreateService();
var sessions = Enumerable.Range(1, 5).Select(i => MakeSession(i, "2.0.1400")).ToList();
SetupSessions(repo, sessions);
for (var i = 0; i < 5; i++)
{
// Working sets 100..500 — top 3 must be sessions 5, 4, 3 in that order.
SetupEvents(repo, sessions[i].SessionId, Snapshot(cpu: 1, ws: (i + 1) * 100));
}
var result = await service.ComputeAsync(days: 43, limit: 100);
var bucket = result.ByVersion.Single();
var wsOffenders = bucket.TopOffenders!.Where(o => o.Dimension == "maxWorkingSetMb").ToList();
Assert.Equal(3, wsOffenders.Count);
Assert.Equal(new[] { 500d, 400d, 300d }, wsOffenders.Select(o => o.Value).ToArray());
Assert.Equal(sessions[4].SessionId, wsOffenders[0].SessionId);
Assert.Equal("PC-5", wsOffenders[0].DeviceName);
}
[Fact]
public async Task Compute_snapshotless_sessions_still_contribute_index_mirrored_latency()
{
var (service, repo) = CreateService();
var sessions = new List<SessionSummary>
{
MakeSession(1, "2.0.1400", avgLatency: 80, requestCount: 200),
MakeSession(2, "2.0.1400", avgLatency: 120, requestCount: 300) // no snapshots
};
SetupSessions(repo, sessions);
SetupEvents(repo, sessions[0].SessionId, Snapshot(cpu: 1, ws: 100));
var result = await service.ComputeAsync(days: 44, limit: 100);
Assert.Equal(2, result.SessionsScanned);
Assert.Equal(1, result.SessionsWithSnapshots);
var bucket = result.ByVersion.Single();
// Latency stats cover BOTH sessions (mirror-sourced), resource stats only the one with snapshots.
Assert.Equal(2, bucket.ApiLatencyMs!.SampleCount);
Assert.Equal(120, bucket.ApiLatencyMs.Max);
Assert.Equal(1, bucket.MaxWorkingSetMb!.SampleCount);
}
[Fact]
public async Task Compute_passes_projection_and_tenant_filter_to_repo_and_echoes_tenant()
{
var (service, repo) = CreateService();
string? capturedTenant = null;
IEnumerable<string>? capturedSelect = null;
repo
.Setup(r => r.GetAllSessionsPageAsync(It.IsAny<string?>(), It.IsAny<int?>(), It.IsAny<int>(), It.IsAny<string?>(), It.IsAny<IReadOnlyCollection<string>?>(), It.IsAny<IEnumerable<string>?>()))
.Callback<string?, int?, int, string?, IReadOnlyCollection<string>?, IEnumerable<string>?>((t, _, _, _, _, sel) => { capturedTenant = t; capturedSelect = sel; })
.ReturnsAsync(new RawPage<SessionSummary>(new List<SessionSummary>(), null));
var result = await service.ComputeAsync(days: 45, limit: 100, tenantId: Tenant);
Assert.Equal(Tenant, capturedTenant);
Assert.Same(AgentEfficiencyMetricsService.SessionIndexProjection, capturedSelect);
Assert.Equal(Tenant, result.TenantId);
Assert.Equal(45, result.WindowDays);
Assert.Equal(100, result.SessionLimit);
}
[Fact]
public async Task Compute_caches_per_days_limit_tenant_key()
{
var (service, repo) = CreateService();
int callCount = 0;
repo
.Setup(r => r.GetAllSessionsPageAsync(It.IsAny<string?>(), It.IsAny<int?>(), It.IsAny<int>(), It.IsAny<string?>(), It.IsAny<IReadOnlyCollection<string>?>(), It.IsAny<IEnumerable<string>?>()))
.Callback(() => callCount++)
.ReturnsAsync(new RawPage<SessionSummary>(new List<SessionSummary>(), null));
var first = await service.ComputeAsync(days: 46, limit: 100);
var second = await service.ComputeAsync(days: 46, limit: 100); // cache hit
await service.ComputeAsync(days: 46, limit: 100, tenantId: Tenant); // miss — tenant-scoped
Assert.Equal(2, callCount);
Assert.False(first.FromCache);
Assert.True(second.FromCache);
}
}
@@ -0,0 +1,93 @@
using System;
using System.Collections.Generic;
using Azure.Data.Tables;
using AutopilotMonitor.Functions.Services;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Pins that the column-projected SessionsIndex scan driving
/// <c>AgentEfficiencyMetricsService</c> maps every field the aggregation reads
/// (SessionId, TenantId, DeviceName, AgentVersion, AvgApiLatencyMs, ApiRequestCount,
/// StartedAt) identically to the full-row drain. Same fixture discipline as
/// <see cref="SessionStatsProjectionEquivalenceTests"/>: the projected row carries the
/// projection's keys ONLY, so a getter for an omitted column returns null exactly as it
/// would against live storage — and dropping a load-bearing column from the production
/// array fails here immediately.
/// </summary>
public class AgentEfficiencyProjectionEquivalenceTests
{
private const string TenantId = "00000000-0000-0000-0000-000000000abc";
private static readonly TableStorageService Sut =
new(new Mock<TableServiceClient>().Object, NullLogger<TableStorageService>.Instance);
private static string IndexRowKey(DateTime startedAt, string sessionId)
=> $"{(DateTime.MaxValue.Ticks - startedAt.Ticks):D19}_{sessionId}";
private static TableEntity FullRow(string sessionId, DateTime startedAt)
=> new TableEntity(TenantId, IndexRowKey(startedAt, sessionId))
{
["SessionId"] = sessionId,
["Status"] = "Succeeded",
["StartedAt"] = new DateTimeOffset(startedAt),
["AgentVersion"] = "2.0.1400",
["DeviceName"] = "PC-EFF",
["AvgApiLatencyMs"] = 84.5,
["ApiRequestCount"] = 321,
// Representative noise the efficiency scan never reads.
["SerialNumber"] = "SN-FULL",
["Manufacturer"] = "Contoso",
["Model"] = "Model-X",
["OsName"] = "Windows 11",
["GeoCountry"] = "DE",
["EventCount"] = 123,
["FailureSnapshotJson"] = "{\"big\":\"" + new string('x', 2000) + "\"}",
};
private static TableEntity Project(TableEntity full)
{
var keep = new HashSet<string>(AgentEfficiencyMetricsService.SessionIndexProjection, StringComparer.Ordinal);
var projected = new TableEntity(full.PartitionKey, full.RowKey);
foreach (var kv in full)
{
if (kv.Key is "PartitionKey" or "RowKey" or "Timestamp" or "odata.etag") continue;
if (keep.Contains(kv.Key)) projected[kv.Key] = kv.Value;
}
return projected;
}
[Fact]
public void Projected_row_maps_every_efficiency_relevant_field_identically()
{
var started = DateTime.UtcNow.AddHours(-2);
var sid = Guid.NewGuid().ToString();
var fullRow = FullRow(sid, started);
var full = Sut.MapIndexEntityToSessionSummary(fullRow);
var projected = Sut.MapIndexEntityToSessionSummary(Project(fullRow));
Assert.Equal(sid, projected.SessionId);
Assert.Equal(full.TenantId, projected.TenantId);
Assert.Equal(full.DeviceName, projected.DeviceName);
Assert.Equal(full.AgentVersion, projected.AgentVersion);
Assert.Equal(full.AvgApiLatencyMs, projected.AvgApiLatencyMs);
Assert.Equal(full.ApiRequestCount, projected.ApiRequestCount);
// StartedAt feeds the repo's cross-tenant merge-sort and next-page cursor.
Assert.Equal(full.StartedAt, projected.StartedAt);
}
[Fact]
public void Projection_contains_the_repo_cursor_columns()
{
// FetchAllSessionsPageInternalAsync merge-sorts on StartedAt and rebuilds the cursor
// from StartedAt + SessionId; a projection without them silently breaks pagination.
Assert.Contains("StartedAt", AgentEfficiencyMetricsService.SessionIndexProjection);
Assert.Contains("SessionId", AgentEfficiencyMetricsService.SessionIndexProjection);
Assert.Contains("PartitionKey", AgentEfficiencyMetricsService.SessionIndexProjection);
Assert.Contains("RowKey", AgentEfficiencyMetricsService.SessionIndexProjection);
}
}
@@ -0,0 +1,162 @@
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// DO ingest fallback (IME ≥ 1.104 removed the [DO TEL] log line; the agent's DO collector
/// only emits <c>do_telemetry</c> for downloads whose completion a poll itself observed).
/// <c>download_progress</c> events carry the full <c>do*</c> set on every poll and must fold
/// into the summary's Do* columns so interrupted / between-poll-completed downloads still
/// reach the Delivery Optimization aggregates:
/// - progress-only sessions populate Do* incl. a valid DoDownloadMode (DoAggregator gate),
/// - a later do_telemetry stays authoritative,
/// - replayed progress events are idempotent (Math.Max),
/// - DownloadBytes prefers transferred bytes over file size.
/// </summary>
public class AppInstallDoFieldsFallbackTests
{
private static readonly DateTime T0 = new(2026, 8, 19, 10, 0, 0, DateTimeKind.Utc);
private static Dictionary<string, AppInstallAggregationState> Aggregate(params EnrollmentEvent[] events)
{
var summaries = new Dictionary<string, AppInstallAggregationState>(StringComparer.OrdinalIgnoreCase);
foreach (var evt in events)
EventIngestProcessor.AggregateAppInstallEvent(evt, "tenant", "session", summaries);
return summaries;
}
private static EnrollmentEvent DownloadProgress(
long bytesDownloaded, long doTotal, long doPeers, long doHttp, int doMode,
long doFileSize = 0, int doPct = 0, long doCacheServer = 0, DateTime? ts = null)
=> new EnrollmentEvent
{
EventType = "download_progress",
Timestamp = ts ?? T0,
Data = new Dictionary<string, object>
{
["appName"] = "App",
["bytesDownloaded"] = bytesDownloaded,
["doFileSize"] = doFileSize,
["doTotalBytesDownloaded"] = doTotal,
["doBytesFromPeers"] = doPeers,
["doBytesFromHttp"] = doHttp,
["doPercentPeerCaching"] = doPct,
["doDownloadMode"] = doMode,
["doBytesFromCacheServer"] = doCacheServer,
},
};
private static EnrollmentEvent DoTelemetry(
long doFileSize, long doTotal, long doPeers, long doHttp, int doMode,
string? duration = null)
{
var data = new Dictionary<string, object>
{
["appName"] = "App",
["doFileSize"] = doFileSize,
["doTotalBytesDownloaded"] = doTotal,
["doBytesFromPeers"] = doPeers,
["doBytesFromHttp"] = doHttp,
["doPercentPeerCaching"] = 50,
["doDownloadMode"] = doMode,
};
if (duration != null) data["doDownloadDuration"] = duration;
return new EnrollmentEvent { EventType = "do_telemetry", Timestamp = T0, Data = data };
}
[Fact]
public void ProgressOnly_PopulatesDoColumns_AndSurvivesDoAggregator()
{
// An interrupted download: progress events observed, do_telemetry never fired.
var s = Aggregate(
DownloadProgress(bytesDownloaded: 1000, doTotal: 1000, doPeers: 600, doHttp: 400, doMode: 1, doFileSize: 5000, doPct: 60),
DownloadProgress(bytesDownloaded: 3000, doTotal: 3000, doPeers: 2000, doHttp: 1000, doMode: 1, doFileSize: 5000, doPct: 66)
)["App"].Summary;
Assert.Equal(3000, s.DoTotalBytesDownloaded);
Assert.Equal(2000, s.DoBytesFromPeers);
Assert.Equal(1000, s.DoBytesFromHttp);
Assert.Equal(5000, s.DoFileSize);
Assert.Equal(66, s.DoPercentPeerCaching);
Assert.Equal(1, s.DoDownloadMode);
Assert.Equal(3000, s.DownloadBytes); // from bytesDownloaded, not the file size
// The whole point: DoDownloadMode >= 0 keeps the row inside the DO aggregate.
var aggregate = DoAggregator.Compute(new[] { s });
Assert.Equal(1, aggregate.DoAppCount);
Assert.Equal(2000, aggregate.BytesFromPeers);
}
[Fact]
public void ProgressReplay_IsIdempotent_MaxFoldNeverRegresses()
{
// Out-of-order / replayed batch: the older (smaller) progress arrives again last.
var s = Aggregate(
DownloadProgress(bytesDownloaded: 3000, doTotal: 3000, doPeers: 2000, doHttp: 1000, doMode: 1),
DownloadProgress(bytesDownloaded: 1000, doTotal: 1000, doPeers: 600, doHttp: 400, doMode: 1)
)["App"].Summary;
Assert.Equal(3000, s.DoTotalBytesDownloaded);
Assert.Equal(2000, s.DoBytesFromPeers);
Assert.Equal(1000, s.DoBytesFromHttp);
}
[Fact]
public void DoTelemetry_StaysAuthoritative_OverEarlierProgress()
{
// Progress saw a partial state; the final telemetry read has the complete numbers
// (which can be LOWER per bucket than a transient progress max — telemetry wins).
var s = Aggregate(
DownloadProgress(bytesDownloaded: 4000, doTotal: 4000, doPeers: 3500, doHttp: 500, doMode: 1, doPct: 87),
DoTelemetry(doFileSize: 5000, doTotal: 5000, doPeers: 3000, doHttp: 2000, doMode: 1, duration: "00:01:40")
)["App"].Summary;
Assert.Equal(5000, s.DoTotalBytesDownloaded);
Assert.Equal(3000, s.DoBytesFromPeers); // telemetry overwrote the higher progress value
Assert.Equal(2000, s.DoBytesFromHttp);
Assert.Equal(50, s.DoPercentPeerCaching);
Assert.Equal("00:01:40", s.DoDownloadDuration);
Assert.Equal(100, s.DownloadDurationSeconds);
}
[Fact]
public void DoTelemetry_WithUnsetMode_DoesNotRegressAKnownMode()
{
// A telemetry event whose DownloadMode property was absent parses as -1; it must not
// re-hide the row from DoAggregator after progress already established the mode.
var s = Aggregate(
DownloadProgress(bytesDownloaded: 1000, doTotal: 1000, doPeers: 600, doHttp: 400, doMode: 3),
DoTelemetry(doFileSize: 1000, doTotal: 1000, doPeers: 600, doHttp: 400, doMode: -1)
)["App"].Summary;
Assert.Equal(3, s.DoDownloadMode);
}
[Fact]
public void DownloadBytes_PrefersTransferredBytes_OverFileSize()
{
// Historic bug: DownloadBytes was max-ed with doFileSize, inflating the transfer
// measure. With a real transfer total present the file size must not win.
var s = Aggregate(
DoTelemetry(doFileSize: 9000, doTotal: 5000, doPeers: 3000, doHttp: 2000, doMode: 1)
)["App"].Summary;
Assert.Equal(5000, s.DownloadBytes);
Assert.Equal(9000, s.DoFileSize); // the file size column itself keeps the real value
}
[Fact]
public void DownloadBytes_FallsBackToFileSize_WhenNoTransferTotal()
{
var evt = new EnrollmentEvent
{
EventType = "do_telemetry",
Timestamp = T0,
Data = new Dictionary<string, object> { ["appName"] = "App", ["doFileSize"] = 7000L, ["doDownloadMode"] = 1 },
};
var s = Aggregate(evt)["App"].Summary;
Assert.Equal(7000, s.DownloadBytes);
}
}
@@ -0,0 +1,37 @@
using AutopilotMonitor.Functions.Services;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Pins the OData filter composition behind <c>GetSessionEventsByTypesAsync</c>:
/// partition scope AND-ed with a parenthesized or-chain, values escaped. The grouping
/// parentheses are load-bearing — without them the or-branches escape the partition
/// scope and the query degrades to a cross-partition table scan.
/// </summary>
public class EventTypesFilterTests
{
[Fact]
public void Single_type_builds_scoped_equality()
{
var filter = TableStorageService.BuildEventTypesFilter("tenant_session", new[] { "agent_metrics_snapshot" });
Assert.Equal("PartitionKey eq 'tenant_session' and (EventType eq 'agent_metrics_snapshot')", filter);
}
[Fact]
public void Multiple_types_or_chain_stays_inside_the_partition_scope()
{
var filter = TableStorageService.BuildEventTypesFilter(
"tenant_session", new[] { "agent_metrics_snapshot", "agent_started", "spool_pressure_detected" });
Assert.Equal(
"PartitionKey eq 'tenant_session' and (EventType eq 'agent_metrics_snapshot' or EventType eq 'agent_started' or EventType eq 'spool_pressure_detected')",
filter);
}
[Fact]
public void Values_with_single_quotes_are_escaped()
{
var filter = TableStorageService.BuildEventTypesFilter("t'x", new[] { "a'b" });
Assert.Equal("PartitionKey eq 't''x' and (EventType eq 'a''b')", filter);
}
}
@@ -9,7 +9,10 @@ namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Verifies PlatformMetricsService passes the requested window + session limit
/// through to GetAllSessionsPageAsync and surfaces both on the response.
/// through to GetAllSessionsPageAsync and surfaces both on the response, that
/// SessionsScanned reports the honest scan count (not the has-snapshots subset),
/// and that the split fetch (full events for the 20 latency-sample sessions,
/// filtered three-type fetch for the rest) preserves the aggregation semantics.
/// </summary>
public class PlatformMetricsServiceWindowTests
{
@@ -20,16 +23,36 @@ public class PlatformMetricsServiceWindowTests
{
var sessionRepo = new Mock<ISessionRepository>();
sessionRepo
.Setup(r => r.GetAllSessionsPageAsync(It.IsAny<string?>(), It.IsAny<int?>(), It.IsAny<int>(), It.IsAny<string?>(), It.IsAny<IReadOnlyCollection<string>?>()))
.Setup(r => r.GetAllSessionsPageAsync(It.IsAny<string?>(), It.IsAny<int?>(), It.IsAny<int>(), It.IsAny<string?>(), It.IsAny<IReadOnlyCollection<string>?>(), It.IsAny<IEnumerable<string>?>()))
.ReturnsAsync(EmptyPage());
sessionRepo
.Setup(r => r.GetSessionEventsAsync(It.IsAny<string>(), It.IsAny<string>(), It.IsAny<int>()))
.ReturnsAsync(new List<EnrollmentEvent>());
sessionRepo
.Setup(r => r.GetSessionEventsByTypesAsync(It.IsAny<string>(), It.IsAny<string>(), It.IsAny<IReadOnlyCollection<string>>(), It.IsAny<IEnumerable<string>?>(), It.IsAny<int>()))
.ReturnsAsync(new List<EnrollmentEvent>());
var service = new PlatformMetricsService(sessionRepo.Object, NullLogger<PlatformMetricsService>.Instance);
return (service, sessionRepo);
}
private static SessionSummary MakeSession(int i) => new SessionSummary
{
SessionId = $"00000000-0000-0000-0000-{i:D12}",
TenantId = "11111111-1111-1111-1111-111111111111",
AgentVersion = "2.0.1400",
StartedAt = DateTime.UtcNow.AddHours(-i),
Status = SessionStatus.Succeeded
};
private static EnrollmentEvent MakeEvent(string eventType, Dictionary<string, object>? data = null, long sequence = 1) => new EnrollmentEvent
{
EventType = eventType,
Data = data ?? new Dictionary<string, object>(),
Sequence = sequence,
Timestamp = DateTime.UtcNow
};
[Theory]
[InlineData(11)]
[InlineData(13)]
@@ -41,8 +64,8 @@ public class PlatformMetricsServiceWindowTests
int? capturedDays = null;
repo
.Setup(r => r.GetAllSessionsPageAsync(It.IsAny<string?>(), It.IsAny<int?>(), It.IsAny<int>(), It.IsAny<string?>(), It.IsAny<IReadOnlyCollection<string>?>()))
.Callback<string?, int?, int, string?, IReadOnlyCollection<string>?>((_, d, _, _, _) => capturedDays = d)
.Setup(r => r.GetAllSessionsPageAsync(It.IsAny<string?>(), It.IsAny<int?>(), It.IsAny<int>(), It.IsAny<string?>(), It.IsAny<IReadOnlyCollection<string>?>(), It.IsAny<IEnumerable<string>?>()))
.Callback<string?, int?, int, string?, IReadOnlyCollection<string>?, IEnumerable<string>?>((_, d, _, _, _, _) => capturedDays = d)
.ReturnsAsync(EmptyPage());
var result = await service.ComputePlatformMetricsAsync(days, limit: 50);
@@ -66,8 +89,8 @@ public class PlatformMetricsServiceWindowTests
int? capturedPageSize = null;
repo
.Setup(r => r.GetAllSessionsPageAsync(It.IsAny<string?>(), It.IsAny<int?>(), It.IsAny<int>(), It.IsAny<string?>(), It.IsAny<IReadOnlyCollection<string>?>()))
.Callback<string?, int?, int, string?, IReadOnlyCollection<string>?>((_, _, ps, _, _) => capturedPageSize = ps)
.Setup(r => r.GetAllSessionsPageAsync(It.IsAny<string?>(), It.IsAny<int?>(), It.IsAny<int>(), It.IsAny<string?>(), It.IsAny<IReadOnlyCollection<string>?>(), It.IsAny<IEnumerable<string>?>()))
.Callback<string?, int?, int, string?, IReadOnlyCollection<string>?, IEnumerable<string>?>((_, _, ps, _, _, _) => capturedPageSize = ps)
.ReturnsAsync(EmptyPage());
var result = await service.ComputePlatformMetricsAsync(days: days, limit: limit);
@@ -83,8 +106,8 @@ public class PlatformMetricsServiceWindowTests
int? capturedDays = null;
repo
.Setup(r => r.GetAllSessionsPageAsync(It.IsAny<string?>(), It.IsAny<int?>(), It.IsAny<int>(), It.IsAny<string?>(), It.IsAny<IReadOnlyCollection<string>?>()))
.Callback<string?, int?, int, string?, IReadOnlyCollection<string>?>((_, d, _, _, _) => capturedDays = d)
.Setup(r => r.GetAllSessionsPageAsync(It.IsAny<string?>(), It.IsAny<int?>(), It.IsAny<int>(), It.IsAny<string?>(), It.IsAny<IReadOnlyCollection<string>?>(), It.IsAny<IEnumerable<string>?>()))
.Callback<string?, int?, int, string?, IReadOnlyCollection<string>?, IEnumerable<string>?>((_, d, _, _, _, _) => capturedDays = d)
.ReturnsAsync(EmptyPage());
var result = await service.ComputePlatformMetricsAsync(0);
@@ -117,7 +140,7 @@ public class PlatformMetricsServiceWindowTests
var (service, repo) = CreateService();
int callCount = 0;
repo
.Setup(r => r.GetAllSessionsPageAsync(It.IsAny<string?>(), It.IsAny<int?>(), It.IsAny<int>(), It.IsAny<string?>(), It.IsAny<IReadOnlyCollection<string>?>()))
.Setup(r => r.GetAllSessionsPageAsync(It.IsAny<string?>(), It.IsAny<int?>(), It.IsAny<int>(), It.IsAny<string?>(), It.IsAny<IReadOnlyCollection<string>?>(), It.IsAny<IEnumerable<string>?>()))
.Callback(() => callCount++)
.ReturnsAsync(EmptyPage());
@@ -128,4 +151,96 @@ public class PlatformMetricsServiceWindowTests
Assert.Equal(3, callCount);
}
[Fact]
public async Task ComputePlatformMetrics_reports_sessionsScanned_even_when_no_session_has_snapshots()
{
// The truncated check must be based on scanned sessions, not the has-snapshots
// subset — a window full of snapshot-less sessions previously read as "not truncated".
var (service, repo) = CreateService();
var sessions = Enumerable.Range(1, 7).Select(MakeSession).ToList();
repo
.Setup(r => r.GetAllSessionsPageAsync(It.IsAny<string?>(), It.IsAny<int?>(), It.IsAny<int>(), It.IsAny<string?>(), It.IsAny<IReadOnlyCollection<string>?>(), It.IsAny<IEnumerable<string>?>()))
.ReturnsAsync(new RawPage<SessionSummary>(sessions, null));
var result = await service.ComputePlatformMetricsAsync(days: 77, limit: 50);
Assert.Equal(7, result.SessionsScanned);
Assert.Empty(result.Sessions);
}
[Fact]
public async Task ComputePlatformMetrics_uses_full_fetch_for_latency_sample_and_filtered_fetch_for_rest()
{
// First 20 sessions form the delivery-latency sample (deltas over ALL events);
// every further session only needs the three metrics event types.
var (service, repo) = CreateService();
var sessions = Enumerable.Range(1, 25).Select(MakeSession).ToList();
repo
.Setup(r => r.GetAllSessionsPageAsync(It.IsAny<string?>(), It.IsAny<int?>(), It.IsAny<int>(), It.IsAny<string?>(), It.IsAny<IReadOnlyCollection<string>?>(), It.IsAny<IEnumerable<string>?>()))
.ReturnsAsync(new RawPage<SessionSummary>(sessions, null));
var fullFetchSessions = new List<string>();
var filteredFetchSessions = new List<string>();
IReadOnlyCollection<string>? capturedEventTypes = null;
repo
.Setup(r => r.GetSessionEventsAsync(It.IsAny<string>(), It.IsAny<string>(), It.IsAny<int>()))
.Callback<string, string, int>((_, sid, _) => { lock (fullFetchSessions) fullFetchSessions.Add(sid); })
.ReturnsAsync(new List<EnrollmentEvent>());
repo
.Setup(r => r.GetSessionEventsByTypesAsync(It.IsAny<string>(), It.IsAny<string>(), It.IsAny<IReadOnlyCollection<string>>(), It.IsAny<IEnumerable<string>?>(), It.IsAny<int>()))
.Callback<string, string, IReadOnlyCollection<string>, IEnumerable<string>?, int>((_, sid, types, _, _) =>
{
lock (filteredFetchSessions) { filteredFetchSessions.Add(sid); capturedEventTypes = types; }
})
.ReturnsAsync(new List<EnrollmentEvent>());
await service.ComputePlatformMetricsAsync(days: 78, limit: 50);
Assert.Equal(20, fullFetchSessions.Count);
Assert.Equal(5, filteredFetchSessions.Count);
Assert.Equal(sessions.Take(20).Select(s => s.SessionId).ToHashSet(), fullFetchSessions.ToHashSet());
Assert.NotNull(capturedEventTypes);
Assert.Contains("agent_metrics_snapshot", capturedEventTypes!);
Assert.Contains("agent_started", capturedEventTypes!);
// Without spool_pressure_detected in the filter the SpoolPressureDetected flag
// silently dies on the filtered path.
Assert.Contains("spool_pressure_detected", capturedEventTypes!);
}
[Fact]
public async Task ComputePlatformMetrics_spool_pressure_and_crash_survive_the_filtered_path()
{
// A session outside the latency sample gets its events via the filtered fetch;
// the spool-pressure flag and crash tally must still work on that stream.
var (service, repo) = CreateService();
var sessions = Enumerable.Range(1, 21).Select(MakeSession).ToList();
repo
.Setup(r => r.GetAllSessionsPageAsync(It.IsAny<string?>(), It.IsAny<int?>(), It.IsAny<int>(), It.IsAny<string?>(), It.IsAny<IReadOnlyCollection<string>?>(), It.IsAny<IEnumerable<string>?>()))
.ReturnsAsync(new RawPage<SessionSummary>(sessions, null));
var session21 = sessions[20].SessionId;
repo
.Setup(r => r.GetSessionEventsByTypesAsync(It.IsAny<string>(), session21, It.IsAny<IReadOnlyCollection<string>>(), It.IsAny<IEnumerable<string>?>(), It.IsAny<int>()))
.ReturnsAsync(new List<EnrollmentEvent>
{
MakeEvent("agent_started", new Dictionary<string, object> { ["previousExitType"] = "exception_crash", ["previousCrashException"] = "System.NullReferenceException" }, sequence: 1),
MakeEvent("agent_metrics_snapshot", new Dictionary<string, object>
{
["agent_cpu_percent"] = 3.5,
["agent_working_set_mb"] = 90.0,
["spool_pending_item_count"] = 12.0
}, sequence: 2),
MakeEvent("spool_pressure_detected", sequence: 3)
});
var result = await service.ComputePlatformMetricsAsync(days: 79, limit: 50);
var metric = Assert.Single(result.Sessions);
Assert.Equal(session21, metric.SessionId);
Assert.True(metric.SpoolPressureDetected);
Assert.Equal(1, result.CrashRate!.ExceptionCrashes);
Assert.Equal("System.NullReferenceException", Assert.Single(result.CrashRate.TopExceptions).ExceptionType);
Assert.Equal(21, result.SessionsScanned);
}
}
@@ -1023,6 +1023,7 @@ public class PolicyEnforcementMiddlewareTests
// the GlobalReadOrDelegatedSubset tier — see GlobalSessions_IsBoundedSubsetTier below.)
[InlineData("GET", "/api/global/audit/logs", TenantScoping.QueryParam)]
[InlineData("GET", "/api/global/metrics/app", TenantScoping.QueryParam)]
[InlineData("GET", "/api/global/metrics/agent-efficiency", TenantScoping.QueryParam)]
[InlineData("GET", "/api/global/metrics/fleet-health", TenantScoping.QueryParam)]
[InlineData("GET", "/api/global/metrics/summary", TenantScoping.QueryParam)]
[InlineData("GET", "/api/global/search/sessions", TenantScoping.QueryParam)]
@@ -48,8 +48,8 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
public Task<RawPage<SessionSummary>> GetSessionsPageAsync(string tenantId, int? days, int pageSize, string? continuation)
=> _storage.GetSessionsPageAsync(tenantId, days, pageSize, continuation);
public Task<RawPage<SessionSummary>> GetAllSessionsPageAsync(string? tenantIdFilter, int? days, int pageSize, string? continuation, IReadOnlyCollection<string>? allowedTenantIds = null)
=> _storage.GetAllSessionsPageAsync(tenantIdFilter, days, pageSize, continuation, allowedTenantIds);
public Task<RawPage<SessionSummary>> GetAllSessionsPageAsync(string? tenantIdFilter, int? days, int pageSize, string? continuation, IReadOnlyCollection<string>? allowedTenantIds = null, IEnumerable<string>? select = null)
=> _storage.GetAllSessionsPageAsync(tenantIdFilter, days, pageSize, continuation, allowedTenantIds, select);
public Task<SessionStats> GetSessionStatsAsync(string tenantId, int days)
=> _storage.GetSessionStatsAsync(tenantId, days);
@@ -201,6 +201,9 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
public Task<List<EnrollmentEvent>> GetSessionEventsByTypeAsync(string tenantId, string sessionId, string eventType, int maxResults = 200)
=> _storage.GetSessionEventsByTypeAsync(tenantId, sessionId, eventType, maxResults);
public Task<List<EnrollmentEvent>> GetSessionEventsByTypesAsync(string tenantId, string sessionId, IReadOnlyCollection<string> eventTypes, IEnumerable<string>? select = null, int maxResults = 2000)
=> _storage.GetSessionEventsByTypesAsync(tenantId, sessionId, eventTypes, select, maxResults);
public Task<RawPage<EnrollmentEvent>> GetSessionEventsPageAsync(string tenantId, string sessionId, int pageSize, string? continuation)
=> _storage.GetSessionEventsPageAsync(tenantId, sessionId, pageSize, continuation);
@@ -0,0 +1,85 @@
using System.Net;
using AutopilotMonitor.Functions.Services;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Functions.Metrics
{
/// <summary>
/// Agent-efficiency aggregate (Global Admin / Global Reader): per-agent-version percentiles
/// for CPU, memory, handles, threads, spool and request metrics plus crash rates — computed
/// server-side so review tooling never has to pull raw snapshot rows. Honors <c>?tenantId=</c>
/// to scope the aggregate to one tenant (TenantScoping.QueryParam in the policy catalog).
/// </summary>
public class GetGlobalAgentEfficiencyFunction
{
private readonly ILogger<GetGlobalAgentEfficiencyFunction> _logger;
private readonly AgentEfficiencyMetricsService _efficiencyService;
public GetGlobalAgentEfficiencyFunction(
ILogger<GetGlobalAgentEfficiencyFunction> logger,
AgentEfficiencyMetricsService efficiencyService)
{
_logger = logger;
_efficiencyService = efficiencyService;
}
[Function("GetGlobalAgentEfficiency")]
public async Task<HttpResponseData> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "global/metrics/agent-efficiency")] HttpRequestData req)
{
try
{
// Authentication + GlobalReadOrAdmin authorization enforced by PolicyEnforcementMiddleware
var query = System.Web.HttpUtility.ParseQueryString(req.Url.Query);
var days = ParseIntParam(query["days"], defaultValue: 30, min: 1, max: 365);
var limit = ParseIntParam(query["limit"], defaultValue: 500, min: 1, max: 2000);
var tenantIdRaw = query["tenantId"];
string? tenantId = null;
if (!string.IsNullOrWhiteSpace(tenantIdRaw))
{
// The repo's per-tenant route throws on malformed GUIDs — turn that
// caller bug into a 400 instead of a 500.
if (!Guid.TryParse(tenantIdRaw, out var parsedTenant))
{
var badRequest = req.CreateResponse(HttpStatusCode.BadRequest);
await badRequest.WriteAsJsonAsync(new { success = false, message = "Invalid tenantId" });
return badRequest;
}
tenantId = parsedTenant.ToString();
}
var metrics = await _efficiencyService.ComputeAsync(days, limit, tenantId);
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(metrics);
return response;
}
catch (Exception ex)
{
_logger.LogError(ex, "Error computing agent efficiency metrics");
var errorResponse = req.CreateResponse(HttpStatusCode.InternalServerError);
await errorResponse.WriteAsJsonAsync(new
{
success = false,
message = "Failed to compute agent efficiency metrics"
});
return errorResponse;
}
}
private static int ParseIntParam(string? raw, int defaultValue, int min, int max)
{
var value = defaultValue;
if (!string.IsNullOrEmpty(raw) && int.TryParse(raw, out var parsed) && parsed > 0)
value = parsed;
if (value < min) value = min;
if (value > max) value = max;
return value;
}
}
}
@@ -242,6 +242,7 @@ builder.Services.AddSingleton<RateLimitService>();
builder.Services.AddSingleton<DistressRateLimitService>();
builder.Services.AddSingleton<UsageMetricsService>();
builder.Services.AddSingleton<PlatformMetricsService>();
builder.Services.AddSingleton<AgentEfficiencyMetricsService>();
builder.Services.AddSingleton<SlaMetricsService>();
builder.Services.AddSingleton<SlaBreachEvaluationService>();
builder.Services.AddSingleton<GlobalAdminService>();
@@ -456,6 +456,7 @@ public static class EndpointAccessPolicyCatalog
new("GET", "global/presence", EndpointPolicy.GlobalReadOrAdmin),
new("GET", "global/tenants-with-deletion-manifests", EndpointPolicy.GlobalReadOrAdmin),
new("GET", "global/metrics/platform", EndpointPolicy.GlobalReadOrAdmin),
new("GET", "global/metrics/agent-efficiency", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
new("GET", "global/metrics/app", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
new("GET", "global/metrics/fleet-health", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
new("GET", "global/metrics/time-attribution", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
@@ -0,0 +1,378 @@
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Services
{
/// <summary>
/// Server-side agent-efficiency aggregation, bucketed by agent version: CPU, working set,
/// private bytes, thread/handle counts, spool depth/file size, API latency/request counts
/// and crash rates — as percentiles per version, so review tooling (MCP) gets the answer
/// directly instead of pulling thousands of raw snapshot rows. Raw per-session data only
/// surfaces as the top offenders per dimension.
/// Scan shape: one projected SessionsIndex page + one filtered, projected per-session
/// event query (agent_metrics_snapshot / agent_started / spool_pressure_detected only),
/// bounded to 32 concurrent storage calls. 5-minute per-(days, limit, tenant) cache.
/// </summary>
public class AgentEfficiencyMetricsService
{
private readonly ISessionRepository _sessionRepo;
private readonly ILogger<AgentEfficiencyMetricsService> _logger;
private static readonly Dictionary<(int days, int limit, string tenantKey), (AgentEfficiencyMetricsResponse metrics, DateTime expiry)> _cachedByKey = new();
private static readonly TimeSpan CacheDuration = TimeSpan.FromMinutes(5);
private static readonly object _cacheLock = new object();
private const int DefaultWindowDays = 30;
private const int DefaultSessionLimit = 500;
private const int MaxSessionLimit = 2000;
private const int PerSessionFetchConcurrency = 32;
private const int TopOffendersPerDimension = 3;
// SessionsIndex columns this scan consumes. PartitionKey (TenantId), RowKey, StartedAt and
// SessionId also feed the repo's merge/cursor mechanics; AgentVersion buckets, DeviceName
// labels the offender rows; AvgApiLatencyMs/ApiRequestCount are the mirrored per-session
// network aggregates (SessionIndexFieldManifest) so latency percentiles need no event
// fetch at all. Everything else on the ~40-column mirror maps to defaults via the
// null-safe getters and is never read here. Pinned by
// AgentEfficiencyProjectionEquivalenceTests.
internal static readonly string[] SessionIndexProjection =
{
"PartitionKey", "RowKey", "SessionId", "StartedAt",
"AgentVersion", "DeviceName", "AvgApiLatencyMs", "ApiRequestCount"
};
public AgentEfficiencyMetricsService(
ISessionRepository sessionRepo,
ILogger<AgentEfficiencyMetricsService> logger)
{
_sessionRepo = sessionRepo;
_logger = logger;
}
public async Task<AgentEfficiencyMetricsResponse> ComputeAsync(
int days = DefaultWindowDays,
int limit = DefaultSessionLimit,
string? tenantId = null)
{
days = Math.Clamp(days, 1, 365);
limit = Math.Clamp(limit, 1, MaxSessionLimit);
var key = (days, limit, tenantId ?? string.Empty);
lock (_cacheLock)
{
if (_cachedByKey.TryGetValue(key, out var entry) && DateTime.UtcNow < entry.expiry)
{
// Shallow copy so flipping FromCache never mutates the instance an earlier
// (fresh) caller may still be holding/serializing. The buckets are shared
// read-only — nothing downstream writes into a response.
return entry.metrics.CloneAsCacheHit();
}
}
_logger.LogInformation("Computing agent efficiency metrics for days={Days} limit={Limit} tenant={Tenant}...",
days, limit, tenantId ?? "(all)");
var stopwatch = Stopwatch.StartNew();
var metrics = await ComputeInternalAsync(days, limit, tenantId);
stopwatch.Stop();
metrics.ComputeDurationMs = (int)stopwatch.ElapsedMilliseconds;
metrics.ComputedAt = DateTime.UtcNow;
metrics.FromCache = false;
metrics.WindowDays = days;
metrics.SessionLimit = limit;
metrics.TenantId = tenantId;
_logger.LogInformation("Agent efficiency metrics computed in {Ms}ms (days={Days} limit={Limit} sessionsScanned={Scanned})",
metrics.ComputeDurationMs, days, limit, metrics.SessionsScanned);
lock (_cacheLock)
{
_cachedByKey[key] = (metrics, DateTime.UtcNow.Add(CacheDuration));
}
return metrics;
}
// Per-session intermediate: everything the bucket aggregation needs from one
// storage round-trip. Snapshot-derived values are meaningless when HasSnapshots
// is false — the aggregation only reads them behind that flag.
private sealed record SessionEfficiencyData(
string SessionId,
string TenantId,
string? DeviceName,
string ResolvedVersion,
bool HasSnapshots,
double AvgCpu,
double MaxCpu,
double MaxWorkingSetMb,
double MaxPrivateBytesMb,
double MaxThreads,
double MaxHandles,
double MaxSpoolDepth,
double MaxSpoolFileBytes,
double ApiLatencyMs,
double ApiRequestCount,
bool SpoolPressure,
List<Dictionary<string, object>> AgentStartedEvents);
private async Task<AgentEfficiencyMetricsResponse> ComputeInternalAsync(int days, int limit, string? tenantId)
{
var sessionPage = await _sessionRepo.GetAllSessionsPageAsync(
tenantIdFilter: tenantId, days: days, pageSize: limit, continuation: null,
allowedTenantIds: null, select: SessionIndexProjection);
var allSessions = sessionPage.Items;
if (allSessions.Count == 0)
{
return new AgentEfficiencyMetricsResponse();
}
var perSession = await AgentMetricsAggregation.RunWithBoundedConcurrencyAsync(
allSessions, PerSessionFetchConcurrency, ProcessSessionAsync);
var byVersion = perSession
.GroupBy(s => s.ResolvedVersion, StringComparer.Ordinal)
.Select(g => BuildBucket(g.Key, g.ToList()))
.OrderByDescending(b => b.SessionsScanned)
.ThenBy(b => b.AgentVersion, StringComparer.Ordinal)
.ToList();
return new AgentEfficiencyMetricsResponse
{
SessionsScanned = allSessions.Count,
SessionsWithSnapshots = perSession.Count(s => s.HasSnapshots),
ByVersion = byVersion,
Overall = BuildBucket(null, perSession)
};
}
private async Task<SessionEfficiencyData> ProcessSessionAsync(SessionSummary session)
{
List<Dictionary<string, object>> snapshots = new();
List<Dictionary<string, object>> agentStarts = new();
var spoolPressure = false;
try
{
var events = await _sessionRepo.GetSessionEventsByTypesAsync(
session.TenantId, session.SessionId, AgentMetricsAggregation.MetricsEventTypes,
TableStorageService.AgentMetricsEventProjection);
foreach (var e in events)
{
if (e.EventType == Shared.Constants.EventTypes.AgentMetricsSnapshot && e.Data != null)
snapshots.Add(e.Data);
else if (e.EventType == Shared.Constants.EventTypes.AgentStarted && e.Data != null)
agentStarts.Add(e.Data);
else if (e.EventType == Shared.Constants.EventTypes.SpoolPressureDetected)
spoolPressure = true;
}
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Failed to fetch efficiency events for session {SessionId}", session.SessionId);
}
// Resolve the bucket version: snapshot payload first (authoritative for the run),
// SessionsIndex mirror second, "unknown" last. All of a session's agent_started
// events tally into this one bucket — a mid-session self-update attributes the
// pre-update exits to the final version, an accepted simplification.
var resolvedVersion = snapshots
.Select(s => AgentMetricsAggregation.GetString(s, "agent_version"))
.FirstOrDefault(v => !string.IsNullOrEmpty(v));
if (string.IsNullOrEmpty(resolvedVersion))
resolvedVersion = string.IsNullOrEmpty(session.AgentVersion) ? "unknown" : session.AgentVersion;
double avgCpu = 0, maxCpu = 0, maxWs = 0, maxPb = 0, maxThreads = 0, maxHandles = 0, maxSpool = 0, maxSpoolFile = 0;
double apiLatency = session.AvgApiLatencyMs ?? 0;
double apiRequests = session.ApiRequestCount ?? 0;
if (snapshots.Count > 0)
{
var cpuValues = snapshots.Select(s => AgentMetricsAggregation.GetDouble(s, "agent_cpu_percent")).ToList();
avgCpu = cpuValues.Average();
maxCpu = cpuValues.Max();
maxWs = snapshots.Max(s => AgentMetricsAggregation.GetDouble(s, "agent_working_set_mb"));
maxPb = snapshots.Max(s => AgentMetricsAggregation.GetDouble(s, "agent_private_bytes_mb"));
maxThreads = snapshots.Max(s => AgentMetricsAggregation.GetDouble(s, "agent_thread_count"));
maxHandles = snapshots.Max(s => AgentMetricsAggregation.GetDouble(s, "agent_handle_count"));
// V2 emits `spool_pending_item_count`; V1 emitted `spool_queue_depth`.
maxSpool = snapshots.Max(s => AgentMetricsAggregation.GetDoubleFirst(s, "spool_pending_item_count", "spool_queue_depth"));
maxSpoolFile = snapshots.Max(s => AgentMetricsAggregation.GetDouble(s, "spool_file_size_bytes"));
// The SessionsIndex mirror carries the exact session average for agents that
// emit the cumulative counters; fall back to computing it from the last
// snapshot for sessions whose mirror write never happened.
if (apiLatency <= 0)
{
var last = snapshots[snapshots.Count - 1];
var totalLatencyMs = AgentMetricsAggregation.GetDouble(last, "net_total_latency_ms");
var totalRequests = AgentMetricsAggregation.GetDouble(last, "net_total_requests");
if (totalLatencyMs > 0 && totalRequests > 0)
apiLatency = totalLatencyMs / totalRequests;
if (apiRequests <= 0)
apiRequests = totalRequests;
}
}
return new SessionEfficiencyData(
session.SessionId, session.TenantId, session.DeviceName, resolvedVersion!,
HasSnapshots: snapshots.Count > 0,
avgCpu, maxCpu, maxWs, maxPb, maxThreads, maxHandles, maxSpool, maxSpoolFile,
apiLatency, apiRequests, spoolPressure, agentStarts);
}
// agentVersion == null builds the cross-version "overall" bucket (version omitted on the wire).
private static AgentVersionEfficiency BuildBucket(string? agentVersion, List<SessionEfficiencyData> sessions)
{
var withSnapshots = sessions.Where(s => s.HasSnapshots).ToList();
var bucket = new AgentVersionEfficiency
{
AgentVersion = agentVersion,
SessionsScanned = sessions.Count,
SessionsWithSnapshots = withSnapshots.Count,
SpoolPressureSessions = sessions.Count(s => s.SpoolPressure),
AvgCpuPercent = BuildStats(withSnapshots.Select(s => s.AvgCpu)),
MaxCpuPercent = BuildStats(withSnapshots.Select(s => s.MaxCpu)),
MaxWorkingSetMb = BuildStats(withSnapshots.Select(s => s.MaxWorkingSetMb)),
MaxPrivateBytesMb = BuildStats(withSnapshots.Select(s => s.MaxPrivateBytesMb)),
// Thread/handle counts only exist on agents that emit them (V2); zero-filtered so
// legacy sessions don't drag the percentiles to 0.
MaxThreadCount = BuildStats(withSnapshots.Select(s => s.MaxThreads).Where(v => v > 0)),
MaxHandleCount = BuildStats(withSnapshots.Select(s => s.MaxHandles).Where(v => v > 0)),
MaxSpoolDepth = BuildStats(withSnapshots.Select(s => s.MaxSpoolDepth)),
MaxSpoolFileBytes = BuildStats(withSnapshots.Select(s => s.MaxSpoolFileBytes)),
// Latency/requests come from the SessionsIndex mirror, so sessions without
// snapshots still contribute; zero means "no data", not "0 ms".
ApiLatencyMs = BuildStats(sessions.Select(s => s.ApiLatencyMs).Where(v => v > 0)),
ApiRequestCount = BuildStats(sessions.Select(s => s.ApiRequestCount).Where(v => v > 0)),
CrashRate = AgentMetricsAggregation.AggregateCrashRate(sessions.SelectMany(s => s.AgentStartedEvents))
};
var offenders = new List<EfficiencyOffender>();
AddTopOffenders(offenders, withSnapshots, "maxCpuPercent", s => s.MaxCpu);
AddTopOffenders(offenders, withSnapshots, "maxWorkingSetMb", s => s.MaxWorkingSetMb);
AddTopOffenders(offenders, withSnapshots, "maxHandleCount", s => s.MaxHandles);
bucket.TopOffenders = offenders.Count > 0 ? offenders : null;
return bucket;
}
private static void AddTopOffenders(
List<EfficiencyOffender> target,
List<SessionEfficiencyData> sessions,
string dimension,
Func<SessionEfficiencyData, double> value)
{
target.AddRange(sessions
.Where(s => value(s) > 0)
.OrderByDescending(value)
.Take(TopOffendersPerDimension)
.Select(s => new EfficiencyOffender
{
SessionId = s.SessionId,
TenantId = s.TenantId,
DeviceName = string.IsNullOrEmpty(s.DeviceName) ? null : s.DeviceName,
Dimension = dimension,
Value = Math.Round(value(s), 1)
}));
}
private static PercentileStats? BuildStats(IEnumerable<double> values)
{
var sorted = values.OrderBy(v => v).ToList();
if (sorted.Count == 0) return null;
return new PercentileStats
{
P50 = MetricsMath.Percentile(sorted, 50),
P95 = MetricsMath.Percentile(sorted, 95),
Max = Math.Round(sorted[sorted.Count - 1], 1),
Avg = Math.Round(sorted.Average(), 1),
SampleCount = sorted.Count
};
}
}
// ── Response DTOs ────────────────────────────────────────────────────────────
public class AgentEfficiencyMetricsResponse
{
public int WindowDays { get; set; }
public int SessionLimit { get; set; }
/// <summary>Echo of the requested tenant filter; null = cross-tenant aggregate.</summary>
public string? TenantId { get; set; }
/// <summary>Sessions the scan covered — compare against <see cref="SessionLimit"/> for truncation.</summary>
public int SessionsScanned { get; set; }
public int SessionsWithSnapshots { get; set; }
public List<AgentVersionEfficiency> ByVersion { get; set; } = new();
public AgentVersionEfficiency? Overall { get; set; }
public DateTime ComputedAt { get; set; }
public int ComputeDurationMs { get; set; }
public bool FromCache { get; set; }
/// <summary>Shallow copy flagged as a cache hit — see the cache-read path.</summary>
internal AgentEfficiencyMetricsResponse CloneAsCacheHit() => new()
{
WindowDays = WindowDays,
SessionLimit = SessionLimit,
TenantId = TenantId,
SessionsScanned = SessionsScanned,
SessionsWithSnapshots = SessionsWithSnapshots,
ByVersion = ByVersion,
Overall = Overall,
ComputedAt = ComputedAt,
ComputeDurationMs = ComputeDurationMs,
FromCache = true
};
}
public class AgentVersionEfficiency
{
/// <summary>Null on the cross-version "overall" bucket (omitted on the wire).</summary>
public string? AgentVersion { get; set; }
public int SessionsScanned { get; set; }
public int SessionsWithSnapshots { get; set; }
public int SpoolPressureSessions { get; set; }
public PercentileStats? AvgCpuPercent { get; set; }
public PercentileStats? MaxCpuPercent { get; set; }
public PercentileStats? MaxWorkingSetMb { get; set; }
public PercentileStats? MaxPrivateBytesMb { get; set; }
public PercentileStats? MaxThreadCount { get; set; }
public PercentileStats? MaxHandleCount { get; set; }
public PercentileStats? MaxSpoolDepth { get; set; }
public PercentileStats? MaxSpoolFileBytes { get; set; }
public PercentileStats? ApiLatencyMs { get; set; }
public PercentileStats? ApiRequestCount { get; set; }
public CrashRateMetrics? CrashRate { get; set; }
public List<EfficiencyOffender>? TopOffenders { get; set; }
}
/// <summary>Distribution of a per-session statistic across a version bucket.</summary>
public class PercentileStats
{
public double P50 { get; set; }
public double P95 { get; set; }
public double Max { get; set; }
public double Avg { get; set; }
public int SampleCount { get; set; }
}
public class EfficiencyOffender
{
public string SessionId { get; set; } = string.Empty;
public string TenantId { get; set; } = string.Empty;
public string? DeviceName { get; set; }
public string Dimension { get; set; } = string.Empty;
public double Value { get; set; }
}
}
@@ -0,0 +1,145 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
namespace AutopilotMonitor.Functions.Services
{
/// <summary>
/// Shared primitives for the agent-metrics aggregation services
/// (<see cref="PlatformMetricsService"/>, <see cref="AgentEfficiencyMetricsService"/>):
/// crash-rate tally over <c>agent_started</c> payloads, bounded fan-out, and the
/// loosely-typed payload getters. One home so the two services cannot drift on the
/// exit-type classification or the numeric coercion rules.
/// </summary>
internal static class AgentMetricsAggregation
{
/// <summary>
/// Event types the metrics aggregations consume. Everything else in a session's
/// Events partition is dead weight for these endpoints — pairs with
/// <see cref="TableStorageService.AgentMetricsEventProjection"/> for a filtered,
/// projected per-session fetch instead of a full-partition drain.
/// </summary>
internal static readonly string[] MetricsEventTypes =
{
Shared.Constants.EventTypes.AgentMetricsSnapshot,
Shared.Constants.EventTypes.AgentStarted,
Shared.Constants.EventTypes.SpoolPressureDetected
};
/// <summary>
/// Tallies crash classifications over <c>agent_started</c> event payloads.
/// Unknown/absent <c>previousExitType</c> counts as a first run (the agent only
/// omits it on its very first boot of a session).
/// </summary>
internal static CrashRateMetrics AggregateCrashRate(IEnumerable<Dictionary<string, object>> agentStartedEvents)
{
var metrics = new CrashRateMetrics();
var exceptionCounts = new Dictionary<string, int>(StringComparer.OrdinalIgnoreCase);
foreach (var data in agentStartedEvents)
{
metrics.TotalStarts++;
var exitType = GetString(data, "previousExitType");
switch (exitType)
{
case "clean":
metrics.CleanExits++;
break;
case "exception_crash":
metrics.ExceptionCrashes++;
var exType = GetString(data, "previousCrashException");
if (!string.IsNullOrEmpty(exType))
{
exceptionCounts.TryGetValue(exType, out var count);
exceptionCounts[exType] = count + 1;
}
break;
case "hard_kill":
metrics.HardKills++;
break;
case "reboot_kill":
metrics.RebootKills++;
break;
default:
metrics.FirstRuns++;
break;
}
}
var nonFirstRuns = metrics.TotalStarts - metrics.FirstRuns;
metrics.CrashRatePercent = nonFirstRuns > 0
? Math.Round((double)metrics.ExceptionCrashes / nonFirstRuns * 100, 1)
: 0;
metrics.TopExceptions = exceptionCounts
.OrderByDescending(kv => kv.Value)
.Take(5)
.Select(kv => new CrashExceptionSummary { ExceptionType = kv.Key, Count = kv.Value })
.ToList();
return metrics;
}
// Bounded fan-out: equivalent to Task.WhenAll over `body(item)` but with
// at most `maxConcurrency` tasks in flight. Without this guard a 1000-
// session metric query would fire 1000 simultaneous storage requests
// and either throttle Azure Tables or run the worker out of file
// handles before responding.
internal static async Task<List<TResult>> RunWithBoundedConcurrencyAsync<TInput, TResult>(
IReadOnlyList<TInput> items,
int maxConcurrency,
Func<TInput, Task<TResult>> body)
{
using var sem = new SemaphoreSlim(maxConcurrency);
var tasks = items.Select(async item =>
{
await sem.WaitAsync().ConfigureAwait(false);
try
{
return await body(item).ConfigureAwait(false);
}
finally
{
sem.Release();
}
});
var results = await Task.WhenAll(tasks).ConfigureAwait(false);
return results.ToList();
}
internal static double GetDouble(Dictionary<string, object> data, string key)
{
if (data.TryGetValue(key, out var value))
{
if (value is double d) return d;
if (value is int i) return i;
if (value is long l) return l;
if (value is float f) return f;
if (double.TryParse(value?.ToString(), out var parsed)) return parsed;
}
return 0;
}
// Returns the value of the first key that is present in `data`. Lets us read a V2 field
// name with a V1 fallback without conflating "key missing" with "key present but zero".
internal static double GetDoubleFirst(Dictionary<string, object> data, params string[] keys)
{
foreach (var key in keys)
{
if (data.ContainsKey(key)) return GetDouble(data, key);
}
return 0;
}
internal static string GetString(Dictionary<string, object> data, string key)
{
if (data.TryGetValue(key, out var value))
{
return value?.ToString() ?? string.Empty;
}
return string.Empty;
}
}
}
@@ -184,52 +184,109 @@ namespace AutopilotMonitor.Functions.Services
: evt.Data?.ContainsKey("bytes_downloaded") == true ? "bytes_downloaded" : null;
if (bytesKey != null && long.TryParse(evt.Data![bytesKey]?.ToString(), out var bytes))
summary.DownloadBytes = Math.Max(summary.DownloadBytes, bytes);
// DO fallback: IME >= 1.104 removed the [DO TEL] log line, and the agent's
// DO collector only emits do_telemetry for downloads whose completion a poll
// itself observed — fast, interrupted or between-poll-completed downloads
// never get one. The progress events carry the full do* set on every poll,
// so fold them in monotonically; a do_telemetry (below) stays authoritative.
if (evt.Data != null)
ApplyDoFields(summary, evt.Data, isAuthoritative: false);
break;
case "do_telemetry":
if (evt.Data != null)
{
if (evt.Data.ContainsKey("doFileSize") && long.TryParse(evt.Data["doFileSize"]?.ToString(), out var doFs))
{
summary.DownloadBytes = Math.Max(summary.DownloadBytes, doFs);
summary.DoFileSize = doFs;
}
if (evt.Data.ContainsKey("doTotalBytesDownloaded") && long.TryParse(evt.Data["doTotalBytesDownloaded"]?.ToString(), out var doTotalDl))
summary.DoTotalBytesDownloaded = doTotalDl;
if (evt.Data.ContainsKey("doBytesFromPeers") && long.TryParse(evt.Data["doBytesFromPeers"]?.ToString(), out var doPeers))
summary.DoBytesFromPeers = doPeers;
if (evt.Data.ContainsKey("doBytesFromHttp") && long.TryParse(evt.Data["doBytesFromHttp"]?.ToString(), out var doHttp))
summary.DoBytesFromHttp = doHttp;
if (evt.Data.ContainsKey("doPercentPeerCaching") && int.TryParse(evt.Data["doPercentPeerCaching"]?.ToString(), out var doPct))
summary.DoPercentPeerCaching = doPct;
if (evt.Data.ContainsKey("doDownloadMode") && int.TryParse(evt.Data["doDownloadMode"]?.ToString(), out var doMode))
summary.DoDownloadMode = doMode;
if (evt.Data.ContainsKey("doDownloadDuration"))
{
var doDurStr = evt.Data["doDownloadDuration"]?.ToString() ?? string.Empty;
summary.DoDownloadDuration = doDurStr;
if (TimeSpan.TryParse(doDurStr, out var doDurTs) && doDurTs.TotalSeconds >= 1)
summary.DownloadDurationSeconds = Math.Max(summary.DownloadDurationSeconds, (int)doDurTs.TotalSeconds);
}
if (evt.Data.ContainsKey("doBytesFromLanPeers") && long.TryParse(evt.Data["doBytesFromLanPeers"]?.ToString(), out var doLan))
summary.DoBytesFromLanPeers = doLan;
if (evt.Data.ContainsKey("doBytesFromGroupPeers") && long.TryParse(evt.Data["doBytesFromGroupPeers"]?.ToString(), out var doGroup))
summary.DoBytesFromGroupPeers = doGroup;
if (evt.Data.ContainsKey("doBytesFromInternetPeers") && long.TryParse(evt.Data["doBytesFromInternetPeers"]?.ToString(), out var doInet))
summary.DoBytesFromInternetPeers = doInet;
if (evt.Data.ContainsKey("doBytesFromLinkLocalPeers") && long.TryParse(evt.Data["doBytesFromLinkLocalPeers"]?.ToString(), out var doLinkLocal))
summary.DoBytesFromLinkLocalPeers = doLinkLocal;
if (evt.Data.ContainsKey("doBytesFromCacheServer") && long.TryParse(evt.Data["doBytesFromCacheServer"]?.ToString(), out var doCache))
summary.DoBytesFromCacheServer = doCache;
if (evt.Data.ContainsKey("doCacheHost"))
summary.DoCacheHost = evt.Data["doCacheHost"]?.ToString() ?? string.Empty;
}
ApplyDoFields(summary, evt.Data, isAuthoritative: true);
break;
}
RecalculateAppDurations(state);
}
/// <summary>
/// Folds the <c>do*</c> Delivery Optimization fields of one event into the summary.
/// Two writers share this so they cannot drift:
/// <list type="bullet">
/// <item><c>do_telemetry</c> (<paramref name="isAuthoritative"/> = true) — the collector's
/// final per-app read: last-write-wins per present key (pre-existing semantics), except
/// <c>DoDownloadMode</c> never regresses to the -1 "unset" sentinel — that would re-hide
/// the row from <c>DoAggregator</c> after a progress event already established a mode.</item>
/// <item><c>download_progress</c> (false) — per-poll observations: byte counters and file
/// size fold via <c>Math.Max</c> (monotonic per download, so replays and out-of-order
/// batches stay idempotent); mode/percent/cacheHost are latest-observation writes.</item>
/// </list>
/// <c>DownloadBytes</c> prefers actually-transferred bytes (<c>doTotalBytesDownloaded</c>)
/// over <c>doFileSize</c> — the file size is only the fallback when no transfer total exists
/// (it used to unconditionally inflate the transfer measure).
/// </summary>
internal static void ApplyDoFields(AppInstallSummary summary, Dictionary<string, object> data, bool isAuthoritative)
{
static bool TryGetLong(Dictionary<string, object> d, string key, out long value)
{
value = 0;
return d.TryGetValue(key, out var raw) && long.TryParse(raw?.ToString(), out value);
}
static bool TryGetInt(Dictionary<string, object> d, string key, out int value)
{
value = 0;
return d.TryGetValue(key, out var raw) && int.TryParse(raw?.ToString(), out value);
}
static long Fold(bool authoritative, long current, long incoming)
=> authoritative ? incoming : Math.Max(current, incoming);
var hasFileSize = TryGetLong(data, "doFileSize", out var doFs);
var hasTotalDl = TryGetLong(data, "doTotalBytesDownloaded", out var doTotalDl);
if (hasFileSize)
summary.DoFileSize = Fold(isAuthoritative, summary.DoFileSize, doFs);
if (hasTotalDl)
summary.DoTotalBytesDownloaded = Fold(isAuthoritative, summary.DoTotalBytesDownloaded, doTotalDl);
// Transfer measure: real transferred bytes when known, file size only as fallback.
// Progress events already fed DownloadBytes via their bytesDownloaded field, so only
// the authoritative telemetry contributes here.
if (isAuthoritative && (hasFileSize || hasTotalDl))
summary.DownloadBytes = Math.Max(summary.DownloadBytes, hasTotalDl && doTotalDl > 0 ? doTotalDl : doFs);
if (TryGetLong(data, "doBytesFromPeers", out var doPeers))
summary.DoBytesFromPeers = Fold(isAuthoritative, summary.DoBytesFromPeers, doPeers);
if (TryGetLong(data, "doBytesFromHttp", out var doHttp))
summary.DoBytesFromHttp = Fold(isAuthoritative, summary.DoBytesFromHttp, doHttp);
if (TryGetLong(data, "doBytesFromLanPeers", out var doLan))
summary.DoBytesFromLanPeers = Fold(isAuthoritative, summary.DoBytesFromLanPeers, doLan);
if (TryGetLong(data, "doBytesFromGroupPeers", out var doGroup))
summary.DoBytesFromGroupPeers = Fold(isAuthoritative, summary.DoBytesFromGroupPeers, doGroup);
if (TryGetLong(data, "doBytesFromInternetPeers", out var doInet))
summary.DoBytesFromInternetPeers = Fold(isAuthoritative, summary.DoBytesFromInternetPeers, doInet);
if (TryGetLong(data, "doBytesFromLinkLocalPeers", out var doLinkLocal))
summary.DoBytesFromLinkLocalPeers = Fold(isAuthoritative, summary.DoBytesFromLinkLocalPeers, doLinkLocal);
if (TryGetLong(data, "doBytesFromCacheServer", out var doCache))
summary.DoBytesFromCacheServer = Fold(isAuthoritative, summary.DoBytesFromCacheServer, doCache);
if (TryGetInt(data, "doPercentPeerCaching", out var doPct))
summary.DoPercentPeerCaching = doPct;
// -1 is the "unset" sentinel DoAggregator filters on — never let it clobber a
// known mode (a telemetry event without the DownloadMode property parses as -1).
if (TryGetInt(data, "doDownloadMode", out var doMode) && doMode >= 0)
summary.DoDownloadMode = doMode;
if (data.TryGetValue("doCacheHost", out var cacheHostRaw))
{
var cacheHost = cacheHostRaw?.ToString();
if (isAuthoritative || !string.IsNullOrEmpty(cacheHost))
summary.DoCacheHost = cacheHost ?? string.Empty;
}
// Only do_telemetry carries a duration; progress events never have the key.
if (data.ContainsKey("doDownloadDuration"))
{
var doDurStr = data["doDownloadDuration"]?.ToString() ?? string.Empty;
summary.DoDownloadDuration = doDurStr;
if (TimeSpan.TryParse(doDurStr, out var doDurTs) && doDurTs.TotalSeconds >= 1)
summary.DownloadDurationSeconds = Math.Max(summary.DownloadDurationSeconds, (int)doDurTs.TotalSeconds);
}
}
internal static void RecalculateAppDurations(AppInstallAggregationState state)
{
var summary = state.Summary;
@@ -115,9 +115,9 @@ namespace AutopilotMonitor.Functions.Services
private async Task<PlatformAgentMetricsResponse> ComputePlatformMetricsInternalAsync(int days, int limit)
{
// Fetch only the newest `limit` sessions in the window. Each session below
// triggers its own GetSessionEventsAsync call; the per-session work is
// bounded by PerSessionFetchConcurrency so even 1000-session limits
// don't fan out 1000 simultaneous storage requests.
// triggers its own event fetch; the per-session work is bounded by
// PerSessionFetchConcurrency so even 1000-session limits don't fan out
// 1000 simultaneous storage requests.
var sessionPage = await _sessionRepo.GetAllSessionsPageAsync(
tenantIdFilter: null, days: days, pageSize: limit, continuation: null);
var allSessions = sessionPage.Items;
@@ -135,7 +135,7 @@ namespace AutopilotMonitor.Functions.Services
.Select(s => s.SessionId)
.ToHashSet(StringComparer.Ordinal);
var perSessionResults = await RunWithBoundedConcurrencyAsync(
var perSessionResults = await AgentMetricsAggregation.RunWithBoundedConcurrencyAsync(
allSessions,
PerSessionFetchConcurrency,
session => ProcessSessionAsync(session, latencySampleSessionIds));
@@ -146,13 +146,18 @@ namespace AutopilotMonitor.Functions.Services
.ToList();
var deliveryLatency = AggregateDeliveryLatency(perSessionResults);
var crashRate = AggregateCrashRate(perSessionResults);
var crashRate = AgentMetricsAggregation.AggregateCrashRate(
perSessionResults.SelectMany(r => r.AgentStartedEvents));
return new PlatformAgentMetricsResponse
{
Sessions = sessionMetrics,
DeliveryLatency = deliveryLatency,
CrashRate = crashRate
CrashRate = crashRate,
// Sessions actually scanned in the window — the honest basis for the
// caller's truncation check. Sessions.Count only counts sessions WITH
// snapshots, so comparing it against the limit understates truncation.
SessionsScanned = allSessions.Count
};
}
@@ -162,11 +167,20 @@ namespace AutopilotMonitor.Functions.Services
{
try
{
var events = await _sessionRepo.GetSessionEventsAsync(session.TenantId, session.SessionId);
// Latency-sample sessions need the FULL event stream: the delivery-latency
// deltas are computed over every event of those 20 sessions, not just the
// metrics types — filtering them would shrink the sample ~50x and change the
// published percentiles. Everyone else gets the three-type projected fetch.
var isLatencySample = latencySampleSessionIds.Contains(session.SessionId);
var events = isLatencySample
? await _sessionRepo.GetSessionEventsAsync(session.TenantId, session.SessionId)
: await _sessionRepo.GetSessionEventsByTypesAsync(
session.TenantId, session.SessionId, AgentMetricsAggregation.MetricsEventTypes,
TableStorageService.AgentMetricsEventProjection);
// ── Latency deltas (only for sample-set sessions) ───────────
List<double> latencyDeltas = new();
if (latencySampleSessionIds.Contains(session.SessionId))
if (isLatencySample)
{
foreach (var e in events)
{
@@ -266,33 +280,6 @@ namespace AutopilotMonitor.Functions.Services
}
}
// Bounded fan-out: equivalent to Task.WhenAll over `body(item)` but with
// at most `maxConcurrency` tasks in flight. Without this guard a 1000-
// session metric query would fire 1000 simultaneous storage requests
// and either throttle Azure Tables or run the worker out of file
// handles before responding.
private static async Task<List<TResult>> RunWithBoundedConcurrencyAsync<TInput, TResult>(
IReadOnlyList<TInput> items,
int maxConcurrency,
Func<TInput, Task<TResult>> body)
{
using var sem = new SemaphoreSlim(maxConcurrency);
var tasks = items.Select(async item =>
{
await sem.WaitAsync().ConfigureAwait(false);
try
{
return await body(item).ConfigureAwait(false);
}
finally
{
sem.Release();
}
});
var results = await Task.WhenAll(tasks).ConfigureAwait(false);
return results.ToList();
}
// Aggregate over the latency deltas already extracted by the per-session
// pass. No additional storage round-trips — the same events the metric
// pass walked have been pre-binned into PerSessionData.LatencyDeltasMs
@@ -325,61 +312,6 @@ namespace AutopilotMonitor.Functions.Services
};
}
// Aggregate crash classifications over the agent_started events the
// per-session pass already collected — same idea as the latency
// aggregator: zero extra storage calls.
private static CrashRateMetrics AggregateCrashRate(IReadOnlyList<PerSessionData> perSession)
{
var metrics = new CrashRateMetrics();
var exceptionCounts = new Dictionary<string, int>(StringComparer.OrdinalIgnoreCase);
foreach (var r in perSession)
{
foreach (var data in r.AgentStartedEvents)
{
metrics.TotalStarts++;
var exitType = GetString(data, "previousExitType");
switch (exitType)
{
case "clean":
metrics.CleanExits++;
break;
case "exception_crash":
metrics.ExceptionCrashes++;
var exType = GetString(data, "previousCrashException");
if (!string.IsNullOrEmpty(exType))
{
exceptionCounts.TryGetValue(exType, out var count);
exceptionCounts[exType] = count + 1;
}
break;
case "hard_kill":
metrics.HardKills++;
break;
case "reboot_kill":
metrics.RebootKills++;
break;
default:
metrics.FirstRuns++;
break;
}
}
}
var nonFirstRuns = metrics.TotalStarts - metrics.FirstRuns;
metrics.CrashRatePercent = nonFirstRuns > 0
? Math.Round((double)metrics.ExceptionCrashes / nonFirstRuns * 100, 1)
: 0;
metrics.TopExceptions = exceptionCounts
.OrderByDescending(kv => kv.Value)
.Take(5)
.Select(kv => new CrashExceptionSummary { ExceptionType = kv.Key, Count = kv.Value })
.ToList();
return metrics;
}
private static double Percentile(List<double> sortedValues, double percentile)
{
if (sortedValues.Count == 0) return 0;
@@ -387,38 +319,16 @@ namespace AutopilotMonitor.Functions.Services
return sortedValues[Math.Max(0, Math.Min(index, sortedValues.Count - 1))];
}
// Payload getters live in AgentMetricsAggregation (shared with the efficiency
// service); these thin wrappers keep the call sites above unchanged.
private static double GetDouble(Dictionary<string, object> data, string key)
{
if (data.TryGetValue(key, out var value))
{
if (value is double d) return d;
if (value is int i) return i;
if (value is long l) return l;
if (value is float f) return f;
if (double.TryParse(value?.ToString(), out var parsed)) return parsed;
}
return 0;
}
=> AgentMetricsAggregation.GetDouble(data, key);
// Returns the value of the first key that is present in `data`. Lets us read a V2 field
// name with a V1 fallback without conflating "key missing" with "key present but zero".
private static double GetDoubleFirst(Dictionary<string, object> data, params string[] keys)
{
foreach (var key in keys)
{
if (data.ContainsKey(key)) return GetDouble(data, key);
}
return 0;
}
=> AgentMetricsAggregation.GetDoubleFirst(data, keys);
private static string GetString(Dictionary<string, object> data, string key)
{
if (data.TryGetValue(key, out var value))
{
return value?.ToString() ?? string.Empty;
}
return string.Empty;
}
=> AgentMetricsAggregation.GetString(data, key);
}
// ── Response DTOs ────────────────────────────────────────────────────────────
@@ -433,6 +343,14 @@ namespace AutopilotMonitor.Functions.Services
public bool FromCache { get; set; }
public int WindowDays { get; set; }
public int SessionLimit { get; set; }
/// <summary>
/// Sessions the scan actually covered in the window (before the has-snapshots filter
/// that shapes <see cref="Sessions"/>). Callers must compare THIS against
/// <see cref="SessionLimit"/> to decide whether the window was truncated —
/// <c>Sessions.Count</c> understates truncation on fleets where many sessions
/// emit no agent_metrics_snapshot.
/// </summary>
public int SessionsScanned { get; set; }
}
public class SessionAgentMetric
@@ -1099,7 +1099,8 @@ namespace AutopilotMonitor.Functions.Services
public async Task<RawPage<SessionSummary>> GetAllSessionsPageAsync(
string? tenantIdFilter, int? days, int pageSize, string? continuation,
IReadOnlyCollection<string>? allowedTenantIds = null)
IReadOnlyCollection<string>? allowedTenantIds = null,
IEnumerable<string>? select = null)
{
if (pageSize < 1) throw new ArgumentOutOfRangeException(nameof(pageSize));
@@ -1111,10 +1112,11 @@ namespace AutopilotMonitor.Functions.Services
// the per-tenant SessionsIndex scan — natively-ordered, cheaper.
if (!string.IsNullOrEmpty(tenantIdFilter))
{
return await GetSessionsPageAsync(tenantIdFilter!, days, pageSize, continuation);
var tenantPage = await FetchSessionsPageInternalAsync(tenantIdFilter!, pageSize, continuation, days, select);
return new RawPage<SessionSummary>(tenantPage.Sessions, tenantPage.HasMore ? tenantPage.NextCursor : null);
}
var page = await FetchAllSessionsPageInternalAsync(maxResults: pageSize, cursor: continuation, days: days, allowedTenantIds: allowedTenantIds);
var page = await FetchAllSessionsPageInternalAsync(maxResults: pageSize, cursor: continuation, days: days, allowedTenantIds: allowedTenantIds, select: select);
return new RawPage<SessionSummary>(page.Sessions, page.HasMore ? page.NextCursor : null);
}
@@ -1317,7 +1319,8 @@ namespace AutopilotMonitor.Functions.Services
/// <see cref="GetAllSessionsPageAsync"/> (single page).
/// </summary>
private async Task<(List<SessionSummary> Sessions, bool HasMore, string? NextCursor)> FetchAllSessionsPageInternalAsync(
int maxResults, string? cursor, int? days, IReadOnlyCollection<string>? allowedTenantIds = null)
int maxResults, string? cursor, int? days, IReadOnlyCollection<string>? allowedTenantIds = null,
IEnumerable<string>? select = null)
{
try
{
@@ -1393,7 +1396,7 @@ namespace AutopilotMonitor.Functions.Services
var tenantSessions = new List<SessionSummary>();
await foreach (var entity in indexTableClient.QueryAsync<TableEntity>(
filter: filter, maxPerPage: Math.Min(fetchPerTenant, 1000)))
filter: filter, maxPerPage: Math.Min(fetchPerTenant, 1000), select: select))
{
tenantSessions.Add(MapIndexEntityToSessionSummary(entity));
if (tenantSessions.Count >= fetchPerTenant) break;
@@ -2612,6 +2615,70 @@ namespace AutopilotMonitor.Functions.Services
}
}
// Columns the metrics aggregation paths (PlatformMetricsService, AgentEfficiencyMetricsService)
// need from an event row: EventType routes the event to its aggregation, DataJson carries the
// snapshot payload, Sequence pins the authoritative sort, and the timestamp trio feeds
// ResolveEventTimestamp (OccurredUtc column → RowKey decode → system Timestamp) plus the
// ReceivedAt side of the delivery-latency delta. Severity/Phase/Message/Source are never read
// by the metrics math and map to defaults via MapToEnrollmentEvent's null-safe getters.
internal static readonly string[] AgentMetricsEventProjection =
{ "PartitionKey", "RowKey", "EventType", "DataJson", "Sequence", "ReceivedAt", BusinessTimestamp.OccurredUtcColumn, "Timestamp" };
/// <summary>
/// Builds the OData filter for <see cref="GetSessionEventsByTypesAsync"/>: partition scope
/// plus an or-chain over the event types. Internal so the composition (escaping, grouping)
/// is unit-testable without a live table.
/// </summary>
internal static string BuildEventTypesFilter(string partitionKey, IReadOnlyCollection<string> eventTypes)
{
var typeClauses = string.Join(" or ", eventTypes.Select(t => $"EventType eq '{ODataSanitizer.EscapeValue(t)}'"));
return $"PartitionKey eq '{ODataSanitizer.EscapeValue(partitionKey)}' and ({typeClauses})";
}
/// <summary>
/// Multi-type sibling of <see cref="GetSessionEventsByTypeAsync"/>: one server-side query
/// matching ANY of <paramref name="eventTypes"/>, with an optional column projection.
/// <paramref name="maxResults"/> is a hard cap — the drain stops once reached (the
/// single-type variant only uses it as the storage page size and drains every page).
/// Fail-soft like the other event readers: storage errors log and return an empty list.
/// </summary>
public async Task<List<EnrollmentEvent>> GetSessionEventsByTypesAsync(
string tenantId, string sessionId, IReadOnlyCollection<string> eventTypes,
IEnumerable<string>? select = null, int maxResults = 2000)
{
SecurityValidator.EnsureValidGuid(tenantId, nameof(tenantId));
SecurityValidator.EnsureValidGuid(sessionId, nameof(sessionId));
if (eventTypes == null || eventTypes.Count == 0)
return new List<EnrollmentEvent>();
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.Events);
var events = new List<EnrollmentEvent>();
var filter = BuildEventTypesFilter($"{tenantId}_{sessionId}", eventTypes);
var query = tableClient.QueryAsync<TableEntity>(
filter: filter,
maxPerPage: Math.Min(maxResults, 1000),
select: select
);
await foreach (var entity in query)
{
events.Add(MapToEnrollmentEvent(entity));
if (events.Count >= maxResults) break;
}
return events.OrderBy(e => e.Sequence).ToList();
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to get events by types for session {SessionId}", sessionId);
return new List<EnrollmentEvent>();
}
}
/// <summary>
/// Literal-row variant of <see cref="GetSessionEventsPageAsync"/> backing the single-session
/// path of <c>/api/raw/events</c>. Emits the raw <c>Events</c> rows verbatim (DataJson as the
@@ -59,12 +59,27 @@ describe('platformWindowEcho', () => {
expect(echo.sessionLimit).toBe(2000);
});
it('flags truncated when the analyzed count reaches the cap', () => {
const echo = platformWindowEcho({ sessionLimit: 100 }, { days: 30, limit: 100 }, 100);
it('decides truncated on sessionsScanned when the backend sends it', () => {
// The formerly-dishonest case: the cap WAS hit (2000 scanned) but only 1400
// sessions carried snapshots — the analyzed-count comparison read false.
const echo = platformWindowEcho({ sessionLimit: 2000, sessionsScanned: 2000 }, { days: 30, limit: 2000 }, 1400);
expect(echo.truncated).toBe(true);
expect(echo.sessionsScanned).toBe(2000);
});
it('does not flag truncated when fewer sessions than the cap were analyzed', () => {
it('does not flag truncated when the scan stayed under the cap, even if every session analyzed', () => {
const echo = platformWindowEcho({ sessionLimit: 100, sessionsScanned: 80 }, { days: 30, limit: 100 }, 80);
expect(echo.truncated).toBe(false);
expect(echo.sessionsScanned).toBe(80);
});
it('legacy fallback: flags truncated on the analyzed count when sessionsScanned is absent (old backend)', () => {
const echo = platformWindowEcho({ sessionLimit: 100 }, { days: 30, limit: 100 }, 100);
expect(echo.truncated).toBe(true);
expect(echo.sessionsScanned).toBeUndefined();
});
it('legacy fallback: does not flag truncated when fewer sessions than the cap were analyzed', () => {
const echo = platformWindowEcho({ sessionLimit: 100 }, { days: 30, limit: 100 }, 42);
expect(echo.truncated).toBe(false);
});
@@ -70,14 +70,37 @@ suite('get_geographic_sessions', () => {
});
suite('get_platform_metrics', () => {
it('should return agent performance metrics', async () => {
it('should return per-session agent metrics with the honest scan count', async () => {
const data = await apiFetch<any>(`/api/global/metrics/platform`);
expect(data).toHaveProperty('sessionsAnalyzed');
expect(data).toHaveProperty('cpu');
expect(data).toHaveProperty('memory');
expect(data).toHaveProperty('network');
expect(data.cpu).toHaveProperty('avgPercent');
expect(data.memory).toHaveProperty('avgWorkingSetMB');
expect(Array.isArray(data.sessions)).toBe(true);
expect(data).toHaveProperty('sessionsScanned');
expect(data).toHaveProperty('windowDays');
expect(data).toHaveProperty('sessionLimit');
if (data.sessions.length > 0) {
const s = data.sessions[0];
expect(s).toHaveProperty('avgCpu');
expect(s).toHaveProperty('avgWorkingSet');
expect(s).toHaveProperty('avgSpoolDepth');
}
});
});
suite('get_agent_efficiency_metrics', () => {
it('should return per-version efficiency buckets with percentiles', async () => {
const data = await apiFetch<any>(`/api/global/metrics/agent-efficiency?days=30&limit=100`);
expect(data).toHaveProperty('sessionsScanned');
expect(data).toHaveProperty('sessionsWithSnapshots');
expect(Array.isArray(data.byVersion)).toBe(true);
if (data.byVersion.length > 0) {
const bucket = data.byVersion[0];
expect(bucket).toHaveProperty('agentVersion');
expect(bucket).toHaveProperty('sessionsScanned');
expect(bucket).toHaveProperty('crashRate');
if (bucket.maxWorkingSetMb) {
expect(bucket.maxWorkingSetMb).toHaveProperty('p50');
expect(bucket.maxWorkingSetMb).toHaveProperty('p95');
}
}
});
});
@@ -109,6 +109,7 @@ describe('role catalog snapshot — privilege-leak guard', () => {
// fails — that failure is the deliberate prompt to decide its role placement.
const GA_FULL = [
'annotate_session',
'get_agent_efficiency_metrics',
'get_api_usage',
'get_app_install_metrics',
'get_audit_logs',
@@ -190,6 +191,7 @@ describe('role catalog snapshot — privilege-leak guard', () => {
// cross-tenant aggregates.
const PLATFORM_ONLY = [
'annotate_session',
'get_agent_efficiency_metrics',
'get_api_usage',
'get_ops_events',
'get_platform_metrics',
@@ -153,17 +153,26 @@ export function buildGeoLocationParams(input: {
* 100), so on busy installs a wider `days` changes nothing — `truncated` flags
* when the cap was hit (more sessions than analyzed likely exist). Prefers the
* backend's clamped echo over the requested values.
*
* Truncation is decided on `sessionsScanned` (sessions the backend actually
* covered) when the backend sends it. The legacy fallback compares
* `sessionsAnalyzed` — the has-snapshots SUBSET — against the cap, which
* understates truncation on fleets where many sessions emit no snapshots;
* it only remains for the deploy window where the MCP is newer than the backend.
*/
export function platformWindowEcho(
raw: { windowDays?: number; sessionLimit?: number },
raw: { windowDays?: number; sessionLimit?: number; sessionsScanned?: number },
requested: { days: number; limit: number },
sessionsAnalyzed: number,
): { windowDays: number; sessionLimit: number; truncated: boolean } {
): { windowDays: number; sessionLimit: number; truncated: boolean; sessionsScanned?: number } {
const sessionLimit = raw.sessionLimit ?? requested.limit;
return {
windowDays: raw.windowDays ?? requested.days,
sessionLimit,
truncated: sessionsAnalyzed >= sessionLimit,
truncated: raw.sessionsScanned != null
? raw.sessionsScanned >= sessionLimit
: sessionsAnalyzed >= sessionLimit,
...(raw.sessionsScanned != null ? { sessionsScanned: raw.sessionsScanned } : {}),
};
}
@@ -408,11 +417,15 @@ export function registerAdminTools(server: McpServer, ga: boolean, strictGa: boo
description:
'Get aggregated platform-level agent performance metrics across recent sessions. ' +
'Returns: avg/max/p95 CPU, memory (working set, private bytes), network (bytes up/down, latency, requests), ' +
'spool queue stats, event delivery latency percentiles, agent crash rates (incl. top exceptions), ' +
'top sessions by CPU/memory, and per-agent-version breakdown. Global Admin only. ' +
'Only the newest maxSessions sessions inside the window are analyzed (default 100), so on a ' +
'busy install widening days alone may not change the result — raise maxSessions to widen the ' +
'sample. The response echoes sessionLimit and a truncated flag when the cap was hit. ' +
'days accepts any value 1-365 (e.g. 5, 7, 12, 30, 90).',
'sample. The response echoes sessionLimit, sessionsScanned, and a truncated flag based on the ' +
'sessions actually scanned. days accepts any value 1-365 (e.g. 5, 7, 12, 30, 90). ' +
'For per-agent-version percentiles computed server-side, prefer get_agent_efficiency_metrics. ' +
'Note totalBytesDown counts the agent\'s own HTTP traffic to the backend, NOT app downloads ' +
'(app/Delivery Optimization bytes live in get_app_install_metrics / get_geographic_metrics).',
inputSchema: {
days: z.coerce.number().int().min(1).max(365).optional().default(30)
.describe('Time window in days (1-365). Defaults to 30.'),
@@ -429,14 +442,39 @@ export function registerAdminTools(server: McpServer, ga: boolean, strictGa: boo
avgCpu: number; maxCpu: number; avgWorkingSet: number; maxWorkingSet: number;
avgPrivateBytes: number; avgLatency: number;
totalBytesUp: number; totalBytesDown: number; totalRequests: number;
avgSpoolDepth: number; maxSpoolDepth: number; peakSpoolDepth: number;
maxSpoolFileBytes: number; totalEventsEmitted: number; spoolPressureDetected: boolean;
};
const raw = await apiFetch(`/api/global/metrics/platform${buildQuery({ days: args.days, limit: args.maxSessions })}`) as
{ sessions?: SessionMetric[]; windowDays?: number; sessionLimit?: number };
// Backend fans out one storage query per session (bounded to 32 concurrent), so
// 2000-session windows take longer than the default 30s client timeout — same
// override precedent as get_session_summary. Warm (cached) calls return instantly.
const raw = await apiFetch(
`/api/global/metrics/platform${buildQuery({ days: args.days, limit: args.maxSessions })}`,
{ signal: AbortSignal.timeout(90_000) }) as
{
sessions?: SessionMetric[]; windowDays?: number; sessionLimit?: number; sessionsScanned?: number;
deliveryLatency?: { p50Ms: number; p95Ms: number; p99Ms: number; avgMs: number; sampleCount: number; clockSkewPercent: number };
crashRate?: {
totalStarts: number; cleanExits: number; exceptionCrashes: number; hardKills: number;
rebootKills: number; firstRuns: number; crashRatePercent: number;
topExceptions?: { exceptionType: string; count: number }[];
};
computedAt?: string; computeDurationMs?: number; fromCache?: boolean;
};
const sessions = raw?.sessions ?? [];
const requested = { days: args.days, limit: args.maxSessions };
// Backend-computed blocks that exist independently of snapshot availability
// (crash rate comes from agent_started events, latency from all sampled events).
const backendAggregates = {
deliveryLatency: raw?.deliveryLatency,
crashRate: raw?.crashRate,
computedAt: raw?.computedAt,
computeDurationMs: raw?.computeDurationMs,
fromCache: raw?.fromCache,
};
if (sessions.length === 0) {
return toolResultText(
{ ...platformWindowEcho(raw, requested, 0), sessionsAnalyzed: 0, message: 'No performance data available' },
{ ...platformWindowEcho(raw, requested, 0), sessionsAnalyzed: 0, ...backendAggregates, message: 'No performance data available' },
MAX_RESULT_SIZE_CHARS.small);
}
const windowEcho = platformWindowEcho(raw, requested, sessions.length);
@@ -475,6 +513,7 @@ export function registerAdminTools(server: McpServer, ga: boolean, strictGa: boo
windowDays: windowEcho.windowDays,
sessionLimit: windowEcho.sessionLimit,
truncated: windowEcho.truncated,
...(windowEcho.sessionsScanned != null ? { sessionsScanned: windowEcho.sessionsScanned } : {}),
sessionsAnalyzed: sessions.length,
cpu: { avgPercent: round(avg(cpus)), maxPercent: round(Math.max(...maxCpus)), p95Percent: round(p95(maxCpus)) },
memory: {
@@ -487,6 +526,15 @@ export function registerAdminTools(server: McpServer, ga: boolean, strictGa: boo
totalRequests: sessions.reduce((a, s) => a + s.totalRequests, 0),
avgLatencyMs: round(avg(lat)),
},
spool: {
avgDepth: round(avg(sessions.map(s => s.avgSpoolDepth ?? 0))),
maxDepth: round(Math.max(...sessions.map(s => s.maxSpoolDepth ?? 0))),
maxPeakDepth: round(Math.max(...sessions.map(s => s.peakSpoolDepth ?? 0))),
maxSpoolFileBytes: Math.max(...sessions.map(s => s.maxSpoolFileBytes ?? 0)),
totalEventsEmitted: sessions.reduce((a, s) => a + (s.totalEventsEmitted ?? 0), 0),
pressureDetectedSessions: sessions.filter(s => s.spoolPressureDetected).length,
},
...backendAggregates,
topSessionsByCpu: topCpu,
topSessionsByMemory: topMem,
agentVersionBreakdown: versionBreakdown,
@@ -498,6 +546,46 @@ export function registerAdminTools(server: McpServer, ga: boolean, strictGa: boo
})
);
// get_agent_efficiency_metrics — Global Admin/Reader only; server-side per-version aggregation.
if (ga) server.registerTool(
'get_agent_efficiency_metrics',
{
title: 'Agent Efficiency Metrics',
description:
'Get agent efficiency metrics aggregated by agent version, computed server-side: ' +
'p50/p95/max/avg percentiles for CPU, working set, private bytes, thread count, handle count, ' +
'spool depth/file size, API latency and request counts, plus crash rates (incl. top exceptions) ' +
'and spool-pressure session counts per version. Includes an overall cross-version bucket and ' +
'the top offender sessions per dimension (maxCpuPercent / maxWorkingSetMb / maxHandleCount) — ' +
'drill into those with get_session_summary only when a threshold is violated, instead of ' +
'pulling raw snapshot rows. Compare sessionsScanned against sessionLimit to detect truncation. ' +
'Global Admin only. Omit tenantId for the cross-tenant aggregate; pass it to scope to one tenant. ' +
'Prefer this over get_platform_metrics for version-comparison and efficiency reviews.',
inputSchema: {
days: z.coerce.number().int().min(1).max(365).optional().default(30)
.describe('Time window in days (1-365). Defaults to 30.'),
maxSessions: z.coerce.number().int().min(1).max(2000).optional().default(500)
.describe('Newest N sessions in the window to analyze (1-2000, default 500).'),
tenantId: z.string().optional()
.describe('Optional tenant GUID to scope the aggregate to a single tenant. Omit for cross-tenant.'),
},
annotations: READ_ONLY,
},
async (args) => withToolTelemetry('get_agent_efficiency_metrics', args, async () => {
try {
// Server-side aggregation is the point of this endpoint — pass the payload through.
// Cold 2000-session windows fan out one filtered storage query per session backend-side,
// so give it more headroom than the 30s default; warm (cached) calls return instantly.
const data = await apiFetch(
`/api/global/metrics/agent-efficiency${buildQuery({ days: args.days, limit: args.maxSessions, tenantId: args.tenantId })}`,
{ signal: AbortSignal.timeout(120_000) });
return toolResultText(data, MAX_RESULT_SIZE_CHARS.small);
} catch (error: unknown) {
return toolError('get_agent_efficiency_metrics', args, error);
}
})
);
// Tool 15: get_usage_metrics
server.registerTool(
'get_usage_metrics',
@@ -720,6 +720,13 @@ namespace AutopilotMonitor.DecisionCore.Engine
/// <see cref="SessionStage.AwaitingHello"/> with the HelloSafety window armed at this
/// signal's instant.
/// </para>
/// <para>
/// sits-d Cloud PCs (2026-08-20): on SkipUser=true flows the observed skip stands in
/// for the final-exit evidence — Windows never renders the user ESP page, so no
/// Shell-Core 62407 and no AccountSetup provisioning category can ever exist, and this
/// handler is the only remaining site that can knock on the (open) arm-B gate. See the
/// inline <c>skipUserExitEquivalent</c> comment for the full derivation.
/// </para>
/// </summary>
private DecisionStep HandleImeUserSessionCompletedV1(DecisionState state, DecisionSignal signal)
{
@@ -768,11 +775,33 @@ namespace AutopilotMonitor.DecisionCore.Engine
// session is still parked pre-Hello (same stage guard as the deferred-promote path
// in HandleAccountSetupProvisioningCompleteV1) — AwaitingHello/AwaitingDesktop/
// Finalizing/terminal stages keep their existing rails untouched.
//
// sits-d Cloud PCs (2026-08-20): on a SkipUser=true flow Windows never renders the
// user ESP page — Shell-Core 62407 does not fire and the AccountSetup provisioning
// categories are never written, so IsPostAccountSetupFinalExit is unsatisfiable by
// construction (there is no exit to record). Arm B of ShouldTransitionToAwaitingHello
// is open from the EspConfigDetected bootstrap onward, but every site that evaluates
// the gate hangs off a signal this flow cannot produce (EspExiting and
// EspPhaseChanged(FinalizingSetup) ride Shell-Core; the desktop fast-path requires
// the Hello policy to be disabled) — five sessions sat at "waiting on:
// hello_resolution" for six hours with the gate open and nothing left to knock.
// On such a flow the observed skip IS the exit evidence: no page exists whose exit
// could ever say more. The other three facts (AccountSetup anchor, genuine IME
// user-session completion at-or-after it, real-user desktop) stay mandatory, which
// keeps this STRICTER than the existing Shell-Core-carried arm-B promotion (pinned
// in ClassicAwaitingHelloGuardTests, which needs no IME/desktop evidence at all).
// This knock is also self-healing against Fix 10's AccountSetup bounce-back: the
// IME phase line always precedes the user-session-complete line in log order, so a
// bounced promotion is re-attempted by the very next user-session re-emission.
// Device Preparation keeps its own rails (arm D + DevicePrepCompletion backstop).
var skipUserExitEquivalent =
state.ScenarioObservations.SkipUserEsp?.Value == true
&& state.ScenarioProfile.Mode != EnrollmentMode.DevicePreparation;
var armCSatisfied =
builder.ImeUserSessionCompletedUtc != null
&& state.AccountSetupEnteredUtc != null
&& builder.ImeUserSessionCompletedUtc.Value >= state.AccountSetupEnteredUtc.Value
&& IsPostAccountSetupFinalExit(state)
&& (IsPostAccountSetupFinalExit(state) || skipUserExitEquivalent)
&& state.DesktopArrivedUtc != null;
var parkedPreHello = state.Stage == SessionStage.SessionStarted
|| state.Stage == SessionStage.EspDeviceSetup
@@ -57,9 +57,13 @@ namespace AutopilotMonitor.Shared.DataAccess
/// <paramref name="allowedTenantIds"/> (when non-null) bounds the cross-tenant fan-out to that
/// subset — used by delegated ("MSP") callers so the aggregate covers only their managed tenants.
/// </summary>
/// <param name="select">Optional column projection for the SessionsIndex scan. <c>null</c>
/// returns the full mirror row (every existing caller). Aggregation paths pass a narrow
/// set so a 2000-session window doesn't materialize the ~40-column row per session.</param>
Task<RawPage<SessionSummary>> GetAllSessionsPageAsync(
string? tenantIdFilter, int? days, int pageSize, string? continuation,
IReadOnlyCollection<string>? allowedTenantIds = null);
IReadOnlyCollection<string>? allowedTenantIds = null,
IEnumerable<string>? select = null);
/// <summary>
/// Server-side aggregation for the dashboard stats cards (per-tenant scope).
@@ -196,6 +200,17 @@ namespace AutopilotMonitor.Shared.DataAccess
/// </summary>
Task<List<EnrollmentEvent>> GetSessionEventsStrictAsync(string tenantId, string sessionId, int maxResults = 1000);
Task<List<EnrollmentEvent>> GetSessionEventsByTypeAsync(string tenantId, string sessionId, string eventType, int maxResults = 200);
/// <summary>
/// Multi-type sibling of <see cref="GetSessionEventsByTypeAsync"/>: one server-side OData
/// query matching ANY of <paramref name="eventTypes"/>, with an optional column
/// <paramref name="select"/> projection. Built for the metrics aggregation paths that only
/// need a handful of periodic event types out of a session's full partition.
/// <paramref name="maxResults"/> is enforced as a hard cap (the query stops draining pages
/// once reached), unlike the single-type variant where it only sizes the storage page.
/// </summary>
Task<List<EnrollmentEvent>> GetSessionEventsByTypesAsync(
string tenantId, string sessionId, IReadOnlyCollection<string> eventTypes,
IEnumerable<string>? select = null, int maxResults = 2000);
/// <summary>
/// Reads a single page of session events. The returned <see cref="RawPage{T}"/>
@@ -11,6 +11,10 @@ import { useNotifications } from "../../../contexts/NotificationContext";
import { scopedApi } from "@/lib/scopedApi";
import { authenticatedFetch, TokenExpiredError } from "@/lib/authenticatedFetch";
import { getErrorCodeEntry, formatErrorCode } from "@/utils/errorCodeMap";
import { trackEvent } from "@/lib/appInsights";
// Static import is safe: appReportData is pure data prep with no jsPDF dependency —
// jsPDF itself only loads with the dynamic import inside handleExportPdf.
import { scopeLabel } from "@/utils/pdf/appReportData";
import { useAggregatedAdminScope } from "@/hooks";
import { GlobalAdminBanner, globalAdminSubtitle } from "@/components/GlobalAdminBanner";
import { TenantScopeSelector } from "@/components/TenantScopeSelector";
@@ -215,6 +219,7 @@ function AppDetailContent() {
const [days, setDays] = useState<7 | 30 | 90>(initialDays as 7 | 30 | 90);
const [analytics, setAnalytics] = useState<AnalyticsResponse | null>(null);
const [loading, setLoading] = useState(true);
const [exporting, setExporting] = useState(false);
const progress = useFetchProgress("appAnalytics.lastFetchMs");
const { begin: progressBegin, finish: progressFinish } = progress;
@@ -370,6 +375,47 @@ function AppDetailContent() {
}
};
const handleExportPdf = async () => {
if (!analytics || exporting) return;
setExporting(true);
const t0 = performance.now();
const baseProps = {
appName,
appType: analytics.appType,
windowDays: days,
scope: scope.isAggregatedGlobalView ? "aggregated" : "tenant",
};
try {
// Lazy import keeps jsPDF out of the route bundle; a failed chunk load
// (deploy skew, offline) lands in the catch below and is tracked too.
const { generateAppReportPdf } = await import("@/utils/pdf/appReportPdf");
const result = generateAppReportPdf({
analytics,
appName,
days,
scopeLabel: scopeLabel(scope),
generatedAt: new Date(),
});
trackEvent("app_report_pdf_exported", {
...baseProps,
success: true,
durationMs: Math.round(performance.now() - t0),
pageCount: result.pageCount,
});
} catch (err) {
trackEvent("app_report_pdf_exported", {
...baseProps,
success: false,
durationMs: Math.round(performance.now() - t0),
errorName: err instanceof Error ? err.name : "UnknownError",
});
console.error("PDF export failed", err);
addNotification("error", "Export Failed", "Unable to generate the PDF report.", "app-pdf-export-error");
} finally {
setExporting(false);
}
};
// Single fetch effect: re-runs when scope, app, days, or tenant selection change.
// Gated on scopeInitialized so we don't waste a backend hit fetching the
// wrong scope before the GA default-to-own-tenant has settled.
@@ -442,6 +488,22 @@ function AppDetailContent() {
{d} Days
</button>
))}
<button
onClick={handleExportPdf}
disabled={loading || !analytics || exporting}
className="inline-flex items-center gap-1.5 px-4 py-2 text-sm rounded-md bg-gray-100 text-gray-700 hover:bg-gray-200 disabled:opacity-40 transition-colors"
title="Download this report as a PDF"
>
<svg className="w-4 h-4" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path
strokeLinecap="round"
strokeLinejoin="round"
strokeWidth={2}
d="M4 16v2a2 2 0 002 2h12a2 2 0 002-2v-2M7 10l5 5 5-5M12 15V3"
/>
</svg>
<span>{exporting ? "Exporting…" : "Export PDF"}</span>
</button>
</div>
</div>
</div>
+224 -12
View File
@@ -16,6 +16,7 @@
"@xyflow/react": "^12.11.3",
"dagre": "^0.8.5",
"fflate": "^0.8.3",
"jspdf": "^4.2.1",
"leaflet": "^1.9.4",
"next": "^16.3.1",
"react": "^19.2.8",
@@ -1423,9 +1424,6 @@
"cpu": [
"arm64"
],
"libc": [
"glibc"
],
"license": "MIT",
"optional": true,
"os": [
@@ -1442,9 +1440,6 @@
"cpu": [
"arm64"
],
"libc": [
"musl"
],
"license": "MIT",
"optional": true,
"os": [
@@ -1461,9 +1456,6 @@
"cpu": [
"x64"
],
"libc": [
"glibc"
],
"license": "MIT",
"optional": true,
"os": [
@@ -1480,9 +1472,6 @@
"cpu": [
"x64"
],
"libc": [
"musl"
],
"license": "MIT",
"optional": true,
"os": [
@@ -2077,6 +2066,19 @@
"undici-types": "~6.21.0"
}
},
"node_modules/@types/pako": {
"version": "2.0.4",
"resolved": "https://registry.npmjs.org/@types/pako/-/pako-2.0.4.tgz",
"integrity": "sha512-VWDCbrLeVXJM9fihYodcLiIv0ku+AlOa/TQ1SvYOaBuyrSKgEcro95LJyIsJ4vSo6BXIxOKxiJAat04CmST9Fw==",
"license": "MIT"
},
"node_modules/@types/raf": {
"version": "3.4.3",
"resolved": "https://registry.npmjs.org/@types/raf/-/raf-3.4.3.tgz",
"integrity": "sha512-c4YAvMedbPZ5tEyxzQdMoOhhJ4RD3rngZIdwC2/qDN3d7JpEhB6fiBRKVY1lg5B7Wk+uPBjn5f39j1/2MY1oOw==",
"license": "MIT",
"optional": true
},
"node_modules/@types/react": {
"version": "19.2.18",
"resolved": "https://registry.npmjs.org/@types/react/-/react-19.2.18.tgz",
@@ -2097,6 +2099,13 @@
"@types/react": "^19.2.0"
}
},
"node_modules/@types/trusted-types": {
"version": "2.0.7",
"resolved": "https://registry.npmjs.org/@types/trusted-types/-/trusted-types-2.0.7.tgz",
"integrity": "sha512-ScaPdn1dQczgbl0QFTeTOmVHFULt394XJgOQNoyVhZ6r2vLnMLJfBPd53SB52T/3G36VI1/g2MZaX0cwDuXsfw==",
"license": "MIT",
"optional": true
},
"node_modules/@typescript-eslint/eslint-plugin": {
"version": "8.65.0",
"resolved": "https://registry.npmjs.org/@typescript-eslint/eslint-plugin/-/eslint-plugin-8.65.0.tgz",
@@ -3196,6 +3205,16 @@
"dev": true,
"license": "MIT"
},
"node_modules/base64-arraybuffer": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/base64-arraybuffer/-/base64-arraybuffer-1.0.2.tgz",
"integrity": "sha512-I3yl4r9QB5ZRY3XuJVEPfc2XhZO6YweFPI+UovAzn+8/hb3oJ6lnysaFcjVpkCPfVWFUDvoZ8kmVDP7WyRtYtQ==",
"license": "MIT",
"optional": true,
"engines": {
"node": ">= 0.6.0"
}
},
"node_modules/baseline-browser-mapping": {
"version": "2.11.0",
"resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.11.0.tgz",
@@ -3369,6 +3388,26 @@
],
"license": "CC-BY-4.0"
},
"node_modules/canvg": {
"version": "3.0.11",
"resolved": "https://registry.npmjs.org/canvg/-/canvg-3.0.11.tgz",
"integrity": "sha512-5ON+q7jCTgMp9cjpu4Jo6XbvfYwSB2Ow3kzHKfIyJfaCAOHLbdKPQqGKgfED/R5B+3TFFfe8pegYA+b423SRyA==",
"license": "MIT",
"optional": true,
"dependencies": {
"@babel/runtime": "^7.12.5",
"@types/raf": "^3.4.0",
"core-js": "^3.8.3",
"raf": "^3.4.1",
"regenerator-runtime": "^0.13.7",
"rgbcolor": "^1.0.1",
"stackblur-canvas": "^2.0.0",
"svg-pathdata": "^6.0.3"
},
"engines": {
"node": ">=10.0.0"
}
},
"node_modules/chai": {
"version": "6.2.2",
"resolved": "https://registry.npmjs.org/chai/-/chai-6.2.2.tgz",
@@ -3499,6 +3538,21 @@
"dev": true,
"license": "MIT"
},
"node_modules/core-js": {
"version": "3.50.0",
"resolved": "https://registry.npmjs.org/core-js/-/core-js-3.50.0.tgz",
"integrity": "sha512-BRWgOLKkFeCgRudR6zrs8p9XJZcE14grzKMMssoYrk6krtuEZ7MTKPIY5RzOnqsEKIR9kst7wNzphttraT+Yqw==",
"hasInstallScript": true,
"license": "MIT",
"optional": true,
"engines": {
"node": "*"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/core-js"
}
},
"node_modules/cross-spawn": {
"version": "7.0.6",
"resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz",
@@ -3514,6 +3568,16 @@
"node": ">= 8"
}
},
"node_modules/css-line-break": {
"version": "2.1.0",
"resolved": "https://registry.npmjs.org/css-line-break/-/css-line-break-2.1.0.tgz",
"integrity": "sha512-FHcKFCZcAha3LwfVBhCQbW2nCNbkZXn7KVUJcsT5/P8YmfsVja0FMPJr0B903j/E69HUphKiV9iQArX8SDYA4w==",
"license": "MIT",
"optional": true,
"dependencies": {
"utrie": "^1.0.2"
}
},
"node_modules/cssesc": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/cssesc/-/cssesc-3.0.0.tgz",
@@ -3912,6 +3976,16 @@
"csstype": "^3.0.2"
}
},
"node_modules/dompurify": {
"version": "3.4.14",
"resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.4.14.tgz",
"integrity": "sha512-dVoH9z+MY+C9IilgGCk3YfFqjLi3fChm2OiKJMzh6axrJ5qwxqWaZamgmHrpv22CN/KdbZJuGEGgfQoL00LTdg==",
"license": "(MPL-2.0 OR Apache-2.0)",
"optional": true,
"optionalDependencies": {
"@types/trusted-types": "^2.0.7"
}
},
"node_modules/dunder-proto": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz",
@@ -4703,6 +4777,17 @@
"dev": true,
"license": "MIT"
},
"node_modules/fast-png": {
"version": "6.4.0",
"resolved": "https://registry.npmjs.org/fast-png/-/fast-png-6.4.0.tgz",
"integrity": "sha512-kAqZq1TlgBjZcLr5mcN6NP5Rv4V2f22z00c3g8vRrwkcqjerx7BEhPbOnWCPqaHUl2XWQBJQvOT/FQhdMT7X/Q==",
"license": "MIT",
"dependencies": {
"@types/pako": "^2.0.3",
"iobuffer": "^5.3.2",
"pako": "^2.1.0"
}
},
"node_modules/fastq": {
"version": "1.20.1",
"resolved": "https://registry.npmjs.org/fastq/-/fastq-1.20.1.tgz",
@@ -5168,6 +5253,20 @@
"dev": true,
"license": "MIT"
},
"node_modules/html2canvas": {
"version": "1.4.1",
"resolved": "https://registry.npmjs.org/html2canvas/-/html2canvas-1.4.1.tgz",
"integrity": "sha512-fPU6BHNpsyIhr8yyMpTLLxAbkaK8ArIBcmZIRiBLiDhjeqvXolaEmDGmELFuX9I4xDcaKKcJl+TKZLqruBbmWA==",
"license": "MIT",
"optional": true,
"dependencies": {
"css-line-break": "^2.1.0",
"text-segmentation": "^1.0.3"
},
"engines": {
"node": ">=8.0.0"
}
},
"node_modules/ignore": {
"version": "5.3.2",
"resolved": "https://registry.npmjs.org/ignore/-/ignore-5.3.2.tgz",
@@ -5229,6 +5328,12 @@
"node": ">=12"
}
},
"node_modules/iobuffer": {
"version": "5.4.0",
"resolved": "https://registry.npmjs.org/iobuffer/-/iobuffer-5.4.0.tgz",
"integrity": "sha512-DRebOWuqDvxunfkNJAlc3IzWIPD5xVxwUNbHr7xKB8E6aLJxIPfNX3CoMJghcFjpv6RWQsrcJbghtEwSPoJqMA==",
"license": "MIT"
},
"node_modules/is-array-buffer": {
"version": "3.0.5",
"resolved": "https://registry.npmjs.org/is-array-buffer/-/is-array-buffer-3.0.5.tgz",
@@ -5772,6 +5877,23 @@
"json5": "lib/cli.js"
}
},
"node_modules/jspdf": {
"version": "4.2.1",
"resolved": "https://registry.npmjs.org/jspdf/-/jspdf-4.2.1.tgz",
"integrity": "sha512-YyAXyvnmjTbR4bHQRLzex3CuINCDlQnBqoSYyjJwTP2x9jDLuKDzy7aKUl0hgx3uhcl7xzg32agn5vlie6HIlQ==",
"license": "MIT",
"dependencies": {
"@babel/runtime": "^7.28.6",
"fast-png": "^6.2.0",
"fflate": "^0.8.1"
},
"optionalDependencies": {
"canvg": "^3.0.11",
"core-js": "^3.6.0",
"dompurify": "^3.3.1",
"html2canvas": "^1.0.0-rc.5"
}
},
"node_modules/jsx-ast-utils": {
"version": "3.3.5",
"resolved": "https://registry.npmjs.org/jsx-ast-utils/-/jsx-ast-utils-3.3.5.tgz",
@@ -6637,6 +6759,22 @@
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/pako": {
"version": "2.2.0",
"resolved": "https://registry.npmjs.org/pako/-/pako-2.2.0.tgz",
"integrity": "sha512-zJq6RP/5q+TO2OpFV3FHzlPnFjmkb7Nc99a5SNjJE+uu/PkpChs+NIZSSzbBoD+6kjiISXjfYdwj1ZRQ81dz/w==",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/puzrin"
},
{
"type": "github",
"url": "https://github.com/sponsors/nodeca"
}
],
"license": "(MIT AND Zlib)"
},
"node_modules/parent-module": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/parent-module/-/parent-module-1.0.1.tgz",
@@ -6684,6 +6822,13 @@
"dev": true,
"license": "MIT"
},
"node_modules/performance-now": {
"version": "2.1.0",
"resolved": "https://registry.npmjs.org/performance-now/-/performance-now-2.1.0.tgz",
"integrity": "sha512-7EAHlyLHI56VEIdK57uwHdHKIaAGbnXPiw0yWbarQZOKaKpvUIgW0jWRVLiatnM+XXlSwsanIBH/hzGMJulMow==",
"license": "MIT",
"optional": true
},
"node_modules/picocolors": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz",
@@ -6964,6 +7109,16 @@
],
"license": "MIT"
},
"node_modules/raf": {
"version": "3.4.1",
"resolved": "https://registry.npmjs.org/raf/-/raf-3.4.1.tgz",
"integrity": "sha512-Sq4CW4QhwOHE8ucn6J34MqtZCeWFP2aQSmrlroYgqAV1PjStIhJXxYuTgUIfkEk7zTLjmIjLmU5q+fbD1NnOJA==",
"license": "MIT",
"optional": true,
"dependencies": {
"performance-now": "^2.1.0"
}
},
"node_modules/react": {
"version": "19.2.8",
"resolved": "https://registry.npmjs.org/react/-/react-19.2.8.tgz",
@@ -7120,6 +7275,13 @@
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/regenerator-runtime": {
"version": "0.13.11",
"resolved": "https://registry.npmjs.org/regenerator-runtime/-/regenerator-runtime-0.13.11.tgz",
"integrity": "sha512-kY1AZVr2Ra+t+piVaJ4gxaFaReZVH40AKNo7UCX6W+dEwBo/2oZJzqfuN1qLq1oL45o56cPaTXELwrTh8Fpggg==",
"license": "MIT",
"optional": true
},
"node_modules/regexp.prototype.flags": {
"version": "1.5.4",
"resolved": "https://registry.npmjs.org/regexp.prototype.flags/-/regexp.prototype.flags-1.5.4.tgz",
@@ -7199,6 +7361,16 @@
"node": ">=0.10.0"
}
},
"node_modules/rgbcolor": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/rgbcolor/-/rgbcolor-1.0.1.tgz",
"integrity": "sha512-9aZLIrhRaD97sgVhtJOW6ckOEh6/GnvQtdVNfdZ6s67+3/XwLS9lBcQYzEEhYVeUowN7pRzMLsyGhK2i/xvWbw==",
"license": "MIT OR SEE LICENSE IN FEEL-FREE.md",
"optional": true,
"engines": {
"node": ">= 0.8.15"
}
},
"node_modules/rolldown": {
"version": "1.1.5",
"resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.1.5.tgz",
@@ -7580,6 +7752,16 @@
"dev": true,
"license": "MIT"
},
"node_modules/stackblur-canvas": {
"version": "2.7.0",
"resolved": "https://registry.npmjs.org/stackblur-canvas/-/stackblur-canvas-2.7.0.tgz",
"integrity": "sha512-yf7OENo23AGJhBriGx0QivY5JP6Y1HbrrDI6WLt6C5auYZXlQrheoY8hD4ibekFKz1HOfE48Ww8kMWMnJD/zcQ==",
"license": "MIT",
"optional": true,
"engines": {
"node": ">=0.1.14"
}
},
"node_modules/std-env": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/std-env/-/std-env-4.0.0.tgz",
@@ -7809,6 +7991,16 @@
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/svg-pathdata": {
"version": "6.0.3",
"resolved": "https://registry.npmjs.org/svg-pathdata/-/svg-pathdata-6.0.3.tgz",
"integrity": "sha512-qsjeeq5YjBZ5eMdFuUa4ZosMLxgr5RZ+F+Y1OrDhuOCEInRMA3x74XdBtggJcj9kOeInz0WE+LgCPDkZFlBYJw==",
"license": "MIT",
"optional": true,
"engines": {
"node": ">=12.0.0"
}
},
"node_modules/tailwindcss": {
"version": "3.4.19",
"resolved": "https://registry.npmjs.org/tailwindcss/-/tailwindcss-3.4.19.tgz",
@@ -7877,6 +8069,16 @@
"node": ">= 6"
}
},
"node_modules/text-segmentation": {
"version": "1.0.3",
"resolved": "https://registry.npmjs.org/text-segmentation/-/text-segmentation-1.0.3.tgz",
"integrity": "sha512-iOiPUo/BGnZ6+54OsWxZidGCsdU8YbE4PSpdPinp7DeMtUJNJBoJ/ouUSTJjHkh1KntHaltHl/gDs2FC4i5+Nw==",
"license": "MIT",
"optional": true,
"dependencies": {
"utrie": "^1.0.2"
}
},
"node_modules/thenify": {
"version": "3.3.1",
"resolved": "https://registry.npmjs.org/thenify/-/thenify-3.3.1.tgz",
@@ -8330,6 +8532,16 @@
"dev": true,
"license": "MIT"
},
"node_modules/utrie": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/utrie/-/utrie-1.0.2.tgz",
"integrity": "sha512-1MLa5ouZiOmQzUbjbu9VmjLzn1QLXBhwpUa7kdLUQK+KQ5KA9I1vk5U4YHe/X2Ch7PYnJfWuWT+VbuxbGwljhw==",
"license": "MIT",
"optional": true,
"dependencies": {
"base64-arraybuffer": "^1.0.2"
}
},
"node_modules/victory-vendor": {
"version": "36.9.2",
"resolved": "https://registry.npmjs.org/victory-vendor/-/victory-vendor-36.9.2.tgz",
@@ -23,6 +23,7 @@
"@xyflow/react": "^12.11.3",
"dagre": "^0.8.5",
"fflate": "^0.8.3",
"jspdf": "^4.2.1",
"leaflet": "^1.9.4",
"next": "^16.3.1",
"react": "^19.2.8",
@@ -4,10 +4,10 @@
*/
export const PAGE_LASTMOD: Record<string, string> = {
"/": "2026-07-30T22:21:18+02:00",
"/about": "2026-07-30T22:21:18+02:00",
"/about": "2026-08-18T15:47:15+02:00",
"/roadmap": "2026-07-06T18:31:00+02:00",
"/changelog": "2026-07-06T18:12:31+02:00",
"/privacy": "2026-07-31T14:30:57+02:00",
"/privacy": "2026-08-11T13:30:44+02:00",
"/terms": "2026-07-31T14:30:57+02:00",
"/docs": "2026-07-06T18:12:31+02:00",
"/docs/private-preview": "2026-07-06T18:12:31+02:00",
@@ -0,0 +1,245 @@
import { describe, it, expect } from "vitest";
import {
appReportFileName,
scopeLabel,
prepareReportModel,
formatReportDate,
type AppReportAnalytics,
type AppReportInput,
} from "../appReportData";
function makeAnalytics(overrides: Partial<AppReportAnalytics> = {}): AppReportAnalytics {
return {
appType: "MSI",
windowDays: 90,
bucket: "week",
summary: {
totalInstalls: 912,
succeeded: 888,
skipped: 20,
unmeasured: 3,
failed: 1,
failureRate: 0.1,
avgDurationSeconds: 55,
p95DurationSeconds: 110,
trend: "stable",
trendDelta: 0,
flakinessScore: 0.02,
},
timeSeries: [
{ bucketStart: "2026-08-03T00:00:00Z", installs: 10, succeeded: 9, failed: 1, failureRate: 10, avgDurationSeconds: 50 },
],
versionBreakdown: [],
topFailureCodes: [],
detectionLiesCount: 0,
deviceModelBreakdown: [],
versionRegressions: [],
...overrides,
};
}
function makeInput(overrides: Partial<AppReportAnalytics> = {}): AppReportInput {
return {
analytics: makeAnalytics(overrides),
appName: "RealmJoin Agent (Device)",
days: 90,
scopeLabel: "All tenants (aggregated)",
generatedAt: new Date(2026, 7, 18), // Aug 18, 2026 local
};
}
describe("appReportFileName", () => {
it("slugifies the app name with window and date", () => {
expect(appReportFileName("RealmJoin Agent (Device)", 90, new Date(2026, 7, 18))).toBe(
"app-report-realmjoin-agent-device-90d-20260818.pdf"
);
});
it("falls back to 'app' when the name has no latin characters", () => {
expect(appReportFileName("日本語アプリ", 30, new Date(2026, 0, 5))).toBe(
"app-report-app-30d-20260105.pdf"
);
});
it("caps overlong names without a trailing dash", () => {
const name = "A".repeat(100) + " suffix";
const file = appReportFileName(name, 7, new Date(2026, 7, 18));
const slug = file.replace("app-report-", "").replace("-7d-20260818.pdf", "");
expect(slug.length).toBeLessThanOrEqual(60);
expect(slug.endsWith("-")).toBe(false);
});
});
describe("scopeLabel", () => {
it("labels the aggregated global view", () => {
expect(
scopeLabel({ isAggregatedGlobalView: true, selectedTenantName: undefined, selectedTenantId: "" })
).toBe("All tenants (aggregated)");
});
it("prefers the friendly tenant name", () => {
expect(
scopeLabel({ isAggregatedGlobalView: false, selectedTenantName: "contoso.com", selectedTenantId: "guid" })
).toBe("Tenant: contoso.com");
});
it("falls back to a shortened tenant id, then to 'Current tenant'", () => {
expect(
scopeLabel({
isAggregatedGlobalView: false,
selectedTenantName: undefined,
selectedTenantId: "12345678-abcd-ef00-0000-000000000000",
})
).toBe("Tenant: 12345678…");
expect(
scopeLabel({ isAggregatedGlobalView: false, selectedTenantName: undefined, selectedTenantId: "" })
).toBe("Current tenant");
});
});
describe("prepareReportModel", () => {
it("builds meta line, file name, and 8 KPIs", () => {
const model = prepareReportModel(makeInput());
expect(model.metaLine).toBe(
"90 day window · weekly buckets · All tenants (aggregated) · Generated Aug 18, 2026"
);
expect(model.fileName).toBe("app-report-realmjoin-agent-device-90d-20260818.pdf");
expect(model.kpis).toHaveLength(8);
expect(model.kpis[0]).toMatchObject({ label: "Total installs", value: "912" });
expect(model.kpis[1].hint).toBe("+20 skipped (not applicable)");
expect(model.kpis[3]).toMatchObject({ value: "0.1%", tone: "default" });
expect(model.kpis[6].value).toBe("stable");
expect(model.kpis[7].value).toBe("2%");
});
it("marks elevated failure rates with warning/danger tones", () => {
const warn = prepareReportModel(
makeInput({ summary: { ...makeAnalytics().summary, failureRate: 7.5 } })
);
expect(warn.kpis[3].tone).toBe("warning");
const danger = prepareReportModel(
makeInput({ summary: { ...makeAnalytics().summary, failureRate: 25 } })
);
expect(danger.kpis[3].tone).toBe("danger");
});
it("renders trend deltas as signed values (Helvetica has no arrow glyphs)", () => {
const improving = prepareReportModel(
makeInput({ summary: { ...makeAnalytics().summary, trend: "improving", trendDelta: -1.23 } })
);
expect(improving.kpis[6].value).toBe("-1.2 pp");
expect(improving.kpis[6].hint).toBe("improving");
const worsening = prepareReportModel(
makeInput({ summary: { ...makeAnalytics().summary, trend: "worsening", trendDelta: 2.5 } })
);
expect(worsening.kpis[6].value).toBe("+2.5 pp");
});
it("truncates failure codes to 10 rows and reports the remainder", () => {
const codes = Array.from({ length: 12 }, (_, i) => ({
code: `0x8007000${i}`,
exitCode: null,
count: 12 - i,
}));
const model = prepareReportModel(makeInput({ topFailureCodes: codes }));
expect(model.failureCodes.rows).toHaveLength(10);
expect(model.failureCodes.moreCount).toBe(2);
});
it("enriches known error codes and labels unknown ones", () => {
const model = prepareReportModel(
makeInput({
topFailureCodes: [
{ code: "0x80070005", exitCode: null, count: 3 },
{ code: "0xDEADBEEF", exitCode: null, count: 1 },
],
})
);
expect(model.failureCodes.rows[0][1]).not.toBe("Unknown code");
expect(model.failureCodes.rows[0][1].length).toBeGreaterThan(0);
expect(model.failureCodes.rows[1][1]).toBe("Unknown code");
});
it("sorts device models by lift and truncates to 10", () => {
const models = Array.from({ length: 12 }, (_, i) => ({
manufacturer: "Contoso",
model: `Model ${i}`,
installs: 10,
failed: i,
failureRate: i * 5,
liftVsBaseline: i * 0.3,
}));
const model = prepareReportModel(makeInput({ deviceModelBreakdown: models }));
expect(model.deviceModels.rows).toHaveLength(10);
expect(model.deviceModels.moreCount).toBe(2);
expect(model.deviceModels.rows[0][1]).toBe("Model 11");
expect(model.deviceModels.rows[0][5]).toBe("3.30x");
});
it("shows version durations only when measured", () => {
const model = prepareReportModel(
makeInput({
versionBreakdown: [
{ appVersion: "1.0", installs: 5, failed: 0, failureRate: 0, measuredInstalls: 5, medianDurationSeconds: 90, p95DurationSeconds: 120 },
{ appVersion: "1.1", installs: 3, failed: 1, failureRate: 33.3, measuredInstalls: 0, medianDurationSeconds: 0, p95DurationSeconds: 0 },
],
})
);
expect(model.versions.rows[0][4]).toBe("2m");
expect(model.versions.rows[1][4]).toBe("—");
expect(model.versions.rows[1][5]).toBe("—");
});
it("emits detection-lies text with singular/plural forms", () => {
expect(prepareReportModel(makeInput({ detectionLiesCount: 0 })).detectionLiesText).toBeNull();
expect(prepareReportModel(makeInput({ detectionLiesCount: 1 })).detectionLiesText).toContain(
"1 install was reported as Succeeded"
);
expect(prepareReportModel(makeInput({ detectionLiesCount: 18 })).detectionLiesText).toContain(
"18 installs were reported as Succeeded"
);
});
it("formats regression lines and caps them at 3", () => {
const reg = {
currentVersion: "2.0",
previousVersion: "1.9",
currentMedianSeconds: 300,
previousMedianSeconds: 120,
currentMeasuredCount: 40,
previousMeasuredCount: 60,
lift: 2.5,
};
const model = prepareReportModel(makeInput({ versionRegressions: [reg, reg, reg, reg] }));
expect(model.regressionLines).toHaveLength(3);
expect(model.regressionLines[0]).toBe(
"Duration regression: median install duration rose from 2.0 to 5.0 min after version 2.0 (40 measured installs vs 60 on 1.9 — lift 2.5x)"
);
});
it("produces a valid model for an app with no data at all", () => {
const model = prepareReportModel(
makeInput({
summary: {
totalInstalls: 0, succeeded: 0, skipped: 0, unmeasured: 0, failed: 0,
failureRate: 0, avgDurationSeconds: 0, p95DurationSeconds: 0,
trend: "stable", trendDelta: null, flakinessScore: 0,
},
timeSeries: [],
})
);
expect(model.kpis[0].value).toBe("0");
expect(model.kpis[4].value).toBe("—");
expect(model.kpis[6].value).toBe("—");
expect(model.timeSeries).toEqual([]);
expect(model.failureCodes.rows).toEqual([]);
expect(model.failureCodes.moreCount).toBe(0);
});
});
describe("formatReportDate", () => {
it("is locale-independent", () => {
expect(formatReportDate(new Date(2026, 0, 5))).toBe("Jan 5, 2026");
expect(formatReportDate(new Date(2026, 11, 31))).toBe("Dec 31, 2026");
});
});
@@ -0,0 +1,74 @@
// Smoke test: assemble a full report document under node (jsPDF is DOM-free for
// document assembly; only the download step in generateAppReportPdf needs a browser).
import { describe, it, expect } from "vitest";
import { buildAppReportDoc } from "../appReportPdf";
import { prepareReportModel, type AppReportInput } from "../appReportData";
function makeInput(): AppReportInput {
const timeSeries = Array.from({ length: 13 }, (_, i) => ({
bucketStart: new Date(Date.UTC(2026, 4, 25 + i * 7)).toISOString(),
installs: 10 + i * 15,
succeeded: 9 + i * 15,
failed: i % 4 === 0 ? 1 : 0,
failureRate: i % 4 === 0 ? 5 : 0,
avgDurationSeconds: 90 - i * 3,
}));
return {
analytics: {
appType: "MSI",
windowDays: 90,
bucket: "week",
summary: {
totalInstalls: 912, succeeded: 888, skipped: 20, unmeasured: 3, failed: 1,
failureRate: 0.1, avgDurationSeconds: 55, p95DurationSeconds: 110,
trend: "stable", trendDelta: 0, flakinessScore: 0.02,
},
timeSeries,
versionBreakdown: [
{ appVersion: "1.2.0", installs: 500, failed: 1, failureRate: 0.2, measuredInstalls: 480, medianDurationSeconds: 52, p95DurationSeconds: 100 },
],
topFailureCodes: [
{ code: "esp_apps_install_failure", exitCode: 1618, count: 1 },
],
detectionLiesCount: 18,
deviceModelBreakdown: [
{ manufacturer: "Dell Inc.", model: "Latitude 5440", installs: 200, failed: 1, failureRate: 0.5, liftVsBaseline: 1.1 },
],
versionRegressions: [],
},
appName: "RealmJoin Agent (Device)",
days: 90,
scopeLabel: "All tenants (aggregated)",
generatedAt: new Date(2026, 7, 18),
};
}
describe("buildAppReportDoc", () => {
it("assembles a multi-section report without throwing", () => {
const input = makeInput();
const doc = buildAppReportDoc(prepareReportModel(input), input.generatedAt);
expect(doc.getNumberOfPages()).toBeGreaterThanOrEqual(1);
const raw = doc.output();
expect(raw.startsWith("%PDF")).toBe(true);
});
it("handles an app with no data at all", () => {
const input = makeInput();
input.analytics = {
...input.analytics,
timeSeries: [],
versionBreakdown: [],
topFailureCodes: [],
detectionLiesCount: 0,
deviceModelBreakdown: [],
versionRegressions: [],
summary: {
totalInstalls: 0, succeeded: 0, skipped: 0, unmeasured: 0, failed: 0,
failureRate: 0, avgDurationSeconds: 0, p95DurationSeconds: 0,
trend: "stable", trendDelta: null, flakinessScore: 0,
},
};
const doc = buildAppReportDoc(prepareReportModel(input), input.generatedAt);
expect(doc.getNumberOfPages()).toBeGreaterThanOrEqual(1);
});
});
@@ -0,0 +1,87 @@
import { describe, it, expect } from "vitest";
import { niceTicks, linearScale, tickIndices, formatBucketLabel } from "../chartMath";
describe("niceTicks", () => {
it("returns a safe default for zero", () => {
expect(niceTicks(0)).toEqual({ max: 1, ticks: [0, 1] });
});
it("returns a safe default for negative and non-finite input", () => {
expect(niceTicks(-5)).toEqual({ max: 1, ticks: [0, 1] });
expect(niceTicks(NaN)).toEqual({ max: 1, ticks: [0, 1] });
expect(niceTicks(Infinity)).toEqual({ max: 1, ticks: [0, 1] });
});
it("rounds a small maximum up to a nice value", () => {
const { max, ticks } = niceTicks(7);
expect(max).toBe(8);
expect(ticks[0]).toBe(0);
expect(ticks[ticks.length - 1]).toBe(8);
});
it("rounds 93 up to 100 with clean steps", () => {
const { max, ticks } = niceTicks(93);
expect(max).toBe(100);
expect(ticks).toEqual([0, 25, 50, 75, 100]);
});
it("keeps an exact multiple as the maximum", () => {
const { max } = niceTicks(100);
expect(max).toBe(100);
});
it("produces float-clean ticks for fractional maxima", () => {
const { ticks } = niceTicks(0.7);
for (const t of ticks) {
// No accumulation artifacts like 0.30000000000000004.
expect(String(t).length).toBeLessThanOrEqual(6);
}
});
});
describe("linearScale", () => {
it("maps the domain onto the range linearly", () => {
const scale = linearScale(100, 50);
expect(scale(0)).toBe(0);
expect(scale(50)).toBe(25);
expect(scale(100)).toBe(50);
});
it("returns a zero scale for an empty domain", () => {
expect(linearScale(0, 50)(10)).toBe(0);
});
});
describe("tickIndices", () => {
it("returns all indices when n fits within maxLabels", () => {
expect(tickIndices(7, 10)).toEqual([0, 1, 2, 3, 4, 5, 6]);
});
it("returns evenly spaced indices including first and last for 90/10", () => {
const idx = tickIndices(90, 10);
expect(idx.length).toBeLessThanOrEqual(10);
expect(idx[0]).toBe(0);
expect(idx[idx.length - 1]).toBe(89);
// Strictly ascending, no duplicates.
for (let i = 1; i < idx.length; i++) {
expect(idx[i]).toBeGreaterThan(idx[i - 1]);
}
});
it("handles empty and degenerate inputs", () => {
expect(tickIndices(0, 10)).toEqual([]);
expect(tickIndices(5, 0)).toEqual([]);
expect(tickIndices(1, 10)).toEqual([0]);
});
});
describe("formatBucketLabel", () => {
it("formats an ISO date as UTC M/D", () => {
expect(formatBucketLabel("2026-08-03T00:00:00Z")).toBe("8/3");
expect(formatBucketLabel("2026-12-25T00:00:00Z")).toBe("12/25");
});
it("returns the raw string for an unparseable date", () => {
expect(formatBucketLabel("not-a-date")).toBe("not-a-date");
});
});
@@ -0,0 +1,293 @@
// Pure data preparation for the app install report PDF — no jsPDF import, so the
// page can statically import the helpers here without pulling jsPDF into the
// route chunk (jsPDF enters only via the lazily imported appReportPdf module).
import { getErrorCodeEntry, formatErrorCode } from "@/utils/errorCodeMap";
// Structural mirror of the fields the report uses from the page's module-local
// AnalyticsResponse. A field rename on the page surfaces as a compile error at
// the generateAppReportPdf call site — intentional, cheap coupling.
export interface ReportTimeSeriesPoint {
bucketStart: string;
installs: number;
succeeded: number;
failed: number;
failureRate: number;
avgDurationSeconds: number;
}
export interface AppReportAnalytics {
appType: string;
windowDays: number;
bucket: "day" | "week";
summary: {
totalInstalls: number;
succeeded: number;
skipped: number;
unmeasured: number;
failed: number;
failureRate: number;
avgDurationSeconds: number;
p95DurationSeconds: number;
trend: "improving" | "worsening" | "stable";
trendDelta: number | null;
flakinessScore: number;
};
timeSeries: ReportTimeSeriesPoint[];
versionBreakdown: Array<{
appVersion: string;
installs: number;
failed: number;
failureRate: number;
measuredInstalls: number;
medianDurationSeconds: number;
p95DurationSeconds: number;
}>;
topFailureCodes: Array<{
code: string;
exitCode: number | null;
count: number;
}>;
detectionLiesCount: number;
deviceModelBreakdown: Array<{
manufacturer: string;
model: string;
installs: number;
failed: number;
failureRate: number;
liftVsBaseline: number;
}>;
versionRegressions: Array<{
currentVersion: string;
previousVersion: string;
currentMedianSeconds: number;
previousMedianSeconds: number;
currentMeasuredCount: number;
previousMeasuredCount: number;
lift: number;
}>;
}
export interface AppReportInput {
analytics: AppReportAnalytics;
appName: string;
days: number;
scopeLabel: string;
generatedAt: Date;
}
export type KpiTone = "default" | "success" | "danger" | "warning";
export interface ReportKpi {
label: string;
value: string;
tone: KpiTone;
hint?: string;
}
export interface ReportTable {
rows: string[][];
/** Rows cut off by the per-table cap — rendered as a trailing "+N more" note. */
moreCount: number;
}
export interface AppReportModel {
appName: string;
appType: string;
metaLine: string;
fileName: string;
kpis: ReportKpi[];
detectionLiesText: string | null;
regressionLines: string[];
bucket: "day" | "week";
timeSeries: ReportTimeSeriesPoint[];
failureCodes: ReportTable;
deviceModels: ReportTable;
versions: ReportTable;
}
const MAX_FAILURE_CODE_ROWS = 10;
const MAX_DEVICE_MODEL_ROWS = 10;
const MAX_VERSION_ROWS = 8;
const MAX_REGRESSION_LINES = 3;
const MAX_SLUG_LENGTH = 60;
const MONTHS = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"];
/** Locale-independent "Aug 20, 2026" (local time of the exporting user). */
export function formatReportDate(date: Date): string {
return `${MONTHS[date.getMonth()]} ${date.getDate()}, ${date.getFullYear()}`;
}
/** `app-report-realmjoin-agent-device-90d-20260818.pdf` */
export function appReportFileName(appName: string, days: number, date: Date): string {
const slug =
appName
.toLowerCase()
.replace(/[^a-z0-9]+/g, "-")
.replace(/^-+|-+$/g, "")
.slice(0, MAX_SLUG_LENGTH)
.replace(/-+$/, "") || "app";
const y = date.getFullYear();
const m = String(date.getMonth() + 1).padStart(2, "0");
const d = String(date.getDate()).padStart(2, "0");
return `app-report-${slug}-${days}d-${y}${m}${d}.pdf`;
}
/** Human-readable tenant scope for the report title block. */
export function scopeLabel(scope: {
isAggregatedGlobalView: boolean;
selectedTenantName?: string;
selectedTenantId: string;
}): string {
if (scope.isAggregatedGlobalView) return "All tenants (aggregated)";
if (scope.selectedTenantName) return `Tenant: ${scope.selectedTenantName}`;
if (scope.selectedTenantId) return `Tenant: ${scope.selectedTenantId.slice(0, 8)}…`;
return "Current tenant";
}
// Same formatting rules as the app detail page.
function formatDuration(s: number): string {
if (!s) return "—";
if (s < 60) return `${Math.round(s)}s`;
if (s < 3600) return `${Math.round(s / 60)}m`;
return `${(s / 3600).toFixed(1)}h`;
}
// PDF variant of the page's trendText: built-in Helvetica (WinAnsi) has no
// arrow glyphs, so improving/worsening render as signed deltas instead.
function trendText(trend: string, delta: number | null): string {
if (delta == null) return "—";
if (trend === "improving") return `-${Math.abs(delta).toFixed(1)} pp`;
if (trend === "worsening") return `+${Math.abs(delta).toFixed(1)} pp`;
return "stable";
}
function failureRateTone(rate: number): KpiTone {
if (rate >= 20) return "danger";
if (rate >= 5) return "warning";
return "default";
}
/** Build the render-ready report model from the page's analytics response. */
export function prepareReportModel(input: AppReportInput): AppReportModel {
const { analytics, appName, days, generatedAt } = input;
const s = analytics.summary;
const kpis: ReportKpi[] = [
{ label: "Total installs", value: String(s.totalInstalls), tone: "default" },
{
label: "Succeeded",
value: String(s.succeeded),
tone: "success",
hint: s.skipped > 0 ? `+${s.skipped} skipped (not applicable)` : undefined,
},
{ label: "Failed", value: String(s.failed), tone: s.failed > 0 ? "danger" : "default" },
{
label: "Failure rate",
value: `${s.failureRate.toFixed(1)}%`,
tone: failureRateTone(s.failureRate),
},
{
label: "Avg duration",
value: formatDuration(s.avgDurationSeconds),
tone: "default",
hint:
s.unmeasured > 0
? `measured installs only — ${s.unmeasured} without observed start`
: "measured installs only",
},
{ label: "P95 duration", value: formatDuration(s.p95DurationSeconds), tone: "default" },
{
label: "Trend",
value: trendText(s.trend, s.trendDelta),
tone: "default",
hint: s.trendDelta != null && s.trend !== "stable" ? s.trend : undefined,
},
{
label: "Flakiness",
value: `${(s.flakinessScore * 100).toFixed(0)}%`,
tone: "default",
hint: "% of installs with retries",
},
];
const failureCodeRows = analytics.topFailureCodes.slice(0, MAX_FAILURE_CODE_ROWS).map((row) => {
const entry = getErrorCodeEntry(row.code);
const exitEntry = row.exitCode != null ? getErrorCodeEntry(row.exitCode) : null;
const exitDisplay =
row.exitCode != null
? exitEntry
? `${row.exitCode} (${exitEntry.description.slice(0, 24)}${exitEntry.description.length > 24 ? "…" : ""})`
: String(row.exitCode)
: "—";
return [
formatErrorCode(row.code),
entry ? entry.description : "Unknown code",
exitDisplay,
String(row.count),
];
});
const deviceModelRows = [...analytics.deviceModelBreakdown]
.sort((a, b) => b.liftVsBaseline - a.liftVsBaseline || b.installs - a.installs)
.slice(0, MAX_DEVICE_MODEL_ROWS)
.map((row) => [
row.manufacturer,
row.model,
String(row.installs),
String(row.failed),
`${row.failureRate.toFixed(1)}%`,
`${row.liftVsBaseline.toFixed(2)}x`,
]);
const versionRows = analytics.versionBreakdown.slice(0, MAX_VERSION_ROWS).map((row) => [
row.appVersion,
String(row.installs),
String(row.failed),
`${row.failureRate.toFixed(1)}%`,
row.measuredInstalls > 0 ? formatDuration(row.medianDurationSeconds) : "—",
row.measuredInstalls > 0 ? formatDuration(row.p95DurationSeconds) : "—",
]);
const regressionLines = (analytics.versionRegressions ?? [])
.slice(0, MAX_REGRESSION_LINES)
.map(
(reg) =>
`Duration regression: median install duration rose from ${(reg.previousMedianSeconds / 60).toFixed(1)} to ` +
`${(reg.currentMedianSeconds / 60).toFixed(1)} min after version ${reg.currentVersion} ` +
`(${reg.currentMeasuredCount} measured installs vs ${reg.previousMeasuredCount} on ${reg.previousVersion} — lift ${reg.lift}x)`
);
const detectionLiesText =
analytics.detectionLiesCount > 0
? `${analytics.detectionLiesCount} install${analytics.detectionLiesCount === 1 ? " was" : "s were"} ` +
"reported as Succeeded but the detection rule did not find the app afterwards. Check the detection rule."
: null;
return {
appName,
appType: analytics.appType,
metaLine:
`${analytics.windowDays} day window · ${analytics.bucket === "day" ? "daily" : "weekly"} buckets · ` +
`${input.scopeLabel} · Generated ${formatReportDate(generatedAt)}`,
fileName: appReportFileName(appName, days, generatedAt),
kpis,
detectionLiesText,
regressionLines,
bucket: analytics.bucket,
timeSeries: analytics.timeSeries,
failureCodes: {
rows: failureCodeRows,
moreCount: Math.max(0, analytics.topFailureCodes.length - MAX_FAILURE_CODE_ROWS),
},
deviceModels: {
rows: deviceModelRows,
moreCount: Math.max(0, analytics.deviceModelBreakdown.length - MAX_DEVICE_MODEL_ROWS),
},
versions: {
rows: versionRows,
moreCount: Math.max(0, analytics.versionBreakdown.length - MAX_VERSION_ROWS),
},
};
}
@@ -0,0 +1,180 @@
// App install report PDF builder — the entry point of the lazily loaded PDF
// chunk. This is the ONLY module that imports jsPDF at runtime; the page loads
// it via `await import("@/utils/pdf/appReportPdf")` on button click, so jsPDF
// never enters the route bundle.
import { jsPDF } from "jspdf";
import { pdfColors, pdfPage, contentW } from "./pdfTheme";
import {
startReport,
ensureSpace,
sectionTitle,
kpiRow,
calloutBox,
drawTable,
finalizeFooters,
truncateToWidth,
type KpiCell,
} from "./pdfPrimitives";
import { drawStackedBarChart, drawLineChart } from "./pdfCharts";
import {
prepareReportModel,
formatReportDate,
type AppReportInput,
type AppReportModel,
type KpiTone,
} from "./appReportData";
export interface AppReportResult {
pageCount: number;
fileName: string;
}
const KPI_TONE_COLORS: Record<KpiTone, string> = {
default: pdfColors.text,
success: pdfColors.success,
danger: pdfColors.danger,
warning: pdfColors.warning,
};
const BAR_CHART_H = 58;
const LINE_CHART_H = 52;
/** Generate the report and hand it to the browser as a file download. */
export function generateAppReportPdf(input: AppReportInput): AppReportResult {
const model = prepareReportModel(input);
const doc = buildAppReportDoc(model, input.generatedAt);
downloadBlob(doc.output("blob"), model.fileName);
return { pageCount: doc.getNumberOfPages(), fileName: model.fileName };
}
/** Document assembly, separated from the browser download for testability. */
export function buildAppReportDoc(model: AppReportModel, generatedAt: Date): jsPDF {
const doc = new jsPDF({ orientation: "portrait", unit: "mm", format: "a4" });
const ctx = startReport(doc, "App Install Report");
// Title block: app name (wrapped) + appType badge + meta line.
doc.setFont("helvetica", "bold");
doc.setFontSize(16);
doc.setTextColor(pdfColors.text);
const nameLines = doc.splitTextToSize(model.appName, contentW - 30) as string[];
ctx.y += 4;
doc.text(nameLines, pdfPage.margin, ctx.y + 4);
if (model.appType) {
const lastLine = nameLines[nameLines.length - 1] ?? "";
const badgeX = pdfPage.margin + doc.getTextWidth(lastLine) + 3;
const badgeY = ctx.y + (nameLines.length - 1) * 6.5;
doc.setFontSize(7);
const badgeText = truncateToWidth(doc, model.appType, 24);
const badgeW = doc.getTextWidth(badgeText) + 4;
doc.setFillColor(pdfColors.badgeBg);
doc.roundedRect(badgeX, badgeY, badgeW, 5, 1, 1, "F");
doc.setTextColor(pdfColors.badgeText);
doc.text(badgeText, badgeX + 2, badgeY + 3.5);
}
ctx.y += nameLines.length * 6.5 + 2;
doc.setFont("helvetica", "normal");
doc.setFontSize(8.5);
doc.setTextColor(pdfColors.subtle);
doc.text(truncateToWidth(doc, model.metaLine, contentW), pdfPage.margin, ctx.y + 3);
ctx.y += 8;
const kpiCells: KpiCell[] = model.kpis.map((kpi) => ({
label: kpi.label,
value: kpi.value,
valueColor: KPI_TONE_COLORS[kpi.tone],
hint: kpi.hint,
}));
kpiRow(ctx, kpiCells, 4);
ctx.y += 2;
if (model.detectionLiesText) {
calloutBox(ctx, `Heads up: ${model.detectionLiesText}`);
}
for (const line of model.regressionLines) {
calloutBox(ctx, line);
}
sectionTitle(ctx, "Installs over time (success vs failure)");
ensureSpace(ctx, BAR_CHART_H + 2);
drawStackedBarChart(doc, { x: pdfPage.margin, y: ctx.y, w: contentW, h: BAR_CHART_H }, model.timeSeries);
ctx.y += BAR_CHART_H + 2;
sectionTitle(ctx, "Avg Install Duration over time");
ensureSpace(ctx, LINE_CHART_H + 2);
drawLineChart(doc, { x: pdfPage.margin, y: ctx.y, w: contentW, h: LINE_CHART_H }, model.timeSeries);
ctx.y += LINE_CHART_H + 2;
sectionTitle(ctx, "Top Failure Codes");
if (model.failureCodes.rows.length > 0) {
drawTable(ctx, {
columns: [
{ header: "Code", width: 30 },
{ header: "Description", width: 88 },
{ header: "Exit code", width: 44 },
{ header: "Count", width: 18, align: "right" },
],
rows: model.failureCodes.rows,
moreCount: model.failureCodes.moreCount,
});
} else {
emptySectionNote(ctx, "No failures recorded in this window.");
}
sectionTitle(ctx, "Device Model Correlation");
if (model.deviceModels.rows.length > 0) {
drawTable(ctx, {
columns: [
{ header: "Manufacturer", width: 34 },
{ header: "Model", width: 66 },
{ header: "Installs", width: 20, align: "right" },
{ header: "Failed", width: 20, align: "right" },
{ header: "Failure rate", width: 22, align: "right" },
{ header: "Lift vs base", width: 18, align: "right" },
],
rows: model.deviceModels.rows,
moreCount: model.deviceModels.moreCount,
});
} else {
emptySectionNote(ctx, "Not enough installs per device model to compute correlation.");
}
if (model.versions.rows.length > 0) {
sectionTitle(ctx, "Version Breakdown");
drawTable(ctx, {
columns: [
{ header: "Version", width: 60 },
{ header: "Installs", width: 22, align: "right" },
{ header: "Failed", width: 22, align: "right" },
{ header: "Failure rate", width: 26, align: "right" },
{ header: "Median dur.", width: 25, align: "right" },
{ header: "P95 dur.", width: 25, align: "right" },
],
rows: model.versions.rows,
moreCount: model.versions.moreCount,
});
}
finalizeFooters(doc, `Generated by Autopilot Monitor · ${formatReportDate(generatedAt)}`);
return doc;
}
function emptySectionNote(ctx: ReturnType<typeof startReport>, text: string) {
ensureSpace(ctx, 8);
ctx.doc.setFont("helvetica", "italic");
ctx.doc.setFontSize(8);
ctx.doc.setTextColor(pdfColors.faint);
ctx.doc.text(text, pdfPage.margin + 2, ctx.y + 3);
ctx.y += 8;
}
// Same blob-anchor pattern as every other export in the app
// (cf. app/admin/components/SessionExportSection.tsx).
function downloadBlob(blob: Blob, fileName: string) {
const url = URL.createObjectURL(blob);
const a = document.createElement("a");
a.href = url;
a.download = fileName;
a.click();
URL.revokeObjectURL(url);
}
@@ -0,0 +1,53 @@
// Pure chart math for the PDF chart renderers — no jsPDF dependency, unit-testable.
/**
* Round a data maximum up to a "nice" axis maximum (1/2/2.5/5 progression) and
* return evenly spaced tick values from 0 to that maximum.
*/
export function niceTicks(maxValue: number, tickCount = 4): { max: number; ticks: number[] } {
if (!Number.isFinite(maxValue) || maxValue <= 0) {
return { max: 1, ticks: [0, 1] };
}
const rawStep = maxValue / tickCount;
const magnitude = Math.pow(10, Math.floor(Math.log10(rawStep)));
const normalized = rawStep / magnitude;
const niceFactor =
normalized <= 1 ? 1 : normalized <= 2 ? 2 : normalized <= 2.5 ? 2.5 : normalized <= 5 ? 5 : 10;
const step = niceFactor * magnitude;
const max = Math.ceil(maxValue / step) * step;
const ticks: number[] = [];
// Float-safe loop: derive each tick from the integer index, not by accumulation.
const count = Math.round(max / step);
for (let i = 0; i <= count; i++) {
ticks.push(Number((i * step).toPrecision(12)));
}
return { max, ticks };
}
/** Map a data value in [0, domainMax] to a length in [0, range]. */
export function linearScale(domainMax: number, range: number): (v: number) => number {
if (!Number.isFinite(domainMax) || domainMax <= 0) return () => 0;
return (v: number) => (v / domainMax) * range;
}
/**
* Pick up to maxLabels evenly spaced indices out of n buckets, always including
* the first and last — so 7 daily buckets label all days while 90 label ~10.
*/
export function tickIndices(n: number, maxLabels: number): number[] {
if (n <= 0 || maxLabels <= 0) return [];
if (n <= maxLabels) return Array.from({ length: n }, (_, i) => i);
const step = (n - 1) / (maxLabels - 1);
const indices = new Set<number>();
for (let i = 0; i < maxLabels; i++) {
indices.add(Math.round(i * step));
}
return [...indices].sort((a, b) => a - b);
}
/** UTC M/D bucket label — mirrors formatBucketTick on the app detail page. */
export function formatBucketLabel(iso: string): string {
const d = new Date(iso);
if (isNaN(d.getTime())) return iso;
return `${d.getUTCMonth() + 1}/${d.getUTCDate()}`;
}
@@ -0,0 +1,164 @@
// jsPDF renderers for the two report charts, drawn directly from timeSeries
// data with a print-friendly light palette (no DOM capture, theme-independent).
import type { jsPDF } from "jspdf";
import { pdfColors } from "./pdfTheme";
import { niceTicks, linearScale, tickIndices, formatBucketLabel } from "./chartMath";
import type { ReportTimeSeriesPoint } from "./appReportData";
export interface ChartRegion {
x: number;
y: number;
w: number;
h: number;
}
const Y_LABEL_W = 12;
const X_LABEL_H = 6;
const LEGEND_H = 5;
const MAX_X_LABELS = 10;
interface PlotArea {
x: number;
y: number;
w: number;
h: number;
}
function plotArea(region: ChartRegion): PlotArea {
return {
x: region.x + Y_LABEL_W,
y: region.y + LEGEND_H,
w: region.w - Y_LABEL_W,
h: region.h - LEGEND_H - X_LABEL_H,
};
}
function drawEmptyPlaceholder(doc: jsPDF, region: ChartRegion) {
doc.setDrawColor(pdfColors.border);
doc.setLineWidth(0.3);
doc.roundedRect(region.x, region.y, region.w, region.h, 1, 1, "S");
doc.setFont("helvetica", "normal");
doc.setFontSize(9);
doc.setTextColor(pdfColors.faint);
doc.text("No data in this window", region.x + region.w / 2, region.y + region.h / 2, {
align: "center",
});
}
/** Gridlines + y tick labels + x bucket labels shared by both charts. */
function drawAxes(
doc: jsPDF,
plot: PlotArea,
yTicks: number[],
yMax: number,
yUnit: string,
points: ReportTimeSeriesPoint[]
) {
const yScale = linearScale(yMax, plot.h);
doc.setFont("helvetica", "normal");
doc.setFontSize(6.5);
for (const tick of yTicks) {
const y = plot.y + plot.h - yScale(tick);
doc.setDrawColor(pdfColors.border);
doc.setLineWidth(0.1);
doc.line(plot.x, y, plot.x + plot.w, y);
doc.setTextColor(pdfColors.subtle);
doc.text(`${tick}${yUnit}`, plot.x - 1.5, y + 1, { align: "right" });
}
// Baseline slightly stronger than the gridlines.
doc.setDrawColor(pdfColors.subtle);
doc.setLineWidth(0.2);
doc.line(plot.x, plot.y + plot.h, plot.x + plot.w, plot.y + plot.h);
const n = points.length;
const slotW = plot.w / n;
doc.setTextColor(pdfColors.subtle);
for (const i of tickIndices(n, MAX_X_LABELS)) {
const cx = plot.x + slotW * (i + 0.5);
doc.text(formatBucketLabel(points[i].bucketStart), cx, plot.y + plot.h + 4, {
align: "center",
});
}
}
function drawLegend(
doc: jsPDF,
region: ChartRegion,
entries: Array<{ label: string; color: string }>
) {
doc.setFont("helvetica", "normal");
doc.setFontSize(6.5);
let x = region.x + region.w;
// Right-aligned: lay the entries out from the right edge backwards.
for (const entry of [...entries].reverse()) {
const labelW = doc.getTextWidth(entry.label);
x -= labelW;
doc.setTextColor(pdfColors.subtle);
doc.text(entry.label, x, region.y + 3);
x -= 3.5;
doc.setFillColor(entry.color);
doc.rect(x, region.y + 0.8, 2.5, 2.5, "F");
x -= 5;
}
}
/** "Installs over time": succeeded (green) with failed (red) stacked on top. */
export function drawStackedBarChart(doc: jsPDF, region: ChartRegion, points: ReportTimeSeriesPoint[]) {
if (points.length === 0) {
drawEmptyPlaceholder(doc, region);
return;
}
const plot = plotArea(region);
const { max, ticks } = niceTicks(Math.max(...points.map((p) => p.succeeded + p.failed)));
drawLegend(doc, region, [
{ label: "Succeeded", color: pdfColors.success },
{ label: "Failed", color: pdfColors.danger },
]);
drawAxes(doc, plot, ticks, max, "", points);
const yScale = linearScale(max, plot.h);
const slotW = plot.w / points.length;
const barW = Math.max(0.4, slotW * 0.7);
points.forEach((p, i) => {
const x = plot.x + slotW * i + (slotW - barW) / 2;
const succeededH = yScale(p.succeeded);
const failedH = yScale(p.failed);
if (succeededH > 0) {
doc.setFillColor(pdfColors.success);
doc.rect(x, plot.y + plot.h - succeededH, barW, succeededH, "F");
}
if (failedH > 0) {
doc.setFillColor(pdfColors.danger);
doc.rect(x, plot.y + plot.h - succeededH - failedH, barW, failedH, "F");
}
});
}
/** "Avg Install Duration over time": blue polyline, dot markers when sparse. */
export function drawLineChart(doc: jsPDF, region: ChartRegion, points: ReportTimeSeriesPoint[]) {
if (points.length === 0) {
drawEmptyPlaceholder(doc, region);
return;
}
const plot = plotArea(region);
const { max, ticks } = niceTicks(Math.max(...points.map((p) => p.avgDurationSeconds)));
drawLegend(doc, region, [{ label: "Avg duration", color: pdfColors.primary }]);
drawAxes(doc, plot, ticks, max, "s", points);
const yScale = linearScale(max, plot.h);
const slotW = plot.w / points.length;
const coords = points.map((p, i) => ({
x: plot.x + slotW * (i + 0.5),
y: plot.y + plot.h - yScale(p.avgDurationSeconds),
}));
doc.setDrawColor(pdfColors.primary);
doc.setLineWidth(0.5);
for (let i = 1; i < coords.length; i++) {
doc.line(coords[i - 1].x, coords[i - 1].y, coords[i].x, coords[i].y);
}
if (points.length <= 31) {
doc.setFillColor(pdfColors.primary);
for (const c of coords) {
doc.circle(c.x, c.y, 0.7, "F");
}
}
}
@@ -0,0 +1,218 @@
// Generic report chrome for jsPDF-generated documents: page header/footer,
// section titles, KPI rows, callout boxes, and a simple fixed-column table.
// App-agnostic on purpose — future report exports (SLA, session report) can
// reuse this module unchanged.
//
// Type-only jsPDF import: the runtime jsPDF instance is created by the caller
// (inside the lazily loaded report builder), so this file adds no bundle weight
// to anything that imports it statically.
import type { jsPDF } from "jspdf";
import { pdfColors, pdfPage, contentW, contentTop, contentBottom } from "./pdfTheme";
export interface PdfContext {
doc: jsPDF;
/** Current write cursor (top of the next content block), in mm. */
y: number;
/** Report title repeated in the header band of every page. */
reportTitle: string;
}
export interface TableColumn {
header: string;
/** Column width in mm; widths should sum to contentW. */
width: number;
align?: "left" | "right";
}
export interface TableSpec {
columns: TableColumn[];
rows: string[][];
/** Trailing italic "+N more …" note when the source data was truncated. */
moreCount?: number;
zebra?: boolean;
}
function drawPageHeader(doc: jsPDF, reportTitle: string) {
const y = pdfPage.margin + 4;
doc.setFont("helvetica", "bold");
doc.setFontSize(10);
doc.setTextColor(pdfColors.brand);
doc.text("Autopilot Monitor", pdfPage.margin, y);
doc.setFont("helvetica", "normal");
doc.setFontSize(9);
doc.setTextColor(pdfColors.subtle);
doc.text(reportTitle, pdfPage.w - pdfPage.margin, y, { align: "right" });
doc.setDrawColor(pdfColors.border);
doc.setLineWidth(0.3);
doc.line(pdfPage.margin, y + 3, pdfPage.w - pdfPage.margin, y + 3);
}
/** Start a new report document: draws the first page's header and returns the cursor context. */
export function startReport(doc: jsPDF, reportTitle: string): PdfContext {
drawPageHeader(doc, reportTitle);
return { doc, y: contentTop + 2, reportTitle };
}
/** Page-break guard: if the next block of `needed` mm doesn't fit, start a new page. */
export function ensureSpace(ctx: PdfContext, needed: number) {
if (ctx.y + needed <= contentBottom) return;
ctx.doc.addPage();
drawPageHeader(ctx.doc, ctx.reportTitle);
ctx.y = contentTop + 2;
}
export function sectionTitle(ctx: PdfContext, text: string) {
ensureSpace(ctx, 14);
ctx.y += 3;
ctx.doc.setFont("helvetica", "bold");
ctx.doc.setFontSize(11);
ctx.doc.setTextColor(pdfColors.text);
ctx.doc.text(text, pdfPage.margin, ctx.y + 4);
ctx.y += 8;
}
/** Truncate text with an ellipsis so it fits maxWidth at the doc's CURRENT font settings. */
export function truncateToWidth(doc: jsPDF, text: string, maxWidth: number): string {
if (doc.getTextWidth(text) <= maxWidth) return text;
let t = text;
while (t.length > 1 && doc.getTextWidth(`${t}…`) > maxWidth) {
t = t.slice(0, -1);
}
return `${t}…`;
}
export interface KpiCell {
label: string;
value: string;
/** Resolved hex color for the value text. */
valueColor: string;
hint?: string;
}
const KPI_CELL_H = 17;
const KPI_GAP = 3;
/** One row of bordered KPI cells, `perRow` across the content width. */
export function kpiRow(ctx: PdfContext, kpis: KpiCell[], perRow = 4) {
const rows = Math.ceil(kpis.length / perRow);
ensureSpace(ctx, rows * (KPI_CELL_H + KPI_GAP));
const cellW = (contentW - (perRow - 1) * KPI_GAP) / perRow;
kpis.forEach((kpi, i) => {
const col = i % perRow;
const row = Math.floor(i / perRow);
const x = pdfPage.margin + col * (cellW + KPI_GAP);
const y = ctx.y + row * (KPI_CELL_H + KPI_GAP);
ctx.doc.setDrawColor(pdfColors.border);
ctx.doc.setLineWidth(0.3);
ctx.doc.roundedRect(x, y, cellW, KPI_CELL_H, 1, 1, "S");
ctx.doc.setFont("helvetica", "normal");
ctx.doc.setFontSize(6.5);
ctx.doc.setTextColor(pdfColors.subtle);
ctx.doc.text(truncateToWidth(ctx.doc, kpi.label.toUpperCase(), cellW - 6), x + 3, y + 5);
ctx.doc.setFont("helvetica", "bold");
ctx.doc.setFontSize(12);
ctx.doc.setTextColor(kpi.valueColor);
ctx.doc.text(truncateToWidth(ctx.doc, kpi.value, cellW - 6), x + 3, y + 11);
if (kpi.hint) {
ctx.doc.setFont("helvetica", "normal");
ctx.doc.setFontSize(6);
ctx.doc.setTextColor(pdfColors.faint);
ctx.doc.text(truncateToWidth(ctx.doc, kpi.hint, cellW - 6), x + 3, y + 15);
}
});
ctx.y += rows * (KPI_CELL_H + KPI_GAP);
}
/** Amber warning callout with a left accent bar; wraps the text to the content width. */
export function calloutBox(ctx: PdfContext, text: string) {
ctx.doc.setFont("helvetica", "normal");
ctx.doc.setFontSize(8.5);
const lines = ctx.doc.splitTextToSize(text, contentW - 10) as string[];
const boxH = lines.length * 4 + 6;
ensureSpace(ctx, boxH + 3);
ctx.doc.setFillColor(pdfColors.warningBg);
ctx.doc.roundedRect(pdfPage.margin, ctx.y, contentW, boxH, 1, 1, "F");
ctx.doc.setFillColor(pdfColors.warningAccent);
ctx.doc.rect(pdfPage.margin, ctx.y, 1.2, boxH, "F");
ctx.doc.setTextColor(pdfColors.warning);
ctx.doc.text(lines, pdfPage.margin + 5, ctx.y + 5);
ctx.y += boxH + 3;
}
const TABLE_HEADER_H = 7;
const TABLE_ROW_H = 6;
function drawTableHeader(ctx: PdfContext, columns: TableColumn[]) {
ctx.doc.setFillColor(pdfColors.headerBand);
ctx.doc.rect(pdfPage.margin, ctx.y, contentW, TABLE_HEADER_H, "F");
ctx.doc.setFont("helvetica", "bold");
ctx.doc.setFontSize(7);
ctx.doc.setTextColor(pdfColors.subtle);
let x = pdfPage.margin;
for (const col of columns) {
const tx = col.align === "right" ? x + col.width - 2 : x + 2;
ctx.doc.text(col.header.toUpperCase(), tx, ctx.y + 4.7, {
align: col.align === "right" ? "right" : "left",
});
x += col.width;
}
ctx.y += TABLE_HEADER_H;
}
/** Fixed-column table with zebra striping and header repetition after page breaks. */
export function drawTable(ctx: PdfContext, spec: TableSpec) {
const { columns, rows, moreCount = 0, zebra = true } = spec;
ensureSpace(ctx, TABLE_HEADER_H + TABLE_ROW_H * 2);
drawTableHeader(ctx, columns);
rows.forEach((row, rowIndex) => {
// Re-draw the header when the row would spill past the footer.
if (ctx.y + TABLE_ROW_H > contentBottom) {
ensureSpace(ctx, TABLE_HEADER_H + TABLE_ROW_H);
drawTableHeader(ctx, columns);
}
if (zebra && rowIndex % 2 === 1) {
ctx.doc.setFillColor(pdfColors.zebra);
ctx.doc.rect(pdfPage.margin, ctx.y, contentW, TABLE_ROW_H, "F");
}
ctx.doc.setFont("helvetica", "normal");
ctx.doc.setFontSize(8);
ctx.doc.setTextColor(pdfColors.text);
let x = pdfPage.margin;
row.forEach((cell, i) => {
const col = columns[i];
if (!col) return;
const text = truncateToWidth(ctx.doc, cell, col.width - 4);
const tx = col.align === "right" ? x + col.width - 2 : x + 2;
ctx.doc.text(text, tx, ctx.y + 4.2, { align: col.align === "right" ? "right" : "left" });
x += col.width;
});
ctx.doc.setDrawColor(pdfColors.border);
ctx.doc.setLineWidth(0.1);
ctx.doc.line(pdfPage.margin, ctx.y + TABLE_ROW_H, pdfPage.margin + contentW, ctx.y + TABLE_ROW_H);
ctx.y += TABLE_ROW_H;
});
if (moreCount > 0) {
ensureSpace(ctx, TABLE_ROW_H);
ctx.doc.setFont("helvetica", "italic");
ctx.doc.setFontSize(7.5);
ctx.doc.setTextColor(pdfColors.faint);
ctx.doc.text(`+${moreCount} more not shown`, pdfPage.margin + 2, ctx.y + 4);
ctx.y += TABLE_ROW_H;
}
ctx.y += 2;
}
/** Stamp the footer on every page — call once, after all content is placed. */
export function finalizeFooters(doc: jsPDF, generatedLabel: string) {
const total = doc.getNumberOfPages();
const y = pdfPage.h - pdfPage.margin;
for (let p = 1; p <= total; p++) {
doc.setPage(p);
doc.setFont("helvetica", "normal");
doc.setFontSize(7);
doc.setTextColor(pdfColors.faint);
doc.text(generatedLabel, pdfPage.margin, y);
doc.text(`Page ${p} of ${total}`, pdfPage.w - pdfPage.margin, y, { align: "right" });
}
}
@@ -0,0 +1,44 @@
// Print palette + A4 layout constants for generated PDF reports.
//
// Deliberately separate from components/charts/chartTheme.ts: those colors are
// tuned for dark on-screen surfaces (gray-400 axis text, #374151 grid) and look
// washed out on white paper. PDFs always render on a fixed light background,
// independent of the user's on-screen theme.
export const pdfColors = {
text: "#111827",
subtle: "#6b7280",
faint: "#9ca3af",
border: "#e5e7eb",
headerBand: "#f3f4f6",
zebra: "#f9fafb",
brand: "#16a34a",
success: "#059669",
danger: "#dc2626",
warning: "#b45309",
warningBg: "#fffbeb",
warningAccent: "#f59e0b",
primary: "#2563eb",
badgeBg: "#dbeafe",
badgeText: "#1e40af",
} as const;
/** A4 portrait geometry in millimetres. */
export const pdfPage = {
w: 210,
h: 297,
margin: 15,
/** Height reserved under the top margin for the repeating page header band. */
headerH: 12,
/** Height reserved above the bottom margin for the page footer. */
footerH: 10,
} as const;
/** Usable content width between the side margins. */
export const contentW = pdfPage.w - 2 * pdfPage.margin;
/** First y a content block may start at (below the page header). */
export const contentTop = pdfPage.margin + pdfPage.headerH;
/** Last y a content block may extend to (above the footer). */
export const contentBottom = pdfPage.h - pdfPage.margin - pdfPage.footerH;